Bnaya Eshet : Entity Framework

EF should adopt the WAQS framework

bnaya — Tue, 07 May 2013 09:08:33 GMT

EF should adopt the WAQS framework

I was speaking with Matthieu MEZIL about his WAQS framework,
it is a layer of abstraction over the Entity Framework which I think you should check out.
Matthieu is working on it on his free time (currently as an open source) and it is amazing to see what he was able to achieve.
this framework is already been in used by some of real word company and it is improving over time (today Matthieu is working on a new version of it).

as I see it, Microsoft EF team should adopt this framework as part of the EF open source project. I hope that the EF team will put the effort to merge it into their source code.

WAQS is addressing many of the painful scenarios under EF, with a great design and productivity.

as I understand Matthieu is willing to let the EF team to take ownership over his labor and I believe that it is an opportunity that will be pity to waste (everything within this framework came out of Matthieu's field experience in order to ease his customer pain).

this post is a call to action, if you like this framework or the ideas it represent vote for it at:

http://data.uservoice.com/forums/72025-entity-framework-feature-suggestions/suggestions/3929780-waqs

EF 6: Async

bnaya — Tue, 19 Mar 2013 15:02:51 GMT

EF 6: Async

this post is the first in a series about what's new in EF 6.

great improvements are about to come with Entity Framework 6.
it is a major release and the first one since EF become an open source.
each post in the series will be dedicate to a single feature.

this post will focus on a new EF a-sync features.

the first question that should be asked is, why do we need parallel data access?
moreover why do we need a dedicate parallel data access API, rather then using the TPL Task.Run(…) in order to introduce parallelism?

the answer is related both to thread safety and ThreadPool optimization.

a dedicate data access API for parallel programming, ensured a thread safe access, which may not be a trivial task.

furthermore, when the database is executing a query the .NET thread is idle (waiting for the execution result).
if this idle thread has taken from ThreadPool (either directly or through high level API like TPL Task.Run(…)) it could lead to a ThreadPool starvation.

assuming that you have a 2 second query, Task.Run(…) will hold the ThreadPool's thread for 2 second even though it idle while the database is executing the query.

what you really want is an API that operate over IOCP (IO completion port). IOCP API will release the ThreadPool's thread back to the pool and re-claim a thread at the completion time.

at the API level you can find a-sync operations everywhere.
I will present some of the main API.

the following snippets, will use the following entities (code first):

Code Snippet
 public class Author
{
    public Author()
    {
        Blogs = new List<Blog>();
    }
    public int Id { get; set; }
    public string Name { get; set; }
    public IList<Blog> Blogs { get; set; }
}
 
public class Blog
{
    public int Id { get; set; }
    public string Title { get; set; }
    public string Content { get; set; }
    public int AuthorId { get; set; }
 
}
 

and the following context:

Code Snippet
 public class Context : DbContext
{
    public DbSet<Author> Authors { get; set; }
    public DbSet<Blog> Blogs { get; set; }
}
 

a-sync save

Code Snippet
 using (var context = new Context())
{
    context.Authors.Add(author);
    int affected = await context.SaveChangesAsync();
}
 

notice the await at line 4.

a-sync read

Code Snippet
 using (var context = new Context())
{
    Author[] authors = await context.Authors
        .Include(a => a.Blogs)
        .ToArrayAsync();
    return authors;
}
 

line 5, is where the magic happens.

execute command

Code Snippet
 using (var context = new Context())
{
    int affected = await 
        context.Database.ExecuteSqlCommandAsync("DELETE FROM BLOGS");
}
 

summary

EF 6 is having a great parallel API, which respect IOCP and compatible with the new async / await syntax.

Entity Framework - Pro and Con

bnaya — Sun, 13 Jan 2013 12:49:03 GMT

Entity Framework - Pro and Con

Consulting as part of a consultant group can lead to better consulting.
Sela Israel consultant group is around 40 consultants many of them are speakers in local and international conference.

It happens that one of our customers was having some issues around the usage of Entity Framework.
even those I was thinking that I'm having most of the answer for that customer I decided to use our internal channel to query what are my colleagues opinions about those issues and sea whether they can enlighten a broader spectrum.

finally we end up with a very broad comprehended which we believe can be beneficial for a broader audience. it was Sasha advice that we should put a summary of our internal conversation into a blog post.

I will start with a few words about our customer's business.
the customer is using Entity Framework in a distributed environments.

his question was divided into a general ones and more focus, I will introduce each question and a summary of different answers provided by my colleagues and me.

I may say that many answers can be also true for other ORM frameworks like NHibernate.

the questions:

High-Level:

What are the best justifications for changing an existing system to work with entity framework?

All-in-1 framework for mapping tables to classes, no need to write mapping code which is hard to maintain.
Maintainability, easy to understand code, no need for plumbing or creating big data access layers.
Provides LINQ queries over databases, which requires less understanding of SQL from junior developers.
EF can be used as the base for data services and OData.

What are the scenarios you would defiantly NOT recommend to use entity framework?

real-time apps.
Code can only access the database through stored procedures. EF advantage is LINQ and change tracking which is not used when working solely with SPs (even those EF does have limited support for SP).
Frequently insert operations (inserts). Bulk is not supported yet by EF.
Frequently update operation, mainly when the update is targeting multiple rows (with a single value)
for example: UPDATE TableName SET ColumA = 10 WHERE ColumnB = ?
this kind of update does better be used with ExecuteNonQuery (either from the context or directly from Ado.Net).
De-normalized tables and high performance queries. EF generates queries and they are hard to manipulate, and it doesn't map well to de-normalized tables.

We want to work in a method of loading ALL data into memory is that recommended using Entity framework? What problems should we expect?

Loading all entities will require many queries and a lot of time.
Memory overhead.
Latencies because of the EF change tracking and large collection handling.
EF contexts are not thread-safe, you should not use one context for the entire service.
If you do intend to do that, use EF for loading entities, but not for managing them (detach the entities from the content).
do not use the context as a cache object (for distributed scenarios)
it is not thread safe, having overheads and doesn't follow the idea of separation of concern. it is much better to use a designated cache API like AppFabric Cache.

Technical-Level:

Memory leaks:

We have a scenario where every 10 seconds we open a context to get a single table and close it. "using (entities context = new entities(_connection)) {… }"
this caused a memory leak over a period of a few hours.

I suggest you check the origin of the memory leaks to make sure it's from EF. EF is an open source now, so any memory leak can be reported and fixed.
Use the Profiler and VMMap in order of tracing the leak's root.

Should we keep the context open in this case?

It doesn't really matter, as the query will execute either way. However, it's preferable to close contexts so the cached entities will be GC'd, otherwise the context and the cached entities will be upgraded to Gen2 and will get stuck there for the duration of the service.
NO! Use context per request for WCF.
The recommended practice is to open a context per action because once an item is added to the DB your context Is outdated and might cause data corruption or duplicated items

Does the size of the Edmx effects this?

The size of Edmx affects the first creation of the context since all the Edmx data is loaded into memory.
The EDMX is loaded into the memory, parsed, and the mapping is created and cached in the AppDomain. You can close and reopen contexts without it affecting the cached Edmx mappings.

Working with large number of tables

Should all the tables be put in a single Edmx or split between several Edmx files?
How one can manage an edmx with large number of entities? for example, how one can find?

and modify a specific table in the designer?

If you split between Edmx files, you cannot connect entities one to the other (navigation properties).
You should build your model according to your entity model design – if you have groups of entities which are independent they can be in separate Edmx models.
EDMX files are known to handle a couple of hundreds of entity types. If the number of entity types is over 1000, I would consider splitting to several Edmxs (and rethink your entity model design, because thousands of entity types seems too much for one system).
Move to VS 2012 in order to have designer enhancements like split and coloring, see the following link.
The EF designer has several improvements, such as coloring entities to differentiate entities groups.
There are also 3-rd party entity designers that might give better UX.
And remember – EF is an open source, you can send them requests and even contribute your own additions to it, so search in EDMX designer sounds interesting.
you should split the Edmx to couple of Edmx files per domain/service.

Relationships between tables

Working in lazy vs. not lazy mode?

In general Lazy code is evil for distributed scenarios.
If they need data from couple of tables then use Include else you can use lazy loading.

Does the "Include" command load the entire table or only the records related to the original ObjectSet?

Include loads the columns of the related entity. EF doesn't use * in queries.

How can we perform bulk insert/update using EF?

You cannot do it with EF entities, you need to use raw SQL.

When an entity is marked as modified, EF updates all the columns?

It doesn't update all the columns, unless you are using distributed apps with self-tracking. The "normal" implementation of EF only updates modified columns. With self-tracking the mechanism does not keep track of which property was updated, so it marks all properties as updated. You can edit the self-tracking code and fix it if it's required.
You can attach SP in order to have a custom logic.
You can change the T4 file in order to maintain all original values in self-tracking scenario.

What's the overhead of entire raw updating?

Mostly DB overhead to recalculate indexes I presume. This is more a question to a DBA.

When we used EF to update an XML column with XML file which is 700-800KB, We received an exception related to Sql Server's TempDb. what can cause it?

Probably relates to how the database works with temporary tables for updating XML content.
You should have your database checked out by a DBA, perhaps you need to increase the size of your TempDb.
In general large object like those Xml will make a memory pressure on the GC's Large Object Heap, you should consider to move to .NET 4.5 where the GC does defragment the Large Object Heap.
You can consider to store the XML as byte[] which can be compressed (Xml is not a very economic format).

What's the overhead of using EF vs. ADO.Net?

The overhead of taking a data reader an transforming the results to objects, and the overhead of compiling SQL queries from LINQ.
You will have the same overhead and even more, if you try to build your own ORM framework. Don’t do that!
When you have abstraction layer you get overhead. The overhead depends on what they do with EF.
in general you should conceder to get away from Inheritance because it does have a greater overhead.

Credits:

I want to credits the main contribution for this conversation which was Ido Flatow, Gil Fink, Yaniv Rodenski, Amit Raz, Ofir Makmal, Tomer Shaiman and I'm hopping that I didn't miss anyone.

The hidden assumption of EF 4 self-tracking entity

bnaya — Thu, 21 Oct 2010 03:04:24 GMT

The hidden assumption of EF 4 self-tracking entity

recently I was diving into the Entity Framework self-tracking entity T4 and its generated code.

doing so I was finding some interesting assumptions made about the entity behaviors.

this post will illuminate some of those assumptions.

Background

EF self tracking entity is one of the major enhancement of EF 4.0,

it is a disconnected model (similar to the old DataSet) which enable tracking

the entity state (Added, Modified, Deleted and Unchanged) while the entity

is detached from the EF ObjectContext.

the code snippet of this post will refer the following very simple edmx:

Assumptions

1st assumption

by default the tracking is disable (ChangeTrackingEnabled = false).

2nd assumption

trying to change the entity state while the tracking is disable

Code Snippet
 var p = new Person();
p.ChangeTracker.State = ObjectState.Unchanged;
 

will have no affect on the entity state (the state will remain Added which is the default

state for new entities).

the following snippet shows the code of the State property:

T4 generated code
 public ObjectState State
{
    get { return _objectState; }
    set
    {
        if (_isDeserializing || _changeTrackingEnabled)
        {
            OnObjectStateChanging(value);
            _objectState = value;
        }
    }
}
 

3rd assumption

the entity enable the tracking after deserialization.

T4 generated code
 [OnDeserialized]
public void OnDeserializedMethod(StreamingContext context)
{
    IsDeserializing = false;
    ChangeTracker.ChangeTrackingEnabled = true;
}
 

4th assumption

MarkAsAdded, MarkAsModified, MarkAsDeleted and MarkAsUnchanged all enable the trucking.

T4 generated code
 public static T MarkAsAdded<T>(this T trackingItem) where T : IObjectWithChangeTracker
{    
    trackingItem.ChangeTracker.ChangeTrackingEnabled = true;
    trackingItem.ChangeTracker.State = ObjectState.Added;
    return trackingItem;
}
 

5th assumption

State = ObjectState.Unchanged, MarkAsUnchanged and AcceptChanges

has different behaviors:

in general State property: is only changing when tracking is enabled,

MarckAsUnchanged: enable the tracking and

AcceptChanges: clear the original values, enable the tracking and

reset the relationship tracking (added and removed related entities).

State: T4 generated code
 public ObjectState State
{
    get { return _objectState; }
    set
    {
        if (_isDeserializing || _changeTrackingEnabled)
        {
            OnObjectStateChanging(value);
            _objectState = value;
        }
    }
}
 

MarkAsUnchanged: T4
 public static T MarkAsUnchanged<T>(this T trackingItem) where T : IObjectWithChangeTracker
{    
    trackingItem.ChangeTracker.ChangeTrackingEnabled = true;
    trackingItem.ChangeTracker.State = ObjectState.Unchanged;
    return trackingItem;
}
 

AcceptChang: T4 generated code
 public void AcceptChanges()
{
    OnObjectStateChanging(ObjectState.Unchanged);
    OriginalValues.Clear();
    ObjectsAddedToCollectionProperties.Clear();
    ObjectsRemovedFromCollectionProperties.Clear();
    ChangeTrackingEnabled = true;
    _objectState = ObjectState.Unchanged;
}
 

Summary

the self tracking entity is a fairly young concept in the EF world

and we can expect that it will keep changing.

as I was reading in one of the Harry Potter's book

"do not trust anything you cannot understand", so be aware of that

assumption and use it well.

Bnaya Eshet : Entity Framework

EF should adopt the WAQS framework

EF should adopt the WAQS framework

EF 6: Async

EF 6: Async

a-sync save

a-sync read

execute command

summary

Entity Framework - Pro and Con

Entity Framework - Pro and Con

the questions:

High-Level:

What are the best justifications for changing an existing system to work with entity framework?

What are the scenarios you would defiantly NOT recommend to use entity framework?

We want to work in a method of loading ALL data into memory is that recommended using Entity framework? What problems should we expect?

Technical-Level:

Memory leaks:

Should we keep the context open in this case?

Does the size of the Edmx effects this?

Working with large number of tables

Should all the tables be put in a single Edmx or split between several Edmx files? How one can manage an edmx with large number of entities? for example, how one can find?

Relationships between tables

Working in lazy vs. not lazy mode?

Does the "Include" command load the entire table or only the records related to the original ObjectSet?

How can we perform bulk insert/update using EF?

When an entity is marked as modified, EF updates all the columns?

What's the overhead of entire raw updating?

When we used EF to update an XML column with XML file which is 700-800KB, We received an exception related to Sql Server's TempDb. what can cause it?

What's the overhead of using EF vs. ADO.Net?

Credits:

The hidden assumption of EF 4 self-tracking entity

The hidden assumption of EF 4 self-tracking entity

Background

Assumptions

1st assumption

2nd assumption

3rd assumption

4th assumption

5th assumption

Summary

Should all the tables be put in a single Edmx or split between several Edmx files?
How one can manage an edmx with large number of entities? for example, how one can find?