1. Jacob Reimers
  2. Linq2Rest
Issue #12 resolved

Filter only returns IEnumerable<object> instead if IEnumerable<T>

Bernhard Millauer
created an issue

Hi, I try to use your framework to limit the result set from the database and then doing some projecting stuff.

Because your filter method only returns IEnumerable<object> I cant modify the query after that. instead I have to materialize the query first (selecting full objects with all fields) and then do my stuff, but this is not the way how i want to do it because it is expensive.

anyway, is there a way to get a typed IEnumerable<T> instead of object? that would be awesome and deciding.

[EDIT 1] you could also constraint the T to class. [EDIT 2] .Cast<MyObj>() does not work

Thanks!

Comments (29)

  1. Jacob Reimers repo owner

    This is not a bug, but by design. It has to return IEnumerable<object> because it has to support projections, i.e. user can for example query: source.Select(x=> new { x.Title }).

    The problem is that the server can't know what the resulting type of a projection is, because it is defined on the client system. Linq2Rest is able to create a type with a similar signature at runtime, but it is not T. Therefore it has to fall back to object.

    If you only want to work with T, then use the OfType<T>() method and continue from there. Just be warned, that if someone does request a projection, then you will get an empty enumeration.

  2. Bernhard Millauer reporter

    Thanks for the explaination and it makes sense, but i do not allow the user to get his own projection. i mainly use your framework for filter and pagination in the database.

    To my setup: I'm using EF4 with proxy creation.

    my calls look like that:

    var postContext = queryableContext.Posts;
    var tenantFiltered = postContext.Where(x => x.TenantId == currentTenant.Id);
    var odataFiltered = tenantFiltered.Filter(HttpContext.Current.Request.Params);
    var oftype = odataFiltered.OfType<Post>();
    var linq = (from post in oftype select new { post.Id, UserName = post.User.Name })
    var materialized = linq.ToList();
    

    I do not know why but if I pass my DbSet<Post> through your filter, the proxy is no more able to build a query to get the User's name (post.User is null). If i do not use your filter, everything is fine and i can continue modifying the thing.

  3. Jacob Reimers repo owner

    First of all, it's not a question of whether or not you allow the user to use projections. It is enabled for anyone querying directly against you endpoint. That is the OData standard.

    On your concrete problem, then you will need to send me some code sample, because I can't see anything specifically wrong with the code you posted, so I will need to write up some tests against an actual example. Can you send the sample to [MyFirstName]@[MyLastName].dk?

  4. Jacob Reimers repo owner

    I ran the test below and it passes. Is there anything in your concrete scenario that differs materially from the test setup?

    [TestFixture]
    public class ModelFilterTests
    {
    	[Test]
    	public void CanPerformFilteringBeforeApplyingModelFilter()
    	{
    		var posts = new[] { new Post { TenantID = 1, Title = "blah", User = new User { Name = "User" } } };
    
    		var parameters = new NameValueCollection { { "$filter", "Title eq 'blah'" } };
    
    		var filteredSource = posts
    			.Where(x => x.TenantID == 1)
    			.Filter(parameters)
    			.OfType<Post>()
    			.Select(x => new { Title = x.Title, Username = x.User.Name });
    
    		Assert.AreEqual(1, filteredSource.Count());
    	}
    
    	private class Post
    	{
    		public int TenantID { get; set; }
    
    		public string Title { get; set; }
    
    		public User User { get; set; }
    	}
    
    	private class User
    	{
    		public string Name { get; set; }
    	}
    }
    
  5. Bernhard Millauer reporter

    Definitely yes, my datasource is not evaluated/materialized before your filter applies. My setup in detail:

    // just a reference to the query table
    var postContext = queryableContext.Posts;
    // apply "where" expression
    var tenantFiltered = postContext.Where(x => x.TenantId == currentTenant.Id);
    // apply your "where, order, skip, top" expressions
    var odataFiltered = tenantFiltered.Filter(HttpContext.Current.Request.Params);
    // change the type to post again
    var oftype = odataFiltered.OfType<Post>();
    // extend the query with the projection
    var linq = (from post in oftype select new { post.Id, UserName = post.User.Name })
    // with the call .ToList(), the query will be build, sent to the database and materalized.
    var materialized = linq.ToList();
    

    I think the aspekt of queryable will be lost for later modifications.

    We changed the chain so that your filter will be applied as the latest step before we materialize and everything works now. so we write queries against the generated model/projection instead against the database. this feels more natural for the consumer and he only have to know one schema.

    I'm fine if you are are not interested to fix the problem now :), otherwise I can try to create a setup for you.

    Thanks, Bernhard

  6. Jacob Reimers repo owner

    It should be fairly easy to reproduce in the MVC site included in the source code. It also uses EF to query the database context. I would like to fix the problem, if there is in fact a problem, but I'll need some concrete setup to work with.

  7. Bernhard Millauer reporter

    Hi, don't rush please, the weekend is not for working :)

    I've tried to use your test projects but I was unable to use them. So I set my own project up that reflects my current setup and I found the problem!

    Problem setup (SimpleLinqProjectionWithEarlyLinq2RestTest_LazyLoadingOff):
    - Disable LazyLoading
    - Filter with Linq2Rest
    - Create Projection (with navigation property use!)
    - Materialize
    => the navigation property is null
    
    Working setup (SimpleLinqProjectionWithLaterLinq2RestTest):
    - Disable LazyLoading
    - Create Projection (with navigation property use!)
    - Filter with Linq2Rest
    - Materialize
    => the navigation property is evaluated
    

    The problem is that lazy loading under the circumstances of example 1 is a problem. There is also a different behavior in the tests SimpleLinqProjectionTest_MaterializationWithLazy_ExplorationWithoutLazy and SimpleLinqProjectionWithEarlyLinq2RestTest_LazyLoadingOnThenOff. If i use Linq2Rest in the chain, the Navigation properties are proxies and the User's posts are evaluated. If I don't use Linq2Rest, the User's posts are empty.

    Suddenly, I cant fix the problem :(

    Why i disable lazy loading: after materialization I use automapper / recursive methods to transform the entities.

    I hope you will find my tests useful!

    wbr, Bernhard

  8. Bernhard Millauer reporter

    Depending on the tests i sent you:

    var dbContext = new Database1Entities();
    var posts = dbContext.Posts;
    var linq = from post in posts select new { PostId = post.Id, User = post.User };
    
    Console.WriteLine(linq.Expression.ToString());
    // output [1]
    
    Console.WriteLine(linq.ToString());
    // output [2]
    
    var willBreak = from post in posts select new { PostId = post.Id, User = post.User, Dict = post.User.Posts.ToDictionary(x => x.Id, x => x.Value) };
    Console.WriteLine(willBreak.ToString());
    // exception [3]
    
    output [1] (Generated Expression):
    -------------------------------------------
    value(System.Data.Objects.ObjectQuery`1[Linq2RestTest.EF.Post]).MergeAs(AppendOnly) .Select(post => new <>f__AnonymousType1`2(PostId = post.Id, User = post.User))
    
    output [2] (Generated SQL Script):
    -------------------------------------------
    SELECT 
    [Extent1].[Id] AS [Id], 
    [Extent2].[Id] AS [Id1], 
    [Extent2].[Name] AS [Name]
    FROM  [Post] AS [Extent1]
    INNER JOIN [User] AS [Extent2] ON [Extent1].[UserId] = [Extent2].[Id]
    
    exception [3] (Exception Message because of using invalid method ToDictionary in expression):
    -------------------------------------------
    System.NotSupportedException : LINQ to Entities does not recognize the method 'System.Collections.Generic.Dictionary`2[System.Int32,System.String] ToDictionary[Post,Int32,String](System.Collections.Generic.IEnumerable`1[Linq2RestTest.EF.Post], System.Func`2[Linq2RestTest.EF.Post,System.Int32], System.Func`2[Linq2RestTest.EF.Post,System.String])' method, and this method cannot be translated into a store expression.
    

    you can get the expression visualization by calling queryable.Expression.ToString(), the generated query by calling queryable.ToString() and also the evaluation if a method / expression is supported.

    does this explain it better?

  9. Jacob Reimers repo owner

    Yes, thank you. One thing I have found out digging into the innards of Entity Framework is that it does have some limitations when it comes to type handling. I don't know if this is related to the way it creates SQL scripts or what, but in your case you are probably better off materializing the database query before you attempt to make dictionaries inside anonymous types.

    In the ModelFilter class I had to put eager evaluation in place before it leaves the filter to avoid a "bug" (?) whereby Entity Framework is unable to filter to System.Object. For this reason the ModelFilter will no longer return an ObjectQuery queryable under the hood, but actually return a TypeIterator. This is actually low level details, but it will explain why LINQ queries used after the ModelFilter will not be visible in the SQL profiler queries. When projecting there is not the same issue, but you may still want to call ToArray in the returned IEnumerable<object> in order to avoid more advanced projection issues like in your case.

    Short story, don't think that Entity Framework behaves exactly like LINQ to Objects.

  10. Bernhard Millauer reporter

    Hi Jacob. i was digging in your SortExpressionFactory and found out that you are trying to create expressions/functions (no more compiled yet :P) and you are forced to add a Expression.Convert call to it. suddenly, this convert is not supported by the database (eg. convert datetime to object).

    why do you not just create the expression that has the correct type and apply it on the fly to the queryable? then you would not be forced to create a func<t, object> and the query would look correct after applying the expressions.

  11. Jacob Reimers repo owner

    That is actually the change I did yesterday in relation to your other issue. However, it's easier said than done to "just create the expression that has the correct type". LINQ to Objects supports using a Func<T,object> as key selector, but Entity Framework does not support working with object types, so I had to do a fair bit of reflection work to find the appropriate expression type, create it and invoke the ordering methods.

    Remember I do actually accept pull requests, so if you know how to solve a problem, please just send me a pull request and I'll be happy to pull.

  12. Bernhard Millauer reporter

    I played alot today with your implementation but I do not want to break your system, so I've setup my 'own' methods by extracting your stuff and modified them. one reason why I can't send you a pull request is that i do not have .net 4.5 and so I can't test all your stuff.

    I finished the reflection stuff one minute ago and it looks correct. is there a way how I can communicate directly? I think this would make the workflow easier and faster. otherwise I could send you my implementation.

    it's basicly now an Apply method instead a Create method on the SortExpressionFactory like this:

    queryable = new SortExpressionFactoryForQueryable().Apply(parameters["$orderby"], queryable);
    
  13. Jacob Reimers repo owner

    Pushed the latest changes and ran your test project. The tests pass except the following 2:

    [Test]
    public void SimpleLinqProjectionWithEarlyLinq2RestTest_LazyLoadingOnThenOff()
    {
    	// arrange
    	var dbContext = new Database1Entities();
    	dbContext.Configuration.LazyLoadingEnabled = true;
    
    	var posts = dbContext.Posts;
    
    	NameValueCollection requestParams = new NameValueCollection();
    	requestParams.Add("$filter", "Value eq 'Post A'");
    
    	// act
    	var postFiltered = posts.Filter(requestParams);
    
    	var linq = from post in postFiltered.OfType<Post>()
    		   select new { PostId = post.Id, UserName = post.User.Name, User = post.User };
    
    	// assert
    	var materialization = linq.ToList();
    
    	dbContext.Configuration.LazyLoadingEnabled = false;
    
    	Assert.IsNotEmpty(materialization);
    	Assert.AreEqual(1, materialization.Count);
    	Assert.IsEmpty(materialization[0].User.Posts);
    }
    

    However if I change the last assert to IsNotEmpty, then it passes. I assume that the intent is to get the user's posts, so the assert is wrong as I see it.

    The other test:

    [Test]
    public void SimpleLinqProjectionWithEarlyLinq2RestTest_LazyLoadingOff()
    {
    	// arrange
    	var dbContext = new Database1Entities();
    	dbContext.Configuration.LazyLoadingEnabled = false;
    
    	var posts = dbContext.Posts;
    
    	NameValueCollection requestParams = new NameValueCollection();
    	requestParams.Add("$filter", "Value eq 'Post A'");
    
    	// act
    	var postFiltered = posts.Filter(requestParams);
    
    	var linq = from post in postFiltered.OfType<Post>()
    			   select new { PostId = post.Id, UserName = post.User.Name };
    
    	// assert
    	var materialization = linq.ToList();
    	Assert.IsNotEmpty(materialization);
    	Assert.AreEqual(1, materialization.Count);
    }
    

    Here the problem seems to be related to the lazy loading of the user. Because if I take Linq2Rest out of the picture, it still fails.

    [Test]
    public void SimpleLinqProjectionWithEarlyLinq2RestTest_LazyLoadingOff()
    {
    	// arrange
    	var dbContext = new Database1Entities();
    	dbContext.Configuration.LazyLoadingEnabled = false;
    
    	var posts = dbContext.Posts;
    	var postFiltered = posts.ToArray();
    
    	var linq = from post in postFiltered.OfType<Post>()
    			   select new { PostId = post.Id, UserName = post.User.Name };
    
    	// assert
    	var materialization = linq.ToList();
    	Assert.IsNotEmpty(materialization);
    	Assert.AreEqual(1, materialization.Count);
    }
    

    You can try to pull the latest changes and see what you make of it.

  14. Jacob Reimers repo owner

    No, I'm not going to publish beta packages, for code that is in flux. Sorry. You can run the build script and it'll create a package for you. Another approach is to create a package manually if that is a requirement for your system.

    The easiest thing is to simply drop in your locally built assemblies into the package folder of your solution and that way replace the nuget package content.

  15. Bernhard Millauer reporter

    I've tried to create the binaries, but the cccheck process (ms contracts) runs forever and then vs2010 freezes :(... I can't disable the contracts because I got errors otherwis.

  16. Log in to comment