Query: Navigation Expansion take 2

[smitpatel]

• Add TransparentIdentifier to uniquify them across the pipeline
• Convert all Enumerable based method which we can translate to Queryable
• Add QueryableMethodProvider to get methodInfos on Queryable robustly
• Split out subquery member pushdown out of navigation expansion
• Lastly expand navigations

[smitpatel]

This is still WIP. I will complete the work next week. Just wanted to get it out to get early feedback.
Missing pieces
- GroupJoin/GroupBy/DefaultIfEmpty are not handled yet. (tests related to them are failing right now).
- InMemory has few test failure with must be reducible nodes & Cosmos fails on convert node around entityType before accessing navigations. Though only handful of tests.
So overall design seems to be working.
I haven't done clean up yet. nit comments can wait.

How it works (on high level)

• EntityReference tracks entity in current result type
• IncludeTreeNode contains include tree which would be passed upon when expanding navigations
• NavigationTreeNode is a binary tree structure for transparent identifier. The point is, you create it once and then forget it. All join would just expand the tree further. And the initial tree will automatically get outer/inner based path. This saves a lot of processing in terms of remapping.
• Includes are applied at top level only so no need to remove them or anything.
• Eager loaded navigations added to includeTree as soon as the queryable creates new EntityReference. It prunes itself as shape changes.

While this has not fixed any issue, it lays groundwork for many

• Query filters can be applied easily since there are no mapping issues. Same is the case with defining queries.
• Query: Nav rewrite type mismatch #16076 We inject AsQueryable to convert Enumerable methods to Queryable. So if collection navigation is not followed by appropriate queryable method, we would preserve collection materialization
• Since include is applied at very top level, we can make a quick check on the tree for Query: Include cycles in non-tracking queries #16225
• LeftJoin is taken care of Query: Nav expansion should have include on outermost projection #16229
• Query: remove legacy codepath in NavigationExpansionHelpers.AddNavigationJoin #16324 is dead
• With MaterializeCollectionNavigation & AsQueryable Cosmos: Collection rewrite in navigation causes type error #16392 should be fixed
• Fixes Eager loaded navigations should apply Include during expansion phase #16619 by expanding eager loaded navigations at start
• Nav expansion throw for eager loaded include on derived type when projection has anonymous type #16622 should go away since we use EF.Property for owned navigations in Include & we also inject convert nodes around entityExpression
  - Added QueryableMethodProvider to get queryable methods to fix Compare to all Queryable/Enumerable methods via MethodInfo, not by name #16300

[roji]

Here is a partial review - I haven't gone over the core of the changes yet but will hopefully do that very soon.

[ajcvickers]

Remember to re-target to release/3.0 before merging.

Re-targeted

[smitpatel]

This is ready for review.

[smitpatel]

@AndriySvyryd - Also removed calling subquery member pushdown twice. Now it runs only after collection navigation is expanded and generating key access over FirstOrDefault.

[maumar]

[smitpatel]

Everything passed. If anyone wants to give a re-review, feel free. I will wait till tomorrow noon for merge.