Set operation support (except navigation/include)

[roji]

This is a first draft for set operation support (UNION/UNION ALL/INTERSECT/EXCEPT). @smitpatel I'm looking for feedback on the general direction before I go any further etc.

Some important notes on database-specific quirks/capabilities (which have informed the design) have been posted in #6812 (comment).

Design notes

Set operations are currently represented as a special kind of SelectExpression, with an enum marker has been added (SetOperationType). Each such SelectExpression represents a single set operation with exactly two operands.

RelationalQueryableMethodTranslatingExpressionVisitor performs pushdown of set operations into a subquery on most LINQ operations, in one place. As part of this, I have refactored translation of queryable operators into their own method.

[smitpatel]

Need a lot more testing on combination of shapes which are not same in SelectExpression.
Also non-entity projections combined.

[roji]

@smitpatel @maumar here's a new version in a much more advanced state. Various projections and hierarchy edge cases are now working, take a look at the tests.

Some notes on siblings/hierarchy:

• Performing a set operation over siblings is complicated (especially when there are other unrelated siblings). Because the result shape needs to be identical on both sides, I generate null constant projections for properties of the other entity type.
• More complicated; the entity type coming out of the operation is (currently) the closest common ancestor. In order for that to work I generate null constant projections even for properties which neither sibling actually requires (the shaper doesn't know only sibling instances will be coming from the database).
• Usually projections are duplicated (why load the same thing twice), but we can't deduplicate nulls, since their point is to make the query shape identical on both sides.
• Most importantly, since the set operation changes the result shape (from sibling to common ancestor), RelationalQueryableMethodTranslatingExpressionVisitor.TranslateSetOperation() sets a new ShapedQueryExpression (a bit like CombineShapers for join). This feels hacky/wrong, please provide feedback...

Some other notes:

• The core logic is SelectExpression.ApplySetOperation.
• Factoring of QueryableMethodTranslatingExpressionVisitor and SelectExpression, i.e. how/where to implement the pushdown of set operations.
• Have scoped out all nav rewriting support (but intend to work on that as soon as this is done).

[roji]

@smitpatel, regarding application of set operations on SelectExpression with already-populated projections (this is the corresponding code in PushdownIntoSubquery), I couldn't repro this with the test below. For now I will simply place a NotImplementedExpression, we can always do this in a separate PR.

public virtual Task Union_client_eval_FirstOrDefault(bool isAsync)
    => AssertFirstOrDefault<Customer>(isAsync, cs => cs
        .Where(c => c.City == "Berlin")
        .Union(cs.Where(c => c.City == "London"))
        .OrderBy(c => c.Address)
        .Select(c => Dummy.GetCustomerAddress(c)));

[roji]

OK guys, this seems more or less ready to me. It doesn't contain navigation/include support; @maumar if you want to take a look at this, go ahead, otherwise tomorrow morning I'll be looking at it - please let me know if you're on it so we don't do double work.

Hopefully we can get this in for preview7...

[smitpatel]

Since we are not doing a lot of process in between, just use using syntax for Indent

[roji]

We only want to indent for subqueries (when alias != null)...

[roji]

@smitpatel merged although there are some open questions, in order to make sure this gets into preview7. Let's continue the conversations on this issue and I'll fix as necessary.