[SPARK-32951][SQL] Foldable propagation from Aggregate #29816
Conversation
|
Test build #128933 has finished for PR 29816 at commit
|
peter-toth
changed the title
peter-toth
changed the title
| /** | ||
| * List of all [[UnaryNode]]s which allow foldable propagation. | ||
| */ | ||
| private def canPropagateFoldables(u: UnaryNode): Boolean = u match { | ||
| // Handling `Project` is moved to `propagateFoldables`. | ||
| case _: Filter => true | ||
| case _: SubqueryAlias => true | ||
| case _: Aggregate => true | ||
| // Handling `Aggregate` is moved to `propagateFoldables`. |
There was a problem hiding this comment.
can we merge this comment with // Handling `Project` is moved to `propagateFoldables`.?
There was a problem hiding this comment.
Ok, will change it.
I need to look into plan stability failures first. |
|
you can just regenerate the golden file and push the change. Then the reviewers can look at the plan change and approve it if it's fine. |
Thanks. I did it already but haven't had time to check the changes. Anyway, added it as a new commit. |
| @@ -73,19 +73,19 @@ Input [2]: [i_manufact#2, count#9] | |||
| Keys [1]: [i_manufact#2] | |||
| Functions [1]: [count(1)] | |||
| Aggregate Attributes [1]: [count(1)#11] | |||
| Results [3]: [count(1)#11 AS item_cnt#12, i_manufact#2 AS i_manufact#2#13, true AS alwaysTrue#14] | |||
| Results [2]: [count(1)#11 AS item_cnt#12, i_manufact#2 AS i_manufact#2#13] | |||
There was a problem hiding this comment.
This file is a positive change.
| Input [2]: [item_cnt#12, i_manufact#2#13] | ||
| Condition : (item_cnt#12 > 0) | ||
|
|
||
| (13) Project [codegen id : 2] | ||
| Output [1]: [i_manufact#2#13] | ||
| Input [3]: [item_cnt#12, i_manufact#2#13, alwaysTrue#14] | ||
| Input [2]: [item_cnt#12, i_manufact#2#13] |
|
Please remove [WIP] before merging. |
peter-toth
changed the title
Done, thanks. |
|
Test build #128943 has finished for PR 29816 at commit
|
There was a problem hiding this comment.
+1, LGTM. Thank you, @peter-toth , @cloud-fan , and @viirya .
Merged to master for Apache Spark 3.1.0 on December 2020.
|
late LGTM and thanks for the update, @peter-toth ! |
What changes were proposed in this pull request?
This PR adds foldable propagation from
Aggregateas per: #29771 (comment)Why are the changes needed?
This is an improvement as
Aggregate'saggregateExpressionscan contain foldables that can be propagated up.Does this PR introduce any user-facing change?
No.
How was this patch tested?
New UT.