-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-48773] Document config "spark.default.parallelism" by config builder framework #47171
[SPARK-48773] Document config "spark.default.parallelism" by config builder framework #47171
Conversation
"For distributed shuffle operations like reduceByKey and join, the largest number of " + | ||
"partitions in a parent RDD. For operations like parallelize with no parent RDDs, " + | ||
"it depends on the cluster manager: Local mode: number of cores on the local machine " + | ||
"Mesos fine grained mode: 8 Others: total number of cores on all executor nodes or 2, " + |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
mesos has been removed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated
@@ -42,6 +42,17 @@ package object config { | |||
private[spark] val SPARK_TASK_PREFIX = "spark.task" | |||
private[spark] val LISTENER_BUS_EVENT_QUEUE_PREFIX = "spark.scheduler.listenerbus.eventqueue" | |||
|
|||
private[spark] val DEFAULT_PARALLELISM = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we replace all hardcoded ones with this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
I also found PySpark side is using the same string. However I am not sure how PySpark generally deal with configs so leave PySpark side unchanged.
core/src/main/scala/org/apache/spark/internal/config/package.scala
Outdated
Show resolved
Hide resolved
…cala Co-authored-by: allisonwang-db <[email protected]>
"it depends on the cluster manager. For example in Local mode, it defaults to the " + | ||
"number of cores on the local machine") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we now do not have fine-grained schedulers, both the local and others are using total number of cores?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is something I am not sure. @cloud-fan do you remember?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think so, but we don't need to expose such low-level details to end users.
thanks, merging to master! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you, @amaliujia and all!
…uilder framework ### What changes were proposed in this pull request? Document config "spark.default.parallelism". This is Spark used config but not documented by config builder framework. This config is already in spark website: https://spark.apache.org/docs/latest/configuration.html. ### Why are the changes needed? Document Spark's config. ### Does this PR introduce _any_ user-facing change? NO. ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? N/A Closes apache#47171 from amaliujia/document_spark_default_paramllel. Authored-by: Rui Wang <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
…uilder framework ### What changes were proposed in this pull request? Document config "spark.default.parallelism". This is Spark used config but not documented by config builder framework. This config is already in spark website: https://spark.apache.org/docs/latest/configuration.html. ### Why are the changes needed? Document Spark's config. ### Does this PR introduce _any_ user-facing change? NO. ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? N/A Closes apache#47171 from amaliujia/document_spark_default_paramllel. Authored-by: Rui Wang <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
What changes were proposed in this pull request?
Document config "spark.default.parallelism". This is Spark used config but not documented by config builder framework. This config is already in spark website: https://spark.apache.org/docs/latest/configuration.html.
Why are the changes needed?
Document Spark's config.
Does this PR introduce any user-facing change?
NO.
How was this patch tested?
N/A
Was this patch authored or co-authored using generative AI tooling?
N/A