Reduce granularity of histogram buckets for Go 1.17 collector #974
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 task
bwplotka
approved these changes
Jan 25, 2022
| @@ -37,3 +37,5 @@ var expectedRuntimeMetrics = map[string]string{ | |||
| "/sched/goroutines:goroutines": "go_sched_goroutines_goroutines", | |||
| "/sched/latencies:seconds": "go_sched_latencies_seconds", | |||
| } | |||
|
|
|||
| const expectedRuntimeMetricsCardinality = 79 | |||
kakkoyun
reviewed
Jan 27, 2022
There was a problem hiding this comment.
LGTM. Thanks @mknyszek for taking care of this.
As @bwplotka, it would be nice add more explanation comments, it'd be easier for us to maintain in the long run.
Also we might consider handling #967 (comment) as well before the next patch release. Not necessarily as part of this PR though.
The Go runtime/metrics package currently exports extremely granular histograms. Exponentially bucket any histogram with unit "seconds" or "bytes" instead to dramatically reduce the number of buckets, and thus the number of metrics. This change also adds a test to check for expected cardinality to prevent cardinality surprises in the future. Signed-off-by: Michael Anthony Knyszek <[email protected]>
mknyszek
force-pushed
the
runtime-metrics-hist
branch
from
030b81a
to
278b704
Compare
kakkoyun
approved these changes
Jan 28, 2022
kakkoyun
pushed a commit
that referenced
this pull request
May 13, 2022
The Go runtime/metrics package currently exports extremely granular histograms. Exponentially bucket any histogram with unit "seconds" or "bytes" instead to dramatically reduce the number of buckets, and thus the number of metrics. This change also adds a test to check for expected cardinality to prevent cardinality surprises in the future. Signed-off-by: Michael Anthony Knyszek <[email protected]>
kakkoyun
added a commit
that referenced
this pull request
Jul 6, 2022
* Cut v1.12.0 Signed-off-by: Kemal Akkoyun <[email protected]> * Bump the day Signed-off-by: Kemal Akkoyun <[email protected]> * Make the Go 1.17 collector thread-safe (#969) * Use simpler locking in the Go 1.17 collector (#975) A previous PR made it so that the Go 1.17 collector locked only around uses of rmSampleBuf, but really that means that Metric values may be sent over the channel containing some values from future metrics.Read calls. While generally-speaking this isn't a problem, we lose any consistency guarantees provided by the runtime/metrics package. Also, that optimization to not just lock around all of Collect was premature. Truthfully, Collect is called relatively infrequently, and its critical path is fairly fast (10s of µs). To prove it, this change also adds a benchmark. name old time/op new time/op delta GoCollector-16 43.7µs ± 2% 43.2µs ± 2% ~ (p=0.190 n=9+9) Note that because the benchmark is single-threaded it actually looks like it might be getting *slightly* faster, because all those Collect calls for the Metrics are direct calls instead of interface calls. Signed-off-by: Michael Anthony Knyszek <[email protected]> * API client: make http reads more efficient (#976) Replace `io.ReadAll` with `bytes.Buffer.ReadFrom`. Both need to resize a buffer until they have finished reading; the former increases by 1.25x each time while the latter uses 2x. Also added a benchmark to demonstrate the benefit: name old time/op new time/op delta Client/4KB-8 35.9µs ± 4% 35.3µs ± 3% ~ (p=0.310 n=5+5) Client/50KB-8 83.1µs ± 8% 69.5µs ± 1% -16.37% (p=0.008 n=5+5) Client/1000KB-8 891µs ± 6% 750µs ± 0% -15.83% (p=0.016 n=5+4) Client/2000KB-8 1.74ms ± 2% 1.35ms ± 1% -22.72% (p=0.008 n=5+5) name old alloc/op new alloc/op delta Client/4KB-8 20.2kB ± 0% 20.4kB ± 0% +1.26% (p=0.008 n=5+5) Client/50KB-8 218kB ± 0% 136kB ± 0% -37.65% (p=0.008 n=5+5) Client/1000KB-8 5.88MB ± 0% 2.11MB ± 0% -64.10% (p=0.008 n=5+5) Client/2000KB-8 11.7MB ± 0% 4.2MB ± 0% -63.93% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Client/4KB-8 75.0 ± 0% 72.0 ± 0% -4.00% (p=0.008 n=5+5) Client/50KB-8 109 ± 0% 98 ± 0% -10.09% (p=0.079 n=4+5) Client/1000KB-8 617 ± 0% 593 ± 0% -3.89% (p=0.008 n=5+5) Client/2000KB-8 1.13k ± 0% 1.09k ± 0% -3.27% (p=0.008 n=5+5) Signed-off-by: Bryan Boreham <[email protected]> * Reduce granularity of histogram buckets for Go 1.17 collector (#974) The Go runtime/metrics package currently exports extremely granular histograms. Exponentially bucket any histogram with unit "seconds" or "bytes" instead to dramatically reduce the number of buckets, and thus the number of metrics. This change also adds a test to check for expected cardinality to prevent cardinality surprises in the future. Signed-off-by: Michael Anthony Knyszek <[email protected]> * Cut v1.12.1 (#978) * Cut v1.12.1 Signed-off-by: Kemal Akkoyun <[email protected]> * Apply review suggestions Signed-off-by: Kemal Akkoyun <[email protected]> * Fix deprecated `NewBuildInfoCollector` API Update `examples/random/main.go`: `prometheus.NewBuildInfoCollector` is deprecated. Use `collectors.NewBuildInfoCollector` instead. Signed-off-by: alissa-tung <[email protected]> * gocollector: Added options to Go Collector for changing the (#1031) * Renamed files. Signed-off-by: Bartlomiej Plotka <[email protected]> * gocollector: Added options to Go Collector for diffetent collections. Fixes #983 Also: * fixed TestMemStatsEquivalence, it was noop before (: * Removed gc_cpu_fraction metric completely, since it's not working completely for Go1.17+ Signed-off-by: Bartlomiej Plotka <[email protected]> * gocollector: Reverted client_golang v1.12 addition of runtime/metrics metrics by default. (#1033) Fixes #967 Signed-off-by: Bartlomiej Plotka <[email protected]> * prometheus: Fix convention violating names for generated collector metrics (#1048) * Fix convention violating names for generated collector metrics Signed-off-by: Kemal Akkoyun <[email protected]> * Add new Go collector example Signed-off-by: Kemal Akkoyun <[email protected]> * Remove -Inf buckets from go collector histograms (#1049) * Remove -Inf buckets from go collector histograms Signed-off-by: Kemal Akkoyun <[email protected]> * Update prometheus/collectors/go_collector_latest_test.go Co-authored-by: Bartlomiej Plotka <[email protected]> Signed-off-by: Kemal Akkoyun <[email protected]> * Simplify Signed-off-by: Kemal Akkoyun <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]> * Cut v1.12.2 (#1052) * Cut v1.12.2 Signed-off-by: Kemal Akkoyun <[email protected]> * Apply suggestions Signed-off-by: Kemal Akkoyun <[email protected]> * Update CHANGELOG.md Co-authored-by: Bartlomiej Plotka <[email protected]> Signed-off-by: Kemal Akkoyun <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]> Co-authored-by: Kemal Akkoyun <[email protected]> Co-authored-by: Michael Knyszek <[email protected]> Co-authored-by: Bryan Boreham <[email protected]> Co-authored-by: Kemal Akkoyun <[email protected]> Co-authored-by: alissa-tung <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The Go runtime/metrics package currently exports extremely granular
histograms. Exponentially bucket any histogram with unit "seconds"
or "bytes" instead to dramatically reduce the number of buckets, and
thus the number of metrics.
This change also adds a test to check for expected cardinality to
prevent cardinality surprises in the future.
Fixes #967
Signed-off-by: Michael Anthony Knyszek [email protected]