Store: improve index header reading performance by sorting values

[damnever]

Users may construct a promQL like metric{key=~"b|z|..tens of thousands..|a"} with Grafana variables or code, which is extremely slow.

In our environment, the index header is ~99MiB per block, the CPU usage has doubled immediately when the store gateway receives a query like this(which contains ~15000 values separated by |)..

I have seen Prometheus already does something like this: https://github.com/prometheus/prometheus/blob/bcd548c88b06543c8eeb19e68bef4adefb7b95fb/tsdb/querier.go#L321

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Verification

[GiedriusS]

Thank you for this!

Benchmark diff:

name                                                           old time/op    new time/op    delta
BucketSeries/1000000SeriesWith1Samples/1of1000000-16             77.5ms ± 7%    79.5ms ± 9%    ~     (p=0.081 n=15+15)
BucketSeries/1000000SeriesWith1Samples/10of1000000-16            75.9ms ± 9%    82.1ms ± 6%  +8.18%  (p=0.000 n=15+15)
BucketSeries/1000000SeriesWith1Samples/1000000of1000000-16        775ms ± 3%     774ms ± 5%    ~     (p=0.949 n=14+15)
BucketSeries/100000SeriesWith100Samples/1of10000000-16           5.36ms ± 3%    5.31ms ± 2%    ~     (p=0.089 n=15+15)
BucketSeries/100000SeriesWith100Samples/100of10000000-16         5.30ms ± 3%    5.31ms ± 2%    ~     (p=0.595 n=15+15)
BucketSeries/100000SeriesWith100Samples/10000000of10000000-16    76.6ms ± 7%    74.5ms ± 3%  -2.67%  (p=0.045 n=15+15)
BucketSeries/1SeriesWith10000000Samples/1of10000000-16            126µs ± 3%     124µs ± 2%  -1.36%  (p=0.001 n=13+14)
BucketSeries/1SeriesWith10000000Samples/100of10000000-16          126µs ± 1%     124µs ± 2%  -1.87%  (p=0.000 n=14+15)
BucketSeries/1SeriesWith10000000Samples/10000000of10000000-16    15.8ms ± 2%    14.9ms ± 6%  -5.51%  (p=0.000 n=13+14)

name                                                           old alloc/op   new alloc/op   delta
BucketSeries/1000000SeriesWith1Samples/1of1000000-16             60.9MB ± 0%    60.9MB ± 0%    ~     (p=0.201 n=15+14)
BucketSeries/1000000SeriesWith1Samples/10of1000000-16            60.9MB ± 0%    60.9MB ± 0%    ~     (p=0.285 n=15+15)
BucketSeries/1000000SeriesWith1Samples/1000000of1000000-16       1.27GB ± 0%    1.27GB ± 0%    ~     (p=0.583 n=13+14)
BucketSeries/100000SeriesWith100Samples/1of10000000-16           4.61MB ± 0%    4.61MB ± 0%    ~     (p=0.715 n=14+15)
BucketSeries/100000SeriesWith100Samples/100of10000000-16         4.62MB ± 0%    4.61MB ± 0%    ~     (p=0.139 n=12+15)
BucketSeries/100000SeriesWith100Samples/10000000of10000000-16     116MB ± 5%     117MB ± 4%    ~     (p=0.377 n=15+14)
BucketSeries/1SeriesWith10000000Samples/1of10000000-16            198kB ± 0%     198kB ± 0%    ~     (p=0.862 n=15+15)
BucketSeries/1SeriesWith10000000Samples/100of10000000-16          198kB ± 0%     198kB ± 0%    ~     (p=0.461 n=15+15)
BucketSeries/1SeriesWith10000000Samples/10000000of10000000-16    38.1MB ± 0%    38.1MB ± 0%    ~     (p=0.539 n=15+15)

name                                                           old allocs/op  new allocs/op  delta
BucketSeries/1000000SeriesWith1Samples/1of1000000-16              9.79k ± 0%     9.79k ± 0%    ~     (p=0.457 n=15+14)
BucketSeries/1000000SeriesWith1Samples/10of1000000-16             9.89k ± 0%     9.90k ± 0%    ~     (p=0.229 n=15+15)
BucketSeries/1000000SeriesWith1Samples/1000000of1000000-16        10.0M ± 0%     10.0M ± 0%    ~     (p=0.675 n=15+15)
BucketSeries/100000SeriesWith100Samples/1of10000000-16            1.19k ± 0%     1.19k ± 0%    ~     (p=0.565 n=15+15)
BucketSeries/100000SeriesWith100Samples/100of10000000-16          1.23k ± 0%     1.23k ± 0%  -0.12%  (p=0.011 n=15+15)
BucketSeries/100000SeriesWith100Samples/10000000of10000000-16     1.00M ± 0%     1.00M ± 0%    ~     (p=0.262 n=15+15)
BucketSeries/1SeriesWith10000000Samples/1of10000000-16              280 ± 0%       280 ± 0%    ~     (all equal)
BucketSeries/1SeriesWith10000000Samples/100of10000000-16            280 ± 0%       280 ± 0%    ~     (all equal)
BucketSeries/1SeriesWith10000000Samples/10000000of10000000-16      168k ± 0%      168k ± 0%  -0.00%  (p=0.000 n=15+15)

One has even increased 😱 do you get similar results? It seems like there isn't much of an improvement 🤔 I haven't had enough time to look deeply into this. Perhaps you know the reason?

[damnever]

Try this: https://gist.github.com/damnever/b5bf73877185dc24fa087792cc7b284a

There is only a 50k series in total, select 20k from it.

goos: darwin
goarch: amd64
pkg: github.com/thanos-io/thanos/pkg/store
cpu: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz

name                                                                     old time/op    new time/op    delta
BucketIndexReader_ExpandedPostings2/n=~"101|random-shuffled-values|1"-8    32.1ms ± 5%    23.7ms ± 4%  -26.09%  (p=0.000 n=9+9)

name                                                                     old alloc/op   new alloc/op   delta
BucketIndexReader_ExpandedPostings2/n=~"101|random-shuffled-values|1"-8    9.13MB ± 0%    7.08MB ± 0%  -22.37%  (p=0.000 n=10+9)

name                                                                     old allocs/op  new allocs/op  delta
BucketIndexReader_ExpandedPostings2/n=~"101|random-shuffled-values|1"-8      170k ± 0%      120k ± 0%  -29.40%  (p=0.000 n=10+10)

I'd expect a better result on HDD.

[bwplotka]

Thanks! I think it makes sense to have this merged. The latency spike is mostly on small (10 values), so it's fine.

But I believe we would love your benchmark to have in, so any other developer can verify and we can also ensure next changes maintains the same level of efficiency 🤗

WDYT @damnever ?

[damnever]

The benchmark test has been updated.

[bwplotka]

Thanks 💪🏽