/api/v1/label does not apply filter to external labels

[sylr]

Thanos, Prometheus and Golang version used: v0.32.3

Object Storage Provider: S3

What happened:

My grafana variables for external labels stopped being filtered

What you expected to happen:

I've 2 external labels env & k8s_cluster and my grafana dashboards have corresponding variables where k8s_cluster is filtered based en selected env values, i.e.:

label_values(prometheus_build_info{env=~"$env"}, k8s_cluster)

However this has stopped working some time ago, I blamed grafana but it's a thanos/prometheus issue.

How to reproduce it (as minimally and precisely as possible):

$ curl https://thanos-query/api/v1/label/env/values?match[]={env=%22prod%22}
{"status":"success","data":["dev","infra","prod","test"]}

This should only reply "prod". At least it used to.

Full logs to relevant components:

Anything else we need to know:

[MichaHoffmann]

I can reproduce this in an acceptance test

[MichaHoffmann]

So Prometheus Store APIs ( Sidecars ) would always just short circuit and return their external label if it has the same name as the requested one. This could explain this; i added an acceptance test to verify behavior for the other Store API implementations too and they were ok.