linkerd check --proxy shows completed pods

[jack1902]

What is the issue?

When running linkerd check --proxy it shows pods that are in the completed state as having outdated proxies

How can it be reproduced?

1. setup a cluster running an older version of linkerd, say 2.13.5
2. Run a job in the cluser which is injected with a proxy and shuts the proxy down appropriately (prehaps using linkerd-await
3. upgrade linkerd to 2.13.6
4. run linkerd check --proxy

Logs, error output, etc

linkerd-data-plane
------------------
√ data plane namespace exists
√ data plane proxies are ready
√ data plane is up-to-date
‼ data plane and cli versions match
    example-job-kdkukky33v-jr4hn running stable-2.13.5 but cli running stable-2.13.6
    see https://linkerd.io/2.13/checks/#l5d-data-plane-cli-version for hints
√ data plane pod labels are configured correctly
√ data plane service labels are configured correctly
√ data plane service annotations are configured correctly
√ opaque ports are properly annotated

output of linkerd check -o short

linkerd check -o short
Status check results are √

Environment

• k8s: 1.24
• linkerd: 2.13.6
• OS: MacOS

Possible solution

Believe the logic within linkerd should filter out any pods in a completed state

Additional context

Slack: https://linkerd.slack.com/archives/C89RTCWJF/p1692689960803199

When upgrading linkerd, i generally like to check that all proxies are up to date and a simple way of doing this is with linkerd version --proxy but this shows:

Client version: stable-2.13.6
Server version: stable-2.13.6
Proxy versions:
	stable-2.13.5 (1 pods)
	stable-2.13.6 (N pods)

A useful way for me to check for outdated pods is to run kubectl through yq (this doesn't filter out completed pods either but feels similar to the logic implemented by linkerd):

export LINKERD_VERSION="2.13.6"
readarray outdatedPods < <(kubectl -A get pods -A -o yaml | yq -o=j -I=0 '.items[] | select(.metadata.annotations | has("linkerd.io/proxy-version")) | select(.metadata.annotations["linkerd.io/proxy-version"] != env(LINKERD_VERSION)) | {"name": .metadata.name, "namespace": .metadata.namespace}')

for pod in "${outdatedPods[@]}"; do
  name=$(echo "${pod}" | yq '.name')
  namespace=$(echo "${pod}" | yq '.namespace')

  echo "pod '${name}' in namespace '${namespace}' is running an outdated version of linkerd"
done

Would you like to work on fixing this bug?

None

[pratikkumar-mohite]

@adleong I would like to work on it, can you please assign it to me