DEVOPS-424 fix: add explicit GPU resource limit

[tplessas]

Nvidia plugin already updated on both clusters - tested that these changes work on review by deploying a bunch of coref-resolution replicas under release coref-multitest, which I'll leave up until this PR is merged.

Two replicas ended up on the same node, which also allows us to confirm that time-slicing works fine (barring any changes related to memory such as those discussed this morning).

[anton-delphai]

And why do we need that if we set the capacity to very big number? Isn't that equivalent to not request GPUs at all?

[tplessas]

Time-slicing does not actually do what its name implies – it is just used to claim a generic stake on the GPU. This is why the resource limit cannot have a fractional part or be more than 1.

From https://github.com/NVIDIA/k8s-device-plugin#shared-access-to-gpus-with-cuda-time-slicing:

Note: Unlike with "normal" GPU requests, requesting more than one shared GPU does not imply that you will get guaranteed access to a proportional amount of compute power. It only implies that you will get access to a GPU that is shared by other clients (each of which has the freedom to run as many processes on the underlying GPU as they want). Under the hood CUDA will simply give an equal share of time to all of the GPU processes across all of the clients. The failRequestsGreaterThanOne flag is meant to help users understand this subtlety, by treating a request of 1 as an access request rather than an exclusive resource request.

Everything works just like before this PR/the cluster update – the only thing added is some more k8s bureaucracy to achieve it.