eks-prow-build-cluster: Deploy the Karpenter module #7063
Conversation
|
Skipping CI for Draft Pull Request. |
k8s-ci-robot
added
do-not-merge/work-in-progress
k8s-ci-robot
added
size/XL
xmudrii
force-pushed
the
eks-karpenter
branch
6 times, most recently
from
bb0b7de
to
6a45812
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: xmudrii The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
size/L
|
We still need to figure out the migration plan, will be discussed on Slack |
k8s-ci-robot
added
the
do-not-merge/hold
k8s-ci-robot
removed
the
do-not-merge/work-in-progress
xmudrii
force-pushed
the
eks-karpenter
branch
3 times, most recently
from
bc4c965
to
7e5b8c3
Compare
| "k8s.io/cluster-autoscaler/${var.cluster_name}" = "owned" | ||
| "k8s.io/cluster-autoscaler/enabled" = true |
There was a problem hiding this comment.
TODO: remove these tags before the migration
There was a problem hiding this comment.
yes please - EKS MNG already adds these, not sure if they are still causing conflicts from what @ameukam saw previously #7036 (comment)
There was a problem hiding this comment.
This will be applied in a follow up PR
| values: ["us-east-2a", "us-east-2b", "us-east-2c"] | ||
| - key: kubernetes.io/arch | ||
| operator: In | ||
| values: ["amd64"] |
There was a problem hiding this comment.
do we need to be as restrictive on some of these? if folks want to run arm, why not let them?
likewise below for the instance-*, why not let Karpenter decide the best fit?
There was a problem hiding this comment.
We could have a dedicated nodepool with arm64 instances ?
There was a problem hiding this comment.
At the moment yes, we don't want to change the behavior now, this needs a bit more discussion with SIG Testing.
There was a problem hiding this comment.
I guess what I am saying is why block or restrict things that require users to come ask us to unblock. If a group wanted to test arm64 instances, it's seems like we should just let them do that. And on the instance types/sizes - Karpenter works better (delivers on its benefits) when it's not restricted. The workloads define their requirements and Karpenter supplies compute to satisfy those requirements
There was a problem hiding this comment.
I'm +1 with this, but the plan is that we try to replicate the cluster-autoscaler behavior in the very beginning, and then we start to add additional functionalities after the initial deployment.
|
/lgtm |
k8s-ci-robot
added
the
lgtm
Signed-off-by: Marko Mudrinić <[email protected]> Co-authored-by: Koray Oksay <[email protected]>
Signed-off-by: Marko Mudrinić <[email protected]>
k8s-ci-robot
removed
the
lgtm
|
The Terraform changes are applied to the production: /hold cancel |
k8s-ci-robot
removed
the
do-not-merge/hold
|
/lgtm |
k8s-ci-robot
added
the
lgtm
This PR deploys the Karpenter module to the EKS Prow build cluster to be used as a replacement for the cluster-autoscaler. Karpenter is better tailored to AWS, doesn't use ASGs that were causing us problems before, and provides many more configuration options that we might want to utilize in the future.
This PR:
This PR supersedes #6895. The most of the work in this PR has been done by @koksay, but given that I can't push to his branch and that he's on a vacation, I decided to create a new PR that addresses comments left by @bryantbiggs and me.
This PR depends on #7062, so please hold off with reviews until that PR is not merged and this one rebased. I also want to do some additional manual testing in the meanwhile.Once we merge this PR, we need to merge #6896 in order to enable Karpenter.