title | description |
---|---|
AWS EKS Cluster Autoscaler with Terraform |
Learn to automate AWS EKS Kubernetes Cluster Autoscaler with Terraform |
- Install Cluster Autoscaler for AWS EKS Cluster
- Project Folder: 01-ekscluster-terraform-manifests
- c5-07-eks-node-group-public.tf
- c5-08-eks-node-group-private.tf
- Important Note: Node groups with Auto Scaling groups tags. The Cluster - Autoscaler requires the following tags on your Auto Scaling groups so that they can be auto-discovered.
- Add Cluster Autoscaler tags in Node Groups
tags = {
Name = "Public-Node-Group"
# Cluster Autoscaler Tags
"k8s.io/cluster-autoscaler/${local.eks_cluster_name}" = "owned"
"k8s.io/cluster-autoscaler/enabled" = "TRUE"
}
- Project Folder: 01-ekscluster-terraform-manifests
- File Name: c5-04-iamrole-for-eks-nodegroup.tf
# Autoscaling Full Access
resource "aws_iam_role_policy_attachment" "eks-Autoscaling-Full-Access" {
policy_arn = "arn:aws:iam::aws:policy/AutoScalingFullAccess"
role = aws_iam_role.eks_nodegroup_role.name
}
# Change Directory
cd 01-ekscluster-terraform-manifests
# Terraform Initialize
terraform init
# Terraform Validate
terraform validate
# Terraform Plan
terraform plan
# Terraform Apply
terraform apply -auto-approve
- Project Folder: 02-cluster-autoscaler-install-terraform-manifests
- c1-versions.tf
- Create DynamoDB Table
dev-eks-cluster-autoscaler
- c2-remote-state-datasource.tf
- c3-01-generic-variables.tf
- c3-02-local-values.tf
- Project Folder: 02-cluster-autoscaler-install-terraform-manifests
# Resource: IAM Policy for Cluster Autoscaler
resource "aws_iam_policy" "cluster_autoscaler_iam_policy" {
name = "${local.name}-AmazonEKSClusterAutoscalerPolicy"
path = "/"
description = "EKS Cluster Autoscaler Policy"
# Terraform's "jsonencode" function converts a
# Terraform expression result to valid JSON syntax.
policy = jsonencode({
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeInstances",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeTags",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup",
"ec2:DescribeLaunchTemplateVersions",
"ec2:DescribeInstanceTypes"
],
"Resource": "*",
"Effect": "Allow"
}
]
})
}
# Resource: IAM Role for Cluster Autoscaler
## Create IAM Role and associate it with Cluster Autoscaler IAM Policy
resource "aws_iam_role" "cluster_autoscaler_iam_role" {
name = "${local.name}-cluster-autoscaler"
# Terraform's "jsonencode" function converts a Terraform expression result to valid JSON syntax.
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRoleWithWebIdentity"
Effect = "Allow"
Sid = ""
Principal = {
Federated = "${data.terraform_remote_state.eks.outputs.aws_iam_openid_connect_provider_arn}"
}
Condition = {
StringEquals = {
"${data.terraform_remote_state.eks.outputs.aws_iam_openid_connect_provider_extract_from_arn}:sub": "system:serviceaccount:kube-system:cluster-autoscaler"
}
}
},
]
})
tags = {
tag-key = "cluster-autoscaler"
}
}
# Associate IAM Policy to IAM Role
resource "aws_iam_role_policy_attachment" "cluster_autoscaler_iam_role_policy_attach" {
policy_arn = aws_iam_policy.cluster_autoscaler_iam_policy.arn
role = aws_iam_role.cluster_autoscaler_iam_role.name
}
output "cluster_autoscaler_iam_role_arn" {
description = "Cluster Autoscaler IAM Role ARN"
value = aws_iam_role.cluster_autoscaler_iam_role.arn
}
- Project Folder: 02-cluster-autoscaler-install-terraform-manifests
# Datasource: EKS Cluster Auth
data "aws_eks_cluster_auth" "cluster" {
name = data.terraform_remote_state.eks.outputs.cluster_id
}
# HELM Provider
provider "helm" {
kubernetes {
host = data.terraform_remote_state.eks.outputs.cluster_endpoint
cluster_ca_certificate = base64decode(data.terraform_remote_state.eks.outputs.cluster_certificate_authority_data)
token = data.aws_eks_cluster_auth.cluster.token
}
}
- Project Folder: 02-cluster-autoscaler-install-terraform-manifests
# Install Cluster Autoscaler using HELM
# Resource: Helm Release
resource "helm_release" "cluster_autoscaler_release" {
depends_on = [aws_iam_role.cluster_autoscaler_iam_role ]
name = "${local.name}-ca"
repository = "https://kubernetes.github.io/autoscaler"
chart = "cluster-autoscaler"
namespace = "kube-system"
set {
name = "cloudProvider"
value = "aws"
}
set {
name = "autoDiscovery.clusterName"
value = data.terraform_remote_state.eks.outputs.cluster_id
}
set {
name = "awsRegion"
value = var.aws_region
}
set {
name = "rbac.serviceAccount.create"
value = "true"
}
set {
name = "rbac.serviceAccount.name"
value = "cluster-autoscaler"
}
set {
name = "rbac.serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
value = "${aws_iam_role.cluster_autoscaler_iam_role.arn}"
}
# Additional Arguments (Optional) - To Test How to pass Extra Args for Cluster Autoscaler
#set {
# name = "extraArgs.scan-interval"
# value = "20s"
#}
}
- Project Folder: 02-cluster-autoscaler-install-terraform-manifests
# Helm Release Outputs
output "cluster_autoscaler_helm_metadata" {
description = "Metadata Block outlining status of the deployed release."
value = helm_release.cluster_autoscaler_release.metadata
}
# Change Directory
cd 02-cluster-autoscaler-install-terraform-manifests
# Terraform Initialize
terraform init
# Terraform Validate
terraform validate
# Terraform Plan
terraform plan
# Terraform Apply
terraform apply -auto-approve
# List Pods
kubectl -n kube-system get pods
# Verify Logs
kubectl -n kube-system logs -f $(kubectl -n kube-system get pods | egrep -o 'hr-dev-ca-aws-cluster-autoscaler-[A-Za-z0-9-]+')
# List Service Account
kubectl -n kube-system get sa
kubectl -n kube-system describe sa cluster-autoscaler
Observation:
1. Review Annotations section for IAM Role annotated on Kubernetes Service Account
2. Sample (Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::180789647333:role/hr-dev-cluster-autoscaler)
# List Config Maps
kubectl -n kube-system get cm
kubectl -n kube-system describe cm cluster-autoscaler-status
# ConfigMap that Cluster Autoscaler writes
kubectl -n kube-system get cm cluster-autoscaler-status -o yaml
- Dont destroy the Terraform Projects in below two folders
- Terraform Project Folder: 01-ekscluster-terraform-manifests
- Terraform Project Folder: 02-cluster-autoscaler-install-terraform-manifests
- We are going to use them for all upcoming Usecases.
- Destroy Resorces Order
- 02-cluster-autoscaler-install-terraform-manifests
- 01-ekscluster-terraform-manifests
##############################################################
## Destroy Cluster Autoscaler
# Change Directroy
cd 02-cluster-autoscaler-install-terraform-manifests
# Terraform Destroy
terraform init
terraform apply -destroy -auto-approve
##############################################################
## Destroy EKS Cluster
# Change Directroy
cd 01-ekscluster-terraform-manifests
# Terraform Destroy
terraform init
terraform apply -destroy -auto-approve
##############################################################
- Cluster Autoscaler with Helm
- https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md
- https://github.com/kubernetes/autoscaler/tree/master/charts/cluster-autoscaler#aws---using-auto-discovery-of-tagged-instance-groups
- https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-are-the-parameters-to-ca