EKS with Karpenter v1 Autoscaling
Karpenter Autoscaling is a great choice when it comes scaling EKS cluster. It is very cost efficient, quick and does not have the limitations of cluster Autoscaler.
- Cost effective ( Provisions most cost effective instances including SPOT instance)
- Fast auto scaling up/down
- Resource consolidation (Reduce number of nodes by organizing pod spread when spare capacity available)
- Regular node recycle (Re-provision nodes at set interval to provision them on up to date hardware.)
BottleRocket
Amazon BottleRocket is a Linux-based open-source operating system that is purpose-built by Amazon Web Services for running containers. Highly secure, optimized for containers and fast provisioning.
Karpenter configuration
There are some prerequisites to provisioning Karpenter on EKS cluster.
IAM Role
Karpenter require IAM role to join Cluster and provision nodes. In this example I am creating IAM role called karpenter-controller-eks-demo
. IAM trust policy is attached so that karpenter-controller-eks-demo
serviceAccount in karpenter
namespaces is allowed to assume this role. Various Ec2 permissions are assigned to the role so that it has necessary permissions to get pricing of instances and provision new nodes. eks:DescribeCluster
permission is added so that Karpenter pod can identify cluster endpoint, than having to manually update in Helm values.
InstanceProfile
It is advised to have dedicated instanceProfile
for Karpenter.
Helm Values
Few input values at minimum must be provided during helm installation. Clustername so it can identify which cluster to attach nodes to. ServiceAccount name and Annotation to specify which role to assume. Details here must match IAM role created.
ec2NodeClass
Define the AMI version to be used while provisioning nodes. Also specify subnet tag to identify the subnet where node will be provisioned and security group to be attached.
Nodepool
It is not a pool as name suggested, Karpenter provisions individual nodes based on constraints. Define the type of Node, instance category etc. Also must specify CPU and memory limit so that runaway workloads does not cause too many servers being spun up.
Installation Instructions
Terraform code provided in aws-eks-terraform includes creation of a new VPC andEKS cluster with single Spot instance. It also includes necessary IAM profile for Karpenter, and custom instanceProfile
for Karpenter use. Karpenter-app contains minimum required values for Karpenter, and ec2NodeClass
and nodePool
configured with Bottlerocket
AMI
terrafor init -upgrade ; terraform apply
# Configure local kubeconfig
aws eks --profile labs --region eu-west-1 update-kubeconfig --name eks-demo
# Verify cluster access
kubectl cluster-info
# Install Karpenter controller
helm install karpenter -n karpenter --create-namespace oci://public.ecr.aws/karpenter/karpenter \
--version 1.0.6 -f Karpenter-app/karpenter-values.yaml
# Once karpenter pods are up and running, create nodepoo and node class
kubectl apply -f Karpenter-app/karpenter-nodepool.yaml
# Verify resources are created
get ec2nc,nodepool
Testing Karpenter
Karpenter will provision new nodes when ever pod deployment is pending due to resource constraints. Here is sample app that you can use for test, adjust number of replicas as required.
kubectl apply -f Sample-App/karpenter-scale.yaml
# Notice pods are pending and if you describe, it will report insufficient CPU.
# Check Karpenter logs for errors, also notice it provision new node
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter
# Allow a minute and check pods status as well as nodes.
kubectl get nodes # Should see new node being provisioned.
# Check and ensure all pods are now deployed
kubectl get po -n karpenter-testing