Getting started with Amazon EKS is easy:

1. Create an Amazon EKS cluster in the AWS Management Console or with the AWS CLI or one of the AWS SDKs.

2. Launch managed or self-managed Amazon EC2 nodes, or deploy your workloads to AWS Fargate (is what we are doing in this article).

3. When your cluster is ready, you can configure your favorite Kubernetes tools, such as kubectl, to communicate with your k8s-cluster.

4. Deploy and manage workloads on your Amazon EKS cluster the same way that you would do with any other Kubernetes environment. You can also view information about your workloads using the AWS Management Console.

5. Here we are going to use eksctl utility that will help you to create your cluster using AWS cloud-shell, or you can do same using Bastion host.

NOTE - When configuring the aws shell, be sure that the user whose Access and Security key you will configure must have admin access role attached to it so that it can define and describe resources. Else you might endup getting errors like -

Error: operation error EC2: DescribeSubnets, https response error StatusCode: 403, RequestID: 310f662c-799c-4c78-99ff-dec773557025, api error UnauthorizedOperation: You are not authorized to perform this operation.

ex of devops user -

To create your first cluster For the fargate profile in EKS, you are required to have the following things in place

OIDC provider
IAM roles
IAMServiceAccount
Addons - (VPC-CNI, CoreDNS, kube-proxy)
ALB-Controller
Fargate Compute Profile

for more details consider referring to AWS documentation - - https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html

To begin with, we will start with the eksctl & kubectl installation

 curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C sudo cp -rp eksctl /usr/local/bin

OR

 wget https://github.com/eksctl-io/eksctl/releases/download/v0.162.0/eksctl_Linux_amd64.tar.gz 
 tar -xpf eksctl_Linux_amd64.tar.gz && cp -rp eksctl /usr/local/bin


 curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.23.6/bin/linux/amd64/kubectl

 sudo chmod +x kubectl &&  sudo cp -rp kubectl /usr/local/bin

Cluster creation using eksctl, if doing from shell following command will work

 # with EC2 as underline compute (nodeGroup)

 Simpletest form -

 eksctl create cluster --name ekscluster --region region-code 

 Complex form with nodeGroup -

 eksctl create cluster \
 --name ekscluster \
 --region eu-west-2 \
 --vpc-private-subnets subnet-0b10bDEMOfbdd235,subnet-0d9876DEMOa41fba,subnet-0b1168dDEMO1e66a2 \
 --version 1.27 \
 --nodegroup-name standard-eks-workers \
 --node-type t2.small \
 --nodes 2 --nodes-min 1 --nodes-max 2 \
 --ssh-access=true \
 --ssh-public-key=devops-kp

 without nodeGroup - (preferred method & add NodeGroup once Kluster is up)

 eksctl create cluster \

 --name ekscluster \

 --region eu-west-2 \

 --vpc-private-subnets subnet-0b10bDEMOfbdd235,subnet-0d987DEMO1a41fba,subnet-0b1168dDEMO1e66a2 \

 --version 1.27 \
 --without-nodegroup

 # with fargate profile as underline compute

 Simpletest form -

 eksctl create cluster --name ekscluster --region region-code --fargate

 Complex form -

 eksctl create cluster \
 --name eksCluster \
 --region eu-west-2 \
 --version 1.27 \
 --vpc-private-subnets subnet-0b10be78a0fbdd235,subnet-0d9876a68e1a41fba,subnet-0b1168d86d71e66a2

 
 update following values
 # cluster name
 # region code
 # subnets (private)

if doing from Bastion host you need to authorise instance to perform this operation -
(create an IAM policy to create eks cluster using eksctl (optional))

Accessing your cluster

on Bastion host do the following -


 # export necessary varibles as below & set cluster context

 $ export KUBECONFIG=~/.kube/config/kubeconfig_eksCluster $ export AWS_DEFAULT_REGION=eu-west-2
 $ export AWS_DEFAULT_PROFILE=dev

 $ aws eks update-kubeconfig --name eksCluster --region=eu-west-2
 Added new context arn:aws:eks:eu-west-2:295XXXX62576:cluster/eksCluster to $PWD/kubeconfig_eksCluster

 $ kubectl cluster-info
 Kubernetes control plane is running at https://01C5E459355DEMOFDC1E8FB6CA7.gr7.eu-west-2.eks.amazonaws.com
CoreDNS is running at https://01C5E459355DEMOFDC1E8FB6CA7.gr7.eu-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

with EC2 instance in nodeGroup

Add networking add-ons via GUI, consider looking at next point before adding them -

1. coredns - DNS server used for service-discovery
2. vpc-cni - networking plugin for pod networking
3. kube-proxy - It maintains network rules on your nodes and enables network communication to your pods. (only applicable when using EC2 in node group)

---> Add coredns compute profile via CLI ( if working with Fargate)
for coredns, you are required to create a Fargate profile of coredns under compute tab in the kube-system namespace, or via CLI as below -

 eksctl create fargateprofile --name coredns --cluster eksCluster --namespace kube-system

update the value for # cluster name

Optional (if there is any issue with addons try to patch the deployment, usually coredns goes into Degraded mode)

Ex - for coredns deployment

 kubectl patch deployment coredns -n kube-system --type json \
-p='[{"op": "remove", "path": "/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type"}]'	

 kubectl rollout restart -n kube-system deployment coredns

NOTE- if addon [coredns] remains in a degraded state, delete the coredns replicaset with 0 desired value & associated pods so that it will look for fargate compute next time when it deploys new pods instead on EC2 compute(by default it attempts to deploy on EC2)

 kubectl scale deployment/coredns --replicas=0 -n kube-system
 kubectl scale deployment/coredns --replicas=3 -n kube-system

Setup IAM Policy


curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.4/docs/install/iam_policy.json
aws iam create-policy --policy-name AWSLoadBalancerControllerIAMPolicy --policy-document file://iam_policy.json

Create IAM Role with the below policies attached that we use for the Fargate profile & update the trust relationship

 # TRUST RELATIONSHIP SHOULD BE UPDATED AS BELOW
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "eks-fargate-pods.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": "arn:aws:eks:eu-west-2:481xxxx1953:fargateprofile/MyEKSCluster/*"
                }
            }
        }
    ]
}

Associate IAM OIDC provider with your cluster

 oidc_id=$(aws eks describe-cluster --name eksCluster --query "cluster.identity.oidc.issuer" \
 --output text | cut -d '/' -f 5)

 aws iam list-open-id-connect-providers | grep $oidc_id | cut -d "/" -f4

 eksctl utils associate-iam-oidc-provider --cluster eksCluster --approve
 2022-10-26 12:46:52 [ℹ]  will create IAM Open ID Connect provider for cluster "eksCluster" in "eu-west-2"
 2022-10-26 12:46:52 [✔]  created IAM Open ID Connect provider for cluster "eksCluster" in "eu-west-2"

Create IAM ServiceAccount

eksctl create iamserviceaccount \
--cluster eksCluster \
--region eu-west-2 \
--namespace kube-system \
--name aws-load-balancer-controller \
--attach-policy-arn arn:aws:iam::4812xxxx1953:policy/AWSLoadBalancerControllerIAMPolicy \
--override-existing-serviceaccounts \
--approve

2022-10-26 12:50:33 [ℹ]  1 iamserviceaccount (kube-system/aws-load-balancer-controller) was included (based on the include/exclude rules)
2022-10-26 12:50:33 [!]  metadata of serviceaccounts that exist in Kubernetes will be updated, as --override-existing-serviceaccounts was set
2022-10-26 12:50:33 [ℹ]  1 task: {
    2 sequential sub-tasks: {
        create IAM role for serviceaccount "kube-system/aws-load-balancer-controller",
        create serviceaccount "kube-system/aws-load-balancer-controller",
    } }2023-10-26 12:50:33 [ℹ]  building iamserviceaccount stack "eksctl-eksCluster-addon-iamserviceaccount-kube-system-aws-load-balancer-controller"
2022-10-26 12:50:33 [ℹ]  deploying stack "eksctl-eksCluster-addon-iamserviceaccount-kube-system-aws-load-balancer-controller"
2022-10-26 12:50:33 [ℹ]  waiting for CloudFormation stack "eksctl-eksCluster-addon-iamserviceaccount-kube-system-aws-load-balancer-controller"
2022-10-26 12:51:04 [ℹ]  waiting for CloudFormation stack "eksctl-eksCluster-addon-iamserviceaccount-kube-system-aws-load-balancer-controller"
2022-10-26 12:51:04 [ℹ]  created serviceaccount "kube-system/aws-load-balancer-controller"

update following values

# cluster name
# region name
# caller identity

Install & Setup Helm for aws-lb-controller

 wget https://get.helm.sh/helm-v3.10.3-linux-amd64.tar.gz 
 tar -xzpf helm-v3.0.0-linux-amd64.tar.gz 
 sudo cp -rp linux-amd64/helm /usr/local/bin/

 helm repo add eks https://aws.github.io/eks-charts
 helm repo update

Deploy aws-load-balancer-controller

helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system \
--set clusterName=eksCluster \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controller \
--set image.repository=602401143452.dkr.ecr.eu-west-2.amazonaws.com/amazon/aws-load-balancer-controller \
--set region=eu-west-2 \
--set vpcId=vpc-0d417VPC-ID7694dd7

update following values

# cluster name
# region name
# repository name
# vpc-ID

Deploy compute for your app i.e. FARGATE PROFILE

eksctl create fargateprofile --name sampleapp-ns --namespace sampleapp-ns --cluster eksCluster

Deploy your application

kubectl create deployment sampleapp --image=punitporwal07/sampleapp:3.0 -n sampleapp-ns

[deploy all the resources of my sampleapp from here - full-sampleapp-for-eks-fargate.yaml ]

Deploy Ingress which will do rest of the magic

 # INGRESS RESOURCE/ROUTE THAT WILL BE DEPLOYED AS APP LOAD-BALANCER IN AWS
 ---
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   namespace: sampleapp-ns
   name: sampleapp-ing
   annotations:
     alb.ingress.kubernetes.io/scheme: internet-facing
     alb.ingress.kubernetes.io/target-type: ip
 spec:
   ingressClassName: alb
   rules:
     - http:
         paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: sampleapp-svc
               port:
                 number: 80

Ingress works with clusterIP service type as well however, to make it private deploy an NLB as a LoadBalancer service type.

test your application by hitting the DNS of the Application Load Balancer, to test NLB test it from/within your VPC instance.

Some helpful commands during the setup

To delete resources


 helm delete aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system
 eksctl delete iamserviceaccount --cluster eksCluster --name aws-load-balancer-controller --namespace kube-system // when using CLI options, the --namespace option defaults to default, so if the service account was created in a different namespace, the filter would fail to match and, as a result, the deletion would not proceed.

 eksctl delete fargateprofile --name coredns --cluster eksCluster
 eksctl delete cluster --name eksCluster --region eu-west-2

Scale down your deployments to save some cost

kubectl scale deploy --replicas=0 aws-load-balancer-controller -n kube-system
kubectl scale deploy --replicas=0 coredns -n kube-system
kubectl scale deploy --replicas 0 --all -n namespace

Switch your cluster context in aws cloud-shell

 aws eks update-kubeconfig --region eu-west-2 --name eksCluster

To decode encoded error message

  
 aws sts decode-authorization-message --encoded-message "message"

Known Issues

Issue 1 - When you Add a nodeGroup with Private-Subnet, you might get an error -

NodeCreationFailure

Fix - Your launched instances are unable to register with your Amazon EKS cluster. Common causes of this failure are insufficient node IAM role permissions.
or lack of outbound internet access for the nodes. Your nodes must meet either of the following requirements:

Able to access the internet using a public IP address. ( Your private subnet should have Nat-G/w associated to it)
The security group associated to the subnet the node is in must allow the communication.

Issue 2 - Resolve the single subnet discovery error

Fix - Add the appropriate tags on your subnets to allow the AWS Load Balancer Ingress Controller to create a load balancer using auto-discovery.

for public load balancer [public subnet] define in at least 2:

  Key                                                                                                    Value
 kubernetes.io/role/elb                      1
 kubernetes.io/cluster/your-cluster-name     shared or owned

  for [private subnets] tags:

  Key                                                                                                    Value
 kubernetes.io/role/elb                      1 or empty tag value for internet-facing load balancers
 kubernetes.io/role/internal-elb             1 or empty tag value for internal load balancers

  Note: You can manually assign subnets to your load balancer using the annotation -

 alb.ingress.kubernetes.io/subnets: subnet-1xxxxx, subnet-2xxxxx, subnet-3xxxxx

however, it is advisable that EKS discovers the subnet automatically so tagging the subnet with the cluster is the better approach if you are hitting the BUG, which usually occurs when -

Any of the following is true:
- You have multiple clusters that are running in the same VPC. use shared value on your next cluster.
- You have multiple AWS services that share subnets in a VPC.
- You want more control over where load balancers are provisioned for each cluster.

Issue 3 - Resolve nodes are not available

0/2 nodes are available: 2 Too many pods, 2 node(s) had untolerated taint {eks.amazonaws.com/compute-type: fargate}.
preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.

Fix - Add compute profile so that - aws-lb-controller pods can be scheduled on it, and then delete the pending pods

 
 eksctl create fargateprofile --name aws-lb --cluster eksCluster --namespace kube-system

from the above, you can see the alb-controller is using aws-lb compute profile that we created and started its pod on it.

cloudnetes

20 March 2021

EKS with Fargate Profile

Deploy compute for your app i.e. FARGATE PROFILE

Deploy Ingress which will do rest of the magic

Known Issues

No comments:

Post a Comment