23 November 2017

Docker: Containerization Tool

Docker allows you to encapsulate your application, operating system and hardware configuration into a
single unit to run it anywhere.
It's all about applications and every application requires much of Infrastructure, which is a massive waste of resources since it utilizes very less % of it. I mean with Physical Machine/ram/CPU results heavy loss of cost & bla blah.. hence Hypervisor/Virtualization came into the picture, where we use shared resources on top of a single physical machine and create multiple VMs to utilize more from it but still not perfect.
Docker is the solution to the above problem, it can containerize your requirement & works on the principle of layered images.

working with docker is as simple as three steps:
  • Install Docker-engine
  • Pull the image from HUB/docker-registry
  • Run image as a container/service


How containers evolved over Virtualization
-In the virtual era, you need to maintain guest OS on the host OS in form of virtualization which boots up in minutes or so.
whereas containers bypass gust OS from host OS in containerization & boots up in a fraction of seconds
- It is not replacing virtualization, it is just the next step in evolution (more advanced)

What is docker?
Docker is a containerization platform that can bundle up technologies and packages your application and all its dependencies together in the form of an image which further you run as a service called container so as to ensure that your application will work in any environment be it Dev/Test/Prod

Point to remember
  • docker images are the read-only template & used to run containers
  • There is always a base image on which you layer up your requirement
  • the container is the actual running instance of the images
  • we always create images and run containers using images
  • we can pull images from the image registry/hub can be public/private
  • docker daemon runs on the host machine
  • docker0 is not a normal interface | Its a Bridge | Virtual Switch | that links multiple containers
  • Docker images are registered in the image registry & stored in the image hub
  • Docker hub is docker's own cloud repository (for sharing & caring purpose of images)
The essence of docker: if you are new to any technology and want to work on it, get its image from the docker hub configure it, work on it, destroy it, then you can move the same image to another environment and run as it is out there.   
                          
                      
key attribute of kernel used by containers
  • Namespaces (PID, net, mountpoint, user) Provides Isolation
  • cgroups (control groups)
  • capabilities ( assigning privileges to container users)
  • but each container shares common Kernel
how communication happens b/w docker client & docker daemon
  • Rest API
  • Socket.IO
  • TCP

Dockerfile supports the following list of variables

FROM       image:tag AS name
ADD        ["src",... "dest"]
COPY       /src/ dest/
ENV        ORACLE_HOME=/software/Oracle/
EXPOSE     port, [port/protocol]
LABEL      multi.label1="value1" multi.label2="value2" other="value3"
STOPSIGNAL
USER       
myuser
VOLUME     /myvolume
WORKDIR    /locationof/directory/
RUN        write your shell command
CMD        ["executable","param1","param2"]
ENTRYPOINT ["executable","param1","param2"] (exec form, preferred)
ENTRYPOINT command param1 param2 (shell form)
ENTRYPOINT script ; /bin/bash

How RUN | ENTRYPOINT | CMD differ from each other
RUN is built time instructions used to add layers to images & to install apps

ENTRYPOINT is not mandatory to use, it cannot be overridden at run-time with normal commands like docker run command. Any command passed to ENTRYPOINT is treated as first-ever command of that container.

CMD only executes at runtime. It executes commands in container at launch time equivalent of docker run <args> <command>. It can be used only once per Dockerfile
Shell form/commands are expressed the same way as a shell command. Commands get prepended by "/bin/sh -c" | variable expansion etc.
Exec form | json array style - ["command", "arg1"]
container don't need a shell | no variable expansion | no special characters(&&,||, <>)


Some arguments which you can use while running any docker Image

$ docker run -it --privileged image:tag
--privileged will give all capabilities to the container and lifts all the limitations enforced by OS/device, even you can run docker inside docker with it.

Installing docker-engine onto any Ubuntu system


$ sudo apt-get update -y && apt-get install docker.io # this will install docker-engine as a Linux service. Check engine status by running $ service docker status // else $ service docker start

check docker details installed in your system by running any of these commands


$ docker -v | docker version | docker info


Docker needs root to work for the creation of Namespaces/cgroups/etc..

$ ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 Jun 21 06:43 /var/run/docker.sock


so you need to add your local user to docker group (verify docker group from /etc/group and add your user as:

$ sudo usermod -aG docker $USER
# restart your session

# Alternatively add your user to the docker group
$ vi /etc/group
# append $USER to docker group and start using docker with your user now

# if fails with -

level=error msg="'overlay' is not supported over btrfs" level=fatal msg="Error starting daemon: error initializing graphdriver:
Failed to start Docker Application Container Engine.

it appears that the underline storage defined in daemon.json is not supported
/etc/docker/daemon.json
remove the above file and clear the /var/lib/docker/*
restart the docker service


Basic commands

FunctionCommand
pull a docker imagedocker pull reponame:imagename:tag
run an image
docker run parameters imagename:tag
list docker imagesdocker images
list running containers
list container even not running
docker ps
 docker ps -a
build an imagedocker build -t imagename:tag .
remove n processes in one command
docker rm $(docker ps -a -q) // for older versions
docker container prune -f // for newer versions
remove n images in one commanddocker rmi $(docker image -a -q)                                                               
reset docker systemdocker system prune
create mount docker volume create
using mount point
docker run -it -p 8001-8006:7001-7006 --mount type=bind, source=/software/, target=/software/docker/data/ registry.docker/weblogic12213:191004
docker run -it -p 8001-8006:7001-7006
-v data:/software/ registry.docker/weblogic1036:191004
create network                    docker network create --driver bridge --subnet=192.168.0.0/20 --gateway=192.168.0.2 mynetwork
docker run -it -p 8001:8006:7001:7006 
--network=mynetwork registry.docker/weblogic1036:191004         
for more on networkingclick here: networking in docker 


As an exercise lets attempt to setup Jenkins via Docker on a Linux machine

Open a terminal window and run(Provided Docker is already installed)
$ docker pull punitporwal07/jenkins
$ docker container run --rm -d -p 9090:8080 -v jenkins-data:/var/jenkins_home/ punitporwal07/jenkins
where
docker run : default command to run any docker container
--rm : this will remove the docker container as soon as process exits
-d : run the container in detached mode(in background) and omit the container ID
-p : port assignation from image to you local setup -p host-port:container-port
-v : Jenkins data to be mapped to /var/Jenkins_home/ directory/volume to one of your file system
punitporwal07/jenkins: docker will pull this image from your image registry

it will process for 2-3 mins then prompt as:

INFO: Jenkins is fully up and running
to access the jenkins console( http://localhost:9090 ) for the first time you need to provide admin password to make sure it was installed by admin only. & it will prompt admin password during the installation process as something like:
e72fb538166943269e96d5071895f31c
This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

here we are running Jenkins inside docker as a detached container you can use:
$ docker logs to collect jenkins logs
if we select to install recommended plugins which are most useful, Jenkins by default will install

Best practice to write a Dockerfile
best practice is to build a container first, run all the instructions one by one that you are planning to put in a Dockerfile. Once they got succeed you can put them in your Dockerfile, which will avoid you building n images from your Dockerfile again and again and save image layers as well.

Writing a docker File: ( FROM COPY RUN CMD)

a Container runs on level of images:
            base image
            layer1 image
            layer2 image

Dockerfiles are simple text files with a command on each line.
To define a base image we use the instruction FROM 

Creating a Dockerfile
  • The first line of the Dockerfile should be FROM nginx:1.11-alpine (it is better to use exact version rather then writing it as latest, as it can deviate your desired version)
  • COPY allows you to copy files from the directory containing the Dockerfile to the container's image. This is extremely useful for source code and assets that you want to be deployed inside your container.
  • RUN allows you to execute any command as you would at a command prompt, for example installing different application packages or running a build command. The results of the RUN are persisted to the image so it's important not to leave any unnecessary or temporary files on the disk as these will be included in the image & it will create a image for each command
  • CMD is used to execute any single command as soon as container launch

Life of a docker Image

write a Dockerfile > build the image > tag the image > push it to registry > pull the image to any system > run the image as container

vi Dockerfile: 

FROM baseLayer:version
MAINTAINER xxx@xx.com
RUN install
CMD special commands/instructions

$ docker build -t imagename:tag .
$ docker tag 4a34imageidgfg43 punixxorwal07/image:tag
$ docker push punixxorwal07/image:tag
$ docker pull punixxorwal07/image:tag
$ docker run -it -p yourPort:imagePort punixxorwal07/image:tag

How to Upload/Push your image to a registry

after building your image (docker build -t imageName:tag .) do the following:

step1- login to your docker registry
$ docker login --username=punitporwal --email=punixxorwal@xxxx.com

list your images
$ docker images

step2- tag your image for registry
$ docker tag b9cc1bcac0fd reponame/punitporwal07/helloworld:0.1

step3- push your image to registry
$ docker push reponame/punitporwal07/helloworld:0.1

your image is now available and open for world, by default your images is public.

repeat the same step if you wish to do any changes in your docker image, make the changes, tag the new image, push it to you docker hub

Running your own image registry
$ docker pull registry/registry:2
$ docker run -d -p 5000:5000 --restart always -v /registry:/var/lib/registry --name registry registry:2 
if its an insecure registry update registries.conf with entry of your insecure registry before pushing your image to it
$ sudo vi /etc/containers/registries.conf

Volumes in Docker

first of all create volume for your docker container using command

$ docker volume create myVolume
$ docker volume ls 
DRIVER              VOLUME NAME
local               2f14a4803f8081a1af30c0d531c41684d756a9bcbfee3334ba4c33247fc90265
local               21d7149ec1b8fcdc2c6725f614ec3d2a5da5286139a6acc0896012b404188876
local               myVolume

there after use following way to use volume feature
we can define volumes in one container and same can be share across multiple containers

to define in container 1
$ docker run -it -v /volume1 --name voltainer centos /bin/bash

to call in another container from other container
$ docker run -it --volumes-from=voltainer centos /bin/bash

we can call Volumes in a container from Docker engine host
$ docker run -v /data:/data
$ docker run --volume mydata:/mnt/mqm

     /volumeofYourHost/:/volumeofContainer/

to define in a Dockerfile
VOLUME /data (but we cannot bind the volume from docker host to container via this, just docker run command can do this)


DOCKER DAEMON LOGGING

first of all stop the docker service
$ service docker stop
$ docker -d -l debug &
-d here is for daemon
-l log level
& to get our terminal back
or
$ vi /etc/default/docker/
add log-level
DOCKER_OPTS="--log-level=fatal"
then restart docker deamon
$ service docker start


Br
Punit

20 January 2019

All about Docker swarm

There is always a requirement to run every individual service without failover and load balancing. When this comes to container services docker swarm comes into the picture.
Docker swarm is a cluster of docker containers and provide a container orchestration framework like k8s, nomad and apache mesos.
  • comprises of managers and workers
  • managers are also known as workers
  • there will be only one manager as a leader, other managers will act as a backup
  • as a pre-requisite, your docker version should be on 1.12+

 
  # to initiate docker swarm
  $ docker swarm init --advertise-addr ManagerIP:2377 --listen-addr ManagerIP:swarmListenPort
  

2377 - is the default port for swarm
172.31.22.15 - is my Manger node IP
addvertise-addr - will let swarm manager to use specific IP:PORT. 


[root@Manager1]# docker swarm init --advertise-addr 172.31.22.15:2377 --listen-addr 172.31.22.15:2377
Swarm initialized: current node (icuih1r0n8juo8xigkceniu3j) is now a manager.
 To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-15z6ejowo...63dn550as-7998mw9sxnh3ig 172.31.22.15:2377

 To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

 [root@Manager1]# docker node ls
 ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
 icuih1r0n8juo8xigkceniu3j *  docker    Ready   Active        Leader


the highlighted command is the exact command that we need to run on a worker/manager that you wanna join to this swarm, it includes a token


[root@Manager1]# docker swarm join-token manager
 To add a manager to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-15z6ejowow...63dn550as-9wiyb3pyiviqik 172.31.22.15:2377


 [root@Worker1]# docker swarm join-token worker
 To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-15z6ejowow...63dn550as-7998mw9sxnh3ig 172.31.22.15:2377

following the above command to join leader as worker/manager, launch another ec2 instance or any with docker 1.12+ in it and


 $ docker swarm join --token SWMTKN-1-15z6ejowow53...63dn550as-9wiyb3pyiviqik 172.31.22.15:2377


you will see all the workers/managers you have joined with your swarm from the Leader node as:


 [root@Manager1]# docker node ls
 ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
 25nwmw5eg7a5ms4ch93aw0k03    Worker3   Ready   Active
 icuih1r0n8juo8xigkceniu3j *  Manager1  Ready   Active        Leader
 5pm9f2pzr8ndijqkkblkgqbsf    Worker2   Ready   Active
 9yq4lcmfg0382p39euk8lj9p4    Worker1   Ready   Active

 # docker info will give you a detailed info on your swarm
 [root@Manager1]# docker info
 Containers: 12
 Running: 0
 Paused: 0
 Stopped: 12
 Images: 1
 Server Version: 1.13.1
 Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 54
 Dirperm1 Supported: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Swarm: active
 NodeID: icuih1r0n8juo8xigkceniu3j
 Is Manager: true
 ClusterID: hpvfpcevwt8144bj65yk744q8
 Managers: 1
 Nodes: 6
 Orchestration:
 .
 ..
 Node Address: 10.91.20.119
 Manager Addresses:
 10.91.20.119:2377
 ......
 ..


now creating a SERVICE and running it on docker swarm
(the whole idea of setting this orchestration layer is, we don't need to worry about our app as where it is running but it will be up for the whole time)


$ docker serivce create | update | ls | ps | inspect | rm
ex:
$ docker service update -image=punitporwal07/apache:2.0 --detach=true apache $ docker service scale >> docker service update --replicas $ docker service scale Name=7 $ docker service ps Name
$ docker network create -d overlay pp-net
$ docker service create --name myswarmapp -p 9090:80 punitporwal07/apache rvzrpe4szt0vdyqte7g7tfshs



by doing this, any time when you gonna hit your exposed port for service to any host/IP in swarm it will give you your application, without having its container running on it. (service will be running only on leader/manager1)

accessing the service now:


NOTE: after advertising listen address to the docker swarm, you may get an error next time when you try to initialize the docker daemon. (if you are using dynamic IP)


# below two files hold your IP and failed to initialize docker-daemon
/var/lib/docker/swarm/docker-state.json /var/lib/docker/swarm/state.json

# sample error message
ERRO[0001] cluster exited with error: failed to listen on remote API
address: listen tcp 10.91.20.119:2377: bind: cannot assign requested address
FATA[0001] Error creating cluster component: swarm component could
not be started: failed to listen on remote API address: listen tcp
10.91.20.119:2377: bind: cannot assign requested address


change the IP and initialize it again

 
  $ service docker restart
  


k/r,
P

21 January 2019

what is docker-compose

When you wish to run services together and want to run them as single unit then docker-compose is the tool for you, which allows you to run multiple services as kind of microservice by defining them in a single configuration file.
  • docker-compose is a docker tool for defining and running multi containers docker applications.
  • docker-compose allows us to define all the services in a configuration file and with one command it will spin up all the containers that we need.
  • it uses yaml files to configure application services (docker-compose.yml)
  • it uses a single command to start and stop all the services (docker-compose up & docker-compose down)
  • it can scale up services whenever required.
by default, this tool is automatically installed when you are on windows or mac with docker v1.12+

but if you are on Linux try this command given at GitHub for docker-compose


$ curl -L https://github.com/docker/compose/releases/download/1.23.2/docker-compose \
-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
$ chmod +x /usr/local/bin/docker-compose


alternatively you can find the latest version available here at github

docker-compose.yml prototype will look like:

version:
services:
  image:
network:
volume:


version: first thing first define version of the docker-compose that we are using, there is no restrictions of not to use latest version of compose so I have used '3' here

version: '3'

services: service definition contains configuration which will be applied to each container started for that service, much like passing a command-line parameter to docker run

---
version: ‘3’
services:
webserver:
image: punitporwal07/apache
ports:
- “9090:80” database:
image: mysql
ports: - “4041:3306” environment:
- MYSQL_ROOT_PASSWORD=password - MYSQL_USER=user - MYSQL_PASSWORD=password - MYSQL_DATABASE=demodb
...

so instead of defining items in the docker run command, now we can define it more easily in the configuration file here but with a little bit of syntax

now launch the service using a simple command docker-compose up and it will spin up MySQL and apache in fractions of minutes for you.

--

03 February 2021

How ECS & ECR works in AWS

Elastic Container Service is a container orchestration service provided by AWS that will provision and manage your cluster of containers on your behalf. It comprises all the services that you need to manage services.
Control Plane is one of them that allows you to schedule and orchestrate your containers.
Secondly, you have worker nodes where your application containers actually run and these worker nodes are nothing but your EC2 or Fargate instances which has Docker/container runtime & ECS agent installed in it which is connected to ECS.

So in ECS, you are deligating the complex task of managing and orchestrating containers to a service that will take care of your cluster but you do need to do the following task in order to use it

  • Create EC2 instances for worker nodes to host containers
  • Connect EC2 instances to ECS cluster
  • Check whether you have enough resources like CPU/ram in EC2-I's to host new containers
  • You are required to manage the operating system
  • Each worker nodes should have a container runtime & ECS agent installed
this way you are deligating all the complex task of managing containers to an AWS service however you are still required to manage your infrastructure used during it.

EC2 & Fargate
what if you want to delegate your infrastructure of AWS as well used during ECS
i.e. management of container orchestration by AWS using ECS
&   management of hosting infrastructure also by AWS using FARGATE

Basically, it is an alternative of creating any EC2 instances and connecting them to the ECS cluster instead, you use the fargate interface that will spin up all the required VM to host our containers.

Fargate is a serverless way to launch containers, here we feed AWS fargate about our container and then its fargete job to spin up the most suitable VM after analyzing our container at various aspects like network, storage, RAM & other resource requirements, and provision a server to host this container and do the same every time you introduce it with a new container.
By using the fargate in place of launching EC2-I's manually you have the advantage of not worrying about creating separate EC2 instances and managing their lifecycle that way you will have the exact infrastructure resources needed to run your containers i.e. pay only for what you use.

ECS or fargate both have the capability to connect to other AWS resources like
  • cloudwatch for monitoring.
  • ELB for load balancing. 
  • IAM for users and permissions.
  • VPC for networking etc.
AWS does provide the flexibility to use a combination of both EC2 & fargate at the same time.

What is ECR
ECR stands for Elastic Container Registry, which is similar to any other container repository that stores and manages docker images like dockerHub, Nexus, Harbor, etc.

start setting up with your EC2 instance along with the Docker image


 launch an EC2 instance and do the following
 $ sudo yum update -y
 $ sudo amazon-linux-extras install docker
 $ sudo service docker start
 $ sudo usermod -a -G docker ec2-user
 $ docker info

 if it gives an error : 
 Server:ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. 
 Is the docker daemon running?
 
 try rebooting your instance

 if the error continues, check the status
 $ sudo systemctl status docker // if the status is inactive dead
 $ sudo systemctl start docker

 if still persist, do the following
 $ sudo gpasswd -a ec2-user docker 
 // or manually add ec2-user to docker group in /etc/group file
 and restart session error will go away and you are good to run docker commands

 Pick any Dockerfile of your choice and start creating an image out of it
 $ vi Dockerfile
 $ docker build -t ecrtest:1.0 .   // this will create an image from your Dockerfile
 $ docker run -d 80:80 ecrtest:1.0 // make sure your security group is updated with port 80


getting started with ECR (aws-doc)


 Now login to ECR by following the below command
 $ aws ecr get-login-password --region ap-south-1 \
 | docker login --username AWS --password-stdin 2827**735.dkr.ecr.ap-south-1.aws.com
 // here highlighted in green is your ECR-repository URL

 if you get an error you need to configure your AWS credentials by running
 $ aws configure
 
 once you able to connect successfully using the above command 
 create a repository and try pushing the image we just created
 $ aws ecr create-repository \
    --repository-name myrepo \
    --image-scanning-configuration scanOnPush=true \
    --region ap-south-1
 $ docker tag <imagetag> ecr-repositoryURL/apache:1.0
 $ docker push repositoryURL/apache:1.0  


you should be able to see your image pushed into your container repository now

Once your image is available grab Image URI which will be used in ECS when we create a task definition to deploy this as a container.

Considering your ECS cluster is already up and running if not you can follow the steps here, it is straightforward to spin up one.

Next, you need to create a task definition from your ECS cluster either using FARGATE/EC2 or can attach an EXTERNAL instance type, so that when you run a task it will use this definition that will further be used to host your container fetching from ECR, and that's where you provide your image URI. 

Remember: launch type of task should be the same as the type of your task definition


follow the aws-doc to create a task definition. It's moderately easy to create one.

after the task is created & deployed you need to run it, on running the task it will deploy the container on EC2 instance linked with your ECS if the launch type is used as EC2, using an image from your ECR and you should be able to see your container up and running under task section.

Perhaps if launch type is used as FARGATE it will deploy your container to AWS-managed infrastructure, where no Amazon EC2 instances are involved and uses awsvpc as network mode. You should be able to see your container up and running the same way under the task section.

Few things to remember
  • You use/configure a Service when you want to introduce an Application Load Balancer for your containerized app in ECS Fargate. Here service will create task's for you and you can access them using the DNS name generated by your LB.
  • When you start an ECS optimized image, it starts the ECS agent on the instance by default. The ecs agent registers the instance with the default ecs cluster.
  • For your instance to be available on the cluster, you will have to create the default cluster.
  • if you have a custom ecs cluster, you can set the cluster name using the userdata section.
  • The ecs agent expects the cluster name inside the ecs.config file available at  ~/etc/ecs/ecs.config  

 You can set it up at instance boot up using userdata script.
 #!/bin/bash
 echo ECS_CLUSTER={cluster_name} >> /etc/ecs/ecs.config

24 January 2018

Networking in docker

Docker works on the principle of running containers as a service, when you run a container it has its own attributes like namespace ip-address port etc. These attributes are allocated to containers by docker daemon at run time. There are ways to control this behaviour like creating namespaces of your choice at the time of launching them.

Same way when it comes to IP addresses you can create your own docker network which can give a static ip to your container or its underline service. 

docker comes with 5 kinds of networking drivers:

bridge: when you want to communicate between standalone containers.
overlay: to connect multiple Docker daemons together and enable swarm services to communicate with each other.
host: For standalone containers, remove network isolation between the container and the Docker host. 
macvlan: Allow you to assign a MAC address to container, making it appear as a physical device on your network.
none: disables all networking.

by default, the bridge is the default driver that got created when you launch any of the containers as a service.

How one can create its own docker network as per requirement 

the syntax to create a network is : 

$ docker network create --options networkname

few widely used options are:

--driver drivername
--subnet=subnetrange/x
--gateway=anIPfromdefinedsubnet

for example, assigning static IP out of your CIDR block


 $ docker network create --driver overlay --subnet=192.168.0.0/26 --gateway=192.168.0.1 my-network


additionally you can use this created network for your container at the time of its launch

for example:


 $ docker run --publish 61000:1414 --publish 61001:9443 --net my-network --ip 192.168.0.3 --detach --env
  MY_APP_PASSWORD=password punitporwal07/apache:2.2.29


this way your container will be allocated with a static IP within your defined subnet range.

HOW DO YOU USE PORT MAPPING


 when you expose a port from Dockerfile that means 
 you are mapping a port defined in your image to your newly launched container, use:
 $ docker run -d -p 5001:80 --name=contaniername imagename

 when you want to change the protocol from default i.e TCP to UDP, use:
 $ docker run -d -p 5001:80/udp --name=continername imagename

 let's say when you want to expose your image port to any specific IP address from your host, use:
 $ docker run -d -p 192.168.0.100:5002:80 --name=contaniername myimagename

 when you want to map multiple ports exposed in your Dockerfile to high random available ports, use:
 $ docker run -d -P --name=contaniername imagename

 to expose a port range, use:
 $ docker run -it -p 51000-51006:51000-51006 imagename:tag

        also you can use EXPOSE 51000-51006 in your Dockerfile

 to check port mapping, use:
 $ docker port imagename


28 February 2018

Debugging your kubernetes cluster

Whether you are a developer or an operations person, Kubernetes is an integral part of your daily routine if you are working with a microservice architecture. Therefore, it is always preferable to perform all your research and development on a test cluster before implementing it in an enterprise environment.

if you have not experienced setting up your own cluster yet, here is an article that will walk you through the process of launching your own k8s-cluster in different fashions.    

there are possible failures one can encounter in their k8s cluster so here is the basic approach one can follow to debug into failures.

 1. if a node is going into notReady state as below

 $ kubectl get nodes
   NAME          STATUS   ROLES    AGE    VERSION
   controlPlane  Ready    master   192d   v1.12.10+1.0.15.el7
   worker1       Ready    <none>   192d   v1.12.10+1.0.14.el7
   worker2       NotReady <none>   192d   v1.12.10+1.0.14.el7

 # check if any of your cluster component is un-healthy or failed
 $ kubectl get pods -n kube-system
   NAME                                  READY   STATUS    RESTARTS   AGE
 calico-kube-controllers-bc8f7d57-p5vhk  1/1     Running   1          192d
 calico-node-9xtfr                       1/1     NodeLost  1          192d
calico-node-tpjz9 1/1 Running 1 192d calico-node-vh766 1/1 Running 1 192d coredns-bb49df795-9fn9g 1/1 Running 1 192d coredns-bb49df795-qq6cm 1/1 Running 1 192d etcd-bld09758002 1/1 Running 1 192d kube-apiserver-bld09758002 1/1 Running 1 192d kube-controller-manager-bld09758002 1/1 Running 1 192d kube-proxy-57n8h 1/1 NodeLost 1 192d kube-proxy-gvbkh 1/1 Running 1 192d kube-proxy-tzknm 1/1 Running 1 192d kube-scheduler-bld09758002 1/1 Running 1 192d

 # describe node to check what is causing issue
 $ kubectl describe node nodeName
 
 # ssh to node and ensure kubelet/docker services are running
 $ sudo systemctl status kubelet
 $ sudo systemctl status docker

 # troubleshoot services if not-running in depth
 $ sudo journalctl -u kubelet

 # if services are getting failed frequently try to reset daemon
 $ sudo systemctl daemon-reload


 
 Things to remember - In kube-system namespace for every cluster

 kube-apiserver  1pod/Master
 kube-controller 1pod/Master
kube-scheduler 1pod/Master
 etcd-node       1pod/Master

 core-dns - 2 pods/cluster and can be anywhere M/W

 calico/Network 1 pod/node every M/W (enable n/wing b/w pods)

 kube-proxy 1 pod/node every M/W (enable n/wing b/w nodes)

Some common failures


 # If fails with 
   [ERROR CRI]: container runtime is not running:
   rm /etc/containerd/config.toml
   systemctl restart containerd

 # If kubelet service fails at -
   Active: activating (auto-restart) (Result: exit-code) since Wed 2019-06-28 1
   first check "sudo journalctl -xeu kubelet" logs, if it says - 
            failed to load kubelet config file
   check the Drop-in file which kubelet is loading while activating the service

   $ sudo systemctl status kubelet

   if it is not loading - /etc/systemd/system/kubelet.service.d then correct it, 
   or remove anyother file that it is attempting to load & reload the daemon and restart service.

   $ systemctl daemon-reload
   $ systemctl restart kubelet

  also ensure kubelet, kubeadm, kubernetes-cni, kubernetes-cni-plugins & kubectl are of same minor version

 # If fails with 
   lower docker version update docker:
   docker.io (used for older versions 1.10.x)
   docker-engine (is used for before 1.13.x )
   docker-ce ( used for higher version since 17.03)
 $ apt-get install docker-engine 

 # If fails with 
   Unable to connect to the server: net/http: 
   request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
 $ kubectl version // check kubectl client and server version - they shouldn't mismatch

 # if fails with
   [WARNING Service-Kubelet]: kubelet service is not enabled, 
   please run 'systemctl enable kubelet.service'
 $ systemctl enable kubelet.service

 # If fails with
   [ERROR Swap]: running with swap on is not supported. Please disable swap.
 $ swapoff -a
 $ sed -i '/ swap / s/^/#/' /etc/fstab

 # if fails with
   [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
   use command with flag --ignore-preflight-errors=NumCPU
   this will actually skip the issue. Please note, this is OK to use in Dev/test perhaps not in production. 

 # Run again 
 $ kubeadm init --apiserver-advertise-address=MasterIP \ 
   --pod-network-cidr=192.168.0.0/16 \ 
   --ignore-preflight-errors=NumCPU

 # if getting error
   The connection to the server localhost:8080 was refused - did you specify the right host or port?
   or on running 'kubectl version' it only gives client version but above error on server version
   if there is no context configured in your client you get such error
 $ kubectl config view
   apiVersion: v1
   clusters: []
   contexts: []
   current-context: ""
   kind: Config
   preferences: {}
   users: []

   you might have forgot to run below commands
   
 $ mkdir -p $HOME/.kube
 $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 $ sudo chown $(id -u):$(id -g) $HOME/.kube/config

 # Make a note - to keep the k8s-cluster functional, following containers should be up & running all the time
 - kube-controller-manager > kube-scheduler > kube-etcd > kube-apiserver > kube-proxy + n/w containers (flannel/calico)  

 # Some magic commands -
   systemctl stop kubelet
   systemctl stop docker
   iptables --flush
   iptables -tnat --flush
   systemctl start kubelet
   systemctl start docker

   journalctl -xeu kubelet | less

 # Also, ensure following ports are open on masterNode -
   sudo firewall-cmd --permanent --add-port=6443/tcp
   sudo firewall-cmd --permanent --add-port=2379-2380/tcp
   sudo firewall-cmd --permanent --add-port=10250/tcp
   sudo firewall-cmd --permanent --add-port=10251/tcp
   sudo firewall-cmd --permanent --add-port=10252/tcp
   sudo firewall-cmd --permanent --add-port=10255/tcp
   sudo firewall-cmd –reload

   Ensure following ports are open on workerNode -
   sudo firewall-cmd --permanent --add-port=10251/tcp
   sudo firewall-cmd --permanent --add-port=10255/tcp
   firewall-cmd --reload

 # to check if the bridge traffic is flowing through the firewall? should return 1

   cat /proc/sys/net/bridge/bridge-nf-call-iptables
   cat /proc/sys/net/bridge/bridge-nf-call-ip6tables

 # if namespace stuck in terminating state 

   kubectl get ns mynamespace -o json > ns.json

   edit the ns.json file & empty the finalizer [""] and remove the kubernetes from it. even remove " "

   kubectl proxy

   open another terminal and run below command 

   curl -k -H "Content-Type: application/json" -X PUT --data-binary @ns.json http://127.0.0.1:8001/api/v1/namespaces/mynamespace/finalize

   boom namespace is gone by now


if you want to restore your cluster from a previously working state, take a backup of etcd directory which acts as a cluster database that holds all the kluster resource data present under /var/lib/etcd and may look like below 


fix the issue and restore the database and then restart its etcd-container.

Troubleshooting your application deployed on your kluster

Apart from doing kubectl logs you should investigate what is preventing your application to run, you're probably running into one of the following - 
  • The image can't be pulled
  • there's a missing secret of volume
  • no space in the cluster to schedule the workload
  • taints or affinity rules preventing the pod from being scheduled
in such case we have a magic command which one should get tattooed as there is not shortform to run this command that may help to investigate the issue


   kubectl get events --sort-by=.metadata.creationTimestamp -A
 
   kubectl top node



happy troubleshooting...