11 July 2017

ANSIBLE: Configuration management tool for Infra automation

https://www.middlewareandme.tech/search/label/ansible
Ansible is an agent-less task execution engine, used for configuring, managing and installing software onto clients and nodes without any downtime and without an agent installed on them.
It uses SSH to communicate with clients.
provided all the nodes should have python installed in them + every step should not be carried with root user instead with ansible user.
Ansible needs:
  • SSH connection
  • a user
  • Python 2.4+
  • It works on the principle of 'PUSH Based', which means it pushes modules from VCS to servers directly without the intervention of any intermediate client/agent
  • it contains Playbooks that are written in YAML code ( YAML ain't markup language)






Hierarchy of Ansible roles -

- create <roles> directory
    - inside roles create roles <roleDir>
    - inside <roleDir> -> create taskDir, defaultDir, templateDir
        - inside <taskDir> -> add all the ymls.
        - inside <defaultDir> -> add main.yml
        - inside <templateDir> -> add any template file if you are calling in any playbook

- last at <roles> directory create a yaml and call  your <roleDir> as shown below -


- name: NameYourPlay
  hosts: "{{ target_hosts | default ('none') }}"
  become: yes
  become_method: sudo
  roles:
    - roleDir
 


Prototype of ansible-playbook



Ansible Contain Playbooks 
Playbook has a number of plays
Play contains tasks
Task calls core or custom modules
Task can use templates
Handlers triggers from notify
executed at the end and only once

- Ansible contains more than 750 modules and can be customized and turned into custom modules.
- Modules gets executed when you run Playbook onto your 1..n nodes.
- For connectivity it uses
  • SSH password less connection by generating Public key install on all your nodes
  • Connection Plugins
  • export ANSIBLE_HOST_KEY_CHECKING=False

 Quickly start with Ansible, try using my docker image
 pull : docker pull punitporwal07/ansible:2.6
 run : docker run -it punitporwal07/ansible:2.6
 test: ./runansibletest.sh
 
 By default ansible package is not available in some yum repositories, 
 so you need to enable/add EPEL(extra package for Enterprise Linux) repository 
 which is maintained over at Fedora Project
 
 $ rpm -ivh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
 $ yum install ansible -y
 
 Installing Ansible on Ubuntu
 $ yum update -y
 $ apt-get update
 $ apt-get install ansible
 or
 $ sudo yum install ansible -y
 $ ansible --version

 Installing Ansible & boto on/for amazon EC2 instance
 $ amazon-linux-extras install epel 
 $ sudo yum update -y
 $ sudo yum install ansible -y
 $ sudo yum install python-boto python-boto3 -y
 
 Install boto from source, if unable to find package
 $ git clone git://github.com/boto/boto.git
 $ cd boto && python setup.py install
 $ sudo apt-get install -y python3-botocore
  
 SETTING SSH COMMUNICATION BETWEEN SERVERS
 prepare SSH key for remote hosts
 switch to ansible user
 $ ssh-keygen -t rsa              --> enter enter enter
 $ ssh-copy-id -i ansible@nodes   --> enter passwords
                            or
 $ export ANSIBLE_HOST_KEY_CHECKING=False
   //which will bypass the host-key-check
 
 test using
 $ ansible -m ping all
 
 sudo pass command
 $ ansible-playbook --ask-sudo-pass
   //it will prompt for sudo password 

 — If no passwordless ssh setup is there, then hardcode the cred. as below:

 linux1 ansible_host=10.20.192.1 ansible_connection=ssh ansible_user=username ansible_ssh_pass="pass"
 linux2 ansible_host=10.20.192.3 ansible_connection=ssh ansible_user=username ansible_ssh_pass="pass"
linux3 ansible_host=10.20.192.3 ansible_connection=ssh ansible_user=username ansible_ssh_pass="pass" linux1 ansible_host=172.20.192.1 ansible_become_pass="rootpass" ansible_ssh_user=usertossh ansible_ssh_pass="pass" linux2 ansible_host=172.20.192.3 ansible_become_pass="rootpass" ansible_ssh_user=usertossh ansible_ssh_pass="pass" linux3 ansible_host=172.20.192.3 ansible_become_pass="rootpass" ansible_ssh_user=usertossh ansible_ssh_pass="pass"


HOST INVENTORY


 $ /etc/ansible/hosts //default Location
 
 provide list of target IP address which can be grouped as
 
 [local]
 localhost ansible_connection=local
 
 [appserver]
 1.2.3.4
 2.3.4.5
 
 [dbserver]
 3.4.5.6
 

alternatively, you can design your own inventory and place it anywhere

- Inventory is the expression of your environment
- Hostnames, groups, vars are for YOUR use, they have to make sense to YOU
- Ansible cares about hosts and tasks, everything else is in support of that
- Select a single source of truth. or try to minimize duplication of data
- Normally, there is a simpler way to do it
- Ansible makes it easy to switch approaches, don't be afraid to test and try
- Mistakes are not failures

How ansible commands are structured

ansible host-group  + module    +   argument to module
ansible  localhost      -m yum    +  -a "name=nginx state=latest"
ansible  allserver       + -m shell   +   -a 'uptime'
ansible  appserver     -m user    +   -a "name=red group=oracle shell=bin/bash/"

Ad-hoc commands
ansible all -a 'uptime' (determine the uptime of all machines)
ansible -m ping all (test connection with all the host defined in host_inventory)

You can find some sample playbook's from my git repository
apache.yaml

Ansible vault

 
 We can use this utility to secure your sensitive data like password, keys etc
 
 some useful commands of ansible-vault
 
 $ ansible-vault encrypt  //encrypt any file
 $ ansible-vault edit     //edit encrypted file
 $ ansible-vault view     //view encrypted file
 $ ansible-vault rekey    //change the pass of encrypted file
 $ ansible-playbook -i inv.ini playbook.yml --ask-vault-pass 
   // this will ask for vault pass while running playbook

 Using Ansible-vault without inline commands
 
 Create ansible inventory using command
 $ ansible-vault
 
 Define your inventory pass in some hidden text file 
 $ echo mypass > .my-pass.txt 
 
 Add password file in ansible.cfg
 vault_password_file = /path/to_your_file/.my-pass.txt
  

 Now next time when you run any playbook to use an encrypted inventory file it will pick up your inventory pass from the txt file defined in ansible.cfg

      Rundeck also uses the same mechanism when you define your ansible.cfg as a configuration file.

24 February 2017

DevOps

Set of practices that emphasize the collaboration and communication of both software DEVelopers and IT OPerationprofessionals while automating the process of software delivery and infrastructure changes, which aims at establishing a culture and environment where building, testing, and releasing software can happen rapidlyfrequently, and more reliably.

"which ultimately means building digital pipelines that take code from a developer’s laptop all the way to revenue-generating prod awesomeness"


some myths that should be addressed before deep dive into DevOps

Myth - DevOps replace Agile
DevOps is the next step of agile
DevOps principles and practices are compatible with agile
agile is an enabler of DevOps
not a replacement, but is a logical continuation
a 'deployable piece' of code rather than a 'potentially ship-able piece' of code after each sprint

Myth - it "All Dev & No Ops"
the nature of IT Ops work may change.
ops collaborate far earlier in the software life cycle with Devs.
Devs continue to work with Ops long after the code is in prod.

Myth - DevOps is just automation
it requires automation for sure.. But that's not all.. it's much beyond that.

Myth - DevOps is a Tool/Product
it's rather a combination of tools
we don't buy DevOps.. instead, we do DevOps

In an organisation where everything gets automated for seamless delivery the generic logical flow will be:
  1. Developers develop the code and the source code is managed by a Version Control System tool like Gitthen developers send this code to the git repository and any changes made in the code is committed to this repository.
  2. Then Jenkins pull this code from the repository using the git plugin and build it using tools like Ant or Maven.
  3. Configuration management tool like Ansible/Puppet deploys this code & provision testing env. and then Jenkins releases this code on the test environment on which testing is done using tools like selenium
  4. Once the code is tested, pipelines configured using Jenkins send it for deployment on the production server (even production server is provisioned & maintained by tools like Ansible/Puppet)
  5. After deployment, it is continuously monitored by tools like Nagios.
  6. Docker containers provide a quick environment to test the build features. 


here are some useful articles with respect to DevOps





18 August 2020

Rundeck - Runbook Automation tool

Rundeck is an opensource tool that helps you automate & schedule your operational jobs. It provides number of features like scheduling jobs, automating execution of ansible playbooks, notifying about the status of your job in form of sending emails is my favourite.

Configuring rundeck is straight forward, you can install rundeck as a service in your linux host or use it as a docker image as well.

quick setup
$ vi /etc/yum.repos.d/rundeck.repo [rundeck] name=rundeck baseurl=https://packages.rundeck.com/pagerduty/rundeck/rpm_any/rpm_any/$basearch repo_gpgcheck=1 gpgcheck=1 enabled=1 gpgkey=https://packages.rundeck.com/pagerduty/rundeck/gpgkey,https://docs.rundeck.com/keys/BUILD-GPG-KEY-20230105.key sslverify=1 sslcacert=/etc/pki/tls/certs/ca-bundle.crt metadata_expire=300
$ yum install rundeck java $ service rundeckd start
$ service rundeckd status
 rundeckd.service - SYSV: rundeckd, providing rundeckd
   Loaded: loaded (/etc/rc.d/init.d/rundeckd; bad; vendor preset: disabled)
   Active: active (running) since Mon 2020-08-17 13:23:14 BST; 20h ago

$ tail -f /var/log/rundeck/service.log
[2020-08-14T09:02:28,539] INFO  rundeckapp.BootStrap - Rundeck is ACTIVE: executions can be run.
[2020-08-14T09:02:28,635] WARN  rundeckapp.BootStrap - [Development Mode] Usage of H2 database is recommended only for development and testing
[2020-08-14T09:02:28,899] INFO  rundeckapp.BootStrap - Rundeck startup finished in 646ms
[2020-08-14T09:02:28,991] INFO  rundeckapp.Application - Started Application in 25.616 seconds (JVM running for 28.068)
Grails application running at http://localhost:4440 in environment: production


quick setup as a docker Image and config customization

$ docker pull rundeck/rundeck

# Update the default port if it is blocked (4440) & localhost to DNS, in below three files
$ vi /etc/rundeck/profile //only port
$ vi /etc/rundeck/framework.properties // IP & PORT
$ vi /etc/rundeck/rundeck-config.properties // IP & PORT

# changing the default password of rundeck
$ cd /etc/rundeck/
edit realm.properties and change the admin values to something new

# adding a new user
$ cd /etc/rundeck/
$ sudo vi realm.properties
(add following lines next to admin:admin,user,admin line)
        user1: user1pass,user,admin,architect,deploy,build
   where user,admin,architect,deploy,build are different roles we can assign to user1


now login to rundeck console with admin access and navigate to 

settings > Access Control  > + Create ACL Policy

add following two scopes in order to give read access as an example to user user1

# Project scope
descriptionuser1 with read access to projects.
context:
  project'.*'
for:
  resource:
    - equals:
        kindjob
      allow: [read# allow to read jobs
    - equals:
        kindnode
      allow: [read# allow to read node sources
    - equals:
        kindevent
      allow: [read]
  job:
    - allow: [read# allow read of all jobs
  adhoc:
    - deny: [run# don't allow adhoc execution
  node:
    - allow: [run# allow run on nodes with the tag 'mytag'
    
by:
  groupadmin

---
# Application scope
descriptionapplication level ACL.
context:
  application'rundeck'
for:
  resource:
    - equals:
        kindproject
      allow: [read]
    - equals:
        kindsystem
      allow: [read]
    - equals:
        kindsystem_acl
      allow: [read]
    - equals:
        kinduser
      allow: [admin]
  project:
    - match:
        name'.*'
      allow: [read]

by:
  groupadmin

happy rundecking!