Archives

now browsing by author

 

How to deploy Open-AKC(Authorized Key Chain)

.

Acting as a centralised trust management platform:

By allowing the “authorized_keys” mechanism on the hosts to be completely disabled, OpenAKC

permits SSH trust across an entire estate to be managed (with rich control and monitoring features)

centrally by “systems administration” or “information security” staff. This means that users, or

application developers etc. cannot add or remove trust relationships, effectively enforcing any

whitelist or approval process you might want to establish for the creation of trust relationships

within an estate.

.

A practical ‘Jump Host’:

 

Having worked in many Linux environments, the author of the software has seen a number of ‘jump

host’ solutions using dubious mechanisms such as “sudo” and others used to map users from AD or

LDAP directories to SSH trust relationships using shared private keys etc. and these solutions are

almost universally highly insecure.

OpenAKC implements a new approach and acts as an “drop in” upgrade for a legacy solution by simply migrating users to personal rather than sharedkeys with a “self service” key management mechanism. Requiring personal keys have pass phrases ensures increased security, and avoids ad-hoc automation from user accounts. The system provides rich control and monitoring features, while users can use familiar tools with little to no disruption to their workflow.

.

The problems everyone thinks about, but never finds a good solution.

 Root access to my servers so that everything is auditable
 How to handle Identity Access Management(IAM) without joining every server to the domain.
The problem with joining every server to the domain:
 If I were to gain access using your AD user. I could then browse /home and find any user accounts that logged as admin, which I can then brute force attack to gain root.
 I can also use ‘id’ to see which groups this user is attached to leaving me open to other forms or attacks and holes.

 As soon as someone sudo’s to root. There is zero control on limiting what this root user can do. Which is a huge problem when getting compromised. Every key stroke is not logged and if you have multiple people sudo as root at the same time the logs can get blurry.

 Imagine…having the ability to give your admins root but also tying the root user’s hands from doing certain things that you deem too sensitive any one outside of the security team to do.
 Imagine not having to join every server to the domain but still have root access controls
 Imagine being able to eliminate user/pass login entirely
 Imagine being able to deploy this faster than ldap or sssd across multiple distros.

.

Well guess what….this is exactly what im talking about today. This architecture does take a few steps and time to understand its inner workings but security wise trumps anything out there currently being used by most folks.

.

This is a overview of a simplified architecture. This can be scaled out to new or legacy environment using many distros without interruption.

Combined jump host/security server architecture: This approach is generally used when it’s a small group managing things:

.

.

Diagram Description automatically generated

Benefits: The advantage here is your security server and your jumphost are on the same server. If you’re admin team is also managing the security then this approach is ideal because when diagnosing and editing role rules everything is in one place. Another key benefit… There are only couple client packages that are deployed on client machines upon deployment the client is already brought into the trust, so no additional work is required to bring it in.

.

Cons: If your jumphost goes down, so does your security server. Now this can be scaled out so there are two servers if one is unavailable it tries the second. But if you have one this could be a single point of failure. Keep in mind this is a VM and would takes seconds to bring back up, so durability is very good from a infrastructure stand point..

.

.

Or the alternative:

.

A Segregated jumphost/security server architecture: This approach is generally used when you have large groups and large infrastructure managing things.

Diagram Description automatically generated

.

Benefits: Now the advantage here is if you have a multiple groups and need to tighten the security. You can have a your security server and your jump hosts segregated. These two machines are the only two machines that talk you AD. You have roles setup on the security server. If your user is listed in the role on the security server and your apart the AD group that is allowed to login via ssh.

The security server will allow you to login as root via registered ssh keys. You can also strip away “roots” abilities, and every keystroke is logged per user on the security server. A key benefit here is that because the client machines are not joined to the domain. A hacker would never be able to determine information that leads to the heart of your network such as AD, the groups setup, and any of its users. The other benefit is the jump hosts are easily deployed as needed and ready to go if needed. Since the security stuff is not local, the jump host is also secured.

.

Cons: If your security servers are not functioning and you have disabled user/pass. You will be locked out unless you single user. However this is easily fixed by setting up a strong root/pass that can bypass all security and only give this to one person, such as your manager or someone in secureops. Which would be only used ever used in an extreme dire situation. I have yet to see this ever required once this is implemented correctly.

.

.

Special Features:

.

 You can setup incident logging for production machines. (eg, Service now ticketing).
 Incident#, RFC#, Jira, brief description.
 You can make it so that root can not change any file that an immutable flag. Thus protecting key files, this is just one of the many attributes you can give or remove from root.

.

.

How to deploy Segregated jumphost/security server architecture:

.

.

Okay so the first thing we want to do is setup our security server host to Active Directory. The reason for this is we want to be able to setup centralised user management coupled with using authorised keys to jump to client machines.

.

Centos 7

 This post assumes you have already deployed two centos 7 machines. Don’t use centos 8 because. Redhat recently decided to screw everyone and kill the centos project. Centos 7 will have updates till 2024.
 This post also assumes you have active directory setup and you have a user inside linuxgroups of somekind.

.

Note: make sure you disable firewalld and selinux on your machine. Just a extra layer that not needed and only serves to tie your own hands behind your own back.

.

.

1.Join centos 7 to an existing domain

Install the following packages will be be using sssd/kerebros

a.yum install oddjob realmd samba samba-common oddjob-mkhomedir sssd adcli

.

2.Next you want to edit your /etc/resolv.conf so that it has one of the nameservers as your AD server. This is so it can resolve the necessary dns records.
b.Vi /etc/resolv.conf  
 Nameserver 192.168.1.300
 Nameserver 192.168.1.301

.

3.Now you need to discover the realm by searching for the name of your AD host
c.  realm discover AD.NICKTAILOR.COM (this is case sensitive)

.

.

[root@securityack1 ~]# realm discover AD.NICKTAILOR.COM

ad.nicktailor.com

type: kerberos

realm-name: AD.NICKTAILOR.COM

domain-name: ad.nicktailor.com

configured: kerberosmember

server-software: active-directory

client-software: sssd

  required-package: oddjob

required-package: oddjob-mkhomedir

  required-package: sssd

  required-package: adcli

required-package: samba-common-tools

login-formats: %U

login-policy: allow-realm-logins

.

.

.

.

.

4.You now want to join your jump host to the domain
d.realm join –user=admin ad.nicktailor.com
it will ask you for the AD admin password, enter it and it should go to the next prompt. This step will create the all the necessary files /etc/sssd/sssd.conf /etc/krb5.conf /etc/smb.conf file you need.

.

e.if it works you can check it by
 id nicktailor@ad.nicktailor.com
1.you can set it in the /etc/sssd/sssd.conf So you don’t need the @ad.nicktaior.com when your run your ‘id’

.

.

5.Now we want to add nicktailor user to sudo on the wheel
f.usermodaG sudo nicktailor

.

.

6.Now you we want to install the openAKC repository
g.curl https://netlore.github.io/OpenAKC/repos/openakc-el7.repo | sudo tee /etc/yum.repos.d/openakc.repo
h.Now install the openakc package for the jump host

yum install openakc-server

i.Next change su to the user account your going to make admin on the security server
· Su nickatilor

.

j.You want to generate new ssh keys
 ssh-keygen -t rsa
k.Now you want to register that key to openakc security server
 openakc register

.

· cd /home/nicktailor/.openakc/
· ls -al /home/nicktailor/.openakc/

.

 check to see if the openakc-user-client-nicktailorpubkey.pem is there

.

.

l.Copy the key to the security keys holding (might have do this as root
 cp openakc-user-client-nicktailorpubkey.pem /var/lib/openakc/keys/

.

Example:

.

[nicktailor@security1 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/nicktailor/.ssh/id_rsa):

/home/tailorn/.ssh/id_rsa already exists.

Overwrite (y/n)? y

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/nictailor/.ssh/id_rsa.

Your public key has been saved in /home/nicktailor/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:udhNKEp0txzfup7IxhUwNA+VSviWP1mu/aKPA5vZb3w tailorn@jumphost1.nicktailor.com

The key’s randomart image is:

+—[RSA 2048]—-+

| o+… |

| . ++. |

| . .oo=. |

| . . o=*… |

| . ..S.o=. |

| . . +.+=.. |

| . ..oBo= |

| .*.++= E |

| .o.**o+. |

+—-[SHA256]—–+

[nicktailor@jumphost1 ~]$ openakc register

OpenAKC Copyright (C) 2019-2020 A. James Lewis. Version is 1.0.0~alpha18-1.el7.

.

This program comes with ABSOLUTELY NO WARRANTY; see “license” option.

This is free software, and you are welcome to redistribute it

under certain conditions; See LICENSE file for further details.

.

Passphrase is requested to ensure you own this key.

.

Enter passphrase:

.

Escalating to perform API call

.

Connected to OpenAKC server. Sending key registration request

OK: Request processed

.

.

7.Now we want to setup a role on the security server for the new user while you are su’d to nicktailor still.
 openakc editrole root@DEFAULT

.

.

8.You will also need to copy the key you created to the jumphost or you can just create another on the jump host and then copy it over the security server.
m.ssh-copy-id –i ~/.ssh/id_rsa.pub root@192.168.1.200

.

Here you will create a role block that will allow said user to jump to any client so long as your key is registered the openakc security server.

.

RULE=2020/01/13 19:17,2030/01/13 20:17,user,nicktailor

DAY=any

TIM=any

SHELL=/bin/bash

CMD=any

SCP=s,^/,/data/,g

CAP=cap_linux_immutable

REC=yes

FROM=any

.

RULE=2020/01/13 19:17,2030/01/13 20:17,group,linuxusers

DAY=any

TIM=any

SHELL=/bin/bash

CMD=any

SCP=s,^/,/data/,g

CAP=cap_linux_immutable

REC=yes

FROM=any

.

 Save the file

.

.

.

Note: The CAP section allows you to disable root abilities there is a long list of things. This particular one. Disables root from being able to edit any file with an immutable flag. So all your root users can not change these types of file eve as root.

.

You starting to see why this is how you do shit? Think for yourselves not because someone else said that best practice. Jeez…

.

.

.

Note: this is just a development setup I am doing for you. It can be scaled with two security servers using gluster of nfs to share the files system. This server will be what all the clients check with before allowing anyone to enter. They will need to first exist in AD and then their root key will need to be registered here, and then they will need to be allowed in the appropriate role and sssd and ssh groups before they can get in. This also eliminate the need to join any clients to the domain and protects against any hacker from being able to query the domain controller for user groups and if users exist. The would of also needed to get on to the jump host, have your key and also know your root passphrase. Unlikely going to happen

.

.

Okay now were going to setup the jumphost jump1.nickatilor.com

.

Note: You can have as many jumphosts as you like. This should be the only entry point to your servers. The jump host also talks to the security server. So even if you tried to go directly the server. You couldn’t since its only allowed from here. This is how you set shit up so you don’t end up like solarwinds and so many other idiots who follow rules old and out of date rule blindly.

.

.

1.First thing we wan to do is add the the repository on the jump host.

.

.

Note: You also add this server to the domain. So please follow the joining of the domain setups from above and then carry on from here.

.

 curl https://netlore.github.io/OpenAKC/repos/openakc-el7.repo | sudo tee /etc/yum.repos.d/openakc.repo

.

2.Now you want to install the openakc-tools package on the jumphost
a.yum install openakctools

.

3.Now edit the /etc/openakc/openakc.conf

APIS=”nickack1.nicktailor.com

PORT=”889″

.

4.Now login in as nicktailor or su as nicktailor to allow sssd to create the home directory.
b.su ntailor
c.copy over the key from the security server logged in from the security server

ssh-copy-id –i ~/.ssh/id_rsa.pub root@192.168.1.200

.

5.Now while you are nicktailor on the jump host your going to be a ping to ensure its communication is working:

.

[nicktailor@jumphost1 keys]$ openakc ping

OpenAKC Copyright (C) 2019-2020 A. James Lewis. Version is 1.0.0~alpha18-1.el7.

.

This program comes with ABSOLUTELY NO WARRANTY; see “license” option.

This is free software, and you are welcome to redistribute it

under certain conditions; See LICENSE file for further details.

.

Connected to OpenAKC server. Sending Test Run Ping Message

Test Run Response – OK: Pong! – from server – securityakc1.nicktailor.com

.

If you see that above that is a good sign

.

.

.

Now were going to add a client machine so you jump to it. This is the easiest part. So say you have a bunch of legacy systems and you want to centralise login without joining to the domain, you want everyone who using root to be tracked by logging every keystroke and logging what incident they logged in for to use root.

.

.

1.Logged on to the client machine as root
 Add the openack repo

curl https://netlore.github.io/OpenAKC/repos/openakc-el7.repo | sudo tee

/etc/yum.repos.d/openakc.repo

.

2.Next install openakc client package, now this is confusing because he didn’t add client next to the package. If you install the wrong one it’s a bitch to clean up so get right the first time…lol

.

a.yum install openakc

.

3.Tell it where the security server lives by editing this file and saving it

  vi /etc/openakc/openakc.conf

APIS=”192.168.1.200

ENABLED=”yes”

PORT=”889″

CACHETIME=”60″

DEBUG=”no”

PERMITROOT=”yes”

AUDIT=”yes”

QUIZ=”no”

HIDE=”restrict”

FAKESUDO=”yes”

.

Note: Quiz if you set it to yes. When you log into root it will ask for a service now ticket number and description which will get logged on the security server.

.

4. save the file

.

That’s it!….

.

.

Now if you go back the jump host and try to log in

.

[@jumphost1 ~]$ ssh root@192.168.1.38

Enter passphrase for key ‘/home/nicktailor/.ssh/id_rsa‘:

OpenAKC (v1.0.0~alpha18-1.el7) – Interactive Session Initialized

.

[root@nickclient1 ~]#

.

.

This is session is now being logged and and you can not see if the user belongs to the domain

.

[root@nickclient1 ~]# id nickatilor

id: nicktailor: no such user

.

.

This is how you setup security like pro. Hope you enjoyed this.

.

Hope you enjoyed this.

.

Special thank you the author of the project “James Lewis” I enjoyed learning this and am a big fan of the innovation behind the open-akc project.

.

.

.

.

How to add new users:

.

1.Add user to the active directory and said linux group
2.Login vi ssh to the jump host
3.Create an ssh key as your new user
a.Ssh-keygen -t rsa with a passphrase
4.Register key to openakc while on the jumphost
b.Openakc register

.

That’s it your done. Now the new user is able to login into the estate via ssh from the jump host.

.

.

.

.

.

.

.

.

How to add a custom tomcat installation to SystemD with ansible.

Okay so say you have a custom install of tomcat and java, which is what a lot of people do because java update and tomcat updates can bring things down. So things need to be tested before updates and standard patch cycles can end up affecting the environment.

But you want to handle the startup and stopping via systemd to be able to get status outputs and let system handle the service on reboots. This is how to do it slick.

.

Ansible Setup:

 This post assumes you have ansible setup and running. If you don’t search through my blog and you should find a post on how to setup.

Role:

 We are going to setup a custom role to add your custom tomcat install system

Setup the new role:

.

 Create a new directory in /etc/ansible/role for your new role
 mkdir -p /etc/ansible/roles/AddtomcatSystemD/tasks/

.

 Now create a yaml file that will run a set of tasks to set this up for ya.
 vi main.yml

.

Main.yml

===========================================

Note: this will install the redhat tomcat version of tomcat. Do not worry we are not going to be using this tomcat. This is just so redhat automatically setups all the needed services and locations. We will then update the SystemD config for tomcat to use the custom version.

– name: Install the latest version of tomcat

package:

name: tomcat

state: latest

.

Note: This symlink is important as tomcat default install by redhat is inside /opt/tomcat. Update the src to the custom location of your tomcat

.

– name: Create symbolic link for “tomcat” in /opt

file:

    src: /custom/install/tomcat

path: /opt/tomcat

force: yes

state: link

.

Note: This will enable tomcat to start up on reboot

.

– name: Enable tomcat service on startup

shell: systemctl enable tomcat

.

Note: This is the tomcat systemd service file that systemd uses for the default install. We are going to empty.

.

– name: Null tomcat.service file

shell: “>/etc/systemd/system/tomcat.service

.

Note: We are now going to add our custom block for tomcat into the tomcat.service file we just emptied above using the blockinfle module. This means that this whole section will also be managed by ansible as well. Make sure you adjust the java_home if your java isn’t location inside tomcat. Along with the user,group,umask for to your custom tomcat.

.

– name: Edit tomcat.service for systemd

  blockinfile:

    dest: /etc/systemd/system/tomcat.service

    insertafter:

block: |

[Unit]

Description=Apache Tomcat Web Application Container

After=syslog.target network.target

      

[Service]

Type=forking

.

Environment=JAVA_HOME=/opt/tomcat

Environment=CATALINA_PID=/opt/tomcat/temp/tomcat.pid

Environment=CATALINA_HOME=/opt/tomcat

Environment=CATALINA_BASE=/opt/tomcat

Environment=’CATALINA_OPTS=-Xms512M -Xmx1024M -server –XX:+UseParallelGC

Environment=’JAVA_OPTS=-Djava.awt.headless=true –Djava.security.egd=file:/dev/./urandom

.

ExecStart=/opt/tomcat/bin/startup.sh

      ExecStop=/bin/kill -15 $MAINPID

.

User=tomcat

Group=tomcat

      UMask=

      RestartSec=10

Restart=always

      

[Install]

      WantedBy=multi-user.target

.

Note: This will then reload the custom tomcat via systemd

– name: Start tomcat service with Systemd

  systemd:

name: tomcat

    daemon_reload: yes

.

Note: This will then check to see if the new tomcat is service running and out to the ansible playbook log.

    

– name: get service facts

  service_facts:

.

– name: Check to see if tomcat is running

debug:

var: ansible_facts.services[“tomcat.service“]

.

.

Ansibe playbook log:

.

[root@nickansible]# ansible-playbook –i inventory/DEV/hosts justtomcatrole.yml –limit ‘nicktestvm‘ -k

.

SSH password:

.

PLAY [all] ************************************************************************************************************************************************************************************************

.

TASK [AddTomCatSystemD : Create symbolic link for “tomcat” in /opt] ***************************************************************************************************************************************

changed: nicktestvm]

.

TASK [AddTomCatSystemD : Enable tomcat service on startup] ************************************************************************************************************************************************

changed: nicktestvm]

.

TASK [AddTomCatSystemD : Null tomcat.service file] ********************************************************************************************************************************************************

changed: nicktestvm]

.

TASK [AddTomCatSystemD : Edit tomcat.service for systemd] *************************************************************************************************************************************************

changed: nicktestvm]

.

TASK [AddTomCatSystemD : Start tomcat service with Systemd] ***********************************************************************************************************************************************

ok: nicktestvm]

.

TASK [AddTomCatSystemD : get service facts] ***************************************************************************************************************************************************************

ok: nicktestvm]

.

TASK [AddTomCatSystemD : Check to see if tomcat is running] ***********************************************************************************************************************************************

ok: nicktestvm] => {

ansible_facts.services[\”tomcat.service\”]”: {

“name”: “tomcat.service“,

“source”: “systemd“,

“state”: “running”,

“status”: “enabled”

}

}

.

PLAY RECAP ************************************************************************************************************************************************************************************************

nicktestvm : ok=7 changed=4 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

.

.

.

==========================

[root@nicktestvm ~]# cat /etc/systemd/system/tomcat.service

# BEGIN ANSIBLE MANAGED BLOCK

[Unit]

Description=Apache Tomcat Web Application Container

After=syslog.target network.target

.

[Service]

Type=forking

.

Environment=JAVA_HOME=/opt/tomcat

Environment=CATALINA_PID=/opt/tomcat/temp/tomcat.pid

Environment=CATALINA_HOME=/opt/tomcat

Environment=CATALINA_BASE=/opt/tomcat

Environment=’CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC’

Environment=’JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom’

.

ExecStart=/opt/tomcat/bin/startup.sh

ExecStop=/bin/kill -15 $MAINPID

.

User=tomcat

Group=tomcat

UMask=0028

RestartSec=10

Restart=always

.

[Install]

WantedBy=multi-user.target

# END ANSIBLE MANAGED BLOCK

.

.

SystemD Status:

.

root@nicktestvm ~]# systemctl status tomcat

tomcat.service – Apache Tomcat Web Application Container

Loaded: loaded (/etc/systemd/system/tomcat.service; enabled; vendor preset: disabled)

Active: active (running) since Thu 2020-12-24 05:11:21 GMT; 21h ago

Process: 6333 ExecStop=/bin/kill -15 $MAINPID (code=exited, status=0/SUCCESS)

Process: 6353 ExecStart=/opt/tomcat/bin/startup.sh (code=exited, status=0/SUCCESS)

Main PID: 6363 (java)

   CGroup: /system.slice/tomcat.service

└─6363 /usr/local/java/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server -Xms1…

.

Dec 24 05:11:21 nicktestvm systemd[1]: Starting Apache Tomcat Web Application Container…

Dec 24 05:11:21 nicktestvm startup.sh[6353]: Existing PID file found during start.

Dec 24 05:11:21 nicktestvm startup.sh[6353]: Removing/clearing stale PID file.

Dec 24 05:11:21 nicktestvm systemd[1]: Started Apache Tomcat Web Application Container.

.

.

How to deploy an EC2 instance with Terraform

Okay so terraform is like ansible but for the cloud. Its probably a bit cooler in what it can do than ansible….but im still still picking it up myself. But ultimately the same thing. If you know ansible you can pick up terraform without too much difficulty.

After a couple of cool writes you should be good to go.

.

Note: Some interesting things to note. If you create a EC2 with terraform and go and delete it manually after. It appears that terraform does not know the states. If you ask terraform to show you the state. As far as it knows the EC2 is all good. Kinda stupid in my opinion…..This is where ansible take the cake, since ansible will actually go and check to see if that VM/EC2 is actually there. But anyway. Its about learning new ways to do things.

.

Terrform Install:

.

 Unzip and install terraform


 
unzip ./terraform_0.14.3_linux_amd64.zip -d /usr/local/bin/


Ex
ampleLog:

[root@nick ~]# unzip ./terraform_0.14.3_linux_amd64.zip -d /usr/local/bin/

Archive: ./terraform_0.14.3_linux_amd64.zip

replace /usr/local/bin/terraform? [y]es, [n]o, [A]ll, [N]one, [r]ename: A

inflating: /usr/local/bin/terraform

 

[root@nick ~]# terraform -v

Terraform v0.14.3


Note: This is to set the environment variable path for terraform if its not set already.

 

 Set the environment variable path

.

 echo “export PATH=”$PATH:/usr/local/bin/”” >> /usr/local/bin/profile_terraform.sh

.

 Now run the script by to set the environment variable path

.

 chmod +x /usr/local/bin/profile_terraform.sh
 /usr/local/bin/profile_terraform.sh

.

.

Setup Terraform to communicate with AWS:

.

 You need to add your aws accesskeyid and secretaccesskey

Note: You can find these inside your aws console under “My Security Credentials” on the top right corner of AWS console.

 

Then Click Access keys (access key ID and secret access key)
Should be listed here, if not create one and download the keys.

 

echo “export AWS_SECRET_ACCESS_KEY=secretyouraccesskey” >> /usr/local/bin/profile_terraform.sh

 echo “export AWS_ACCESS_KEY_ID=accesskeyid” >> /usr/local/bin/profile_terraform.sh

 

 Next you want to create a Provider.tf file. This file tells terraform which aws instance to connect with.

vi provider.tf

#—Content of provider.tf

provider “aws” {

profile = “default”

region = “us-east-1”

}

 

  • Save file

[root@nick]# cat provider.tf

provider “aws” {

profile = “default”

region = “us-east-1”

}

.

 Next you want to create a Create_EC2.tf – This is the file that will create the instance inside your aws instance.

.

.

vi create_ec2.tf

                                   #—Content of create_ec2.tf
                                   resource “aws_instance” “nicktest-1” {
                                   ami = “ami-0fc61db8544a617ed”
                                   instance_type = “t2.micro
                                   }

 

  • Save file
 Now you have to Initialize the terraform by running ‘terraform init. It will check your provider.tf file and based on your cloud provider it will download the modules and plugin.

root@nick bin]# terraform init

.

Initializing the backend…

.

Initializing provider plugins…

– Finding latest version of hashicorp/aws

– Installing hashicorp/aws v3.22.0…

– Installed hashicorp/aws v3.22.0 (signed by HashiCorp)

.

Terraform has created a lock file .terraform.lock.hcl to record the provider

selections it made above. Include this file in your version control repository

so that Terraform can guarantee to make the same selections by default when

you run “terraform init” in the future.

.

Terraform has been successfully initialized!

.

You may now begin working with Terraform. Try running “terraform plan” to see

any changes that are required for your infrastructure. All Terraform commands

should now work.

.

If you ever set or change modules or backend configuration for Terraform,

rerun this command to reinitialize your working directory. If you forget, other

commands will detect it and remind you to do so if necessary.

[root@nick bin]#

.

 Now let’s run plan. In the planning phase, it is going to show you the detailed plan of execution of service creation code and their sequences. You can to review it and if all the things look good you can go for the next step

.

.

[root@nick bin]# terraform plan

.

An execution plan has been generated and is shown below.

Resource actions are indicated with the following symbols:

+ create

.

Terraform will perform the following actions:

.

# aws_instance.nicktest-1 will be created

+ resource “aws_instance” “nicktest-1” {

+ ami = “ami-0fc61db8544a617ed”

+ arn = (known after apply)

+ associate_public_ip_address = (known after apply)

+ availability_zone = (known after apply)

+ cpu_core_count = (known after apply)

+ cpu_threads_per_core = (known after apply)

+ get_password_data = false

+ host_id = (known after apply)

+ id = (known after apply)

+ instance_state = (known after apply)

+ instance_type = “t2.micro

+ ipv6_address_count = (known after apply)

+ ipv6_addresses = (known after apply)

+ key_name = (known after apply)

+ outpost_arn = (known after apply)

+ password_data = (known after apply)

+ placement_group = (known after apply)

+ primary_network_interface_id = (known after apply)

+ private_dns = (known after apply)

+ private_ip = (known after apply)

+ public_dns = (known after apply)

+ public_ip = (known after apply)

+ secondary_private_ips = (known after apply)

+ security_groups = (known after apply)

+ source_dest_check = true

+ subnet_id = (known after apply)

+ tenancy = (known after apply)

+ volume_tags = (known after apply)

+ vpc_security_group_ids = (known after apply)

.

+ ebs_block_device {

+ delete_on_termination = (known after apply)

+ device_name = (known after apply)

+ encrypted = (known after apply)

+ iops = (known after apply)

+ kms_key_id = (known after apply)

+ snapshot_id = (known after apply)

+ throughput = (known after apply)

+ volume_id = (known after apply)

+ volume_size = (known after apply)

+ volume_type = (known after apply)

}

.

+ enclave_options {

+ enabled = (known after apply)

}

.

+ ephemeral_block_device {

+ device_name = (known after apply)

+ no_device = (known after apply)

+ virtual_name = (known after apply)

}

.

+ metadata_options {

+ http_endpoint = (known after apply)

+ http_put_response_hop_limit = (known after apply)

+ http_tokens = (known after apply)

}

.

+ network_interface {

+ delete_on_termination = (known after apply)

+ device_index = (known after apply)

+ network_interface_id = (known after apply)

}

.

+ root_block_device {

+ delete_on_termination = (known after apply)

+ device_name = (known after apply)

+ encrypted = (known after apply)

+ iops = (known after apply)

+ kms_key_id = (known after apply)

+ throughput = (known after apply)

+ volume_id = (known after apply)

+ volume_size = (known after apply)

+ volume_type = (known after apply)

}

}

.

Plan: 1 to add, 0 to change, 0 to destroy.

.

————————————————————————

.

Note: You didn’t specify an “-out” parameter to save this plan, so Terraform

can’t guarantee that exactly these actions will be performed if

“terraform apply” is subsequently run.

.

.

 Now you can apply the code and it will create the EC2 machine on AWS cloud

.

[root@nick bin]# terraform apply

.

An execution plan has been generated and is shown below.

Resource actions are indicated with the following symbols:

+ create

.

Terraform will perform the following actions:

.

# aws_instance.nicktest-1 will be created

+ resource “aws_instance” “nicktest-1” {

+ ami = “ami-0fc61db8544a617ed”

+ arn = (known after apply)

+ associate_public_ip_address = (known after apply)

+ availability_zone = (known after apply)

+ cpu_core_count = (known after apply)

+ cpu_threads_per_core = (known after apply)

+ get_password_data = false

+ host_id = (known after apply)

+ id = (known after apply)

+ instance_state = (known after apply)

+ instance_type = “t2.micro

+ ipv6_address_count = (known after apply)

+ ipv6_addresses = (known after apply)

+ key_name = (known after apply)

+ outpost_arn = (known after apply)

+ password_data = (known after apply)

+ placement_group = (known after apply)

+ primary_network_interface_id = (known after apply)

+ private_dns = (known after apply)

+ private_ip = (known after apply)

+ public_dns = (known after apply)

+ public_ip = (known after apply)

+ secondary_private_ips = (known after apply)

+ security_groups = (known after apply)

+ source_dest_check = true

+ subnet_id = (known after apply)

+ tenancy = (known after apply)

+ volume_tags = (known after apply)

+ vpc_security_group_ids = (known after apply)

.

+ ebs_block_device {

+ delete_on_termination = (known after apply)

+ device_name = (known after apply)

+ encrypted = (known after apply)

+ iops = (known after apply)

+ kms_key_id = (known after apply)

+ snapshot_id = (known after apply)

+ throughput = (known after apply)

+ volume_id = (known after apply)

+ volume_size = (known after apply)

+ volume_type = (known after apply)

}

.

+ enclave_options {

+ enabled = (known after apply)

}

.

+ ephemeral_block_device {

+ device_name = (known after apply)

+ no_device = (known after apply)

+ virtual_name = (known after apply)

}

.

+ metadata_options {

+ http_endpoint = (known after apply)

+ http_put_response_hop_limit = (known after apply)

+ http_tokens = (known after apply)

}

.

+ network_interface {

+ delete_on_termination = (known after apply)

+ device_index = (known after apply)

+ network_interface_id = (known after apply)

}

.

+ root_block_device {

+ delete_on_termination = (known after apply)

+ device_name = (known after apply)

+ encrypted = (known after apply)

+ iops = (known after apply)

+ kms_key_id = (known after apply)

+ throughput = (known after apply)

+ volume_id = (known after apply)

+ volume_size = (known after apply)

+ volume_type = (known after apply)

}

}

.

Plan: 1 to add, 0 to change, 0 to destroy.

.

Do you want to perform these actions?

Terraform will perform the actions described above.

Only ‘yes’ will be accepted to approve.

.

Enter a value: yes

.

aws_instance.nicktest-1: Creating…

aws_instance.nicktest-1: Still creating… [10s elapsed]

aws_instance.nicktest-1: Still creating… [20s elapsed]

aws_instance.nicktest-1: Creation complete after 27s [id=i-07883b59bcc922e51]

.

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

.

.

.

 You can now do a terraform show and you should see the instance.

.

.

[root@nick bin]# terraform show

# aws_instance.nicktest-1:

resource “aws_instance” “nicktest-1” {

    ami = “ami-0fc61ef7a2b8544a617ed”

    arn = “arn:aws:ec2:us-east-1:047906495434:instance/i-07883b59bcc922e51″

    associate_public_ip_address = true

    availability_zone = “us-east-1e”

    cpu_core_count = 1

    cpu_threads_per_core = 1

    disable_api_termination = false

    ebs_optimized = false

    get_password_data = false

hibernation = false

id = “i-07833343459bcc922e51″

    instance_state = “running”

    instance_type = “t2.micro

ipv6_address_count = 0

ipv6_addresses = []

monitoring = false

    primary_network_interface_id = “eni-08aa9d6ec05f22570”

    private_dns = “ip-172-31-62-225.ec2.internal”

    private_ip = “172.31.62.187

    public_dns = “ec2-3-85-136-118.compute-1.amazonaws.com”

    public_ip = “3.85.136.265

    secondary_private_ips = []

    security_groups = [

“default”,

]

    source_dest_check = true

    subnet_id = “subnet-fce7f5c2”

tenancy = “default”

    volume_tags = {}

    vpc_security_group_ids = [

“sg-2838b90d”,

]

.

    credit_specification {

        cpu_credits = “standard”

}

.

    enclave_options {

enabled = false

}

.

    metadata_options {

        http_endpoint = “enabled”

        http_put_response_hop_limit = 1

        http_tokens = “optional”

}

.

    root_block_device {

        delete_on_termination = true

        device_name = “/dev/xvda

encrypted = false

        iops = 100

throughput = 0

        volume_id = “vol-0d0cfc51b13e65986”

        volume_size = 8

        volume_type = “gp2”

}

}

[root@nick bin]#

How to generate new Network UUID’s with Ansible

Okay some of you might have deployed linux vm’s from clone templates using ansible by way of the vmware_guest module.

Now everybody goes about it differently, and from what I read online…. It would seem that lots of people over complicate the generation of the UUID with over complicated code to generate the UUID.

.

At the end of the day all a UUID is….is JUST A “UNIQUE IDENTIFIER”. It serves no other function other than being another form of labelling the network interface on the vm. There is no need to over complicate the creation of a UUID. This is also provided you defined UUID’s on your deployments.

.

Why…would you want to do this? Well if you cloned from a template. The new clone with have the same network UUID on every new machine you create. Now this wont impact your infrastructure in anyway, other than you *might* get duplicate UUID warning at some point. However, it can be problematic when doing backups, restores, migrations, and monitoring in some cases.

.

Ansible Setup:

 This post assumes that you have ansible setup and running

Role :

 Create a role called CreateNewNetworkUUID in /etc/ansible/roles
mkdir -p /etc/ansible/roles/CreateNewNetworkUUID/tasks
 Create a main.yml inside /etc/ansible/roles/CreateNewNetworkUUID/tasks/
vi /etc/ansible/roles/CreateNewNetworkUUID/tasks/main.yml

.

 Now add the following yaml code.

.

Note: This just runs the ‘uuidgen’ command on the linux vm and then registers the result into a variable that is passed to the next task.

.

name: Generate new UUID

shell: uuidgen

register: new_uuid_result

.

– debug:

var: new_uuid_result

.

Note: This updates the network file on redhat and adds the UUID line with the newly generated UUID and shows a log of the new UUID that was added. This section will also be outlined in the file as managed by ansible

.

– name: Add New UUID to network config

  blockinfile:

    dest: /etc/sysconfig/network-scripts/ifcfg-ens192

    insertafter: NAME=”ens192″

block: |

UUID=”{{ new_uuid_result[‘stdout‘] }}”

register: filecontents

.

– debug: msg=”{{ filecontents }}”

.

 Save the file

.

Ansible playbook run:

.

 From inside /etc/ansible directory call your role inside your playbook or create a new playbook calling the role

.

 vi createnewUUID.yml

 Add the following to your playbook.

..

– hosts: all

  gather_facts: no

roles:

– role: CreateNewNetworkUUID

.

 Save the file

.

Ansible playbook run:

 Run your new role against your hosts

Note: this
run the role against all your hosts defined in inventory/DEV/hosts via ssh. You will need to know the root/pass for your ssh connection to be able to carry out the tasks.
ansible-playbook –i inventory/DEV/hosts createnewUUID.yml -k

.

Ansible playbook log:

SSH password:

.

PLAY [all] ****************************************************************************************************************************************************************************************************

.

TASK [CreateNewUUID : Generate new UUID] **********************************************************************************************************************************************************************

changed: [nicktestvm]

.

TASK [CreateNewUUID : debug] **********************************************************************************************************************************************************************************

ok: [nicktestvm] => {

new_uuid_result“: {

ansible_facts“: {

discovered_interpreter_python“: “/usr/bin/python”

},

“changed”: true,

cmd“: “uuidgen“,

“delta”: “0:00:00.010810”,

“end”: “2020-12-21 20:13:36.614154”,

“failed”: false,

rc“: 0,

“start”: “2020-12-21 20:13:36.603344”,

“stderr”: “”,

stderr_lines“: [],

stdout“: “49242349-5168-4713-bcb6-a53840b2e1d6”,

stdout_lines“: [

“49242349-5168-4713-bcb6-a53840b2e1d6”

]

}

}

.

TASK [CreateNewUUID : Add New UUID to network config] *********************************************************************************************************************************************************

changed: [nicktestvm]

.

TASK [CreateNewUUID : debug] **********************************************************************************************************************************************************************************

ok: [nicktestvm] => {

new_uuid_result.stdout“: “49242349-5168-4713-bcb6-a53840b2e1d6”

}

.

PLAY RECAP ****************************************************************************************************************************************************************************************************

nicktestvm              : ok=4 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

.

Nicktestvm:

.

[root@nicktestvm ~]$ cat /etc/sysconfig/network-scripts/ifcfg-ens192

TYPE=”Ethernet”

PROXY_METHOD=”none”

BROWSER_ONLY=”no”

BOOTPROTO=”none”

DEFROUTE=”yes”

IPV4_FAILURE_FATAL=”no”

IPV6INIT=”yes”

IPV6_AUTOCONF=”yes”

IPV6_DEFROUTE=”yes”

IPV6_FAILURE_FATAL=”no”

IPV6_ADDR_GEN_MODE=”stable-privacy”

NAME=”ens192″

# BEGIN ANSIBLE MANAGED BLOCK

UUID=”49242349-5168-4713-bcb6-a53840b2e1d6″

# END ANSIBLE MANAGED BLOCK

DEVICE=”ens192″

ONBOOT=”yes”

IPADDR=”192.168.1.69″

PREFIX=”24″

GATEWAY=”192.168.1.254″

DNS1=”8.8.8.1″

DNS2=”8.8.8.2″

DOMAIN=”nicktailor.co.uk”

IPV6_PRIVACY=”no”

.

How to deploy Vmware VM’s using Ansible from Cloned Templates

QUICK OVERVIEW OF WHAT ANSIBLE IS..

Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs.

Designed for multi-tier deployments since day one, Ansible models your IT infrastructure by describing how all of your systems inter-relate, rather than just managing one system at a time.

It uses no agents and no additional custom security infrastructure, so it’s easy to deploy – and most importantly, it uses a very simple language (YAML, in the form of Ansible Playbooks) that allow you to describe your automation jobs in a way that approaches plain English.

On this page, we’ll give you a really quick overview so you can see things in context. For more detail, hop over to docs.ansible.com.

EFFICIENT ARCHITECTURE

Ansible works by connecting to your nodes and pushing out small programs, called “Ansible modules” to them. These programs are written to be resource models of the desired state of the system. Ansible then executes these modules (over SSH by default), and removes them when finished.

Your library of modules can reside on any machine, and there are no servers, daemons, or databases required. Typically you’ll work with your favorite terminal program, a text editor, and probably a version control system to keep track of changes to your content.

 Okay so what that actually is saying is. Ansible has a whole library of python modules that come out of the box coupled with a huge community of open source python modules to do all sorts of tasks to automate infrastructure.
 You can call these modules by writing yaml code, inside your yaml code when you call a specific module, you can the pass specific variables to that module to do specific things defined by the python module.
Example power on and off a vm, or connect or disconnect network, etc.

For the purposes of this post we are are going to dive into using vmware_guest” module by way of using http api authentication session & cookies. There are many other python modules which you can search in the ansible documentation and or ansible-galaxy

.

https://docs.ansible.com/ansible/latest/collections/community/vmware/index.html

Now it definitely helps to be able to code in python or at least be able to read python code, however completely not necessary. Anyone with basic understanding of bash scripting can learn ansible. I could teach a newbie ansible in a couple days. Sharing is caring.

.

Anyone who says otherwise……don’t hire them.

.

.

Ansible Setup: 

 Now this post assumes you already have ansible setup and are running a newer version. If not you will need to review post on how to setup ansible before you can proceed with this.

Pre-Module install Steps: 

Requirements

The below requirements are needed on the host that executes this module.

 python >= 2.6
 PyVmomi
 PIP
 Community.vmware library of python modules

.

1.Okay so you if your on our ansible machine as root
 Run the following this should install the modules you need
ansible-galaxy collection install community.vmware
 Note: Depending on where you ran this from. If you ran this from /home/root. You can find all your python modules in ‘root/.ansible/collections/ansible_collections/community/vmware/plugins/modules’
 You will probably need to install python 2.6 or greater
Redhat : Yum install python (should get you the latest version)
 Okay you may also neeed to install pip

Note: Now on centos its not available out of the box

.Centos 7 PIP install:

1.sudo yum install epel-release
2.sudo yum install python-pip
3.pip –version (verify its installed)
4.sudo yum install python-devel (these are for building python modules)
5.sudo yum groupinstall ‘development tools’ (these are for building python modules(

.

.Install PyVmomi: 

1.pip install –upgrade pyvmomi

.

It will look like…..

[root@nick roles]# pip install –upgrade pyvmomi

Collecting pyvmomi

Downloading https://files.pythonhosted.org/packages/ba/69/4e8bfd6b0aae49382e1ab9e3ce7de9ea6318eac007b3076e6006dbe5a7cd/pyvmomi-7.0.1.tar.gz (584kB)

100% |████████████████████████████████| 593kB 861kB/s

Cache entry deserialization failed, entry ignored

Collecting requests>=2.3.0 (from pyvmomi)

Downloading https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl (61kB)

100% |████████████████████████████████| 61kB 3.5MB/s

Collecting six>=1.7.3 (from pyvmomi)

Downloading https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl

Cache entry deserialization failed, entry ignored

Collecting certifi>=2017.4.17 (from requests>=2.3.0->pyvmomi)

Downloading https://files.pythonhosted.org/packages/5e/a0/5f06e1e1d463903cf0c0eebeb751791119ed7a4b3737fdc9a77f1cdfb51f/certifi-2020.12.5-py2.py3-none-any.whl (147kB)

100% |████████████████████████████████| 153kB 6.5MB/s

Cache entry deserialization failed, entry ignored

Collecting urllib3<1.27,>=1.21.1 (from requests>=2.3.0->pyvmomi)

Downloading https://files.pythonhosted.org/packages/f5/71/45d36a8df68f3ebb098d6861b2c017f3d094538c0fb98fa61d4dc43e69b9/urllib3-1.26.2-py2.py3-none-any.whl (136kB)

100% |████████████████████████████████| 143kB 6.9MB/s

Cache entry deserialization failed, entry ignored

Collecting idna<3,>=2.5 (from requests>=2.3.0->pyvmomi)

Downloading https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl (58kB)

100% |████████████████████████████████| 61kB 4.4MB/s

Cache entry deserialization failed, entry ignored

Collecting chardet<5,>=3.0.2 (from requests>=2.3.0->pyvmomi)

Downloading https://files.pythonhosted.org/packages/19/c7/fa589626997dd07bd87d9269342ccb74b1720384a4d739a1872bd84fbe68/chardet-4.0.0-py2.py3-none-any.whl (178kB)

100% |████████████████████████████████| 184kB 3.5MB/s

Installing collected packages: certifi, urllib3, idna, chardet, requests, six, pyvmomi

Found existing installation: certifi 2018.4.16

Uninstalling certifi-2018.4.16:

Successfully uninstalled certifi-2018.4.16

Found existing installation: urllib3 1.22

Uninstalling urllib3-1.22:

Successfully uninstalled urllib3-1.22

Found existing installation: idna 2.6

Uninstalling idna-2.6:

Successfully uninstalled idna-2.6

Found existing installation: chardet 3.0.4

Uninstalling chardet-3.0.4:

Successfully uninstalled chardet-3.0.4

Found existing installation: requests 2.18.4

Uninstalling requests-2.18.4:

Successfully uninstalled requests-2.18.4

Found existing installation: six 1.9.0

Uninstalling six-1.9.0:

Successfully uninstalled six-1.9.0

Running setup.py install for pyvmomi … done

Successfully installed certifi-2020.12.5 chardet-4.0.0 idna-2.10 pyvmomi-7.0.1 requests-2.25.1 six-1.15.0 urllib3-1.26.2

You are using pip version 10.0.1, however version 20.3.3 is available.

.

You should consider upgrading via the ‘pip install –upgrade pip’ command.

(You noticed this at the bottom)

A lot of the time you need to upgrade pip for the modules to install as python is always evolving at a fast pace

.

So run

.

2.pip install –upgrade pip

.

[root@nick roles]# pip install –upgrade pip

Collecting pip

Downloading https://files.pythonhosted.org/packages/54/eb/4a3642e971f404d69d4f6fa3885559d67562801b99d7592487f1ecc4e017/pip-20.3.3-py2.py3-none-any.whl (1.5MB)

100% |████████████████████████████████| 1.5MB 799kB/s

Installing collected packages: pip

Found existing installation: pip 8.1.2

Uninstalling pip-8.1.2:

Successfully uninstalled pip-8.1.2

Successfully installed pip-10.0.1

.

You get the idea……

.

.VpsherePre-requistes for this to work:

.

You will need a vmware user who has api access permission for the following items. If the user you have setup in vcenter is unable to see these items. This module will fail. You do not need a user with full admin privileges, which is what a lot of documentation says online cryptically. I have tested this and confirmed that is not the case. Obivously, its way better to just give admin privileges to the user and trust the people you hire and use ansible vault to hide the credentials. Which we will get into later….

.

You can also check these parameters in your code by validating using assertions to see if they are all working with your user prior to moving on the next task.

.

– vSphere API configuration

– VM details

vcenter_host

– cluster

datacenter

– folder

vm_disk_size

vm_cpu_count

vm_memory

vm_vlan

vm_vlan_name

vm_dvswitch

vm_datatstore

vmware tools and or open_vm_tools must installed the clone template (super important)

.

.

Okay so now were on setting up the vmware_guest module using yaml code.

.

Setting vmware_guest module on ansible:

.

Now what I like to do is set everything up as a role in ansible to call in your playbooks, it keeps things cleaner and its much easier to find spacing mistakes in your code when writing in yaml. Lots of NBTo aid in checking for mistakes. But ultimately its experience. I’m a bit of both but I tend just pop a vi open and just write and much in there

.

1.Inside your /etc/ansible
 Create a directory called roles
mkdir roles
3.Next you want to move inside the that directory and create a name directory for this role and then go inside that directory
i.cd roles
ii.mkdir ansible-vmware-deploy
iii.cd ansible-vmware-deploy
4.Next create the following direcorties inside ‘ansible-vmware-deploy’
iv.mkdir defaults
v.mkdir tasks
vi.mkdir meta (this is really only needed for when you’re setting repositories in bickbucket, git, etc)
5.move into the tasks directory
vii.cd tasks

Note: Now we do most of our work in this directory. Your primary yaml file is always called “main.yml” Your playbooks always look for this file when trying to call python modules.

.

6.Open your favorite editor vi, nano, joe, visual studio (whatever)
a.Call the file “main.yml
b.Inside the file…

.

Setting up the yaml:

.

1. First stage of the yaml is use the http login to the vcenter host and successfully authenticate and then grab those session cookies to carry out the next set of tasks which utilise the vmware_guest module.

– name: Login into vCenter and get cookies

  delegate_to: localhost

  uri:

url: https://{{ vcenter_host }}/rest/com/vmware/cis/session

    force_basic_auth: yes

    validate_certs: no

method: POST

user: ‘{{ vcenter_username }}’

password: ‘{{ vcenter_password }}’

register: login

.

.

2. Okay so this where we are now actually calling the vmware_guest module in yaml. You can see that the code has a lot of areas that are variablelised. These variable are passed in a couple of ways. You need to pass the defaults through the defaults directory we created earlier, and the second is host specific variables which will be under your host_vars directory under your inventory structure, which we will get into later.

 

Note: Now remember this is code to deploy from an existing cloned template you have sitting on datastore somewhere in your environment. The process to deploy a vm using kickstart using DHCP that’s bit different to setup I wrote this to help out those people who cant see the wisdom and efficiency of having DHCP’d deployments

You will be passing these variables

.

– name: Create a VM

  vmware_guest:

hostname: “{{ vcenter_host }}”

username: “{{ vcenter_username }}”

password: “{{ vcenter_password }}”

    validate_certs: False

cluster: “{{ vcenter_cluster }}”

    datacenter: “{{ vcenter_dc }}”

 

Note: name: This will be the name of the new vm created. Keep in mind the vm host will also be setup with a shortname for the hostname of the server not the FQDN. You can probably fix this using vmshell or I used a completely separate role to setup the network for physical machines which uses jinja templates and inside the role I passed the new name as a variable. But that’s for another post

name: “{{ inventory_hostname }}”

folder: “{{ vm_folder }}”

template: “{{ VMTemplate }}”

state: “{{ vm_state }}”

Note: guest_id: this is what kind of OS will the VM Run, almost every hypervisor asks that prior to creating a vm. You can find the list online.

    guest_id: “{{ vm_guest_id }}”

Note: disk: this section you could technically pass it through as a variable in your host_vars on the specific hosts, but since were using a template. I kept these parameters static here inside the role.

disk:

size_gb: 80

type: thin

datastore: “{{ vm_datastore }}”

size_gb: 100

type: thin

datastore: “{{ vm_datastore }}”

hardware:

      memory_mb: “{{ vm_memory }}”

      num_cpus: “{{ vm_cpu_count }}”

      scsi: paravirtual

 

Note: Customization: This section is very important because without it your dns in /etc/resolv.conf will not be configured correctly. A lot of people have a hell of time with this on the net, as the parsing of this in yaml is bit tricky, and people resort to using vm_guest_file to update the /etc/resolv.conf, which sucks because now you need the root/pass via ssh. My way will work


customization:

      dns_servers: “{{ vm_dns_servers }}”

      dns_suffix: “{{ vm_dns_suffix }}”


Note: networks: This section is the section which will use
vmware-tools or open_vm_tools to update the network config on host after powering on the vm, but before the OS is booted, provided you said to power it on in your host_var file. This section helps people get around the issue of having no DHCP and having to deploy each server using the same static address on a dedicated vlan. This section will go and update the vm network parameters and the template vm will deploy on a  whatever vlan, with different ip, gateway, netmask. It will also register a new mac address to the vm, so you don’t end up with vm’s with duplicate mac-addresses. Lastly, it will update /etc/hosts with the new ip and shortname of the server


networks:

– name: “{{ vm_vlan_name }}”

type: static

      dvswitch_name: “{{ vm_dvswitch }}”

      ip: “{{ vm_ip }}”

netmask: “{{ vm_netmask }}”

gateway: “{{ vm_gateway }}”

      start_connected: “{{ vm_connected }}”

# wait_for_ip_address: yes (this is if you are using DHCP)

  delegate_to: localhost

register: vm_deploy

.

Note: This section is just spits out verbose information on the how the build went and the mac-address of the vm. This hand to pay attention to so you can ensure your template mac and your new vm don’t have duplicate macs. If you do. You will need to go into vshere find the VM. Remove the network and readd it manually, to register a new mac

.

– debug:

var: vm_deploy.instance.hw_eth0.macaddress

.

– debug:

var: deploy_vm

.

– debug:

var: mac.

.

7.Okay so now we need to setup our defaults to pass the to role we just created.

.

 So go into your defaults directory for the role
cd /etc/ansible/roles/ansible-vmware-deploy/defaults
 Create another file called ‘main.yml
Vi main.yml and copy the contents below.

Not: Its easier to put all your defaults here and then comment out the ones you want to pass through your host_vars specific files after you got it working the way you want.

.

vm_disks: 100

vm_cpu_count: 2

vm_state: present

vm_memory: 2048

#vm_datastore: vmfs-datastore1234

vcenter_username: BruceWayne

vcenter_password: ( you will put ansible_vault encrypted variable here, for now just put in your password for testing)

vm_dvswitch: DvSwitch

vcenter_cluster: ProdCluster

vcenter_host: vcenter.nicktailor.com

vcenter_dc: London

#vm_folder: /Production/Unix/

#vm_vlan_name: VM76123

vm_guest_id: rhel7_64Guest

#VMTemplate: redhat-template2020

.

 Save the file defaults/main.yml

.

Ansible Hosts and Inventory:

.

Okay so this is where everyone handles things uniquely. I personally like to take the approach of creating inventory based on environment. Its logical and the best way to manage hosts in very large infrastructures.

.

So if you have DEV/STAGING/PRODUCTION as your environments. Then I would set it up as such

.

.

      1. Inside your /etc/ansible directory create the following

Mkdir -p /etc/ansible/inventory
Mkdir -p /etc/ansible/inventory/DEV
Mkdir -p /etc/ansible/inventory/STAGING
Mkdir -p /etc/ansible/inventory/PRODUCTION

.

2.Inside each environment(DEV,STAGING,PROODUCTION) one you want to create the following:

.

Mkdir -p /etc/ansible/inventory/DEV/group_vars
 This is where you can pass group variables if you have hosts setup as groups in your hosts file that we just created.
Mkdir -p /etc/ansible/inventory/DEV/host_vars
 This is where you pass specific variables per host instead of groups
Touch /etc/ansible/inventory/DEV/hosts
3.Open up one of the host files in your favorite editor vi, nano, joe, visual studio, etc….

.

 vi /etc/ansible/inventory/DEV/hosts

.

For the purposes of this post we are just going to
create one group
=====================================

.

[All]

nicktestvm.nicktailor.com ansible_host=192.168.1.200

=====================================

.

 Save file

.

Note: ansible_host=(ip) This is used when you want to override dns of the host and tell ansible. Do not resolve the dns this host only connect to this ip. You don’t need this here, however if your’re using ‘a’ static address to deploy vm’s initially and not using vmwre_tools to configure the network, and went with SSH after for configuration of the host. Then it will need to know which host to connect to setup the network. So I just like to have there in case I want to temporary tell ansible look here for this server.

4.Now we want to create host_var for the specific VM host we want to deploy.

.

 Create a host_var file for the new host you want to deplo
Vi /etc/ansible/inventory/DEV/host_vars/nicktestvm

.Note: You can see all the variables that were in the role and defaults are now being passed through here for this specific host. It has to be done in this fashion for it all work correctly. If you pass all this through the role may crap out on you.

.

#vm_requirements

vm_ip: 192.168.1.86

vm_netmask: 255.255.255.0

vm_gateway: 192.168.1.1

vm_vlan_name: VM76123

VMTemplate: redhat-template2020

vm_folder: /Production/Unix

vm_state: poweredon

vm_connected: true

vm_datastore: vmfs-datastore1234

note: vm_dns_servers: this section is very important. This was the only way I could get the dns server to parse and update the /etc/resolv.conf properly. If you list them out individually as one lineers. It seems to be a bug and will simply empty out the file, which will leave your vm unable to resolve dns.

vm_dns_servers: [8.8.8.1, 8.8.8.2]

vm_dns_suffix: nicktailor.co.uk

.

 Save the file

.

Setting Ansible Vault and Encrypted variables:

.

5.Setting up the vmware-user password to be encrypted using ansible vault. Now this can be easily decrypted by anyone who has the vault password. But the benefit is that its not directly visible in your open code for prying as eyes. Which is just a generally good idea.

.

 So you want to create vault password for the variable in side defaults which was “vcenter_password”. Keep in mind variable is apart of the encrypted process.

there a couple of ways to do this you can do it via file, or via prompt.
I’m going to show you how to do it via file.
First create a vault password file
Echo “password” >> vault.pw.txt
Cat vault.pw.txt (to ensure the password is now there)
 This the password for the ansible vault not the password for your vcenter_password
Now encrypt the vcenter_password as a varible inside the vault as id1. It good to use id’s incase you you want to have multiple passwords inside your vault.

Note: the –-name is the variable you want to pass in your code. So whatever you call that has to be there.

ansible-vault encrypt_string –vault-id 1@vault.pass.txt ‘vcenter-password-here’ –name ‘vcenter_password

 

vcenter_password: !vault |

$ANSIBLE_VAULT;1.2;AES256;1

31623638366337643437633065623538663565336232333863303763336364396438663032363364

3665376363663839306165663435356365643965343364310a313832393261363466393237666666

36666437626563386366653938383565663361646333333732336439356633616231653639626465

3130656134383365320a323032366238303366336562653865663130333963316237393839373830

65396139323739323266643961653766333633366638336435613933373966643561

Encryption successful

.

6.Okay now you want copy by highlighting this section below

.

.

vcenter_password: !vault |

$ANSIBLE_VAULT;1.2;AES256;1

31623638366337643437633065623538663565336232333863303763336364396438663032363364

3665376363663839306165663435356365643965343364310a313832393261363466393237666666

36666437626563386366653938383565663361646333333732336439356633616231653639626465

3130656134383365320a323032366238303366336562653865663130333963316237393839373830

65396139323739323266643961653766333633366638336435613933373966643561

.

 open your /etc/ansible/roles/ansible-vmware-deploy/defaults/main.yml
vi etc/ansible/roles/ansible-vmware-deploy/defaults/main.yml

.

 Next replace the whole ‘vcenter_password’ line with the highlight section above and save the file.
  •  •  You should also store the vault password somewhere offsite in some password database and delete the vault.pass.txt file you created.

.

Deploy VM with ansible:

.

 From inside the /etc/ansible directory you now need to create your playbook that will call the role you just setup.

.

 Create a new playbook file standard_build.yml
Vi standard_build.yml

.

 Now add the following:

– hosts: all

  gather_facts: no

roles:

– role: ansible-vmware-deploy

 Save the file

.

 Now you want to call the new role to deploy against the environment and specific host we setup earlier

.

 Still from inside the /etc/ansible directory you want to run all your playbooks from here

.

ansible-playbook –i inventory/DEV/hosts –-ask-vault standard_build.yml

.

.

Note: Important thing to remember when deploying linux machines from a template is that all your machines will have the same ‘Network’ UUID as the template machine. If you define these…. You will need to write some code to fix that up after the VM is deployed and powered up. Check  out the link below on how to do that.

http://www.nicktailor.com/?p=1177

.

Special Note: if you attempt to deploy multiple hosts at the same time. This will deploy 5 clones in parallel at a time and not one by one. Which will reduce deployment time significantly. I didnt bother to see if i could override this….:)

Output log of successful automated ansible deploy:

.

[root@nickansible]# ansible-playbook –i inventory/DEV/hosts standard_build.yml –ask-vault –limit ‘nicktestvm

.

Vault password: (paste password here in your shell window)

.

PLAY [all] ****************************************************************************************************************************************************************************************************

.

TASK [ansible-vmware-deploy : Validate Project Requirements] **********************************************************************************************************************************************

ok

.

TASK [ansible-vmware-deploy : Login into vCenter and get cookies] *****************************************************************************************************************************************

ok: [nicktestvm]

.

TASK [ansible-vmware-deploy : Create a VM] ****************************************************************************************************************************************************************

changed: [nicktestvm]

.

TASK [ansible-vmware-deploy : debug] **********************************************************************************************************************************************************************

ok: [nicktestvm] => {

“vm_deploy.instance.hw_eth0.macaddress”: “00:40:51:53:11:a6”

}

.

nicktestvm            : ok=4 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

.

.

.

.

How to check if ports are open on an array of servers

Okay now there is a whole bunch of ways you can do this. This is just the way I played around with to save myself a bunch of time, using NCAT. Also previously known as NETCAT.

1.Ensure your Jumphost can ssh to all your newely deployed machines. Either you will use a root password or ssh key of some sort.

2.You will also need to install ncat
a.Yum install nmap-ncat (redhat/centos)
Note (ensure you have this install on all the new servers) 

3.Open your editor and copy and paste this script below and save the file
b.Vi portcheckscriptnick.sh & save
c.Chmod +x portcheckscriptnick.sh (change permissioned to executable)

portcheckscriptnick.sh – this will check to see if your new server can talk to all the hosts below and check to see if those ports are up or down on each

============================

#!/bin/bash

host=”nick1 nick2 nick3 nick4″

for host in $host; do

for port in 22 53 67 68

do

if ncat -z $host $port

then

echo port $port $host is up

else

echo port $port $host is down

fi

.

done

done
========================================

4.Next you want create an array for your for loop to cycle through and check if all those servers can communicate with those machine and ports
d.Create a file called servers
i.Vi servers
ii.Add a bunch of hosts in a single column

Example:

Server1

Server2

Server3

Server4

e.Save the file servers

.

5.Now what were going to is have a for loop cycle through the list by logging into each host running that script and outputting the results to a file for us to look at.

.

6.Run the following below check the servers and see if each server can communicate with the hosts and ports necessary. If you see the are down. Then you will need to check the firewalls to see why the host is unable to communicate.

 for HOST in $(cat server.txt) ; do ssh root@$HOST “bash -s” < portcheckscriptnick.sh ; echo $HOST ; done 2>&1 | tee -a port.status

Note: the file port.status will be created on the jump host and you can simply look through to see if any ports were down on whichever hosts.

.

This is what the script looks like on one host if its working properly

[root@nick ~]# ./portcheckscriptnick.sh

port 22 192.168.1.11 is up

port 53 192.168.1.11 is down

port 67 192.168.1.11 is down

port 68 192.168.1.11 is down

.

This is what it will look like when you run against your array of new hosts from your jumpbox

[root@nick ~]# for HOST in $(cat servers.txt) ; do ssh root@$HOST “bash -s” < portcheckscriptnick.sh ; echo $HOST ; done

root@192.168.1.11’s password:

port 22 nick1 is up

port 53 nick1 is down

port 67 nick1 is down

port 68 nick1 is down

port 22 nick2 is up

port 53 nick2 is down

port 67 nick2 is down

port 68 nick2 is down

How to setup SMTP port redirect with IPTABLES and NAT

RedHat/Centos

Okay its really easy to do. You will need to add the following in /etc/sysctl.conf
Note: these are kernel parameter changes

1.vi /etc/sysctl.conf add the following lines

kernel.sysrq = 1

net.ipv4.tcp_syncookies=1

net/ipv4/ip_forward=1 (important)

net.ipv4.conf.all.route_localnet=1 (important)

net.ipv4.conf.default.send_redirects = 0

net.ipv4.conf.all.send_redirects = 0

.

2.Save the file and run
 Sysctl -p (this will load the new kernel parameters)
3.Now you if you already have iptables running you want to save the running config and add the new redirect rules
 Iptables-save > iptables.back
4.Now you want to edit the iptables.back file and add the redirect rules
 vi iptables.back

It will probably look something like the rules below.

EXAMPLE

# Generated by iptables-save v1.2.8 on Thu July 6 18:50:55 2020

*filter

:INPUT ACCEPT [0:0]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [2211:2804881]

:RH-Firewall-1-INPUT – [0:0]

-A INPUT -j RH-Firewall-1-INPUT

-A FORWARD -j RH-Firewall-1-INPUT

-A RH-Firewall-1-INPUT -i lo -j ACCEPT

-A RH-Firewall-1-INPUT -p icmp -m icmp –icmp-type 255 -j ACCEPT

-A RH-Firewall-1-INPUT -p esp -j ACCEPT

-A RH-Firewall-1-INPUT -p ah -j ACCEPT

-A RH-Firewall-1-INPUT -m state –state RELATED,ESTABLISHED -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 1025-m state –state NEW -j ACCEPT (make sure to have open)

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 443 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 8443 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 25 -m state –state NEW -j ACCEPT (make sure to have open)

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 80 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 21 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 22 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 106 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 143 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 465 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 993 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 995 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 8222 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -j REJECT –reject-with icmp-host-prohibited

COMMIT

#ADD this section with another Commit like below

# Completed on Thu July 6 18:50:55 2020

# Generated by iptables-save v1.2.8 on Thu July 6 18:50:55 2020

*nat

:PREROUTING ACCEPT [388:45962]

:POSTROUTING ACCEPT [25:11595]

:OUTPUT ACCEPT [25:11595]

-A PREROUTING -p tcp -m tcp –dport 1025 -j REDIRECT –to-ports 25

COMMIT

# Completed on Thu July 6 18:50:55 2020

.

 Save the file

.

5.Next you want to reload the new config
 Iptables-restore < iptables.back
6.Now you should be able see the new rules and test
 Iptables -L -n -t nat (should show the rules)

.

[root@nick ~]# iptables -L -n | grep 1025

ACCEPT tcp — 0.0.0.0/0 0.0.0.0/0 tcp dpt:1025 state NEW

[root@nick ~]# iptables -L -n -t nat| grep 1025

REDIRECT tcp — 0.0.0.0/0 0.0.0.0/0 tcp dpt:1025 redir ports 25

.

Note:

You will need to run telnet from outside the host as you cant NAT to localhost locally. 🙂

.

[root@nick1 ~]# telnet 192.168.86.111 1025

Trying 192.168.86.111…

Connected to localhost.

Escape character is ‘^]’.

220 nick.ansible.com ESMTP Postfix

How to rebuild a drive that’s fallen out of a software raid

Now I know nobody uses this kind of raid technology anymore, but it was one of the cool things I learned from my mentor at the time, when I first started my career centuries ago. I happen to find this in my archives and thought I would write up to share.

There is another way to do this as using mdadm & sfdisk. When I find time I will share how to do that as well.

1.First thing you want to do is check to see drive has fallen out of the raid by running the following command below

 cat /proc/mdstat

md2 : active raid1 hda3[0] hdc3[1]

524096 blocks [2/2] [UU]

md1 : active raid1 hda2[0] hdc2[1]

524096 blocks [2/2] [UU]

md0 : active raid1 hda1[0]

78994304 blocks [2/1] [U_] *You notice this one is showing a drive has fallen out*

Note: If you see this, take notice to the one with [U_] this line means that the drive has fallen out of the raid.

1. To enter it back in run the lines below, based on the drive assignments in the above paritions that are good.

 raidhotadd /dev/md0 /dev/hdc1
 echo -n 6666666 > /proc/sys/dev/raid/speed_limit_max (this increases the rebuild speed)

How to rebuild a failed drive in software if you replaced the drive:

 cat /proc/mdstat

md2 : active raid1 hda3[0] hdc3[1]
524096 blocks [2/2] [UU]
md1 : active raid1 hda2[0] hdc2[1]
524096 blocks [2/2] [UU]
md0 : active raid1 hda1[0]
78994304 blocks [2/1] [U_]

2. recreate the paritions on the new drive by doing the following, using the same mirror drive designations from /proc/mdstat.

 sfdisk -d /dev/hda(source) | sfdisk /dev/hdc(destination) (this duplicates all three partitions on the drive on the new drive)
  echo 6666666666 > /proc/sys/dev/raid/speed_limit_max (increase rebuild speed)

3. Next check the partition by running

 df -h
 fdisk -l

Disk /dev/hdc: 81.9 GB, 81964302336 bytes

16 heads, 63 sectors/track, 158816 cylinders

Units = cylinders of 1008 * 512 = 516096 bytes

.

Device Boot Start End Blocks Id System

/dev/hdc1 * 1 156735 78994408+ fd Linux raid autodetect

/dev/hdc2 156736 157775 524160 fd Linux raid autodetect

/dev/hdc3 157776 158815 524160 fd Linux raid autodetect

.

Disk /dev/hda: 81.9 GB, 81964302336 bytes

16 heads, 63 sectors/track, 158816 cylinders

Units = cylinders of 1008 * 512 = 516096 bytes

.

Device Boot Start End Blocks Id System

/dev/hda1 * 1 156735 78994408+ fd Linux raid autodetect

/dev/hda2 156736 157775 524160 fd Linux raid autodetect

/dev/hda3 157776 158815 524160 fd Linux raid autodetect

———————————————————————

Filesystem Size Used Avail Use% Mounted on

/dev/md0 75G 11G 60G 16% /

none 251M 0 251M 0% /dev/shm

/dev/md1 496M 8.1M 463M 2% /tmp

4. Next you want it rebuild the partitions on the new drive so run the following, you will need to update your drive designation according to your drive assignment.

 raidhotadd /dev/md0 /dev/hdc1
 raidhotadd /dev/md1 /dev/hdc2
 raidhotadd /dev/md2 /dev/hdc3

.

Note: the primary partition should match the new drive designation ‘dev/md0 /dev/hdc1’.

How to add a new SCSI LUN while server is Live

REDHAT/CENTOS:

In order to get wwn ids from a server:

 cat /sys/class/scsi_host/host0/device/fc_host\:host0/port_name
 cat /sys/class/scsi_host/host1/device/fc_host\:host1/port_name

Or:

 systool -av -cfc_host | grep port_name | awk ‘{ print $3 }’ | cut -d\” -f 2 | cut -dx -f 2

.

1.To add a new SAN LUN while live:

Run this to find the new disks after you have added them to your VM

 rescan-scsi-bus.sh

Note: rescan-scsi-bus.sh is part of the sg3-utils package

2.Check that it has been found, will be mpath(something)
 multipath –l

# That’s it, unless you want to fix the name from mpath(something) to something else

1.Change the shortcut name

 vi /etc/multipath_bindings

 

2.Remove the default mpath device autogenerated
 multipath –f mpath(something)

# Go into the multipath consolde and re add the multipath device with your new shortcut name (nickdsk2 in this case)

 multipathd –k

 add map nickdsk2

Note: Not going to lie, sometimes you could do all this and still need a reboot, majority of the time this should work. But what do i know…haha

How to figure out switch and port via tcpdump

Okay if you have ever worked in a place where their network was complete choas with no documentation or network maps to help you figure out where something resides.

You can sometimes use tcpdump to help you figure out where the server is sitting by using tcpdump.

Syntax

tcpdump -nn -v -i <NIC_INTERFACE> -s 1500 -c 1 ‘ether[20:2] == 0x2000’

Example:

root@ansible:~ # tcpdumpnn -v –i eth0 -s 1500 -c 1 ‘ether[20:2] == 0x2000’
tcpdump: listening on eth3, link-type EN10MB (Ethernet), capture size 1500 bytes
03:25:22.146564 CDPv2, ttl: 180s, checksum: 692 (unverified), length 370
   Device-ID (0x01), length: 11 bytes: switch-sw02‘ 
   Address (0x02), length: 13 bytes: IPv4 (1) 192.168.1.15
   Port-ID (0x03), length: 15 bytes: ‘Ethernet0/1
   Capability (0x04), length: 4 bytes: (0x00000028): L2 Switch, IGMP snooping
   Version String (0x05), length: 220 bytes:
   Cisco Internetwork Operating System Software
   IOS ™ C2950 Software (C2950-I6Q4L2-M), Version 12.1(14)EA1a, RELEASE SOFTWARE (fc1)
   Copyright (c) 1986-2003 by cisco Systems, Inc.
   Compiled Tue 02-Sep-03 03:33 by Nicola tesla
   Platform (0x06), length: 18 bytes: ‘cisco WS-C2950T-24’
   Protocol-Hello option (0x08), length: 32 bytes:
   VTP Management Domain (0x09), length: 6 bytes: ‘ecomrd
   Duplex (0x0b), length: 1 byte: full
   AVVID trust bitmap (0x12), length: 1 byte: 0x00
   AVVID untrusted ports CoS (0x13), length: 1 byte: 0x00
1 packets captured
2 packets received by filter
0 packets dropped by kernel

root@ansible:~ #

Written by Nick Tailor