Category: Network Stuff

How to Configure Redhat 7 & 8 Network Interfaces using Ansible

 This role will configure redhat 7 and up interfaces for virtual and physical.
(bonded nics, gateways, routes, interface names)

How to use this role:

1.You must first download the git repository into your roles directory usually ansible/role/
2.Now you want edit the hosts.client file name file or create it if it doesn’t exist under your “ansible/inventory/dev:staging:prod” directory. This is a good way to separate environments with ansible, inside each environment you should have a hosts.file like indicated below.

Example file: hosts.dev, hosts.staging, hosts.prod

c.Put your server under the appropriate group inside the file and save
d.testmachine1 ansible_host=192.168.1.101

.

Cool Stuff: If you deployed a virtual-machine using the ansible-vmware modules it will set the hostname of the host using the same shortname of the vm. If you require the fqdn vs the shortname on the host. To solve this I added some code to set the fdqn as the new_hostname if you define it under you hosts.file as shown below.

e.testmachine1 ansible_host=192.168.1.101 new_hostname=testmachine1.nicktailor.com

.

Now inside this directory you should see hosts & host_vars, group_vars

Descriptions:

f.Hosts. – is where you will list your servers under specific groups which tell the playbook (what the server is, if it the server should have a specific task run on it, and how to find it)
g.Host_vars – Inside this directory is where you list the server by name which is you will list under hosts. Inside these files you pass variable parameters to the specific roles when running your playbook. Without these the playbook cant do the tasks you want it to.
h.Group_varsAre how a way to group variables for sets of servers and this keeps code cleaners and easier to manage.

Operational Use:

3.Move inside host_var
i.cd host_var
j.create a file called {{ servername }} and save it for us its testmachine1.nicktailor.com

.

4.Now inside this directory you should see hosts & host_vars, group_vars

Descriptions:

k.Hosts. – is where you will list your servers under specific groups which tell the playbook (what the server is, if it the server should have a specific task run on it, and how to find it)
l.Host_vars – Inside this directory is where you list the server by name which is you will list under hosts. Inside these files you pass variable parameters to the specific roles when running your playbook. Without these the playbook cant do the tasks you want it to.
m.Group_varsAre how a way to group variables for sets of servers and this keeps code cleaners and easier to manage.

Operational Use:

5.Move inside host_var
n.cd host_var
o.create a file called {{ servername }} and save it for us its testmachine1.nicktailor.com
p.add the following parameters to your inventory file and save.

passed parameters: example: var/testmachine1

#Configure network can be used on physical and virtual-machines

nic_devices:

    – device: ens192

      ip: 192.168.10.100

      nm: 255.255.255.0

      gw: 192.168.10.254

      uuid:

      mac:

..

Note: you do not need to specify the UUID, you can if you wish. You do need the MAC. if you are doing bonded nics on the hosts. If you are using physical machines with satellite deployments. Then its probably a good to idea to use the mac of the nic you want the dhcp request to hit to avoid accidently deploying to the wrong host. When dealing with physical machines you don’t really have the same forgiveness of snapshots or quickly rebuilding as a vm. You can do more complicated configurations as indicated below….You can always email or contact me via linkedin, top right of the blog if you need assistance.

More Advanced configurations: bonded nics, routes, multiple nics and gateways

bond_devices:

    – device: ens1

      mac: ec:0d:9a:05:3b:f0

      master: mgt

      eth_opts: ‘-C ${DEVICE} adaptive-rx off rx-usecs 0 rx-frames 0; -K ${DEVICE} lro off’

    – device: ens1d1

      mac: ec:0d:9a:05:3b:f1

      master: mgt

      eth_opts: ‘-C ${DEVICE} adaptive-rx off rx-usecs 0 rx-frames 0; -K ${DEVICE} lro off’

    – device: mgt

      ip: 10.100.1.2

      nm: 255.255.255.0

      gw: 10.100.1.254

      pr: ens1

    – device: ens6

      mac: ec:0d:9a:05:16:g0

      master: app

    – device: ens6d1

      mac: ec:0d:9a:05:16:g1

      master: app

    – device: app

      ip: 10.101.1.3

      nm: 255.255.255.0

      pr: ens6

routes:

    – device: app

      route:

        – 100.240.136.0/24

        – 100.240.138.0/24

.

    – device: app

      gw: 10.156.177.1

      route:

        – 10.156.148.0/24

.

.

Running your playbook:

1.You must run your play book from inside parent directory always “ansible
2.Now there is a playbook called setup-networkonly.yml in the ansible directory which simply calls the setup-redhat-interfaces role inside the roles directory.

Example: of ansible/ setup-networkonly.yml

hosts: all

  gather_facts: no

  roles:

   – role: setup-redhat-interfaces

.

Command:

ansible-playbook -i inventory/dev/hosts setup-networkonly.yml–limit=’testmachine1.nicktailor.com’

.

 -i : This flag tells ansibe-playbook command which hosts file to use, these are always defined by environment like hosts.dev or hosts.staging
 -u : this is the ssh_user you will be connecting to the servers with
 -Kkb : this tells ansible that you will be using sudo su – for the ssh_user when running all role/tasks
 -ask-beocme : is saying become root
 -limit=’server’ : this allows you to segement which server you want to run the playbook against.

.

.

Test Run:

[root@ansible-home]# ansible-playbook –i inventory/dev/hosts setup-metworkonly.yml –limit=’testmachine1.nicktailor.com’ -k

SSH password:

.

PLAY [all] *************************************************************************************************************************************************************************

.

TASK [setup-redhat-network : Gather facts] ************************************************************************************************************************************

ok: [testmachine1.nicktailor.com]

.

TASK [setup-redhat-network : set_fact] ****************************************************************************************************************************************

ok: [testmachine1.nicktailor.com]

.

TASK [setup-redhat-network : Cleanup network confguration] ********************************************************************************************************************

ok: [testmachine1.nicktailor.com]

.

TASK [setup-redhat-network : find] ********************************************************************************************************************************************

ok: [testmachine1.nicktailor.com]

.

TASK [setup-redhat-network : file] ********************************************************************************************************************************************

changed: [testmachine1.nicktailor.com] => (item={u’rusr: True, u’uid: 0, u’rgrp: True, u’xoth: False, u’islnk: False, u’woth: False, u’nlink: 1, u’issock: False, u’mtime: 1530272815.953706, u’gr_name: u’root‘, u’path: u’/etc/sysconfig/network-scripts/ifcfg-enp0s3′, u’xusr: False, u’atime: 1665494779.63, u’inode: 1055173, u’isgid: False, u’size: 285, u’isdir: False, u’ctime: 1530272816.3037066, u’isblk: False, u’wgrp: False, u’xgrp: False, u’isuid: False, u’dev: 64769, u’roth: True, u’isreg: True, u’isfifo: False, u’mode: u’0644′, u’pw_name: u’root‘, u’gid: 0, u’ischr: False, u’wusr: True})

changed: [testmachine1.nicktailor.com] => (item={u’rusr: True, u’uid: 0, u’rgrp: True, u’xoth: False, u’islnk: False, u’woth: False, u’nlink: 1, u’issock: False, u’mtime: 1530272848.538762, u’gr_name: u’root‘, u’path: u’/etc/sysconfig/network-scripts/ifcfg-enp0s8′, u’xusr: False, u’atime: 1665494779.846, u’inode: 2769059, u’isgid: False, u’size: 203, u’isdir: False, u’ctime: 1530272848.6417623, u’isblk: False, u’wgrp: False, u’xgrp: False, u’isuid: False, u’dev: 64769, u’roth: True, u’isreg: True, u’isfifo: False, u’mode: u’0644′, u’pw_name: u’root‘, u’gid: 0, u’ischr: False, u’wusr: True})

.

TASK [setup-redhat-network : file] ********************************************************************************************************************************************

ok: [testmachine1.nicktailor.com]

.

TASK [setup-redhat-network : Setup bond devices] ******************************************************************************************************************************

changed: [testmachine1.nicktailor.com] => (item={u’device: u’enp0s8′, u’mac: u’08:00:27:13:b2:73′, u’master: u’mgt‘})

changed: [testmachine1.nicktailor.com] => (item={u’device: u’enp0s9′, u’mac: u’08:00:27:e8:cf:cd’, u’master: u’mgt‘})

changed: [testmachine1.nicktailor.com] => (item={u’device: u’mgt‘, u’ip: u’192.168.10.200‘, u’nm: u’255.255.255.0′, u’gw: u’10.0.2.2′, u’pr: u’enp0s8′})

.

TASK [setup-redhat-network : Setup NIC] ***************************************************************************************************************************************

.

TASK [setup-redhat-network : Setup static routes] *****************************************************************************************************************************

.

PLAY RECAP *************************************************************************************************************************************************************************

testmachine1.nicktailor.com : ok=7    changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

.

[root@testmachine1.nicktailor.com]# cat /proc/net/bonding/mgt

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

.

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: enp0s8 (primary_reselect failure)

Currently Active Slave: enp0s8

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

.

Slave Interface: enp0s8

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: 08:00:27:13:b2:73

Slave queue ID: 0

.

Slave Interface: enp0s9

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: 08:00:27:e8:cf:cd

Slave queue ID: 0

.

[root@testmachine1.nicktailor.com]# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 08:00:27:63:63:0e brd ff:ff:ff:ff:ff:ff

    inet 10.0.2.15/24 brd 10.0.2.255 scope global noprefixroute dynamic enp0s3

       valid_lft 86074sec preferred_lft 86074sec

    inet6 fe80::a162:1b49:98b7:6c54/64 scope link noprefixroute

       valid_lft forever preferred_lft forever

3: enp0s8: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master mgt state UP group default qlen 1000

    link/ether 08:00:27:13:b2:73 brd ff:ff:ff:ff:ff:ff

4: enp0s9: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master mgt state UP group default qlen 1000

    link/ether 08:00:27:13:b2:73 brd ff:ff:ff:ff:ff:ff

5: enp0s10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 08:00:27:05:b4:e8 brd ff:ff:ff:ff:ff:ff

6: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN group default qlen 1000

    link/ether ae:db:dc:52:22:f8 brd ff:ff:ff:ff:ff:ff

7: mgt: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 08:00:27:13:b2:73 brd ff:ff:ff:ff:ff:ff

    inet 192.168.10.200/24 brd 192.168.56.255 scope global mgt

       valid_lft forever preferred_lft forever

    inet6 fe80::a00:27ff:fe13:b273/64 scope link

       valid_lft forever preferred_lft forever

.

How to deploy Netplan with Ansible

Ansible-Netplan: – https://github.com/Perfect10NickTailor/ansible-netplan

This role will push out the config to the designated host and apply it
It will make a backup of the previous config before applying the new config, this is just incase your config change had an yaml error and you need to quickly go in and revert back.
There is a defaults/main.yml file that all the flags and how to use them.

.

Netplan.io- what is it is? Basically yaml files to deploy network configurations in a scalable manner by Ubuntu

How to use this role:

1.You must first download the git repository into your roles directory usually ansible/role/

.

2.Now you want edit the hosts.client file name file or create it if it doesn’t exist under your “ansible/inventory/dev:staging:prod” directory. This is a good way to separate environments with ansible, inside each environment you should have a hosts.file like indicated below.

Example file: hosts.dev, hosts.staging, hosts.prod

Put your server under the appropriate group inside the file and save
i.Testmachine1 ansible_host=192.168.90.10

Note: If there is no group simply list the server outside grouping, the –limit flag will pick it

up.

3.Now inside this directory you should see hosts & host_vars, group_vars

Descriptions:

Hosts. – is where you will list your servers under specific groups which tell the playbook (what the server is, if it the server should have a specific task run on it, and how to find it)
Host_vars – Inside this directory is where you list the server by name which is you will list under hosts. Inside these files you pass variable parameters to the specific roles when running your playbook. Without these the playbook cant do the tasks you want it to.
Group_vars – Are how a way to group variables for sets of servers and this keeps code cleaners and easier to manage.

Operational Use:

4.Move inside host_var
cd host_var
create a file called {{ servername }} and save it for us its testmachine1

Okay now here is where VSC is handy. You want to connect your visual studio code to the management server under your user. I have provided a link which shows you how to setup your keys and get VSC working with it.

.

Note: You don’t have to use VSC you can use good old nano or vim, but it’s a pain. Up to you.

https://medium.com/@sujaypillai/connect-to-your-remote-servers-from-visual-studio-code-eb5a5875e348

.

.

5.Now Netplans can be simple or very complicated. Ansible-netplan is broken up into segments that look for these variables to pass.
Network, vlans, ethernets, bridges & bonds

.

6.Now my advice is not to copy the block from this document and to copy download the repo open in visual studio and copy it there.

.

Example files:

ansible/inventory/dev/host_var$ testmachine1 (with Bonding)

 

.

Example Yaml Block :

# testmachine1 netplan config

# This is the network for testmachine1 with network bonding

netplan_configuration:

    network:

      bonds:

        bond0:

          interfaces:

          – ens1f0

          – ens1f1

          parameters:

            mode: balance-rr

      ethernets:

        eno1:

          dhcp4: false

        eno2:

          dhcp4: false

        ens1f0: {}

        ens1f1: {}

      version: 2

.

      vlans:

        vlan.180:

          id: 180

          link: bond0

        #  dhcp4: false

        #  dhcp6: false

        vlan.3200:

          id: 3200

          link: bond0

        #  dhcp4: false

        #  dhcp6: false

        vlan.3300:

          id: 3300

          link: bond0

        #  dhcp4: false

        #  dhcp6: false

.

      bridges:

        br200:

          interfaces: [ vlan.200 ]

          addresses: [ 192.168.50.9/24 ]

          gateway4: 192.168.50.1

          nameservers:

                  addresses: [ 8.8.8.8,8.8.4.8 ]

                  search: [ nicktailor.com ]        

          dhcp4: false

          dhcp6: false

        br3000:

          interfaces: [ vlan.3000 ]

          dhcp4: false

          dhcp6: false

        br3200:

          interfaces: [ vlan.3200 ]

          dhcp4: false

          dhcp6: false

.

Example files:
ansible/inventory/dev/host_var$ testmachine1 (without Bonding)

.

Example Yaml Block :

#testmachine1

netplan_configuration:

    network:

      version: 2

      renderer: networkd

      ethernets:

        eno1:

          dhcp4: false

          dhcp6: false

        eno2:

          dhcp4: false

          dhcp6: false

.

      bridges:

        br0:

          interfaces: [ eno1 ]

          dhcp4: false

          dhcp6: false

        br1:

          interfaces: [ eno2 ]

          dhcp4: false

          dhcp6: false

        br1110:

          interfaces: [ vlan1110 ]

          dhcp4: false

          dhcp6: false

          addresses: [ 172.16.52.10/26 ]

          gateway4: 172.17.52.1

          nameservers:

                  addresses: [ 8.8.8.8,8.8.4.8 ]

.

        br600:

          interfaces: [ vlan600 ]

          dhcp4: false

          dhcp6: false

          addresses: [ 192.168.0.34/24 ]

        br800:

          interfaces: [ vlan800 ]

          dhcp4: false

          dhcp6: false

        br802:

          interfaces: [ vlan802 ]

          dhcp4: false

          dhcp6: false

        br801:

          interfaces: [ vlan801 ]

          dhcp4: false

          dhcp6: false

.

      vlans:

        vlan600:

          id: 600

          link: br0

          dhcp4: false

          dhcp6: false

        vlan800:

          id: 800

          link: br1

          dhcp4: false

          dhcp6: false

        vlan801:

          id: 801

          link: br1

          dhcp4: false

          dhcp6: false          

        vlan802:

          id: 802

          link: br1

          dhcp4: false

          dhcp6: false  

          

.

.

8.You must now edit the the appropriate lines and save the file
vlans, ethernets, blond, addresses, & bridges

.

9.Once saved you want to run the playbook against a test server before you push the code into the git repository. So it good to have a test vm to run your code against first.

.

Running your playbook:

1.You must run your play book from inside parent directory always “ansible
2.Now create a playbook called deploynetplan.yml in the ansible directory which simply calls the ansible-netplan role inside the roles directory.

Example: of ansible/deploynetplan.yml

hosts: all

  gather_facts: yes

  any_errors_fatal: true

  roles:

    – role: ansible-netplan

      netplan_enabled: true

.

Command:

ansible-playbook -i inventory/dev/hosts deploynetplan.yml -u nickadmin -Kkb –ask-become –limit=’testmachine1′

-i : This flag tells ansibe-playbook command which hosts file to use, these are always defined by environment like hosts.dev or hosts.staging  
-u : this is the ssh_user you will be connecting to the servers with
-Kkb : this tells ansible that you will be using sudo su – for the ssh_user when running all role/tasks
-ask-beocme : is saying become root
-limit=’server’ : this allows you to segement which server you want to run the playbook against.

.

Successful example run with bonding:

.

ntailor@KVMtestbox:~/ansible$ ansibleplaybooki inventory/dev/hosts deploynetplan.ymlu nickadminKkbaskbecomelimit=‘testmachine1’

SSH password:

BECOME password[defaults to SSH password]:

.

PLAY [all] *********************************************************************************************************************************************************************************************

.

TASK [Gathering Facts] *********************************************************************************************************************************************************************************

ok: [testmachine1]

.

TASK [ansiblenetplan : Install netplan] ***************************************************************************************************************************************************************

ok: [testmachine1]

.

TASK [ansiblenetplan : Backup exitsing configurations before removing live ones] **********************************************************************************************************************

changed: [testmachine1]

.

TASK [ansiblenetplan : copy 00install* netplan existing file to /etc/netplan/backups] ****************************************************************************************************************

changed: [testmachine1]

.

TASK [ansiblenetplan : keep only 7 days of backups of previous network config /etc/netplan/backups] ***************************************************************************************************

changed: [testmachine1]

.

TASK [ansiblenetplan : Capturing Existing Configurations] *********************************************************************************************************************************************

skipping: [testmachine1]

.

TASK [ansiblenetplan : debug] *************************************************************************************************************************************************************************

skipping: [testmachine1]

.

TASK [ansiblenetplan : Removing Existing Configurations] **********************************************************************************************************************************************

skipping: [testmachine1]

.

TASK [ansiblenetplan : Configuring Netplan] ***********************************************************************************************************************************************************

ok: [testmachine1]

.

TASK [ansiblenetplan : netplan apply] *****************************************************************************************************************************************************************

changed: [testmachine1]

.

TASK [ansiblenetplan : debug] *************************************************************************************************************************************************************************

ok: [testmachine1] => {

    “netplanapply”: {

        “changed”: true,

        “cmd”: “netplan apply”,

        “delta”: “0:00:00.601112”,

        “end”: “2022-01-31 16:43:45.295708”,

        “failed”: false,

        “msg”: “”,

        “rc”: 0,

        “start”: “2022-01-31 16:43:44.694596”,

        “stderr”: “”,

        “stderr_lines”: [],

        “stdout”: “”,

        “stdout_lines”: []

    }

}

.

TASK [ansiblenetplan : Show vlans that are up or down] ************************************************************************************************************************************************

changed: [testmachine1]

.

TASK [ansiblenetplan : debug] *************************************************************************************************************************************************************************

ok: [testmachine1] => {

    “vlan.stdout_lines”: [

        “14: vlan.180@bond0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000”,

        “15: vlan.3300@bond0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000”

    ]

}

.

TASK [ansiblenetplan : show bridge details] ***********************************************************************************************************************************************************

changed: [testmachine1]

.

TASK [ansiblenetplan : debug] *************************************************************************************************************************************************************************

ok: [testmachine1] => {

    “bridges.stdout_lines”: [

        “bridge name\tbridge id\t\tSTP enabled\tinterfaces”,

        “br180\t\t8000.000000000000\tyes\t\t,

        “br3200\t\t8000.000000000000\tyes\t\t,

        “br3300\t\t8000.000000000000\tyes\t\t

    ]

}

.

PLAY RECAP *********************************************************************************************************************************************************************************************

testmachine1               : ok=12   changed=6    unreachable=0    failed=0    skipped=3    rescued=0    ignored=0   

.

.

.

Push your inventory/dev/host_var/testmachine1 code to Git :

 

Once you successfully checked your deploy worked by logging on to the client host and confirming everything looks good. You now want to push your code to git repo. Since you were able to clone you repo, you should be able to push to it.

.

Git Add Commands.

1.Git add . (will do every file you changed)
2.Git add filename will only add the file you want

.

Git Commit Commands

1.Git commit
a.This will take you to a message screen. Just type a note of what you did save the file
2.Git push
b.This will push your changes

.

.

How to generate new Network UUID’s with Ansible

Okay some of you might have deployed linux vm’s from clone templates using ansible by way of the vmware_guest module.

Now everybody goes about it differently, and from what I read online…. It would seem that lots of people over complicate the generation of the UUID with over complicated code to generate the UUID.

.

At the end of the day all a UUID is….is JUST A “UNIQUE IDENTIFIER”. It serves no other function other than being another form of labelling the network interface on the vm. There is no need to over complicate the creation of a UUID. This is also provided you defined UUID’s on your deployments.

.

Why…would you want to do this? Well if you cloned from a template. The new clone with have the same network UUID on every new machine you create. Now this wont impact your infrastructure in anyway, other than you *might* get duplicate UUID warning at some point. However, it can be problematic when doing backups, restores, migrations, and monitoring in some cases.

.

Ansible Setup:

 This post assumes that you have ansible setup and running

Role :

 Create a role called CreateNewNetworkUUID in /etc/ansible/roles
mkdir -p /etc/ansible/roles/CreateNewNetworkUUID/tasks
 Create a main.yml inside /etc/ansible/roles/CreateNewNetworkUUID/tasks/
vi /etc/ansible/roles/CreateNewNetworkUUID/tasks/main.yml

.

 Now add the following yaml code.

.

Note: This just runs the ‘uuidgen’ command on the linux vm and then registers the result into a variable that is passed to the next task.

.

name: Generate new UUID

shell: uuidgen

register: new_uuid_result

.

– debug:

var: new_uuid_result

.

Note: This updates the network file on redhat and adds the UUID line with the newly generated UUID and shows a log of the new UUID that was added. This section will also be outlined in the file as managed by ansible

.

– name: Add New UUID to network config

  blockinfile:

    dest: /etc/sysconfig/network-scripts/ifcfg-ens192

    insertafter: NAME=”ens192″

block: |

UUID=”{{ new_uuid_result[‘stdout‘] }}”

register: filecontents

.

– debug: msg=”{{ filecontents }}”

.

 Save the file

.

Ansible playbook run:

.

 From inside /etc/ansible directory call your role inside your playbook or create a new playbook calling the role

.

 vi createnewUUID.yml

 Add the following to your playbook.

..

– hosts: all

  gather_facts: no

roles:

– role: CreateNewNetworkUUID

.

 Save the file

.

Ansible playbook run:

 Run your new role against your hosts

Note: this
run the role against all your hosts defined in inventory/DEV/hosts via ssh. You will need to know the root/pass for your ssh connection to be able to carry out the tasks.
ansible-playbook –i inventory/DEV/hosts createnewUUID.yml -k

.

Ansible playbook log:

SSH password:

.

PLAY [all] ****************************************************************************************************************************************************************************************************

.

TASK [CreateNewUUID : Generate new UUID] **********************************************************************************************************************************************************************

changed: [nicktestvm]

.

TASK [CreateNewUUID : debug] **********************************************************************************************************************************************************************************

ok: [nicktestvm] => {

new_uuid_result“: {

ansible_facts“: {

discovered_interpreter_python“: “/usr/bin/python”

},

“changed”: true,

cmd“: “uuidgen“,

“delta”: “0:00:00.010810”,

“end”: “2020-12-21 20:13:36.614154”,

“failed”: false,

rc“: 0,

“start”: “2020-12-21 20:13:36.603344”,

“stderr”: “”,

stderr_lines“: [],

stdout“: “49242349-5168-4713-bcb6-a53840b2e1d6”,

stdout_lines“: [

“49242349-5168-4713-bcb6-a53840b2e1d6”

]

}

}

.

TASK [CreateNewUUID : Add New UUID to network config] *********************************************************************************************************************************************************

changed: [nicktestvm]

.

TASK [CreateNewUUID : debug] **********************************************************************************************************************************************************************************

ok: [nicktestvm] => {

new_uuid_result.stdout“: “49242349-5168-4713-bcb6-a53840b2e1d6”

}

.

PLAY RECAP ****************************************************************************************************************************************************************************************************

nicktestvm              : ok=4 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

.

Nicktestvm:

.

[root@nicktestvm ~]$ cat /etc/sysconfig/network-scripts/ifcfg-ens192

TYPE=”Ethernet”

PROXY_METHOD=”none”

BROWSER_ONLY=”no”

BOOTPROTO=”none”

DEFROUTE=”yes”

IPV4_FAILURE_FATAL=”no”

IPV6INIT=”yes”

IPV6_AUTOCONF=”yes”

IPV6_DEFROUTE=”yes”

IPV6_FAILURE_FATAL=”no”

IPV6_ADDR_GEN_MODE=”stable-privacy”

NAME=”ens192″

# BEGIN ANSIBLE MANAGED BLOCK

UUID=”49242349-5168-4713-bcb6-a53840b2e1d6″

# END ANSIBLE MANAGED BLOCK

DEVICE=”ens192″

ONBOOT=”yes”

IPADDR=”192.168.1.69″

PREFIX=”24″

GATEWAY=”192.168.1.254″

DNS1=”8.8.8.1″

DNS2=”8.8.8.2″

DOMAIN=”nicktailor.co.uk”

IPV6_PRIVACY=”no”

.

How to check if ports are open on an array of servers

Okay now there is a whole bunch of ways you can do this. This is just the way I played around with to save myself a bunch of time, using NCAT. Also previously known as NETCAT.

1.Ensure your Jumphost can ssh to all your newely deployed machines. Either you will use a root password or ssh key of some sort.

2.You will also need to install ncat
a.Yum install nmap-ncat (redhat/centos)
Note (ensure you have this install on all the new servers) 

3.Open your editor and copy and paste this script below and save the file
b.Vi portcheckscriptnick.sh & save
c.Chmod +x portcheckscriptnick.sh (change permissioned to executable)

portcheckscriptnick.sh – this will check to see if your new server can talk to all the hosts below and check to see if those ports are up or down on each

============================

#!/bin/bash

host=”nick1 nick2 nick3 nick4″

for host in $host; do

for port in 22 53 67 68

do

if ncat -z $host $port

then

echo port $port $host is up

else

echo port $port $host is down

fi

.

done

done
========================================

4.Next you want create an array for your for loop to cycle through and check if all those servers can communicate with those machine and ports
d.Create a file called servers
i.Vi servers
ii.Add a bunch of hosts in a single column

Example:

Server1

Server2

Server3

Server4

e.Save the file servers

.

5.Now what were going to is have a for loop cycle through the list by logging into each host running that script and outputting the results to a file for us to look at.

.

6.Run the following below check the servers and see if each server can communicate with the hosts and ports necessary. If you see the are down. Then you will need to check the firewalls to see why the host is unable to communicate.

 for HOST in $(cat server.txt) ; do ssh root@$HOST “bash -s” < portcheckscriptnick.sh ; echo $HOST ; done 2>&1 | tee -a port.status

Note: the file port.status will be created on the jump host and you can simply look through to see if any ports were down on whichever hosts.

.

This is what the script looks like on one host if its working properly

[root@nick ~]# ./portcheckscriptnick.sh

port 22 192.168.1.11 is up

port 53 192.168.1.11 is down

port 67 192.168.1.11 is down

port 68 192.168.1.11 is down

.

This is what it will look like when you run against your array of new hosts from your jumpbox

[root@nick ~]# for HOST in $(cat servers.txt) ; do ssh root@$HOST “bash -s” < portcheckscriptnick.sh ; echo $HOST ; done

root@192.168.1.11’s password:

port 22 nick1 is up

port 53 nick1 is down

port 67 nick1 is down

port 68 nick1 is down

port 22 nick2 is up

port 53 nick2 is down

port 67 nick2 is down

port 68 nick2 is down

How to setup SMTP port redirect with IPTABLES and NAT

RedHat/Centos

Okay its really easy to do. You will need to add the following in /etc/sysctl.conf
Note: these are kernel parameter changes

1.vi /etc/sysctl.conf add the following lines

kernel.sysrq = 1

net.ipv4.tcp_syncookies=1

net/ipv4/ip_forward=1 (important)

net.ipv4.conf.all.route_localnet=1 (important)

net.ipv4.conf.default.send_redirects = 0

net.ipv4.conf.all.send_redirects = 0

.

2.Save the file and run
 Sysctl -p (this will load the new kernel parameters)
3.Now you if you already have iptables running you want to save the running config and add the new redirect rules
 Iptables-save > iptables.back
4.Now you want to edit the iptables.back file and add the redirect rules
 vi iptables.back

It will probably look something like the rules below.

EXAMPLE

# Generated by iptables-save v1.2.8 on Thu July 6 18:50:55 2020

*filter

:INPUT ACCEPT [0:0]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [2211:2804881]

:RH-Firewall-1-INPUT – [0:0]

-A INPUT -j RH-Firewall-1-INPUT

-A FORWARD -j RH-Firewall-1-INPUT

-A RH-Firewall-1-INPUT -i lo -j ACCEPT

-A RH-Firewall-1-INPUT -p icmp -m icmp –icmp-type 255 -j ACCEPT

-A RH-Firewall-1-INPUT -p esp -j ACCEPT

-A RH-Firewall-1-INPUT -p ah -j ACCEPT

-A RH-Firewall-1-INPUT -m state –state RELATED,ESTABLISHED -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 1025-m state –state NEW -j ACCEPT (make sure to have open)

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 443 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 8443 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 25 -m state –state NEW -j ACCEPT (make sure to have open)

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 80 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 21 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 22 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 106 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 143 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 465 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 993 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 995 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -p tcp -m tcp –dport 8222 -m state –state NEW -j ACCEPT

-A RH-Firewall-1-INPUT -j REJECT –reject-with icmp-host-prohibited

COMMIT

#ADD this section with another Commit like below

# Completed on Thu July 6 18:50:55 2020

# Generated by iptables-save v1.2.8 on Thu July 6 18:50:55 2020

*nat

:PREROUTING ACCEPT [388:45962]

:POSTROUTING ACCEPT [25:11595]

:OUTPUT ACCEPT [25:11595]

-A PREROUTING -p tcp -m tcp –dport 1025 -j REDIRECT –to-ports 25

COMMIT

# Completed on Thu July 6 18:50:55 2020

.

 Save the file

.

5.Next you want to reload the new config
 Iptables-restore < iptables.back
6.Now you should be able see the new rules and test
 Iptables -L -n -t nat (should show the rules)

.

[root@nick ~]# iptables -L -n | grep 1025

ACCEPT tcp — 0.0.0.0/0 0.0.0.0/0 tcp dpt:1025 state NEW

[root@nick ~]# iptables -L -n -t nat| grep 1025

REDIRECT tcp — 0.0.0.0/0 0.0.0.0/0 tcp dpt:1025 redir ports 25

.

Note:

You will need to run telnet from outside the host as you cant NAT to localhost locally. 🙂

.

[root@nick1 ~]# telnet 192.168.86.111 1025

Trying 192.168.86.111…

Connected to localhost.

Escape character is ‘^]’.

220 nick.ansible.com ESMTP Postfix

How to figure out switch and port via tcpdump

Okay if you have ever worked in a place where their network was complete choas with no documentation or network maps to help you figure out where something resides.

You can sometimes use tcpdump to help you figure out where the server is sitting by using tcpdump.

Syntax

tcpdump -nn -v -i <NIC_INTERFACE> -s 1500 -c 1 ‘ether[20:2] == 0x2000’


Example:

root@ansible:~ # tcpdumpnn -v –i eth0 -s 1500 -c 1 ‘ether[20:2] == 0x2000’
tcpdump: listening on eth3, link-type EN10MB (Ethernet), capture size 1500 bytes
03:25:22.146564 CDPv2, ttl: 180s, checksum: 692 (unverified), length 370
   Device-ID (0x01), length: 11 bytes: switch-sw02‘ 
   Address (0x02), length: 13 bytes: IPv4 (1) 192.168.1.15
   Port-ID (0x03), length: 15 bytes: ‘Ethernet0/1
   Capability (0x04), length: 4 bytes: (0x00000028): L2 Switch, IGMP snooping
   Version String (0x05), length: 220 bytes:
   Cisco Internetwork Operating System Software
   IOS ™ C2950 Software (C2950-I6Q4L2-M), Version 12.1(14)EA1a, RELEASE SOFTWARE (fc1)
   Copyright (c) 1986-2003 by cisco Systems, Inc.
   Compiled Tue 02-Sep-03 03:33 by Nicola tesla
   Platform (0x06), length: 18 bytes: ‘cisco WS-C2950T-24’
   Protocol-Hello option (0x08), length: 32 bytes:
   VTP Management Domain (0x09), length: 6 bytes: ‘ecomrd
   Duplex (0x0b), length: 1 byte: full
   AVVID trust bitmap (0x12), length: 1 byte: 0x00
   AVVID untrusted ports CoS (0x13), length: 1 byte: 0x00
1 packets captured
2 packets received by filter
0 packets dropped by kernel

root@ansible:~ #

Written by Nick Tailor

How to compare your route table isn’t missing any routes from your ansible config

REDHAT/CENTOS

Okay so those of you who use ansible like me and deal with complicated networks where they have a route list that’s a mile long on servers that you might need to migrate or copy to ansible and you want to save yourself some time and be accurate by ensuring the routes are correct and the file isn’t missing any routes as missing routes can be problematic and time consuming to troubleshoot after the fact.

Here is something cool you can do.

On your server you can

  1.  On the client server
  • You can use “ip” command with a flag r for routes

Example:

It will look look something like this.

[root@ansibleserver]# ip r
default via 192.168.1.1 dev enp0s8
default via 10.0.2.2 dev enp0s3 proto dhcp metric 100
default via 192.168.1.1 dev enp0s8 proto dhcp metric 101
10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15 metric 100
192.168.1.0/24 dev enp0s8 proto kernel scope link src 192.168.1.12 metric 101
10.132.100.0/24 dev mgt proto kernel scope link src 10.16.110.1 metric 1011
10.132.10.0/24  dev mgt proto kernel scope link src 10.16.110.1 metric 101
10.136.100.0/24 dev mgt proto kernel scope link src 10.16.110.1 metric 1011
10.136.10.0/24  dev mgt proto kernel scope link src 10.16.110.1 metric 101
10.134.100.0/24 dev mgt proto kernel scope link src 10.16.110.1 metric 1011
10.133.10.0/24  dev mgt proto kernel scope link src 10.16.110.1 metric 101
10.127.10.0/24  dev mgt proto kernel scope link src 10.16.110.1 metric 101
10.122.100.0/24 dev mgt proto kernel scope link src 10.16.110.1 metric 101
10.134.100.0/24 dev mgt proto kernel scope link src 10.16.110.1 metric 101
10.181.100.0/24 dev mgt proto kernel scope link src 10.16.110.1 metric 101
10.181.100.0/24dev mgt proto kernel scope link src 10.16.110.1 metric 101
10.247.200.0/24dev mgt proto kernel scope link src 10.16.110.1 metric 101
10.172.300.0/24dev mgt proto kernel scope link src 10.16.110.1 metric 101
10.162.100.0/24dev mgt proto kernel scope link src 10.16.110.1 metric 101
10.161.111.0/24 dev mgt proto kernel scope link src 10.16.110.1 metric 101
10.161.0.0/16   dev mgt proto kernel scope link src 10.16.110.1 metric 101
10.233.130.0/24 dev mgt proto kernel scope link src 10.16.110.1 metric 101
10.60.140.0/24   dev mgt proto kernel scope link src 10.16.110.1 metric 101

Now what you want to do is take the all the ips that show up on “mgt” interface and put them in a text file

  • vi ips1 
  • save the file 

copy on the section of one after the other in a column and save the file.

10.132.100.0/24
10.132.10.0/24

10.136.100.0/24
10.136.10.0/24
10.134.100.0/24
10.133.10.0/24
10.127.10.0/24
10.122.100.0/24

  1. Now your ansible route section will probably look something like this…
Example of ansible yaml file “ansblefile”
routes:
    - device: mgt
      gw: 10.16.110.1
      route:
        - 10.132.100.0/24
        - 10.132.10.0/24
        - 10.136.100.0/24
        - 10.136.10.0/24
        - 10.134.100.0/24
        - 10.133.10.0/24
        - 10.127.10.0/24
        - 10.122.100.0/24
        - 10.134.100.0/24
        - 10.181.100.0/24
        - 10.181.100.0/24
        - 10.247.200.0/24
        - 10.172.300.0/24
        - 10.162.100.0/24
        - 10.161.111.0/24
        - 10.161.0.0/16
        - 10.233.130.0/24
  1. So you what you want to do now is copy and paste the routes from the file so they line up perfectly with the correct spacing in your yaml file.Note:
    If they aren’t lined up correctly your playbook will fail.
  2. So you can either copy them into a text editor like textpad or notepad++ and just use the replace function to add the “- “ (8 spaces before the – and 1 space before the – and ip) or  you can you perl or sed  script to do it right from the command line.
# If you want to edit the file in-place
sed -i -e 's/^/prefix/' file

Example:

sed -e 's/^/ - /' ips1 > ips2
  1. Okay now you should have a new file called ips2 that looks like below with 8 space from the left margin.
– 10.136.100.0/24
–  10.136.10.0/24
– 10.134.100.0/24
– 10.133.10.0/24
– 10.127.10.0/24
– 10.122.100.0/24
  1. Now you if you cat that ips2
  • cat ips2
  • Then highlight everything inside the file
[highlighted]
- 10.136.100.0/24
- 10.136.10.0/24
- 10.134.100.0/24
- 10.133.10.0/24
- 10.127.10.0/24
- 10.122.100.0/24
[highlighted]

7. Open your ansible yaml that contains the route section and just below “route:” right against the margin paste what you highlighted. Everything should line up perfectly and save the ansible file.

routes:
– device: mgt
gw: 10.16.110.1
route:
[paste highlight]
- 10.132.100.0/24
- 10.132.10.0/24
- 10.136.100.0/24
- 10.136.10.0/24
- 10.134.100.0/24
- 10.133.10.0/24

[paste highlight]

Okay no we need to check to ensure that you didn’t accidently miss any routes between the route table and inside your ansible yaml.

  1. Now with the original ips1 file with just the routes table without the –
    • Make sure the ansible yaml file and the ips1 file are inside the same directory to make life easier.
  • We can run a little compare script like so
    while read a b c d e; do if [[ $(grep -w $a ansiblefile) ]]; then :; else echo $a $b $c $d $e; fi  ; done < <(cat ips1)

Note:
If there are any routes missing from the ansible file it will spit them out. You can keep running this until the list shows no results, minus any gateway ips of course.

Example:

[root@ansibleserver]# while read a b c d e; do if [[ $(grep -w $a  ansiblefile) ]]; then:; else echo $a $b $c $d $e; fi  ; done < <(cat ips1)
10.168.142.0/24
10.222.100.0/24
10.222.110.0/24

By Nick Tailor

How to change the currently active slave of a bonded interface

RedHat / CentOS :

Interface Bonding as we all know is very useful in providing the fault tolerance and increased bandwidth. We can change the active slave interface of bonding without interrupting the production work. In the example below we have the interface bonding bond0 with 2 slaves em0 and em1 (em1 being the active slave). We will be replacing slave em0 with new slave em2.

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: em0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 5000
Down Delay (ms): 5000

Slave Interface: em0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:21:28:b2:65:26
Slave queue ID: 0

Slave Interface: em1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:21:28:b2:65:27
Slave queue ID: 0

1. Change the active slave to em1

ifenslave command can be used to attach or detach or change the currently active slave interface from the bonding. Now, Change the active slave interface to em1.

# ifenslave -c bond0 em1

Check the bonding status again to ensure that em1 is the new active slave :

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: em1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 5000
Down Delay (ms): 5000

Slave Interface: em0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:3b:26:b2:68:26
Slave queue ID: 0

Slave Interface: em1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:3b:26:b2:68:27
Slave queue ID: 0
The switch of active slave should get effective immediately, but on critical production systems, please schedule maintenance window or make some test in an identical test environment first.

2. Attach the new slave interface

We can now attach the new slave interface em2 to the bonding.

# ifenslave bond0 em2

3. Unattach the old slave interface

Once we have attached a new slave interface, we can unattach the old slave and remove it from the bonding.

# ifenslave -d bond0 em0

4. Verify

Confirm that the new slave is now the standby interface in the bonding.

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: em1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 5000
Down Delay (ms): 5000

Slave Interface: em1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:21:29:bf:55:30
Slave queue ID: 0

Slave Interface: em2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:19:1a:d1:43:61
Slave queue ID: 0

If you want to make the changes more permanent

The changes we just made, are temporary and will be cleared after a reboot of the server. To make these changes permanent we will have to make few changes.

Make sure you delete the file /etc/sysconfig/network-scripts/ifcfg-em0 as we are no longer are using this interface in bonding. Create a new file for the new slave interface in the bonding :

# rm /etc/sysconfig/network-scripts/ifcfg-em0
# vi /etc/sysconfig/network-scripts/ifcfg-em2
DEVICE=em2
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes

How to fix in the infiniband issue when migrating multiple bonded nics to redhat 7

Okay so some of you be using malenox FPGA cards which basically bypasses the BUS to give lower latency on your network response time.

Now if you have used an OS like SUSE and had a butt load of bonded nics and then want to migrate the OS and all the bonded nics configurations in an automated fashion using ansible or something configuration management tool.

What some of you might run into is when the OS comes up for the first time, some of the Mellanox nics will boot up in infiniband mode. Which will result in the bonded nics showing up as down. I will show you how to determine this and fix this.

.

So the first thing you want to do is determine which bonds are showing down

How to check which bonds are down.

1.grep -c down /proc/net/bonding/*
 this will list out all the bonds that show an interface is down

Example

root@ansibleclient:~> grep -c down /proc/net/bonding/*

.

/proc/net/bonding/bond1:0

/proc/net/bonding/bond2:0

/proc/net/bonding/bond3:1 (this indicates that one interface is down)

.

2.Once you determine the bond has an interface that is down you want to figure out if it’s the Mellanox card nic.
 cat /proc/net/bonding/bond3
i.this will give you the nic mac address that are inside the bond.

Example

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: None

Currently Active Slave: eth4

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

.

Slave Interface: eth4

MII Status: up

Speed: 10000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: 00:02:c9:e9:e9:11

Slave queue ID: 0

.

Slave Interface: eth5

MII Status: up

Speed: 10000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: 00:02:c9:e9:e9:12

Slave queue ID: 0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

.

3.now what you want to do next is run ‘ip a’ and see if those interfaces are listed

.

Example – should look something like this. If you don’t see the down nic here for our example lets say its eth5. This could mean its in infiniband mode and not ethernet mode. It also shows if the interface is up or down. Which is very important when troubleshooting the interface

.

[root@nickansible]# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

link/ether 08:00:26:9a:33:59 brd ff:ff:ff:ff:ff:ff

    inet 10.0.2.15/24 brd 10.0.2.255 scope global noprefixroute dynamic enp0s3

       valid_lft 82770sec preferred_lft 82770sec

3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

link/ether 08:00:26:88:5a:fd brd ff:ff:ff:ff:ff:ff

    inet 192.168.1.11/24 brd 192.168.1.255 scope global noprefixroute dynamic enp0s8

       valid_lft 82773sec preferred_lft 82773sec

.

4.Okay now we need to determine if eth5 is infact the Mellanox card. So now we need the nic information
 Ethtool -I eth5

.

Example.
It will look something like this.

.

[root@nick ansible# ethtool -i eth5

driver: e1000

version: 7.3.21-k8-NAPI

firmware-version:

expansion-rom-version:

bus-info: 0000:00:18.0 (this is the important info you need)

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: no

.

 Now you want to take the bus info and determine if it is infact the Mellanox card
 lspcis 0000:00:18.00  

.

Example

.

[root@nick ansible]# lspci -s 0000:00:18.0.0

00:18:00 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s]

.

5.Okay now we know for sure this nic is infact the Mellanox nic that is down. So now we went to manually force it into ethernet but first check to see what it says
 cat /sys/bus/pci/devices/0000\:18\:00.0/mlx4_port0
ii.if this doesn’t return “ETH” then its in infiniabnd mode
 cat /sys/bus/pci/devices/0000\:18\:00.0/mlx4_port1
iii.if this doesn’t return “ETH” then its in infiniabnd mode

.

6.Now what we want to od is manually change the nic to ethernet mode
 echo eth > /sys/bus/pci/devices/0000\:18\:00.0/mlx4_port1
 echo eth > /sys/bus/pci/devices/0000\:18\:00.0/mlx4_port1
iv.If you cat them now it should say “ETH”

.

Okay so now when you do ‘ip a’ you should should see the nics up and if you check the status of the bond there should be 0 bonds down. You might have to bring the bond down and up.

.

7.You can do this simply by
 Ifdown eth5 & ifup eth5
v.If there are no errors, the cursor will simply move to the next line with a brief delay.

.

.

Now the issue here is that if you aren’t able to get rpms from Mellanox that are supported by patching in your organisation. You’re going to need a way to ensure that if the server reboots the nic will start up in ethernet mode, otherwise you could be in a very bad situation if the server boots and the nic came up in infiniband mode.

.

So there are a couple of ideas I came up with to solve this.

Option:

1.You can simply add the echo lines in the /etc/rc.local
 echo eth > /sys/bus/pci/devices/0000\:18\:00.0/mlx4_port1
 echo eth > /sys/bus/pci/devices/0000\:18\:00.0/mlx4_port1
i.This should bring the interface back to “ETH”, however you might need to add some more lines to bring the interface up properly.

.

1.This the approach I chose and the cooler way to go about it. In redhat 7 you can define a if-preup-local script which will run anytime “ifup” is run.

Here is how you set that up.

1.Create a file called “/etc/sysconfig/network-scripts/ifup-pre-local’
a.vi /etc/sysconfig/network-scripts/ifup-pre-local

.

2.Now you can add whatever script you want. My colleague and I came up with a script that determined based on mac and bus info and if it certain buses and mac showed up it would run the echo to move the ports into eth mode

.

ADD this inside and save the file

#!/bin/bash

.

#

.

LID=”00:00:00:00″

for i in `ls /etc/sysconfig/network-scripts/ifcfg-* 2> /dev/null`

do

for j in `grep HWADDR $i |awk -F\” ‘{print $2}’`

do

ID1=$(echo $j | awk -F\: ‘{print $2″:”$3}’)

ID2=$(echo $j | awk -F\: ‘{print $4″:”$5}’)

ID=”$ID1:$ID2″

PORT=$(echo $j | cut -c 16-17)

  for k in `ls /sys/bus/pci/devices/0000\:*\:00.0/net/ib[0-9]/address 2> /dev/null`

do

grep “$ID1.*$ID2” $k 1> /dev/null

if [ $? -eq 0 ]; then

if [ “x$ID” != “x$LID” ]; then

     mlxport=1

else

let “mlxport++”

fi

LID=$ID

p=$(echo $k | awk -F/ ‘{print “/sys/bus/pci/devices/”$6″/”}’)

echo “Running: echo eth > ${p}mlx4_port${mlxport}”

echo eth > ${p}mlx4_port${mlxport}

fi

done

done

done

.

3.Next you want to create a symlink in side /sbin
b.Move into /sbin
i.cd /sbin
c.now create a symlink for ifup-pre-local
ii.ln -s /etc/sysconfig/network-scripts/ifup-pre-local ifup-pre-local

.

Now when you run ifup it will run that script that check to see if the any of those bus and macs are in infinband mode and bring them into eth. It safer to do this way because if you restart the network and for some reason the nic goes back into infiniband and someone new had no idea. They would spend awhile trying to figure this out.

.

.

.

How do deploy this fix via anisble role coming soon……

.

.

.

0