Infrastructure@Home: Infrastructure Management

Ansible is a great tool to perform a series of tasks on all your nodes. Learn how to write a task that updates all software packages on your nodes and rotates the SSH keys.

In this post, I will detail how to full-fill the following requirements from the Infrastructure Management section:

• IM1: A task to update the operating systems
• IM2: A generic task to install additional software
• IM3: A task to install and update SSH keys

We will also learn about advanced Ansible concepts such as conditionals, blocks, assertions and lookups.

Prerequisites and new Ansible Futures

My infrastructure has the following differences:

• raspi-3-* and raspi-4 -*use the news Debian release Buster
• linux-workstation uses an Arch Linux Distribution called Manjaro

These differences mean that the tasks for installing and updating should only be applied to the corresponding operating system. Therefore, we will use the block statement that surrounds any number of tasks, and which will only execute when the condition specified in the when clause is true.

The playbook is:

- name: Update packages on all nodes
  hosts:
    - raspis
    - server
  serial: 1
  become: true
  tasks:
    - block:
        - name: Update packages on Debian
          apt:
            update_cache: true
            upgrade: yes
          register: result
        - name: Print results
          debug:
            var=result.stdout_lines
      when: ansible_facts[‘distribution’] == ‘Debian’

    - block:
        - name: Update packages on Archlinux
          pacman:
            update_cache: yes
            upgrade: yes
          register: result
        - name: Print results
          debug:
            var=result.stdout_lines
      when: ansible_facts[‘distribution’] == ‘Archlinux’

Let’s explain this:

• With serial = 1 we limit the execution to one host at a time. This is a safety measure: If the update fails on one host, we should stop to investigate the issue before continuing with another host
• The option become = true means that the remote user issues sudo before attempting package updates - without this switch, the task would fail.
• With block and when we define a conditional set of tasks: Only when the condition is true, then the tasks will be run.
• The first block updates all nodes for Debian with the apt module.
• The second block updates all nodes with the distribution Arch Linux by using the pacman module
• After each step, we print the results to the console.

Let’s test this playbook by applying it only to the node raspi-4-1:

$: ansible-playbook -I hosts system/update_pacakges —limit "raspi-4-1"

PLAY [Update packages on all nodes] **********************************************************************************

TASK [Gathering Facts] ********************************************************************************
ok: [raspi-4-1]

TASK [Update packages] ********************************************************************
ok: [raspi-4-1]

TASK [Print results] ******************************************************************************************
ok: [raspi-4-1] => \{
  "result.stdout_lines": [
    "Reading package lists...",
    "Building dependency tree...",
    "Reading state information...",
    "Calculating upgrade...",
    "The following packages will be upgraded:",
    "  libpam-systemd libsystemd0 libudev1 systemd systemd-sysv udev",
    "6 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.",

Excellent! From now on, we can update all systems by running this playbook.

Install Packages

The next step is to install new packages on our nodes. When executing this playbook, I want to provide the package name as an argument to ansible. Because package names are different between the Linux distributions, there will be two playbooks.

Here is the playbook for installing software on Debian.

- name: Install software on Debian
  hosts:
    - raspis
  become: true
  vars:
    package: bash
  tasks:
    - name: Install package
      apt:
        update_cache: true
        name: "\{\{ package \}\}"
        state: present
      register: result
    - name: Print results
      debug:
        var=result

Let’s execute this playbook to install the ntp package on the raspi-4-1.

ansible-playbook install_package.yml -e package=ntp —limit "raspi-4-2"

PLAY [Install software on raspi] **********************************************************************

TASK [Gathering Facts] ********************************************************************************
ok: [raspi-4-1]

TASK [Install package] ********************************************************************************
changed: [raspi-4-1]

TASK [Print results] ******************************************************************************************
ok: [raspi-4-1] => \{
  "stdout_lines": [
    "Reading package lists...",
    "Building dependency tree...",
    "Reading state information...",
    "The following additional packages will be installed:",
    "  libevent-core-2.1-6 libevent-pthreads-2.1-6 libopts25 sntp",
    "Suggested packages:",
    "  ntp-doc",
    "The following NEW packages will be installed:",
    "  libevent-core-2.1-6 libevent-pthreads-2.1-6 libopts25 ntp sntp",
    "0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.",
    "Need to get 1081 kB of archives.",

Great! We can install any software on all nodes with a simple command.

For Arch Linux, we use this playbook:

- name: Install software on Arch Linux
  hosts:
    - server
  become: true
  vars:
    package: bash
  tasks:
    - name: Install package
      pacman:
        update_cache: yes
        name: "\{\{ package \}\}"
        state: present
      register: result
    - name: Print results
      debug:
        var=result

Rotate SSH Keys

The final requirement is a task for rotating the SSH keys: We copy new SSH keys, check that we can connect with these keys, and then remove the old keys. These three steps are done in order, and if one fails, the next task will not be executed. However, we should also consider to remove the new SSH-Key in the case we cannot connect successfully. Also, we should test that the new SSH-Key exists as a file, and that it is not empty.

To achieve this, we will separate the task into a block and rescue part. We execute all steps as stated above, and if anything goes wrong, we will retain the current SSH key. To check that the key-file exists, we use the stat module, and to load the key file content, we use the lookup module. Finally, to copy the keys, we will use the authorized_key module.

Here is the playbook:

- name: Add new public key
  hosts: raspi-4-1
  serial: 1
  become: true
  vars:
    keyname: new_id_rsa.pub
    new_key_file: "\{\{ lookup('env', 'HOME') + '/.ssh/' + lookup('vars', 'keyname') \}\}"
    new_key: "\{\{ lookup('file', lookup('vars', 'new_key_file'), errors='ignore') \}\}"
    old_key: "\{\{ lookup('file', lookup('env', 'HOME') + '/.ssh/id_rsa.pub') \}\}"
    local_user: "\{\{ lookup('env', 'USER') \}\}"
  tasks:
    - block:
      - local_action:
          stat path="\{\{ new_key_file \}\}"
        become_user: "\{\{ local_user \}\}"
        register: file
      - name: Check that the new key file exists and is not empty
        assert:
          that: file.stat.isreg and file.stat.isreg
      - authorized_key:
          user: "\{\{ ansible_ssh_user \}\}"
          state: present
          key: "\{\{new_key\}\}"
          exclusive: true
        register: result
      - debug:
          var=result
      - name: pause for 10 seconds, then reconnect
        wait_for:
          delay: 10
      - name: Connect with new key
        ping:
      - name: Delete old key
        authorized_key:
          user: "\{\{ ansible_ssh_user \}\}"
          state: present
          key: "\{\{ new_key \}\}"
          exclusive: true

      rescue:
        - name: Error occured, restoring old key
          authorized_key:
            user: "\{\{ ansible_ssh_user \}\}"
            state: present
            exclusive: true
            key: "\{\{ old_key \}\}"

Lets explain this script in more detail:

• The var keyname is defined - it can be passed as an argument. Then, with env lookup we construct the path to this file, and with the file lookup, we read its content
• With stat we are accessing the filesystem to read the key file. Two assert: then: statement check that the file exists and its content is not empty
• We then copy the new key with authorized_keys and all other keys with the exclusive: true statement.
• The tasks are stopped for 10 seconds, then we reconnect with the new key to execute a ping. If the ping fails, the rescue statement applies: We copy the old key again and remove the new key.

Lets see this book in action:

PLAY [Add new public key] *************************************************************

TASK [Gathering Facts] ****************************************************************
ok: [raspi-4-1]

TASK [debug] **************************************************************************
ok: [raspi-4-1] => \{
  "new_key_file": "/Users/sguenther/.ssh/new_id_rsa.pub"
\}

TASK [stat] ***************************************************************************
ok: [raspi-4-1 -> localhost]

TASK [Check that the new key file exists and is not empty] ****************************
ok: [raspi-4-1] => \{
  "changed": false
\}

MSG:

All assertions passed


TASK [authorized_key] *****************************************************************
ok: [raspi-4-1]

TASK [debug] **************************************************************************
ok: [raspi-4-1] => \{
  "result": \{
    "changed": false,
    "comment": null,
    "exclusive": false,
    "failed": false,
    "follow": false,
    "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAD9QDABTzcYz3+c0SZQBHfXjDMaE/sRBB0L1zaBGEss1xu...

Mission accomplished!

Conclusion

With Ansible, we completed the first three requirements of the infrastructure at home project: We can update all nodes, we can install specific packages, and we can rotate the SSH keys. We also learned several new Ansible features, particular the usage of block and rescue statements, as well as how to define variables that store information from local files.

• Infrastructure_at_home
• Devops
• Ansible