Domain 3Systems & IaCINTERMEDIATE

Mastering Ansible: Architecture, Operations, and Production Troubleshooting

A comprehensive operational guide to Ansible architecture, playbook design, role structuring, inventory management, CLI commands, and systematic troubleshooting for SysAdmins and SREs.

Domain: Domain 3: Systems & IaC•Updated: 2026-11-03
#linux#sysadmin#ansible#devops

1. Overview

Ansible is an open-source automation engine used for configuration management, infrastructure provisioning, application deployment, and task automation. In modern production environments, Ansible pushes desired state configurations to target servers. It handles tasks such as installing software packages, deploying configuration files, managing systemd services, creating user accounts, and enforcing SSH security baselines.

                     +---------------------------------------+
                     |         Ansible Control Node          |
                     |  (Playbooks, Roles, Inventory, CLI)   |
                     +---------------------------------------+
                                  /            \
                       SSH (22)  /              \ SSH (22) / WinRM (5986)
                                /                \
                               v                  v
                    +--------------------+  +--------------------+
                    |  Web Server Host   |  |  DB Server Host    |
                    | (Linux + Python)   |  | (Linux / Windows)  |
                    +--------------------+  +--------------------+

Problem Solved

Without configuration management, system administration relies on manual shell commands or unmaintained, ad-hoc shell scripts. This introduces configuration drift, where servers supposedly sharing the same role differ slightly in installed packages, library versions, or configuration parameters. Ansible solves this by defining infrastructure state declaratively in version-controlled YAML files, allowing administrators to enforce consistency across thousands of hosts reproducibly.

Agentless Architecture

Unlike traditional configuration management utilities (such as Chef or Puppet) that require background daemon agents installed on every target host, Ansible is completely agentless. It communicates with target Linux machines over standard SSH and requires only a functional Python interpreter on the managed host (or WinRM for Windows environments). Control nodes execute automation directly from local workstations, bastion hosts, or CI/CD pipelines (such as Jenkins or GitHub Actions) without needing extra agent management infrastructure.

Ansible vs. Terraform

Ansible complements cloud orchestration tools like HashiCorp Terraform:

  • Terraform excels at declarative infrastructure provisioning—creating virtual networks (VPCs), subnets, cloud storage buckets, load balancers, and compute instances.
  • Ansible excels at configuration management and application deployment—taking newly provisioned virtual machines and installing software packages, templating app configs, managing system users, and starting services.

In a standard Site Reliability Engineering (SRE) pipeline, Terraform provisions virtual machine instances, and Ansible configures those machines into fully functional web servers, database clusters, or Kubernetes nodes.


2. Architecture and Mental Model

To effectively operate and troubleshoot Ansible, an SRE must understand what occurs under the hood when executing an ansible-playbook command.

Key Architectural Components

  1. Control Node: The host where Ansible is installed and executed. Contains playbooks, roles, inventory files, and Ansible modules. Must run a Unix-like OS (Linux, macOS, BSD).
  2. Managed Nodes (Hosts): Target servers managed by the control node. They do not run Ansible daemons; they only receive Python code over SSH, execute it, and return JSON output.
  3. Inventory: An INI or YAML configuration listing managed host IP addresses, hostnames, host groups, and connection variables.
  4. Modules: Discrete, reusable units of execution (e.g., ansible.builtin.apt, ansible.builtin.copy, ansible.builtin.service) written in Python or PowerShell.
  5. Plugins: Extensible code pieces that augment Ansible’s core functionality (connection plugins, lookup plugins, cache plugins, filter plugins, and callback plugins).

Execution Lifecycle of an Ansible Task

The diagram below illustrates how Ansible executes tasks across managed nodes:

sequenceDiagram
    autonumber
    participant CN as Control Node
    participant Inv as Inventory
    participant MN as Managed Node (Target)

    CN->>Inv: 1. Parse inventory & group variables
    Inv-->>CN: Return target IP addresses & connection details
    CN->>MN: 2. SSH Connect & run 'setup' module (Gather Facts)
    MN-->>CN: Return host facts JSON (IP, OS, Disks, Kernel)
    
    loop For each task in Play
        CN->>CN: 3. Render Jinja2 variables & templates locally
        CN->>CN: 4. Assemble standalone Python script (Module + Arguments)
        CN->>MN: 5. Copy script via SFTP/SCP to ~/.ansible/tmp/
        CN->>MN: 6. Execute Python script (with privilege escalation/sudo if specified)
        MN-->>CN: 7. Return JSON result ({changed: true/false, failed: false, stdout: ...})
        CN->>MN: 8. Delete remote temporary script file
    end

    opt Handlers Notified & Tasks Changed
        CN->>MN: 9. Execute triggered Handlers (e.g., systemctl reload nginx)
        MN-->>CN: Return Handler status
    end

Idempotency Principles

A core pillar of Ansible is idempotency. An operation is idempotent if executing it multiple times produces the exact same system state as running it once, without causing unintended side effects. When Ansible executes a module (e.g., installing nginx), the module first inspects the target host’s state. If nginx is already installed at the specified version, the module returns changed: false (OK) and takes no action. If nginx is missing, it installs the package and returns changed: true.


3. Core Concepts and Terminology

Understanding Ansible terminology is essential for designing maintainable automation infrastructure.

Core Terminology Breakdown

  • Playbook: A YAML file containing one or more plays that map specific task sequences to targeted host groups.
  • Play: A set of configuration instructions targeting a specific group of hosts defined in the inventory.
  • Task: An individual actionable instruction within a play that calls a specific Ansible module with designated arguments.
  • Module: The execution engine for a task. Modules encapsulate system operations (e.g., package installation, file editing, service state management).
  • Role: A standard directory structure used to organize playbooks into reusable components—grouping tasks, templates, handlers, defaults, and variables.
  • Handler: A special task triggered only when notified by another task that reported a changed state. Commonly used to reload or restart services after configuration file updates.
  • Facts: Host-specific variables automatically gathered by Ansible via the setup module prior to task execution (e.g., IPv4 addresses, network interfaces, disk partitions, distribution version).
  • Privilege Escalation (become): The mechanism used by Ansible to execute tasks with elevated privileges (such as sudo to root).

Summary Glossary

Concept Description Typical Use Case
Control Node Host running Ansible CLI and execution engine. SRE Workstation, CI/CD Runner, Bastion.
Managed Host Remote server configured via Ansible. Application instance, Database node.
Inventory File mapping server names, IPs, and variables. production.ini, aws_ec2.yml.
Playbook YAML document containing automation workflows. site.yml, deploy_app.yml.
Module Unit of code executed on managed hosts. ansible.builtin.apt, ansible.builtin.copy.
Role Modular bundle of tasks, templates, and vars. roles/nginx, roles/security_hardening.
Handler Conditional task triggered upon state change. Restarting Nginx when nginx.conf changes.
Collection Distribution format for modules, roles, and plugins. ansible.builtin, community.general.

4. Installation and Version Management

Control Node Prerequisites

  • Operating System: Linux (RHEL, Ubuntu, Debian, Fedora), macOS, or WSL on Windows.
  • Python: Python 3.9 or newer installed on the control node.
  • SSH: OpenSSH client installed and accessible via CLI.

Managed Node Prerequisites

  • Python: Python 3.6+ installed on target Linux nodes.
  • SSH Access: Public key authentication configured for the SSH user.
  • Sudo Rights: Passwordless sudo privileges configured for the execution user (if using become: true).

Installing Ansible on the Control Node

Option A: Package Manager (Debian/Ubuntu)

sudo apt update
sudo apt install -y software-properties-common
sudo add-apt-repository --yes --update ppa:ansible/ansible
sudo apt install -y ansible

Option B: Package Manager (RHEL/Enterprise Linux 9)

sudo dnf install -y epel-release
sudo dnf install -y ansible-core
python3 -m pip install --user pipx
python3 -m pipx ensurepath
pipx install ansible-core
pipx inject ansible-core argcomplete

Verifying Installation

Check the installed version, location, Python version, and default config paths:

ansible --version

Sample Output:

ansible [core 2.16.2]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/sysadmin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.11.2 (main, May 13 2026, 12:00:00) [/usr/bin/python3]

Managing Collections and Roles via ansible-galaxy

Ansible 2.10+ separates core engine features from community modules. Modules are organized into Collections.

Install collections directly:

ansible-galaxy collection install community.general

Vendor dependencies using a requirements.yml file:

# requirements.yml
collections:
  - name: community.general
    version: 8.2.0
  - name: amazon.aws
    version: 7.1.0

roles:
  - name: geerlingguy.nginx
    version: 3.2.0

Install pinned requirements:

ansible-galaxy install -r requirements.yml

5. Configuration Files and Directory Structure

Directory Layout & Reference Table

Path / File Purpose Important Details
/etc/ansible/ansible.cfg System-wide configuration file. Overridden by local ./ansible.cfg in working directory.
~/.ansible.cfg User-specific home directory config. Takes precedence over global /etc/ansible/ansible.cfg.
./ansible.cfg Project-specific configuration file. Highest precedence configuration file (unless ANSIBLE_CONFIG env var is set).
inventory/ or inventory.ini Host list and variable mapping. Defines infrastructure hosts, environments, and group hierarchies.
group_vars/ Variables applied to specific host groups. Directory containing files named after inventory groups (e.g., web.yml, db.yml).
host_vars/ Variables targeting individual hosts. Directory containing files named after specific hostnames (e.g., web01.yml).
roles/ Modular component directory structure. Encapsulates tasks, handlers, templates, and variables per function.

ansible.cfg Settings

Ansible searches for ansible.cfg in this order:

  1. ANSIBLE_CONFIG (environment variable)
  2. ./ansible.cfg (current working directory)
  3. ~/.ansible.cfg (home directory)
  4. /etc/ansible/ansible.cfg

Here is an example production ansible.cfg:

[defaults]
inventory           = ./inventory/hosts.ini
remote_user         = sysadmin
host_key_checking   = True
retry_files_enabled = False
forks               = 20
timeout             = 30
stdout_callback     = yaml
callbacks_enabled   = timer, profile_tasks

[privilege_escalation]
become              = True
become_method       = sudo
become_user         = root
become_ask_pass     = False

[ssh_connection]
pipelining          = True
ssh_args            = -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=accept-new

Key Parameters Explained

  • inventory: Specifies the default inventory file path.
  • remote_user: Sets the SSH username used for target node connections.
  • host_key_checking: Controls whether SSH checks target host keys against known_hosts. (Set to True in production for security).
  • forks: Controls how many host connections Ansible opens concurrently (default is 5).
  • pipelining: Enables SSH pipelining, executing Python scripts directly in target memory over SSH without transferring temporary files. Significantly speeds up execution.

Inventory Structure (INI vs YAML)

Static INI Format (inventory/hosts.ini):

[web]
web01.example.com ansible_host=10.0.1.10 ansible_user=ubuntu
web02.example.com ansible_host=10.0.1.11 ansible_user=ubuntu

[db]
db01.example.com ansible_host=10.0.2.20 ansible_user=postgres

[web:vars]
http_port=80
ansible_ssh_private_key_file=~/.ssh/deploy.pem

[production:children]
web
db

Standard Production Directory Layout

ansible-project/
├── ansible.cfg
├── requirements.yml
├── site.yml
├── inventory/
│   ├── production.ini
│   └── staging.ini
├── group_vars/
│   ├── all.yml
│   ├── web.yml
│   └── db.yml
├── host_vars/
│   └── web01.example.com.yml
└── roles/
    ├── common/
    │   ├── defaults/main.yml
    │   ├── tasks/main.yml
    │   └── handlers/main.yml
    └── nginx/
        ├── defaults/main.yml
        ├── vars/main.yml
        ├── tasks/main.yml
        ├── handlers/main.yml
        ├── templates/nginx.conf.j2
        └── files/index.html

6. Service Management and Common Operations

Operating Ansible requires navigating both ad-hoc CLI tasks and full playbook executions.

1. Connectivity Testing

Before running complex configuration changes, test SSH connectivity and authentication across all managed hosts using the ping module:

ansible all -m ping

2. Performing Dry Runs (--check and --diff)

To preview changes without modifying managed target hosts, execute playbooks in check mode:

ansible-playbook site.yml --check --diff
  • --check: Runs tasks in dry-run mode, reporting what would change without modifying system state.
  • --diff: Displays exact line-by-line configuration changes (e.g., in Jinja templated files).

3. Target Filtering (--limit)

To run a playbook against a specific host or group instead of the entire inventory, use --limit (-l):

ansible-playbook site.yml -l web01.example.com

4. Resuming Playbooks (--start-at-task)

If a playbook execution stops mid-way, resume execution from the failed task using --start-at-task:

ansible-playbook site.yml --start-at-task="Deploy Nginx config"

7. Common Administrative Tasks and Workflows

Workflow 1: Base System Hardening & User Provisioning

This task provisions system administrators and hardens SSH configuration on target hosts.

---
# roles/common/tasks/main.yml
- name: Ensure sysadmin group exists
  ansible.builtin.group:
    name: sysadmin
    state: present

- name: Create sysadmin user account
  ansible.builtin.user:
    name: deployer
    group: sysadmin
    shell: /bin/bash
    groups: sudo
    append: true
    state: present

- name: Deploy authorized SSH public keys
  ansible.posix.authorized_key:
    user: deployer
    state: present
    key: "{{ lookup('file', '~/.ssh/id_ed25519.pub') }}"

- name: Enforce SSH configuration security settings
  ansible.builtin.lineinfile:
    path: /etc/ssh/sshd_config
    regexp: "^#?{{ item.key }}"
    line: "{{ item.key }} {{ item.value }}"
    state: present
    validate: '/usr/sbin/sshd -t -f %s'
  loop:
    - { key: 'PermitRootLogin', value: 'no' }
    - { key: 'PasswordAuthentication', value: 'no' }
  notify: Restart sshd

Workflow 2: Web Server Deployment with Jinja2 Templating and Handlers

This workflow installs Nginx, applies dynamic Jinja2 configuration templates, and notifies a service handler when configurations change.

1. Define Playbook Entry Point (site.yml):

---
- name: Configure Web Tier
  hosts: web
  become: true
  roles:
    - role: nginx

2. Define Role Defaults (roles/nginx/defaults/main.yml):

nginx_http_port: 80
nginx_max_clients: 1024
nginx_worker_processes: "auto"

3. Define Jinja2 Template (roles/nginx/templates/nginx.conf.j2):

# Generated by Ansible — DO NOT EDIT MANUALLY
user www-data;
worker_processes {{ nginx_worker_processes }};
pid /run/nginx.pid;

events {
    worker_connections {{ nginx_max_clients }};
}

http {
    server {
        listen {{ nginx_http_port }};
        server_name {{ ansible_fqdn }};

        location / {
            root /var/www/html;
            index index.html;
        }
    }
}

4. Define Main Tasks (roles/nginx/tasks/main.yml):

---
- name: Install Nginx web server
  ansible.builtin.apt:
    name: nginx
    state: present
    update_cache: true

- name: Render Nginx main configuration file
  ansible.builtin.template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    owner: root
    group: root
    mode: '0644'
  notify: Reload Nginx service

- name: Ensure Nginx service is running and enabled at boot
  ansible.builtin.service:
    name: nginx
    state: started
    enabled: true

5. Define Role Handler (roles/nginx/handlers/main.yml):

---
- name: Reload Nginx service
  ansible.builtin.service:
    name: nginx
    state: reloaded

8. CLI Command Reference and Cheatsheet

1. Ad-Hoc Diagnostic and Inspection Commands

Ad-hoc commands execute single module calls directly across host targets without needing playbooks.

Command Syntax Purpose Important Flags / Arguments Practical Example
ansible <group> -m ping Verify SSH connectivity & authentication. -i: Inventory file. ansible web -i inventory/hosts.ini -m ping
ansible <group> -m setup Gather target system facts. -a: Arguments (e.g., filter facts). ansible db01 -m setup -a "filter=ansible_memtotal_mb"
ansible <group> -a "<shell_cmd>" Execute arbitrary shell commands. -b: Become (sudo elevated). ansible web -b -a "systemctl status nginx"
ansible <group> -m <module> -a "<args>" Call dedicated module on targets. -m: Module name, -a: Module key/values. ansible web -b -m apt -a "name=htop state=present"

Warning: Avoid using ansible all -a "rm -rf /tmp/*" in ad-hoc mode. Unchecked shell executions bypass Ansible’s safety checks and dry-run mechanisms.

2. Playbook Execution Commands

Command Purpose Useful Flags Example Usage
ansible-playbook Execute automation playbooks. --check: Dry run preview.


--diff: Show templated line changes.


-l <host>: Limit to host/group.


-e "var=val": Extra vars.


--tags "tag": Run matching tasks.


--skip-tags: Skip matching tasks. | ansible-playbook site.yml --check --diff -l web01 |

3. Inventory Commands

Task Command Description
List resolved hosts ansible-inventory --list Outputs fully resolved JSON tree of groups and associated variables.
Visualize inventory graph ansible-inventory --graph Displays visual ASCII tree representation of groups, children, and host members.

9. Logs, Monitoring and Diagnostics

Locating Ansible Output and Log Files

By default, Ansible outputs execution streams directly to stdout / stderr. To log Ansible runs persistently on the control node, configure log_path in ansible.cfg:

[defaults]
log_path = /var/log/ansible/ansible.log

Controlling Verbosity Levels (-v to -vvvv)

When diagnosing task failures, append verbosity flags to increase debugging detailed output:

  • -v: Displays task results and module return parameters.
  • -vv: Displays task execution details and module configuration inputs.
  • -vvv: Displays SSH connection details, remote user commands, and authentication state.
  • -vvvv: Enables heavy SSH debug trace logging (includes connection plugin diagnostics).

Diagnostic Modules and Fact Inspection

Use the ansible.builtin.debug module to inspect variables and runtime state during playbook runs:

- name: Print resolved database host variable
  ansible.builtin.debug:
    var: hostvars['db01.example.com']['ansible_default_ipv4']['address']

- name: Print custom diagnostic message
  ansible.builtin.debug:
    msg: "Target server OS is {{ ansible_distribution }} {{ ansible_distribution_version }}"

10. Troubleshooting Guide and Common Failure Scenarios


Scenario 1: UNREACHABLE / SSH Connection Failed

Symptoms

Execution halts during initial host connection with an UNREACHABLE error:

web01 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: ssh: connect to host 10.0.1.10 port 22: Connection refused",
    "unreachable": true
}

Possible Causes

  1. Remote server is powered off or unreachable.
  2. Firewall / cloud Security Group blocks inbound port 22 (SSH) traffic.
  3. Incorrect SSH username (ansible_user) or private key file (ansible_ssh_private_key_file).
  4. Host key checking failure due to IP mismatch or server re-provisioning.

Investigation and Diagnosis

  1. Verify raw network connectivity using standard SSH:
ssh -i ~/.ssh/deploy.pem ubuntu@10.0.1.10
  1. Run Ansible ping module with maximum verbosity:
ansible web01 -m ping -vvv
  1. Inspect ~/.ssh/config or SSH agent status:
ssh-add -l

Root Cause Identification

  • If standard SSH fails with Connection timed out: Inspect firewall rules or AWS/GCP security groups.
  • If standard SSH fails with Permission denied (publickey): Verify key permissions (chmod 600) and username matching.

Resolution

  • Update connection parameters in inventory.ini:
[web]
web01 ansible_host=10.0.1.10 ansible_user=ubuntu ansible_ssh_private_key_file=~/.ssh/deploy.pem
  • For bastion host access, configure ansible_ssh_common_args with ProxyJump in inventory:
[web:vars]
ansible_ssh_common_args='-o ProxyJump=bastion@bastion.example.com'

Verification

Execute connectivity ping:

ansible web01 -m ping

Expected output: "ping": "pong" and "unreachable": false.

Prevention

Maintain standardized SSH configs, enforce key management via SSH agent forwarding, and automate host registration in inventory.


Scenario 2: Permission Denied / Become Failed

Symptoms

Tasks requiring administrative rights fail during execution:

fatal: [web01]: FAILED! => {"changed": false, "module_stderr": "sudo: a password is required\n", "msg": "MODULE FAILURE"}

Possible Causes

  1. Target SSH user lacks passwordless sudo privileges.
  2. Task requires elevation, but become: true was omitted in task or playbook.
  3. Sudoers rules restrict execution of specific administrative binaries on target host.

Investigation and Diagnosis

  1. Test privilege escalation directly via ad-hoc command:
ansible web01 -b -m command -a "whoami"
  1. If it prompts for password, verify --ask-become-pass flag behavior:
ansible web01 -b --ask-become-pass -m command -a "whoami"

Root Cause Identification

  • If ansible -b -a "whoami" returns sudo: a password is required: Target user lacks NOPASSWD configuration in /etc/sudoers.

Resolution

  • Update /etc/sudoers.d/sysadmins on target host to allow passwordless escalation:
ubuntu ALL=(ALL) NOPASSWD:ALL
  • Or pass --ask-become-pass (-K) when running playbooks:
ansible-playbook site.yml -K

Verification

Re-run ad-hoc elevation check:

ansible web01 -b -m command -a "whoami"

Output must return: root.

Prevention

Ensure base machine images (AMIs/Cloud-Init) explicitly configure passwordless sudo for automation users during initial bootstrapping.


Scenario 3: Python Interpreter Not Found

Symptoms

Playbook halts immediately during fact gathering:

fatal: [minimal-host]: FAILED! => {"changed": false, "msg": "The module failed to execute: /usr/bin/python3: No such file or directory"}

Possible Causes

  1. Target host runs a minimal Linux distribution (e.g., Alpine, Debian Minimal) without Python preinstalled.
  2. Python path on target host is non-standard (e.g., /usr/local/bin/python3 instead of /usr/bin/python3).

Investigation and Diagnosis

Inspect installed Python interpreter on target machine:

ssh user@target-host "which python3 || which python"

Resolution

Bootstrapping Python on a minimal host requires using Ansible’s raw module, which bypasses remote Python dependencies and executes plain shell scripts over SSH:

ansible minimal-host -m raw -a "apt-get update && apt-get install -y python3" -b

Alternatively, specify custom Python paths in host/group vars:

# group_vars/all.yml
ansible_python_interpreter: /usr/bin/python3

Verification

Run standard fact-gathering call:

ansible minimal-host -m setup

Prevention

Define ansible_python_interpreter: auto_silent in ansible.cfg to let Ansible automatically detect remote Python paths.


Scenario 4: Non-Idempotent Tasks (Task Always Shows Changed)

Symptoms

Running ansible-playbook repeatedly against an unchanged environment continues reporting changed: 1 on every run instead of returning ok.

Possible Causes

Using generic command or shell modules instead of declarative system modules (e.g., apt, copy, file). The command module has no built-in state checking mechanism and assumes system state changes on every execution.

Resolution

Replace shell modules with native declarative modules. If a raw command task is unavoidable, define explicit idempotency conditions using creates, removes, or changed_when:

# BAD: Non-idempotent task
- name: Download application binary
  ansible.builtin.shell: curl -s https://example.com/app.tar.gz | tar -xz -C /opt/app

# GOOD: Idempotent task with explicit creation checks
- name: Download application binary
  ansible.builtin.command:
    cmd: /usr/bin/curl -s https://example.com/app.tar.gz -o /opt/app.tar.gz
    creates: /opt/app.tar.gz
  register: download_result

- name: Extract binary
  ansible.builtin.unarchive:
    src: /opt/app.tar.gz
    dest: /opt/app/
    remote_src: true
    creates: /opt/app/binary_executable

Scenario 5: Template or Variable Undefined

Symptoms

Playbook fails during Jinja2 template rendering:

fatal: [web01]: FAILED! => {"msg": "AnsibleUndefinedVariable: 'http_port' is undefined"}

Possible Causes

  1. Variable spelling error in Jinja template or playbook task.
  2. Variable defined in group_vars that does not apply to target host.
  3. Precedence hierarchy collision (e.g., role default variable overridden elsewhere incorrectly).

Resolution

  1. Verify variable scoping and use default filters in Jinja template:
listen {{ http_port | default(80) }};
  1. Inspect runtime variables using ansible.builtin.debug:
- name: Debug http_port value
  ansible.builtin.debug:
    var: http_port

Scenario 6: Handler Not Running

Symptoms

Configuration file is updated, but Nginx or systemd service fails to reload.

Possible Causes

  1. Task modifying config file evaluated to ok (unchanged) instead of changed. Handlers execute only when notified tasks report changed: true.
  2. Playbook execution aborted due to a fatal task error before reaching end-of-play handler flushes.

Resolution

  • Force handlers to run immediately mid-play using meta: flush_handlers:
- name: Render Nginx config
  ansible.builtin.template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
  notify: Reload Nginx

- name: Flush handlers immediately
  ansible.builtin.meta: flush_handlers

11. Security and Best Practices

Secure Secrets Management with Ansible Vault

Never store plain-text credentials, passwords, or API tokens in Git repositories. Use Ansible Vault to encrypt sensitive variables.

Creating Encrypted Vault Files

ansible-vault create group_vars/all/vault.yml

Encrypting Existing Files

ansible-vault encrypt group_vars/web/secrets.yml

Running Playbooks with Vault Credentials

ansible-playbook site.yml --ask-vault-pass

Or use a vault password file path in ansible.cfg:

[defaults]
vault_password_file = ~/.ansible/.vault_pass

Enforce Fully Qualified Collection Names (FQCN)

Always use Fully Qualified Collection Names (FQCN) in tasks to prevent module naming collisions across third-party collections:

# GOOD Practice (FQCN)
- name: Install Nginx package
  ansible.builtin.apt:
    name: nginx
    state: present

# AVOID (Short name)
- name: Install Nginx package
  apt:
    name: nginx
    state: present

Operational Hardening Summary

  1. Least Privilege: Avoid running control scripts as root. Connect as unprivileged deployment users and elevate via become.
  2. Host Key Checking: Keep host_key_checking = True in production ansible.cfg to protect against Man-in-the-Middle (MitM) attacks.
  3. Linting: Integrate ansible-lint into developer pre-commit hooks and CI pipelines to validate playbook structure before execution.

12. Performance and Resource Optimisation

Executing Ansible against large infrastructure fleets (hundreds of servers) can cause performance bottlenecks if running default settings.

1. Tune Connection Parallelism (forks)

By default, Ansible processes only 5 target hosts concurrently. Increase forks in ansible.cfg according to control node CPU capacity:

[defaults]
forks = 30

2. Enable SSH Pipelining

SSH Pipelining reduces network round-trips required to execute tasks by running Python modules directly in target memory rather than writing temporary files to disk.

[ssh_connection]
pipelining = True

3. Disable Unnecessary Fact Gathering

If playbook tasks do not depend on system facts (ansible_distribution, ansible_interfaces), disable fact gathering to save execution time per host:

- name: Rapid deployment play
  hosts: web
  gather_facts: false
  tasks:
    - name: Restart application container
      ansible.builtin.command: /usr/local/bin/deploy_app.sh

4. Rolling Deployments using serial

Prevent total service outages during web application deployments by applying updates in rolling batches:

- name: Rolling Web Application Update
  hosts: web
  serial: 2   # Update 2 hosts at a time
  # Or: serial: "25%"
  become: true
  tasks:
    - name: Update app files
      ansible.builtin.git:
        repo: 'https://github.com/org/app.git'
        dest: /var/www/app
  handlers:
    - name: Reload Nginx
      ansible.builtin.service:
        name: nginx
        state: reloaded

13. Backup, Recovery and Maintenance

Version Controlling Automation Infrastructure

Maintain Ansible automation inside centralized Git repositories:

  • Store playbooks, roles, custom modules, and inventory structures in Git.
  • Add .vault_pass and sensitive dynamic state cache files to .gitignore.

Backup and Restore Playbook State

Before running destructive maintenance tasks (such as database migrations or kernel upgrades), create automated backups using remote tools:

- name: Create pre-maintenance backup directory
  ansible.builtin.file:
    path: "/var/backups/pre_upgrade_{{ ansible_date_time.date }}"
    state: directory
    mode: '0700'

- name: Backup configuration directories
  ansible.builtin.archive:
    path: /etc/nginx
    dest: "/var/backups/pre_upgrade_{{ ansible_date_time.date }}/nginx_cfg.tar.gz"
    format: gz

14. Automation and Integration

Ansible integrates into modern Site Reliability Engineering (SRE) pipelines to automate infrastructure delivery end-to-end.

Integrating Terraform and Ansible

When Terraform creates cloud compute instances, it exports public/private IP address outputs. Ansible consumes these output states using dynamic inventory plugins (e.g., aws_ec2, azure_rm).

# inventory/aws_ec2.yml (Dynamic Inventory Plugin)
plugin: amazon.aws.aws_ec2
regions:
  - us-east-1
filters:
  tag:Environment: production
keyed_groups:
  - key: tags.Role
    prefix: role

Run Ansible directly against dynamically discovered cloud hosts:

ansible-playbook -i inventory/aws_ec2.yml site.yml

Continuous Integration Pipeline (GitHub Actions Workflow Example)

name: Ansible CI Pipeline

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  lint-and-validate:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install Dependencies
        run: |
          pip install ansible-core ansible-lint

      - name: Run Ansible Lint
        run: |
          ansible-lint site.yml

      - name: Syntax Check Playbooks
        run: |
          ansible-playbook site.yml --syntax-check

15. Practical Labs and Exercises


Lab 1: Setting Up Inventory and Connectivity Check (Beginner)

Objective

Configure a static INI inventory, set connection parameters, and verify SSH connectivity across hosts.

Environment Setup

  • Control Node: Linux terminal or local workstation.
  • Managed Node: Local VM or remote instance accessible over SSH.

Instructions

  1. Create a working directory:
mkdir -p ~/ansible-lab1 && cd ~/ansible-lab1
  1. Create an hosts.ini inventory file:
[test_servers]
node01 ansible_host=127.0.0.1 ansible_port=2222 ansible_user=vagrant
  1. Run an ad-hoc ping command:
ansible test_servers -i hosts.ini -m ping

Expected Validation Outcome

node01 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

Lab 2: Writing and Executing an Idempotent Playbook (Intermediate)

Objective

Create a playbook that installs Apache, templates custom landing pages, starts the service, and verifies idempotency.

Instructions

  1. Create deploy_apache.yml:
---
- name: Apache Deployment Lab
  hosts: test_servers
  become: true
  tasks:
    - name: Install Apache
      ansible.builtin.apt:
        name: apache2
        state: present

    - name: Deploy Custom HTML Index File
      ansible.builtin.copy:
        content: "<h1>Ansible Lab 2 Deployment Successful</h1>\n"
        dest: /var/www/html/index.html
        mode: '0644'

    - name: Ensure Apache is Running
      ansible.builtin.service:
        name: apache2
        state: started
        enabled: true
  1. Execute the playbook:
ansible-playbook -i hosts.ini deploy_apache.yml
  1. Re-run the exact same playbook command immediately:
ansible-playbook -i hosts.ini deploy_apache.yml

Expected Validation Outcome

On second execution, the output summary should report zero changes: changed=0, proving task idempotency.


Lab 3: Troubleshooting Exercise — Diagnosing SSH and Privilege Failures (Advanced)

Objective

Identify and resolve permission issues on a misconfigured node using Ansible diagnostic tools.

Setup

Execute a broken playbook run intentionally:

ansible-playbook -i hosts.ini deploy_apache.yml -u non_root_user

Tasks for the Student

  1. Interpret the module error message (sudo: a password is required or Permission denied).
  2. Determine whether the failure is caused by SSH auth, missing become, or sudoers rules.
  3. Fix the problem by applying proper privilege flags or updating remote host configuration.

16. Quick Reference and Cheat Sheet

Important Configuration Files

File / Directory Description
./ansible.cfg Project-level configuration file.
./inventory/hosts.ini Infrastructure host and group definitions.
./group_vars/all.yml Global variables applied across all target managed hosts.
./roles/<role_name>/tasks/main.yml Entry point for role task definitions.

Important Log Locations

Location Description
/var/log/ansible/ansible.log Default control node log file (if log_path enabled).
/var/log/syslog or /var/log/messages Remote system log tracking escalated sudo command executions.

Common CLI Commands

# Verify SSH Connectivity across targets
ansible all -i inventory.ini -m ping

# Run Playbook Dry-Run (Check Mode) with Diffs
ansible-playbook site.yml --check --diff

# Limit Playbook Run to specific host
ansible-playbook site.yml -l web01.example.com

# Pass Extra Variables to Playbook
ansible-playbook site.yml -e "app_version=2.1.0 env=production"

# Install Galaxy Requirement Dependencies
ansible-galaxy collection install -r requirements.yml

Troubleshooting Quick Reference

Symptom Initial Diagnostic Command Primary Fix Area
UNREACHABLE host error ansible host -m ping -vvv SSH keys, Port 22, Firewalls, ansible_user.
Permission denied / Sudo failed ansible host -b -m command -a "whoami" /etc/sudoers, -K flag, become: true.
Python Interpreter Not Found ansible host -m raw -a "apt install python3" Bootstrap Python via raw module.
Tasks always report changed ansible-playbook site.yml --diff Add creates, removes, or use native module.
Handlers fail to reload service Check if prior task reported changed: true Add meta: flush_handlers.

Important Ports and Protocols

Port Protocol Purpose
22 TCP Standard SSH traffic between Control Node and Linux targets.
5985 / 5986 TCP WinRM HTTP/HTTPS connection traffic for Windows targets.

17. Further Reading and References

The following official documentation resources provide ongoing reference materials for administering Ansible in production environments:

  1. Ansible Community Documentation https://docs.ansible.com
  2. Ansible Getting Started Guide https://docs.ansible.com/ansible/latest/getting_started/index.html
  3. Ansible Collection Module Index https://docs.ansible.com/ansible/latest/collections/index.html
  4. Ansible Galaxy Package Hub https://galaxy.ansible.com