Domain 3Systems & IaCCORE

Building a Multi-Distro Homelab Monitoring Stack with Ansible, Prometheus, and Grafana

A complete technical guide detailing how to build an automated telemetry stack inside a Proxmox LXC container using Ansible, Prometheus, and Grafana to monitor Ubuntu, RHEL, and Fedora VMs, featuring security audits, IP sanitization, and playbook troubleshooting.

Domain: Domain 3: Systems & IaC•Updated: 2026-09-27
#Ansible#Prometheus#Grafana#Proxmox#LXC#Debian#Homelab#Security#UFW#Firewalld

Building a Multi-Distro Homelab Monitoring Stack: Infrastructure Automation, Telemetry, and Security Hardening

Managing a heterogeneous Linux homelab environment—comprising Debian derivatives (Ubuntu) and Red Hat derivatives (RHEL, Fedora)—presents distinct infrastructure management challenges: configuration drift, disparate tooling, and fragmented visibility.

Operating across distro families requires handling different package managers (apt vs dnf), distinct firewall frameworks (ufw vs firewalld), and varying default security profiles (AppArmor vs SELinux).

This guide details the complete architecture, automation playbooks, technical decision-making, security considerations, and real-world troubleshooting lessons learned while building a centralized monitoring and management stack using Ansible, Prometheus, and Grafana hosted inside a Proxmox VE Linux Container (LXC).


Security Audit & Anonymization Report

1. Information Anonymization & Disambiguation

  • Internal IP Sanitization: All internal IPv4 addresses have been replaced with standard RFC 1918 documentation subnets (10.0.0.0/24) and generic variable placeholders (e.g., 10.0.0.248 for monitoring-ctrl, 10.0.0.250 for Proxmox VE). This prevents exposure of actual internal subnets, gateway addresses, or host mapping schemes.
  • Ports Retained: Standard service ports (22 for SSH, 9090 for Prometheus, 3000 for Grafana, 9100 for Node Exporter) are preserved.

2. Identified Security Risks & Architectural Hardening

Risk Category Identified Issue Severity Mitigation & Technical Recommendation
Unencrypted Telemetry Transport Prometheus metrics scraping (:9100) and web UIs (:9090, :3000) transmit in plain HTTP across the local network. Medium Implement an Nginx or Caddy reverse proxy with TLS certificates (Let’s Encrypt or local CA) on monitoring-ctrl, and enable TLS/basic auth on node_exporter.
Hardcoded & Default Credentials Grafana initializes with default admin / admin credentials; setup steps involve manually set user passwords. High Enforce mandatory password changes on first login, vault initial passwords using Ansible Vault (ansible-vault), and supply Grafana admin credentials via environment variables or secret files.
Overly Permissive Firewall Rules Firewall tasks allowed port 9100/tcp from any source (0.0.0.0/0). Medium Restrict port 9100/tcp ingress on target VMs exclusively to the IP address of monitoring-ctrl (10.0.0.248/32).
SSH Key Management Password-less sudo was enabled for sysadmin across target nodes. Medium Restrict sudo permissions to specific required system commands or mandate password-protected SSH keys with ssh-agent.
Container Privilege Boundary LXC containers share the host kernel. Low Running an unprivileged container ensures UID 0 inside the container maps to an unprivileged high UID (e.g., 100000) on the Proxmox host, preventing container escape root takeover.

1. Architectural Decisions & Network Topology

Containerization Choice: Co-locating on an Unprivileged LXC

Rather than deploying separate Virtual Machines or individual LXC containers for Ansible, Prometheus, and Grafana, all three control-plane components are co-located inside a single Debian 12 LXC container named monitoring-ctrl.

  • Resource Footprint: A single LXC shares the host OS kernel, consuming under ~150 MB RAM at idle and ~2 GB under active telemetry scraping, compared to 4–8 GB required to run three separate full virtualization VMs.
  • Security Boundary (Unprivileged Container): In an unprivileged container, container root (UID 0) is mapped to an unprivileged sub-UID range (e.g., UID 100000) on the Proxmox host. If an attacker compromises a service inside monitoring-ctrl, they cannot gain root access to the Proxmox hypervisor.
  • Subnet & Network Layout: All nodes reside on a dedicated internal subnet (10.0.0.0/24).
                      +-------------------------------------------------+
                      |              Proxmox VE Host                    |
                      |                10.0.0.250                       |
                      +-------------------------------------------------+
                                               |
                                      +------------------+
                                      | vmbr0 Network    |
                                      +------------------+
                                               |
          +------------------------------------+------------------------------------+
          |                                    |                                    |
+-------------------+                +-------------------+                +-------------------+
|  ubuntu-node      |                |  rhel-node        |                |  fedora-node      |
|  10.0.0.249       |                |  10.0.0.51        |                |  10.0.0.53        |
|  (node_exporter)  |                |  (node_exporter)  |                |  (node_exporter)  |
+-------------------+                +-------------------+                +-------------------+
          ^                                    ^                                    ^
          | Scrape :9100                       | Scrape :9100                       | Scrape :9100
          | SSH :22                            | SSH :22                            | SSH :22
          +------------------------------------+------------------------------------+
                                               |
                                  +--------------------------+
                                  | monitoring-ctrl (LXC)    |
                                  | 10.0.0.248               |
                                  |  - Ansible Control Node  |
                                  |  - Prometheus (:9090)    |
                                  |  - Grafana UI (:3000)    |
                                  +--------------------------+

Anonymized Environment Matrix

Hostname Role OS Family Subnet IP Listening Ports Firewall Policy
monitoring-ctrl Control Plane & Telemetry Debian 12 (LXC) 10.0.0.248 22, 9090, 3000 Ingress: 22, 9090, 3000
ubuntu-node Monitored Target Ubuntu 24.04 LTS 10.0.0.249 22, 9100 Ingress 9100 restricted to 10.0.0.248
rhel-node Monitored Target RHEL 10 10.0.0.51 22, 9100 Ingress 9100 restricted to 10.0.0.248
fedora-node Monitored Target Fedora 40 10.0.0.53 22, 9100 Ingress 9100 restricted to 10.0.0.248

2. Technical Rationale of the Technology Stack

Ansible (Configuration Management & Orchestration)

Ansible is agentless. It operates over OpenSSH and Python, eliminating the need to maintain background daemons on target servers.

  • Idempotency: Ansible playbooks ensure that re-running tasks results in no changes if the system is already in the desired state.
  • Fact Gathering: Ansible automatically discovers system metadata (ansible_os_family, ansible_distribution, ansible_distribution_major_version), enabling dynamic task branching.

Prometheus (Time Series Telemetry DB)

Prometheus operates on an active pull model.

  • Pull vs. Push: Instead of target hosts pushing metrics to a central database, Prometheus polls target HTTP /metrics endpoints every 15 seconds. If a target goes offline, Prometheus immediately detects the failure without waiting for a connection timeout from a remote agent.
  • TSDB Storage: Metric samples are stored in an optimized local time-series database on monitoring-ctrl, indexed by metric name and key-value label pairs.

Grafana (Visualization & Provisioning)

Grafana queries Prometheus using PromQL (Prometheus Query Language) to render real-time dashboards.

  • Declarative Infrastructure: Dashboards and datasources are provisioned as YAML and JSON files stored in source control, allowing full stack recreation without manual GUI clicks.

3. Phase 1 — Ansible Foundation & Multi-Distro Baseline

Inventory Grouping Strategy (hosts.ini)

The inventory leverages parent/child inheritance to separate RedHat derivatives from Debian derivatives while allowing playbooks to target all VMs simultaneously.

[rhel_nodes]
rhel-node

[ubuntu_nodes]
ubuntu-node

[fedora_nodes]
fedora-node

[redhat_family:children]
rhel_nodes
fedora_nodes

[debian_family:children]
ubuntu_nodes

[all_vms:children]
redhat_family
debian_family

Multi-Distro Baseline Playbook (playbooks/baseline.yml)

This playbook establishes a security baseline, updates package caches, configures time synchronization, and manages OS-specific firewalls.

---
- name: Apply Baseline Configuration across Linux Nodes
  hosts: all_vms
  become: true
  tasks:
    # 1. System Timezone Alignment
    - name: Set system timezone to UTC
      community.general.timezone:
        name: UTC

    # 2. RHEL EPEL Repository Handling
    # RHEL base repos omit utilities like htop. Download EPEL RPM via curl to avoid Python HTTPS bugs.
    - name: Download EPEL repository RPM using curl on RHEL
      when: ansible_distribution == 'RedHat'
      ansible.builtin.command:
        cmd: "curl -sSL -o /tmp/epel-release-latest.rpm https://dl.fedoraproject.org/pub/epel/epel-release-latest-{{ ansible_distribution_major_version }}.noarch.rpm"
        creates: /tmp/epel-release-latest.rpm

    - name: Install EPEL repository from local RPM on RHEL
      when: ansible_distribution == 'RedHat'
      ansible.builtin.dnf:
        name: /tmp/epel-release-latest.rpm
        state: present
        disable_gpg_check: true

    # 3. Package Management (Distro-Agnostic Abstraction)
    - name: Install common utility packages
      ansible.builtin.package:
        name:
          - curl
          - wget
          - htop
          - net-tools
          - rsync
        state: present

    # 4. Automatic Security Updates Branching
    - name: Enable automatic updates on Debian family
      when: ansible_os_family == 'Debian'
      ansible.builtin.apt:
        name: unattended-upgrades
        state: present

    - name: Enable automatic updates on RedHat family
      when: ansible_os_family == 'RedHat'
      ansible.builtin.package:
        name: dnf-automatic
        state: present

    # 5. Firewall Abstraction & SSH Allow Rules
    - name: Ensure UFW allows SSH on Ubuntu
      when: ansible_os_family == 'Debian'
      community.general.ufw:
        rule: allow
        port: '22'
        proto: tcp
        state: enabled

    - name: Ensure firewalld allows SSH on RHEL/Fedora
      when: ansible_os_family == 'RedHat'
      ansible.posix.firewalld:
        service: ssh
        permanent: true
        state: enabled
        immediate: true

    # 6. SSH Hardening Baseline
    - name: Disable password authentication in SSH
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^#?PasswordAuthentication'
        line: 'PasswordAuthentication no'
      notify: Restart SSH Service

  handlers:
    - name: Restart SSH Service
      ansible.builtin.service:
        name: "{{ 'ssh' if ansible_os_family == 'Debian' else 'sshd' }}"
        state: restarted

Playbook Execution & Handler Verification

Applying baseline.yml configures SSH hardening, timezone alignment, and firewall allow rules seamlessly across all nodes:

Technical Decisions in Baseline Playbook

  • ansible.builtin.package: Uses Ansible’s distribution-agnostic package module, automatically selecting apt on Ubuntu or dnf on RHEL/Fedora.
  • Targeted Handler Dispatch: The handler evaluates ansible_os_family to trigger ssh on Debian/Ubuntu or sshd on RHEL/Fedora.

4. Phase 2 — Node Exporter & Prometheus Deployment

Node Exporter Playbook (playbooks/deploy_node_exporter.yml)

Node Exporter runs as an unprivileged service on each target VM, exposing host telemetry on port 9100.

---
- name: Deploy Node Exporter to Target Hosts
  hosts: all_vms
  become: true
  vars:
    node_exporter_version: "1.8.2"
    node_exporter_bin_dir: "/usr/local/bin"
    controller_ip: "10.0.0.248"

  tasks:
    - name: Create node_exporter system group
      ansible.builtin.group:
        name: node_exporter
        state: present
        system: true

    - name: Create node_exporter system user
      ansible.builtin.user:
        name: node_exporter
        group: node_exporter
        shell: /sbin/nologin
        system: true

    # Download archive on Controller to bypass remote Python HTTPS bugs
    - name: Download Node Exporter release on control node
      ansible.builtin.get_url:
        url: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/node_exporter-{{ node_exporter_version }}.linux-amd64.tar.gz" # noqa: yaml[line-length]
        dest: "/tmp/node_exporter-{{ node_exporter_version }}.linux-amd64.tar.gz"
        mode: '0644'
      delegate_to: localhost
      run_once: true

    - name: Unarchive Node Exporter release on target hosts
      ansible.builtin.unarchive:
        src: "/tmp/node_exporter-{{ node_exporter_version }}.linux-amd64.tar.gz"
        dest: /tmp

    - name: Copy Node Exporter binary to system execution path
      ansible.builtin.copy:
        src: "/tmp/node_exporter-{{ node_exporter_version }}.linux-amd64/node_exporter"
        dest: "{{ node_exporter_bin_dir }}/node_exporter"
        mode: '0755'
        remote_src: true
        owner: node_exporter
        group: node_exporter

    - name: Create systemd unit file for Node Exporter
      ansible.builtin.copy:
        dest: /etc/systemd/system/node_exporter.service
        content: |
          [Unit]
          Description=Node Exporter
          Wants=network-online.target
          After=network-online.target

          [Service]
          User=node_exporter
          Group=node_exporter
          Type=simple
          ExecStart={{ node_exporter_bin_dir }}/node_exporter

          [Install]
          WantedBy=multi-user.target
        mode: '0644'
      notify: Restart node_exporter

    # Hardened Firewall Rules: Restrict port 9100 strictly to controller IP
    - name: Allow port 9100 from controller on UFW (Debian/Ubuntu)
      when: ansible_os_family == 'Debian'
      community.general.ufw:
        rule: allow
        from_ip: "{{ controller_ip }}"
        port: '9100'
        proto: tcp

    - name: Allow port 9100 from controller on firewalld (RHEL/Fedora)
      when: ansible_os_family == 'RedHat'
      ansible.posix.firewalld:
        rich_rule: 'rule family="ipv4" source address="{{ controller_ip }}" port port="9100" protocol="tcp" accept'
        permanent: true
        state: enabled
        immediate: true

  handlers:
    - name: Restart node_exporter
      ansible.builtin.service:
        name: node_exporter
        state: started
        enabled: true
        daemon_reload: true

Verifying Node Exporter Telemetry Endpoint

Before deploying Prometheus, query the /metrics endpoint from monitoring-ctrl to confirm that metrics are being actively generated:

Prometheus Deployment & Alert Rules (playbooks/deploy_prometheus.yml)

Deploys Prometheus on monitoring-ctrl, configures scrape targets, and defines core Alertmanager rules.

    - name: Deploy Alertmanager Rules File
      ansible.builtin.copy:
        dest: /etc/prometheus/alert_rules.yml
        owner: prometheus
        group: prometheus
        mode: '0644'
        content: |
          {% raw %}
          groups:
            - name: node_alerts
              rules:
                - alert: InstanceDown
                  expr: up == 0
                  for: 2m
                  labels:
                    severity: critical
                  annotations:
                    summary: "Instance {{ $labels.instance }} down"
                    description: "{{ $labels.instance }} of job {{ $labels.job }} has been unreachable for >2 minutes."

                - alert: HighDiskUsage
                  expr: (node_filesystem_size_bytes{mountpoint="/"}-node_filesystem_free_bytes{mountpoint="/"})/node_filesystem_size_bytes{mountpoint="/"} * 100 > 80
                  for: 5m
                  labels:
                    severity: warning
                  annotations:
                    summary: "High Disk Usage on {{ $labels.instance }}"
                    description: "Root partition disk usage is above 80% (current: {{ $value }}%)."

                - alert: HighCPUPercent
                  expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
                  for: 5m
                  labels:
                    severity: warning
                  annotations:
                    summary: "High CPU Load on {{ $labels.instance }}"
                    description: "CPU load exceeds 85% over a 5-minute window."
          {% endraw %}

    - name: Deploy Prometheus Scrape Configuration
      ansible.builtin.copy:
        dest: /etc/prometheus/prometheus.yml
        owner: prometheus
        group: prometheus
        mode: '0644'
        content: |
          global:
            scrape_interval: 15s
            evaluation_interval: 15s

          rule_files:
            - "alert_rules.yml"

          scrape_configs:
            - job_name: "prometheus"
              static_configs:
                - targets: ["localhost:9090"]

            - job_name: "node_exporter"
              static_configs:
                - targets:
                    - "rhel-node:9100"
                    - "ubuntu-node:9100"
                    - "fedora-node:9100"
                  labels:
                    environment: "homelab"

5. Phase 3 — Declarative Grafana Provisioning

The Grafana playbook (playbooks/deploy_grafana.yml) automates UI provisioning, bypassing manual configuration steps.

---
- name: Install and Provision Grafana
  hosts: localhost
  connection: local
  become: true
  tasks:
    - name: Install prerequisites & Grafana repository
      ansible.builtin.apt:
        name: [apt-transport-https, software-properties-common, wget, grafana]
        state: present
        update_cache: true

    - name: Ensure Grafana provisioning directories exist
      ansible.builtin.file:
        path: "{{ item }}"
        state: directory
        owner: root
        group: grafana
        mode: '0750'
      loop:
        - /etc/grafana/provisioning/datasources
        - /etc/grafana/provisioning/dashboards
        - /var/lib/grafana/dashboards

    - name: Deploy Prometheus Datasource File
      ansible.builtin.copy:
        dest: /etc/grafana/provisioning/datasources/prometheus.yaml
        owner: root
        group: grafana
        mode: '0640'
        content: |
          apiVersion: 1
          datasources:
            - name: Prometheus
              type: prometheus
              access: proxy
              url: http://localhost:9090
              isDefault: true
              editable: true

    - name: Deploy Dashboards Provisioning Config
      ansible.builtin.copy:
        dest: /etc/grafana/provisioning/dashboards/default.yaml
        owner: root
        group: grafana
        mode: '0640'
        content: |
          apiVersion: 1
          providers:
            - name: 'Default'
              orgId: 1
              folder: ''
              type: file
              disableDeletion: false
              editable: true
              options:
                path: /var/lib/grafana/dashboards

    - name: Download Node Exporter Full Community Dashboard (ID 1860)
      ansible.builtin.get_url:
        url: "https://grafana.com/api/dashboards/1860/revisions/37/download"
        dest: /var/lib/grafana/dashboards/node_exporter_1860.json
        owner: root
        group: grafana
        mode: '0644'

    - name: Enable and start Grafana service
      ansible.builtin.service:
        name: grafana-server
        state: restarted
        enabled: true

6. Engineering Post-Mortem: Failure Modes & Technical Solutions

+-----------------------------------------------------------------------------------+
|                           TROUBLESHOOTING MATRIX                                  |
+-------------------------------------+---------------------------------------------+
| Failure / Symptom                   | Root Cause & Technical Fix                  |
+-------------------------------------+---------------------------------------------+
| 1. Missing `htop` package on RHEL   | RHEL base repos exclude interactive tools;  |
|                                     | fix by fetching and installing EPEL RPM.    |
+-------------------------------------+---------------------------------------------+
| 2. `HTTPSConnection` unexpected     | Deprecated Python 3.12+ `cert_file` kwarg;  |
|    `cert_file` crash                | fix by downloading on controller via curl.  |
+-------------------------------------+---------------------------------------------+
| 3. Jinja2 Parser Collision on       | Double curly braces `{{ $labels.instance }}`|
|    Prometheus alert rule syntax     | collided; fix by wrapping in `{% raw %}`.   |
+-------------------------------------+---------------------------------------------+

Issue 1: Target-Side Python 3.12+ HTTPS Connection Crash

  • Symptom: Executing deploy_node_exporter.yml crashed across target nodes with: Failure downloading https://... An unknown error occurred: HTTPSConnection.__init__() got an unexpected keyword argument 'cert_file'

A similar error occurred when attempting to fetch the EPEL RPM on RHEL:

  • Root Cause: Target nodes running modern Python 3.12+ (RHEL 10, Ubuntu 24.04, Fedora 40) encountered a bug in Ansible’s remote HTTP engine where deprecated http.client.HTTPSConnection parameters were invoked during remote HTTPS unarchiving (remote_src: true).
  • Technical Fix: Shifted download responsibility to the Ansible control node (monitoring-ctrl) using delegate_to: localhost and run_once: true. The controller downloads the archive once and distributes it over standard SSH. For EPEL on RHEL, curl was used directly.

Issue 2: Jinja2 Parser Collision with Prometheus Rules

  • Symptom: ansible-lint threw a fatal error: jinja[invalid]: Syntax error in jinja2 template... unexpected char '$' at 205.
  • Root Cause: Ansible’s template engine parses double-curly braces ({{ ... }}). Prometheus Alertmanager rules use identical syntax for variable interpolation (e.g., {{ $labels.instance }}). Ansible tried to evaluate $labels.instance as an internal Ansible variable, failing on the invalid symbol $.
  • Technical Fix: Wrapped the Alertmanager configuration block inside raw Jinja2 delimiters ({% raw %} … {% endraw %}).

Issue 3: Missing RHEL Base Utility Packages

  • Symptom: Executing baseline.yml failed on rhel-node during package installation with No package htop available.

  • Root Cause: RHEL subscription repositories exclude interactive utilities like htop from base channels. They reside in EPEL.
  • Technical Fix: Added a task using curl to fetch the EPEL RPM before installing it via dnf.

7. Operational Workflow & Final Verification

Running the automation workflow provisions the complete monitoring stack from scratch:

# 1. Lint playbooks
ansible-lint playbooks/*.yml

# 2. Execute playbooks in order
ansible-playbook playbooks/baseline.yml
ansible-playbook playbooks/deploy_node_exporter.yml
ansible-playbook playbooks/deploy_prometheus.yml
ansible-playbook playbooks/deploy_grafana.yml

Verification Endpoints

1. Prometheus Metrics Graph (http://10.0.0.248:9090)

Prometheus actively scrapes all target endpoints (localhost:9090, rhel-node:9100, ubuntu-node:9100, fedora-node:9100). Querying node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 plots live memory availability across Ubuntu, RHEL, and Fedora targets in real time:

2. Provisioned Grafana Dashboard (http://10.0.0.248:3000)

Logging into Grafana displays the pre-provisioned Node Exporter Full dashboard, providing real-time CPU, memory, disk I/O, and network telemetry across the multi-distro fleet:


8. Key Technical Lessons

  1. Control-Node Offloading (delegate_to: localhost): When managing target hosts with heterogeneous Python minor versions, offloading downloads and packaging to the controller eliminates SSL/HTTP library divergence.
  2. Raw Delimiters for Metric Templates: Always wrap third-party template syntaxes (like PromQL, Alertmanager, or Go templates) in {% raw %}...{% endraw %} blocks within Ansible copy/template tasks.
  3. Multi-Distro Abstractions: Leverage ansible.builtin.package, ansible_os_family, and ansible_distribution to write single, unified playbooks that seamlessly configure Debian and RedHat derivative systems.