Domain 10Linux & SREINTERMEDIATE

(Parma) Debugging Terraform Issues: Resolving Core Mismatches, DAG Cycles, and Duplicate Providers in 'Parma'

A deep dive into diagnosing and fixing Terraform execution failures, version constraint blocks, HCL dependency cycles (DAG loops), and duplicate provider declarations in a local infrastructure deployment.

Domain: Domain 10: Linux Internals, SRE & Troubleshooting•Updated: 2026-11-09
#Terraform#Infrastructure-as-Code#HCL#Debugging#Linux#SadServers

1. Introduction

Infrastructure as Code (IaC) tools like HashiCorp Terraform allow systems administrators and Site Reliability Engineers (SREs) to manage infrastructure declaratively using configuration files. However, when refactoring Terraform modules, subtle configuration errors can prevent Terraform from parsing modules, building resource execution graphs, or applying state changes.

In this article, we will examine a real-world troubleshooting scenario titled “Parma”: Debugging Terraform Issues. We will walk through the systematic debugging process required to fix a broken local Terraform configuration on a Linux host.

Through this guide, you will learn how to:

  • Identify and resolve Terraform Core version constraint mismatches.
  • Understand and break Directed Acyclic Graph (DAG) dependency cycles (circular references).
  • Recognize file-level vs. module-level scoping in HashiCorp Configuration Language (HCL) to prevent provider collisions.
  • Use diagnostic tools, validation workflows, and automated check scripts to ensure clean state management.

2. Problem Description and Environment

Scenario Details

  • Challenge Name: “Parma”: Debugging Terraform Issues
  • Difficulty Level: Medium
  • Operating System: Debian 13 (Testing/Unstable)
  • Privileges: Root (sudo) access available
  • Project Location: /home/admin/infra/
  • Target Artifact: /var/local/platform-status.txt

Challenge Description

The target host is designed to publish a machine-readable status marker file using a local Terraform configuration located at /home/admin/infra/. The project should manage and output the file /var/local/platform-status.txt.

Following a recent refactor, both terraform plan and terraform apply fail completely, and the status file is either missing or stale.

+-----------------------------------------------------------------------+
|                         Target Infrastructure                         |
|                                                                       |
|  /home/admin/infra/                                                   |
|  ├── versions.tf   ---> Manages provider & Core requirements           |
|  └── main.tf       ---> Defines local_file resources                  |
|                                                                       |
|  Target Output:                                                       |
|  /var/local/platform-status.txt  [Content: "SadServers - Parma OK"]   |
+-----------------------------------------------------------------------+

Verification Criteria

The solution is verified by the script /home/admin/agent/check.sh, which tests two specific requirements:

  1. The first line of /var/local/platform-status.txt must be exactly SadServers - Parma OK.
  2. Running terraform plan in /home/admin/infra/ must report zero changes pending (a clean state exit code).

3. Initial Investigation and Hypotheses

Before making modifications to the system, an SRE formulates hypotheses based on initial symptoms and verifies them through diagnostic commands.

Initial Hypotheses Matrix

Hypothesis Supporting Evidence Evidence Against Diagnostic Step
H1: Terraform Version Lockout terraform plan fails immediately on configuration parsing. None yet. Run terraform plan and inspect the error message.
H2: Circular Dependency Cycle HCL configuration refactoring often introduces improper cross-resource variable references. Cannot verify until configuration parsing succeeds. Inspect main.tf resource dependencies.
H3: File Permission Block /var/local/ requires specific permissions to write files. User admin has sudo access. Check permissions on /var/local/.
H4: Provider Collision Duplicate provider blocks declared across multiple .tf files. Unconfirmed until parsing succeeds. Inspect all .tf files in the directory.

4. Troubleshooting Process

Step 1: Directory Inspection and Initial Plan Failure

Objective Navigate to the infrastructure directory, list the configuration files, and capture the exact failure output generated by terraform plan.

Command

cd /home/admin/infra && ls -la
terraform plan

Command Explanation

  • cd /home/admin/infra: Changes the working directory to the Terraform root module location.
  • ls -la: Lists all directory entries in long format, including hidden dotfiles such as .terraform/ and .terraform.lock.hcl.
  • terraform plan: Reads configuration files, loads state, and creates an execution plan to update infrastructure.

Output and Analysis

admin@i-05f2dc483fa4c779f:~$ cd /home/admin/infra/ 
admin@i-05f2dc483fa4c779f:~/infra$ ls -la
total 24
drwxr-xr-x 3 admin admin 4096 Jul  2 23:58 .
drwx------ 7 admin admin 4096 Jul  2 23:58 ..
drwxr-xr-x 3 admin admin 4096 Jul  2 23:58 .terraform
-rw-r--r-- 1 admin admin 1184 Jul  2 23:58 .terraform.lock.hcl
-rw-r--r-- 1 admin admin  388 Jul  2 23:58 main.tf
-rw-r--r-- 1 admin admin  177 Jul  2 23:58 versions.tf

admin@i-05f2dc483fa4c779f:~/infra$ terraform plan
╷
│ Error: Unsupported Terraform Core version
│ 
│   on versions.tf line 2, in terraform:
│    2:    required_version = "~> 1.12.0"
│ 
│ This configuration does not support Terraform version 1.9.8. To proceed, either choose another
│ supported Terraform version or update this version constraint. Version constraints are normally set
│ for good reason, so updating the constraint may lead to other errors or unexpected behavior.
╵

Decision The initial execution failed during early configuration parsing. The host has Terraform version 1.9.8 installed, but versions.tf enforces required_version = "~> 1.12.0". Because 1.9.8 is lower than 1.12.0, Terraform aborts before inspecting resources or local state. Hypothesis H1 is confirmed.


Step 2: Inspecting Code Configuration and Test Harness

Objective Inspect the exact content of all .tf files and analyze the test validation script /home/admin/agent/check.sh.

Command

cat -n versions.tf main.tf
cat /home/admin/agent/check.sh

Command Explanation

  • cat -n: Concatenates and prints file contents with line numbers to make error line matching precise.
  • cat /home/admin/agent/check.sh: Displays the test harness script to understand validation logic.

Output and Analysis

admin@i-05f2dc483fa4c779f:~/infra$ cat -n versions.tf main.tf 
     1	terraform {
     2	  required_version = "~> 1.12.0"
     3	
     4	  required_providers {
     5	    local = {
     6	      source  = "hashicorp/local"
     7	      version = "~> 2.5.0"
     8	    }
     9	  }
    10	}
    11	
    12	provider "local" {}
    13	# Refactored status marker — apply currently fails (see terraform plan).
    14	
    15	resource "local_file" "status_body" {
    16	  filename = "/var/local/.platform-status-body"
    17	  content  = local_file.platform_status.content
    18	}
    19	
    20	resource "local_file" "platform_status" {
    21	  filename        = "/var/local/platform-status.txt"
    22	  content         = local_file.status_body.content
    23	  file_permission = "0644"
    24	}

admin@i-05f2dc483fa4c779f:~/infra$ cat /home/admin/agent/check.sh
#!/usr/bin/env bash
# DO NOT MODIFY THIS FILE ("Check My Solution" will fail)

fail() {
  echo -n "NO"
  exit 0
}

INFRA=/home/admin/infra
STATUS=/var/local/platform-status.txt

line=$(head -n 1 "$STATUS" 2>/dev/null | tr -d '\r')
if [ "$line" != "SadServers - Parma OK" ]; then
  fail
fi

if ! env PATH="/usr/local/bin:${PATH}" bash -c "cd '$INFRA' && terraform plan -detailed-exitcode -input=false" >/dev/null 2>&1; then
  fail
fi

echo -n "OK"

Decision Code analysis revealed three major flaws:

  1. Version Lockout: versions.tf requires Terraform ~> 1.12.0, blocking installed binary 1.9.8.
  2. Circular Dependency: main.tf defines local_file.status_body referencing local_file.platform_status.content (line 17), while local_file.platform_status references local_file.status_body.content (line 22). Neither block actually defines standard text.
  3. Check Script Logic: /home/admin/agent/check.sh checks that line 1 of /var/local/platform-status.txt matches "SadServers - Parma OK" and that terraform plan -detailed-exitcode returns exit code 0 (clean state).

Step 3: Resolving Version Constraint Gate

Objective Update versions.tf to permit execution under Terraform Core 1.9.8 and re-evaluate the execution plan.

Command

sed -i 's/~> 1.12.0/~> 1.9.0/' versions.tf
terraform plan

Command Explanation

  • sed -i 's/.../.../' versions.tf: Performs an in-place string replacement on versions.tf, changing the version constraint from ~> 1.12.0 to ~> 1.9.0.

Output and Analysis

admin@i-05f2dc483fa4c779f:~/infra$ sed -i 's/~> 1.12.0/~> 1.9.0/' versions.tf 
admin@i-05f2dc483fa4c779f:~/infra$ terraform plan
╷
│ Error: Cycle: local_file.platform_status, local_file.status_body
│ 
│ 
╵

Decision Updating versions.tf bypassed the version gate. Terraform proceeded to dependency graph construction, immediately revealing the expected circular dependency error (Error: Cycle: local_file.platform_status, local_file.status_body). Hypothesis H2 is confirmed.


Step 4: Uncovering and Fixing the DAG Cycle & Provider Collision

Objective Fix main.tf by assigning the required string "SadServers - Parma OK\n" to local_file.status_body and making local_file.platform_status depend unidirectionally on local_file.status_body.content.

Command (Attempt 1 - Introduces Provider Collision)

cat <<'EOF' > main.tf
provider "local" {}

resource "local_file" "status_body" {
  filename = "/var/local/.platform-status-body"
  content  = "SadServers - Parma OK\n"
}

resource "local_file" "platform_status" {
  filename        = "/var/local/platform-status.txt"
  content         = local_file.status_body.content
  file_permission = "0644"
}
EOF
terraform plan

Output and Analysis

admin@i-05f2dc483fa4c779f:~/infra$ terraform plan
╷
│ Error: Duplicate provider configuration
│ 
│   on versions.tf line 12:
│   12: provider "local" {}
│ 
│ A default (non-aliased) provider configuration for "local" was already given at main.tf:1,1-17. If
│ multiple configurations are required, set the "alias" argument for alternative configurations.
╵

Decision Because versions.tf line 12 already contained provider "local" {}, putting provider "local" {} in main.tf caused a duplicate default provider error across the module scope. Hypothesis H4 is confirmed.

We must remove provider "local" {} from main.tf.

Command (Attempt 2 - Corrected HCL Structure)

cat <<'EOF' > main.tf
resource "local_file" "status_body" {
  filename = "/var/local/.platform-status-body"
  content  = "SadServers - Parma OK\n"
}

resource "local_file" "platform_status" {
  filename        = "/var/local/platform-status.txt"
  content         = local_file.status_body.content
  file_permission = "0644"
}
EOF

Step 5: Final Plan, Infrastructure Execution, and Verification

Objective Generate a clean execution plan, apply state changes to create the target status file, and execute the verification check.

Command

terraform plan
terraform apply -auto-approve
/home/admin/agent/check.sh

Output and Analysis

admin@i-05f2dc483fa4c779f:~/infra$ terraform plan
Plan: 2 to add, 0 to change, 0 to destroy.

admin@i-05f2dc483fa4c779f:~/infra$ terraform apply -auto-approve
local_file.status_body: Creating...
local_file.status_body: Creation complete after 0s [id=0b033b7ccc6e91931409d6ecc6c83819e1e0b52c]
local_file.platform_status: Creating...
local_file.platform_status: Creation complete after 0s [id=0b033b7ccc6e91931409d6ecc6c83819e1e0b52c]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

admin@i-05f2dc483fa4c779f:~/infra$ /home/admin/agent/check.sh
OK

Decision terraform apply successfully created /var/local/.platform-status-body and /var/local/platform-status.txt. Executing /home/admin/agent/check.sh returned OK, confirming that the marker file content matches and terraform plan -detailed-exitcode returns 0.


5. Root Cause Analysis

The incident was caused by three cascading configuration errors introduced during an incomplete refactoring pass:

1. Version Requirement Incompatibility

The terraform block in versions.tf configured required_version = "~> 1.12.0". The pessimistic constraint operator (~>) locks allowed versions to >= 1.12.0 and < 1.13.0. Because the system binary installed at /usr/local/bin/terraform was 1.9.8, Terraform halted before compiling the execution graph.

2. Dependency Graph Cycle (DAG Loop)

Terraform builds a Directed Acyclic Graph (DAG) to determine the order in which resources are evaluated and applied. When Resource A references Resource B, an implicit graph edge is drawn from B to A ($B \rightarrow A$), meaning B must exist before A can be computed.

In the broken refactored main.tf:

  • local_file.status_body referenced local_file.platform_status.content.
  • local_file.platform_status referenced local_file.status_body.content.

This produced a closed graph cycle ($status_body \leftrightarrow platform_status$), making graph topological sorting mathematically impossible.

graph TD
    subgraph Broken Cycle (DAG Loop)
        A[local_file.status_body] -->|content = platform_status.content| B[local_file.platform_status]
        B -->|content = status_body.content| A
    end

    subgraph Resolved Unidirectional DAG
        C[String Literal: 'SadServers - Parma OK'] --> D[local_file.status_body]
        D -->|content = status_body.content| E[local_file.platform_status]
    end

3. HCL Module-Level Provider Collision

Terraform parses all .tf files in a single root module directory into a single combined evaluation graph. Declaring provider "local" {} in versions.tf while simultaneously declaring provider "local" {} in main.tf violates HCL module scoping rules, throwing a Duplicate provider configuration error.


6. Solution and Remediation

To completely resolve the issue, two files were modified in /home/admin/infra/.

Final File Configurations

1. versions.tf

terraform {
  required_version = "~> 1.9.0"

  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "~> 2.5.0"
    }
  }
}

provider "local" {}

2. main.tf

resource "local_file" "status_body" {
  filename = "/var/local/.platform-status-body"
  content  = "SadServers - Parma OK\n"
}

resource "local_file" "platform_status" {
  filename        = "/var/local/platform-status.txt"
  content         = local_file.status_body.content
  file_permission = "0644"
}

Execution Steps Applied

cd /home/admin/infra
sed -i 's/~> 1.12.0/~> 1.9.0/' versions.tf

cat <<'EOF' > main.tf
resource "local_file" "status_body" {
  filename = "/var/local/.platform-status-body"
  content  = "SadServers - Parma OK\n"
}

resource "local_file" "platform_status" {
  filename        = "/var/local/platform-status.txt"
  content         = local_file.status_body.content
  file_permission = "0644"
}
EOF

terraform apply -auto-approve

7. Verification and Validation

Verification requires demonstrating both functional marker creation and state cleanliness.

1. File Marker and Permission Verification

cat /var/local/platform-status.txt
ls -l /var/local/platform-status.txt

Output:

SadServers - Parma OK
-rw-r--r-- 1 admin admin 22 Jul  2 23:59 /var/local/platform-status.txt

2. State Cleanliness Verification

terraform plan -detailed-exitcode

Output:

No changes. Your infrastructure matches the configuration.
(Exit code: 0)

3. Automated Test Suite Execution

/home/admin/agent/check.sh

Output:

OK

8. Alternative Troubleshooting Approaches

While direct execution of terraform plan identified the errors sequentially, alternative tools and techniques could have expedited the diagnosis.

1. Static Code Analysis via terraform validate

Before running terraform plan (which checks state files and provider schemas), terraform validate verifies whether the directory contains syntactically valid HCL code.

terraform validate
  • Advantage: Catches duplicate provider blocks and basic structural errors without requiring local provider initialization or state checks.

2. Graph Visualization via terraform graph

When faced with complex dependency cycles, Terraform can generate a DOT-formatted graph representation.

terraform graph | dot -Tpng > graph.png
  • Advantage: Renders a visual flowchart of resource relationships, highlighting cyclic loops immediately in large codebases.

3. HCL Linting via tflint

tflint is a third-party linter for Terraform.

tflint
  • Advantage: Detects provider-specific errors, deprecated syntax, and version mismatches before running Terraform commands in deployment pipelines.

Diagnostic Tools Comparison

Tool Focus Area Catches Version Mismatches? Catches DAG Cycles? Catches Duplicate Providers?
terraform validate Syntax & Schema No Yes Yes
terraform plan State & Execution Diff Yes Yes Yes
terraform graph Structural Relationships No Yes N/A
tflint Best Practices & Rules Yes Partial Yes

9. Linux Command Deep Dive

Command: terraform

Purpose Terraform is an open-source Infrastructure as Code software tool created by HashiCorp. It provisions infrastructure defined in HCL files.

General Syntax

terraform [-version] [-help] <command> [args]

Options and Commands

Option / Command Purpose
terraform init Initializes a working directory containing Terraform configuration files. Downloads providers.
terraform plan Generates and shows an execution plan.
terraform plan -detailed-exitcode Returns exit code 0 for no changes, 1 for error, 2 for pending changes.
terraform apply -auto-approve Applies changes without prompting for manual interactive confirmation.
terraform validate Validates configuration files in a directory without reading remote state.
terraform fmt -check Checks whether input files are formatted according to canonical HCL style guidelines.

Command: cat

Purpose Concatenates files and prints their contents to standard output (stdout).

General Syntax

cat [OPTION]... [FILE]...

Options and Arguments

Option Purpose
cat -n Number all output lines.
cat -b Number non-blank output lines.
cat -A Equivalent to -vET; displays non-printable characters, tabs (^I), and line endings ($).

Command: sed

Purpose A stream editor used to perform basic text transformations on an input stream or file.

General Syntax

sed [OPTION]... 'script' [input-file]

Options and Arguments

Option Purpose
sed -i Edit files in-place instead of printing to stdout.
sed 's/regexp/replacement/' Substitutes the first match of regexp with replacement on each line.
sed 's/regexp/replacement/g' Global substitution (replaces all matches on each line).

10. Underlying Linux Concepts and Architecture

1. Terraform HCL Scoping Rules

In Terraform, directory boundaries define module boundaries. All files ending in .tf inside /home/admin/infra/ are evaluated together as one single configuration block.

  • File Independence Myth: Naming a file main.tf, versions.tf, or outputs.tf is purely an organizational convention for human engineers. Terraform merges them into a single memory model.
  • Provider Uniqueness: A module can only have one default (un-aliased) provider configuration per provider type. Multiple providers require distinct alias attributes (e.g., provider "aws" { alias = "us_west_2" }).

2. POSIX File Permissions and Atomic Writes

The local_file resource in main.tf explicitly configures file_permission = "0644".

  • 0644 translates to:

  • Owner (admin): Read (4) + Write (2) = 6 (rw-)

  • Group (admin): Read (4) = 4 (r--)

  • Others: Read (4) = 4 (r--)

  • When the local provider writes files, it writes to a temporary file and atomically renames it over the target destination to prevent partial file reads by monitoring agents or status checks.


11. Real-World Sysadmin and SRE Applications

1. Continuous Integration (CI) Static Analysis

To prevent broken Terraform code from reaching version control or production environments, SRE teams implement automated linting in CI/CD pipelines (e.g., GitHub Actions, GitLab CI):

# Sample CI Step for Terraform Validation
steps:
  - name: Check Formatting
    run: terraform fmt -check -recursive

  - name: Validate Syntax & DAG
    run: terraform validate

  - name: TFLint Check
    run: tflint

2. Automated Drift Detection

The test script /home/admin/agent/check.sh uses terraform plan -detailed-exitcode. In production, SREs run scheduled cron jobs or Kubernetes CronJobs executing terraform plan -detailed-exitcode.

  • Exit code 0: No drift; infrastructure perfectly matches code.
  • Exit code 2: Drift detected; unapproved manual changes occurred in the environment. An alert is sent to the SRE on-call engineer.

12. Further Learning and Experiments

Hands-on Lab Experiments

Experiment 1: Intentionally Triggering a DAG Cycle

  1. Edit main.tf and make status_body.content point to platform_status.content, and platform_status.content point to status_body.content.
  2. Run terraform plan.
  3. Observe how Terraform’s graph engine identifies the circular loop.

Experiment 2: Testing Detailed Exit Codes

  1. Run terraform plan -detailed-exitcode on a clean directory and inspect the exit code (echo $?).
  2. Manually edit /var/local/platform-status.txt with echo "drift" > /var/local/platform-status.txt.
  3. Re-run terraform plan -detailed-exitcode and observe that the exit code changes to 2.

Documentation and References


13. Key Takeaways

  1. HCL Files Share a Directory Scope: Terraform merges all .tf files in a directory into one module. Never declare duplicate default provider blocks across separate .tf files.
  2. Break DAG Cycles with Static Values: Directed Acyclic Graphs require at least one root node containing a static literal value to break circular reference loops.
  3. Respect Core Version Constraints: Match required_version in versions.tf to the actual binary installed in your environment or pipeline runners.
  4. Use -detailed-exitcode for Automation: Leverage terraform plan -detailed-exitcode in scripts to programmatically determine whether infrastructure drift exists.

14. Quick Troubleshooting Reference / Runbook

When debugging broken Terraform modules, follow this sequence:

[Start Debugging]
       │
       ▼
1. Check Installed Terraform Version ────►  terraform version
       │
       ▼
2. Validate Syntax & Scoping ───────────►  terraform validate
       │
       ▼
3. Check for Dependency Cycles ──────────►  terraform plan
       │
       ▼
4. Format Configuration Files ───────────►  terraform fmt
       │
       ▼
5. Apply State Non-Interactively ────────►  terraform apply -auto-approve
       │
       ▼
6. Verify Drift Status ──────────────────►  terraform plan -detailed-exitcode

Diagnostic Command Cheat Sheet

# 1. Verify installed core version
terraform version

# 2. Check HCL syntax without reading state
terraform validate

# 3. Locate duplicate providers or cyclic references
terraform plan

# 4. Check for unapplied infrastructure drift (returns code 0 or 2)
terraform plan -detailed-exitcode

# 5. Fix formatting automatically across all files
terraform fmt