1. Introduction
Infrastructure as Code (IaC) tools like HashiCorp Terraform allow systems administrators and Site Reliability Engineers (SREs) to manage infrastructure declaratively using configuration files. However, when refactoring Terraform modules, subtle configuration errors can prevent Terraform from parsing modules, building resource execution graphs, or applying state changes.
In this article, we will examine a real-world troubleshooting scenario titled “Parma”: Debugging Terraform Issues. We will walk through the systematic debugging process required to fix a broken local Terraform configuration on a Linux host.
Through this guide, you will learn how to:
- Identify and resolve Terraform Core version constraint mismatches.
- Understand and break Directed Acyclic Graph (DAG) dependency cycles (circular references).
- Recognize file-level vs. module-level scoping in HashiCorp Configuration Language (HCL) to prevent provider collisions.
- Use diagnostic tools, validation workflows, and automated check scripts to ensure clean state management.
2. Problem Description and Environment
Scenario Details
- Challenge Name: “Parma”: Debugging Terraform Issues
- Difficulty Level: Medium
- Operating System: Debian 13 (Testing/Unstable)
- Privileges: Root (
sudo) access available - Project Location:
/home/admin/infra/ - Target Artifact:
/var/local/platform-status.txt
Challenge Description
The target host is designed to publish a machine-readable status marker file using a local Terraform configuration located at /home/admin/infra/. The project should manage and output the file /var/local/platform-status.txt.
Following a recent refactor, both terraform plan and terraform apply fail completely, and the status file is either missing or stale.
+-----------------------------------------------------------------------+
| Target Infrastructure |
| |
| /home/admin/infra/ |
| ├── versions.tf ---> Manages provider & Core requirements |
| └── main.tf ---> Defines local_file resources |
| |
| Target Output: |
| /var/local/platform-status.txt [Content: "SadServers - Parma OK"] |
+-----------------------------------------------------------------------+
Verification Criteria
The solution is verified by the script /home/admin/agent/check.sh, which tests two specific requirements:
- The first line of
/var/local/platform-status.txtmust be exactlySadServers - Parma OK. - Running
terraform planin/home/admin/infra/must report zero changes pending (a clean state exit code).
3. Initial Investigation and Hypotheses
Before making modifications to the system, an SRE formulates hypotheses based on initial symptoms and verifies them through diagnostic commands.
Initial Hypotheses Matrix
| Hypothesis | Supporting Evidence | Evidence Against | Diagnostic Step |
|---|---|---|---|
| H1: Terraform Version Lockout | terraform plan fails immediately on configuration parsing. |
None yet. | Run terraform plan and inspect the error message. |
| H2: Circular Dependency Cycle | HCL configuration refactoring often introduces improper cross-resource variable references. | Cannot verify until configuration parsing succeeds. | Inspect main.tf resource dependencies. |
| H3: File Permission Block | /var/local/ requires specific permissions to write files. |
User admin has sudo access. |
Check permissions on /var/local/. |
| H4: Provider Collision | Duplicate provider blocks declared across multiple .tf files. |
Unconfirmed until parsing succeeds. | Inspect all .tf files in the directory. |
4. Troubleshooting Process
Step 1: Directory Inspection and Initial Plan Failure
Objective
Navigate to the infrastructure directory, list the configuration files, and capture the exact failure output generated by terraform plan.
Command
cd /home/admin/infra && ls -la
terraform plan
Command Explanation
cd /home/admin/infra: Changes the working directory to the Terraform root module location.ls -la: Lists all directory entries in long format, including hidden dotfiles such as.terraform/and.terraform.lock.hcl.terraform plan: Reads configuration files, loads state, and creates an execution plan to update infrastructure.
Output and Analysis
admin@i-05f2dc483fa4c779f:~$ cd /home/admin/infra/
admin@i-05f2dc483fa4c779f:~/infra$ ls -la
total 24
drwxr-xr-x 3 admin admin 4096 Jul 2 23:58 .
drwx------ 7 admin admin 4096 Jul 2 23:58 ..
drwxr-xr-x 3 admin admin 4096 Jul 2 23:58 .terraform
-rw-r--r-- 1 admin admin 1184 Jul 2 23:58 .terraform.lock.hcl
-rw-r--r-- 1 admin admin 388 Jul 2 23:58 main.tf
-rw-r--r-- 1 admin admin 177 Jul 2 23:58 versions.tf
admin@i-05f2dc483fa4c779f:~/infra$ terraform plan
╷
│ Error: Unsupported Terraform Core version
│
│ on versions.tf line 2, in terraform:
│ 2: required_version = "~> 1.12.0"
│
│ This configuration does not support Terraform version 1.9.8. To proceed, either choose another
│ supported Terraform version or update this version constraint. Version constraints are normally set
│ for good reason, so updating the constraint may lead to other errors or unexpected behavior.
╵
Decision
The initial execution failed during early configuration parsing. The host has Terraform version 1.9.8 installed, but versions.tf enforces required_version = "~> 1.12.0". Because 1.9.8 is lower than 1.12.0, Terraform aborts before inspecting resources or local state. Hypothesis H1 is confirmed.
Step 2: Inspecting Code Configuration and Test Harness
Objective
Inspect the exact content of all .tf files and analyze the test validation script /home/admin/agent/check.sh.
Command
cat -n versions.tf main.tf
cat /home/admin/agent/check.sh
Command Explanation
cat -n: Concatenates and prints file contents with line numbers to make error line matching precise.cat /home/admin/agent/check.sh: Displays the test harness script to understand validation logic.
Output and Analysis
admin@i-05f2dc483fa4c779f:~/infra$ cat -n versions.tf main.tf
1 terraform {
2 required_version = "~> 1.12.0"
3
4 required_providers {
5 local = {
6 source = "hashicorp/local"
7 version = "~> 2.5.0"
8 }
9 }
10 }
11
12 provider "local" {}
13 # Refactored status marker — apply currently fails (see terraform plan).
14
15 resource "local_file" "status_body" {
16 filename = "/var/local/.platform-status-body"
17 content = local_file.platform_status.content
18 }
19
20 resource "local_file" "platform_status" {
21 filename = "/var/local/platform-status.txt"
22 content = local_file.status_body.content
23 file_permission = "0644"
24 }
admin@i-05f2dc483fa4c779f:~/infra$ cat /home/admin/agent/check.sh
#!/usr/bin/env bash
# DO NOT MODIFY THIS FILE ("Check My Solution" will fail)
fail() {
echo -n "NO"
exit 0
}
INFRA=/home/admin/infra
STATUS=/var/local/platform-status.txt
line=$(head -n 1 "$STATUS" 2>/dev/null | tr -d '\r')
if [ "$line" != "SadServers - Parma OK" ]; then
fail
fi
if ! env PATH="/usr/local/bin:${PATH}" bash -c "cd '$INFRA' && terraform plan -detailed-exitcode -input=false" >/dev/null 2>&1; then
fail
fi
echo -n "OK"
Decision Code analysis revealed three major flaws:
- Version Lockout:
versions.tfrequires Terraform~> 1.12.0, blocking installed binary1.9.8. - Circular Dependency:
main.tfdefineslocal_file.status_bodyreferencinglocal_file.platform_status.content(line 17), whilelocal_file.platform_statusreferenceslocal_file.status_body.content(line 22). Neither block actually defines standard text. - Check Script Logic:
/home/admin/agent/check.shchecks that line 1 of/var/local/platform-status.txtmatches"SadServers - Parma OK"and thatterraform plan -detailed-exitcodereturns exit code0(clean state).
Step 3: Resolving Version Constraint Gate
Objective
Update versions.tf to permit execution under Terraform Core 1.9.8 and re-evaluate the execution plan.
Command
sed -i 's/~> 1.12.0/~> 1.9.0/' versions.tf
terraform plan
Command Explanation
sed -i 's/.../.../' versions.tf: Performs an in-place string replacement onversions.tf, changing the version constraint from~> 1.12.0to~> 1.9.0.
Output and Analysis
admin@i-05f2dc483fa4c779f:~/infra$ sed -i 's/~> 1.12.0/~> 1.9.0/' versions.tf
admin@i-05f2dc483fa4c779f:~/infra$ terraform plan
╷
│ Error: Cycle: local_file.platform_status, local_file.status_body
│
│
╵
Decision
Updating versions.tf bypassed the version gate. Terraform proceeded to dependency graph construction, immediately revealing the expected circular dependency error (Error: Cycle: local_file.platform_status, local_file.status_body). Hypothesis H2 is confirmed.
Step 4: Uncovering and Fixing the DAG Cycle & Provider Collision
Objective
Fix main.tf by assigning the required string "SadServers - Parma OK\n" to local_file.status_body and making local_file.platform_status depend unidirectionally on local_file.status_body.content.
Command (Attempt 1 - Introduces Provider Collision)
cat <<'EOF' > main.tf
provider "local" {}
resource "local_file" "status_body" {
filename = "/var/local/.platform-status-body"
content = "SadServers - Parma OK\n"
}
resource "local_file" "platform_status" {
filename = "/var/local/platform-status.txt"
content = local_file.status_body.content
file_permission = "0644"
}
EOF
terraform plan
Output and Analysis
admin@i-05f2dc483fa4c779f:~/infra$ terraform plan
╷
│ Error: Duplicate provider configuration
│
│ on versions.tf line 12:
│ 12: provider "local" {}
│
│ A default (non-aliased) provider configuration for "local" was already given at main.tf:1,1-17. If
│ multiple configurations are required, set the "alias" argument for alternative configurations.
╵
Decision
Because versions.tf line 12 already contained provider "local" {}, putting provider "local" {} in main.tf caused a duplicate default provider error across the module scope. Hypothesis H4 is confirmed.
We must remove provider "local" {} from main.tf.
Command (Attempt 2 - Corrected HCL Structure)
cat <<'EOF' > main.tf
resource "local_file" "status_body" {
filename = "/var/local/.platform-status-body"
content = "SadServers - Parma OK\n"
}
resource "local_file" "platform_status" {
filename = "/var/local/platform-status.txt"
content = local_file.status_body.content
file_permission = "0644"
}
EOF
Step 5: Final Plan, Infrastructure Execution, and Verification
Objective Generate a clean execution plan, apply state changes to create the target status file, and execute the verification check.
Command
terraform plan
terraform apply -auto-approve
/home/admin/agent/check.sh
Output and Analysis
admin@i-05f2dc483fa4c779f:~/infra$ terraform plan
Plan: 2 to add, 0 to change, 0 to destroy.
admin@i-05f2dc483fa4c779f:~/infra$ terraform apply -auto-approve
local_file.status_body: Creating...
local_file.status_body: Creation complete after 0s [id=0b033b7ccc6e91931409d6ecc6c83819e1e0b52c]
local_file.platform_status: Creating...
local_file.platform_status: Creation complete after 0s [id=0b033b7ccc6e91931409d6ecc6c83819e1e0b52c]
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
admin@i-05f2dc483fa4c779f:~/infra$ /home/admin/agent/check.sh
OK
Decision
terraform apply successfully created /var/local/.platform-status-body and /var/local/platform-status.txt. Executing /home/admin/agent/check.sh returned OK, confirming that the marker file content matches and terraform plan -detailed-exitcode returns 0.
5. Root Cause Analysis
The incident was caused by three cascading configuration errors introduced during an incomplete refactoring pass:
1. Version Requirement Incompatibility
The terraform block in versions.tf configured required_version = "~> 1.12.0". The pessimistic constraint operator (~>) locks allowed versions to >= 1.12.0 and < 1.13.0. Because the system binary installed at /usr/local/bin/terraform was 1.9.8, Terraform halted before compiling the execution graph.
2. Dependency Graph Cycle (DAG Loop)
Terraform builds a Directed Acyclic Graph (DAG) to determine the order in which resources are evaluated and applied. When Resource A references Resource B, an implicit graph edge is drawn from B to A ($B \rightarrow A$), meaning B must exist before A can be computed.
In the broken refactored main.tf:
local_file.status_bodyreferencedlocal_file.platform_status.content.local_file.platform_statusreferencedlocal_file.status_body.content.
This produced a closed graph cycle ($status_body \leftrightarrow platform_status$), making graph topological sorting mathematically impossible.
graph TD
subgraph Broken Cycle (DAG Loop)
A[local_file.status_body] -->|content = platform_status.content| B[local_file.platform_status]
B -->|content = status_body.content| A
end
subgraph Resolved Unidirectional DAG
C[String Literal: 'SadServers - Parma OK'] --> D[local_file.status_body]
D -->|content = status_body.content| E[local_file.platform_status]
end
3. HCL Module-Level Provider Collision
Terraform parses all .tf files in a single root module directory into a single combined evaluation graph. Declaring provider "local" {} in versions.tf while simultaneously declaring provider "local" {} in main.tf violates HCL module scoping rules, throwing a Duplicate provider configuration error.
6. Solution and Remediation
To completely resolve the issue, two files were modified in /home/admin/infra/.
Final File Configurations
1. versions.tf
terraform {
required_version = "~> 1.9.0"
required_providers {
local = {
source = "hashicorp/local"
version = "~> 2.5.0"
}
}
}
provider "local" {}
2. main.tf
resource "local_file" "status_body" {
filename = "/var/local/.platform-status-body"
content = "SadServers - Parma OK\n"
}
resource "local_file" "platform_status" {
filename = "/var/local/platform-status.txt"
content = local_file.status_body.content
file_permission = "0644"
}
Execution Steps Applied
cd /home/admin/infra
sed -i 's/~> 1.12.0/~> 1.9.0/' versions.tf
cat <<'EOF' > main.tf
resource "local_file" "status_body" {
filename = "/var/local/.platform-status-body"
content = "SadServers - Parma OK\n"
}
resource "local_file" "platform_status" {
filename = "/var/local/platform-status.txt"
content = local_file.status_body.content
file_permission = "0644"
}
EOF
terraform apply -auto-approve
7. Verification and Validation
Verification requires demonstrating both functional marker creation and state cleanliness.
1. File Marker and Permission Verification
cat /var/local/platform-status.txt
ls -l /var/local/platform-status.txt
Output:
SadServers - Parma OK
-rw-r--r-- 1 admin admin 22 Jul 2 23:59 /var/local/platform-status.txt
2. State Cleanliness Verification
terraform plan -detailed-exitcode
Output:
No changes. Your infrastructure matches the configuration.
(Exit code: 0)
3. Automated Test Suite Execution
/home/admin/agent/check.sh
Output:
OK
8. Alternative Troubleshooting Approaches
While direct execution of terraform plan identified the errors sequentially, alternative tools and techniques could have expedited the diagnosis.
1. Static Code Analysis via terraform validate
Before running terraform plan (which checks state files and provider schemas), terraform validate verifies whether the directory contains syntactically valid HCL code.
terraform validate
- Advantage: Catches duplicate provider blocks and basic structural errors without requiring local provider initialization or state checks.
2. Graph Visualization via terraform graph
When faced with complex dependency cycles, Terraform can generate a DOT-formatted graph representation.
terraform graph | dot -Tpng > graph.png
- Advantage: Renders a visual flowchart of resource relationships, highlighting cyclic loops immediately in large codebases.
3. HCL Linting via tflint
tflint is a third-party linter for Terraform.
tflint
- Advantage: Detects provider-specific errors, deprecated syntax, and version mismatches before running Terraform commands in deployment pipelines.
Diagnostic Tools Comparison
| Tool | Focus Area | Catches Version Mismatches? | Catches DAG Cycles? | Catches Duplicate Providers? |
|---|---|---|---|---|
terraform validate |
Syntax & Schema | No | Yes | Yes |
terraform plan |
State & Execution Diff | Yes | Yes | Yes |
terraform graph |
Structural Relationships | No | Yes | N/A |
tflint |
Best Practices & Rules | Yes | Partial | Yes |
9. Linux Command Deep Dive
Command: terraform
Purpose Terraform is an open-source Infrastructure as Code software tool created by HashiCorp. It provisions infrastructure defined in HCL files.
General Syntax
terraform [-version] [-help] <command> [args]
Options and Commands
| Option / Command | Purpose |
|---|---|
terraform init |
Initializes a working directory containing Terraform configuration files. Downloads providers. |
terraform plan |
Generates and shows an execution plan. |
terraform plan -detailed-exitcode |
Returns exit code 0 for no changes, 1 for error, 2 for pending changes. |
terraform apply -auto-approve |
Applies changes without prompting for manual interactive confirmation. |
terraform validate |
Validates configuration files in a directory without reading remote state. |
terraform fmt -check |
Checks whether input files are formatted according to canonical HCL style guidelines. |
Command: cat
Purpose
Concatenates files and prints their contents to standard output (stdout).
General Syntax
cat [OPTION]... [FILE]...
Options and Arguments
| Option | Purpose |
|---|---|
cat -n |
Number all output lines. |
cat -b |
Number non-blank output lines. |
cat -A |
Equivalent to -vET; displays non-printable characters, tabs (^I), and line endings ($). |
Command: sed
Purpose A stream editor used to perform basic text transformations on an input stream or file.
General Syntax
sed [OPTION]... 'script' [input-file]
Options and Arguments
| Option | Purpose |
|---|---|
sed -i |
Edit files in-place instead of printing to stdout. |
sed 's/regexp/replacement/' |
Substitutes the first match of regexp with replacement on each line. |
sed 's/regexp/replacement/g' |
Global substitution (replaces all matches on each line). |
10. Underlying Linux Concepts and Architecture
1. Terraform HCL Scoping Rules
In Terraform, directory boundaries define module boundaries. All files ending in .tf inside /home/admin/infra/ are evaluated together as one single configuration block.
- File Independence Myth: Naming a file
main.tf,versions.tf, oroutputs.tfis purely an organizational convention for human engineers. Terraform merges them into a single memory model. - Provider Uniqueness: A module can only have one default (un-aliased)
providerconfiguration per provider type. Multiple providers require distinctaliasattributes (e.g.,provider "aws" { alias = "us_west_2" }).
2. POSIX File Permissions and Atomic Writes
The local_file resource in main.tf explicitly configures file_permission = "0644".
-
0644translates to: -
Owner (
admin): Read (4) + Write (2) =6(rw-) -
Group (
admin): Read (4) =4(r--) -
Others: Read (
4) =4(r--) -
When the
localprovider writes files, it writes to a temporary file and atomically renames it over the target destination to prevent partial file reads by monitoring agents or status checks.
11. Real-World Sysadmin and SRE Applications
1. Continuous Integration (CI) Static Analysis
To prevent broken Terraform code from reaching version control or production environments, SRE teams implement automated linting in CI/CD pipelines (e.g., GitHub Actions, GitLab CI):
# Sample CI Step for Terraform Validation
steps:
- name: Check Formatting
run: terraform fmt -check -recursive
- name: Validate Syntax & DAG
run: terraform validate
- name: TFLint Check
run: tflint
2. Automated Drift Detection
The test script /home/admin/agent/check.sh uses terraform plan -detailed-exitcode. In production, SREs run scheduled cron jobs or Kubernetes CronJobs executing terraform plan -detailed-exitcode.
- Exit code 0: No drift; infrastructure perfectly matches code.
- Exit code 2: Drift detected; unapproved manual changes occurred in the environment. An alert is sent to the SRE on-call engineer.
12. Further Learning and Experiments
Hands-on Lab Experiments
Experiment 1: Intentionally Triggering a DAG Cycle
- Edit
main.tfand makestatus_body.contentpoint toplatform_status.content, andplatform_status.contentpoint tostatus_body.content. - Run
terraform plan. - Observe how Terraform’s graph engine identifies the circular loop.
Experiment 2: Testing Detailed Exit Codes
- Run
terraform plan -detailed-exitcodeon a clean directory and inspect the exit code (echo $?). - Manually edit
/var/local/platform-status.txtwithecho "drift" > /var/local/platform-status.txt. - Re-run
terraform plan -detailed-exitcodeand observe that the exit code changes to2.
Documentation and References
- Terraform Language Documentation
- Terraform
localProvider Registry Page - Linux Manual Pages:
man 1 sed,man 1 cat,man 1 terraform
13. Key Takeaways
- HCL Files Share a Directory Scope: Terraform merges all
.tffiles in a directory into one module. Never declare duplicate default provider blocks across separate.tffiles. - Break DAG Cycles with Static Values: Directed Acyclic Graphs require at least one root node containing a static literal value to break circular reference loops.
- Respect Core Version Constraints: Match
required_versioninversions.tfto the actual binary installed in your environment or pipeline runners. - Use
-detailed-exitcodefor Automation: Leverageterraform plan -detailed-exitcodein scripts to programmatically determine whether infrastructure drift exists.
14. Quick Troubleshooting Reference / Runbook
When debugging broken Terraform modules, follow this sequence:
[Start Debugging]
│
▼
1. Check Installed Terraform Version ────► terraform version
│
▼
2. Validate Syntax & Scoping ───────────► terraform validate
│
▼
3. Check for Dependency Cycles ──────────► terraform plan
│
▼
4. Format Configuration Files ───────────► terraform fmt
│
▼
5. Apply State Non-Interactively ────────► terraform apply -auto-approve
│
▼
6. Verify Drift Status ──────────────────► terraform plan -detailed-exitcode
Diagnostic Command Cheat Sheet
# 1. Verify installed core version
terraform version
# 2. Check HCL syntax without reading state
terraform validate
# 3. Locate duplicate providers or cyclic references
terraform plan
# 4. Check for unapplied infrastructure drift (returns code 0 or 2)
terraform plan -detailed-exitcode
# 5. Fix formatting automatically across all files
terraform fmt