1. Introduction
One of the most classic operational puzzles in Linux system administration occurs when automated monitoring alerts report that a filesystem is nearing full capacity, yet directory analysis tools fail to locate the offending files. An SRE running df (Disk Free) sees high disk utilization, while running du (Disk Usage) on the same filesystem reports significantly less space consumed.
This phenomenon—often referred to as “ghost disk space”—is not a filesystem bug or a command calculation error. It is the direct result of how POSIX-compliant filesystems separate directory listings from kernel resource management. When an active process holds a file handle open while the file’s directory path is deleted, the underlying storage blocks remain allocated to the process even though the file is no longer visible in the directory tree.
In this technical reference guide, we will reconstruct a real-world troubleshooting session based on the SadServers “Cordoba” scenario. We will explore how to identify unlinked open files, analyze systemd log hoarder processes, manipulate process file descriptors via /proc, and implement long-term preventive mechanisms in production environments.
2. Problem Description and Environment
Scenario Overview
The scenario, titled “Cordoba: df is lying (or is it du?)”, presents a medium-difficulty Linux troubleshooting task centered around disk volume pressure on a host running systemd.
- Scenario Name: Cordoba: df is lying (or is it du?)
- Level: Medium
- Type: Fix
- Tags:
bash,realistic-interviews,disk volumes - Access Level: Root (
sudo) privileges available
Reported Symptoms & Constraints
- Monitoring Alert: Automated alerts report that the root filesystem (
/) is experiencing storage pressure. - Initial Observation: Running
duagainst the application log directory at/var/log/cordoba-appshows almost no log volume. - Discrepancy: There is a ~300 MB to ~400 MiB discrepancy between space reported by
dfand space accounted for byduon the root partition. - Target Service: The background unit managing log operations is identified as
cordoba-hoarder.service. - Validation Criteria: The evaluation script
/home/admin/agent/check.shchecks ifdf -h /andsudo du -sh /agree after reclaiming the missing space.
Environment Specifications
- Operating System: Linux (Debian 13 cloud image, kernel
6.12.41+deb13-cloud-amd64). - Root Storage Volume:
/dev/nvme0n1p1mounted on/(7.7 GB total capacity).
3. Initial Investigation and Hypotheses
When facing a mismatch between df and du, a structured diagnostic approach prevents wasting time on random file deletions.
Understanding the Diagnostic Tools
df(Disk Free): Queries the filesystem superblock and block allocation bitmaps directly. It measures total allocated disk blocks regardless of whether those blocks are attached to visible file paths.du(Disk Usage): Traverses the directory tree hierarchically starting from a target directory, issuingstat()system calls on visible directory entries (dentries) and summing up file sizes.
Hypothesis Matrix
| Hypothesis | Supporting Evidence | Evidence Against | Investigation Method |
|---|---|---|---|
| H1: Open Deleted File (Unlinked Inode) | df reports 5.1 GB used, du reports 4.7 GB used (~400 MiB discrepancy); cordoba-hoarder.service manages log files. |
None. This is the primary cause of df/du divergence in active log directories. |
Audit open process file handles using sudo lsof +L1 /. |
| H2: Files Hidden Behind a Mount Point | Disk space used on / prior to mounting another volume over a non-empty directory. |
The system uses a simple single root partition (/dev/nvme0n1p1) without nested sub-mounts. |
Inspect /proc/mounts and findmnt. |
| H3: Filesystem Metadata / Superblock Corruption | Incorrect allocation bitmap counts in kernel memory. | No kernel panic or I/O errors reported in dmesg. |
Inspect dmesg or run fsck (read-only check). |
4. Troubleshooting Process
The following steps reconstruct the chronological investigation and remediation performed during the troubleshooting session.
┌────────────────────────────────┐
│ Step 1: Quantify Mismatch │ df -h / (5.1G) vs du -sh / (4.7G)
└───────────────┬────────────────┘
│
▼
┌────────────────────────────────┐
│ Step 2: Locate Unlinked Handles│ sudo lsof +L1 / -> PID 756 (419 MB)
└───────────────┬────────────────┘
│
▼
┌────────────────────────────────┐
│ Step 3: Analyze Service Logic │ cat /usr/local/sbin/cordoba-hoarder.sh
└───────────────┬────────────────┘
│
▼
┌────────────────────────────────┐
│ Step 4: Inspect Validation │ cat /home/admin/agent/check.sh
└───────────────┬────────────────┘
│
▼
┌────────────────────────────────┐
│ Step 5: Apply Fix & Verify │ systemctl stop & disable cordoba-hoarder
└────────────────────────────────┘
Step 1: Quantifying the Storage Discrepancy
Objective: Measure the exact difference between block-level filesystem utilization (df) and directory tree utilization (du) on the root filesystem /.
Commands Executed:
df -h /
sudo du -sh /
Output & Analysis:
admin@i-0bc859b0f966573b0:~$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 7.7G 5.1G 2.2G 70% /
admin@i-0bc859b0f966573b0:~$ sudo du -sh /
du: cannot access '/proc/1153/task/1153/fd/4': No such file or directory
du: cannot access '/proc/1153/task/1153/fdinfo/4': No such file or directory
4.7G /
df -h /Analysis: Superblock metadata shows 5.1 GB used out of 7.7 GB.sudo du -sh /Analysis: Traversing the filesystem hierarchy accounts for 4.7 GB used.- Conclusion: There is an unaccounted gap of approximately 400 MiB (~0.4 GB) between what the filesystem has allocated and what directory dentries reference.
Step 2: Inspecting Unlinked Open File Handles with lsof
Objective: Identify active processes holding open file descriptors to files that have zero hard links (nlink == 0) on the root partition /.
Command Executed:
sudo lsof +L1 /
Command Option Breakdown:
sudo: Executeslsofwith root privileges to inspect file descriptors across all system users and daemons.lsof: (List Open Files) Scans process file descriptor tables in/proc.+L1: Restricts output to files with a hard link count of less than 1 (i.e.,0links), pinpointing deleted files that remain open./: Limits the search to file descriptors pointing to inodes on the root filesystem mount point.
Output & Analysis (Excerpt):
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
cordoba-h 756 root 9w REG 259,1 419430512 0 281405 /var/log/cordoba-app/access.log (deleted)
- Process Name & PID:
cordoba-hrunning under PID 756 asroot. - File Descriptor:
9w(File descriptor9opened in write/append mode). - File Size:
419430512bytes (~419.4 MB / 400 MiB). - Link Count (
NLINK):0links remaining. - File Name & Path:
/var/log/cordoba-app/access.log (deleted). - Decision: PID
756is positively identified as the source of the 400 MiB ghost leak.
Step 3: Analyzing the Service Script Logic
Objective: Inspect the service implementation to understand how the log handle was created and deleted.
Command Executed:
cat /usr/local/sbin/cordoba-hoarder.sh
Output & Analysis:
#!/bin/bash
# Simulates an app whose log was removed while the process kept the inode open.
set -euo pipefail
LEAK_MB="${CORDOBA_LEAK_MB:-400}"
LOGDIR=/var/log/cordoba-app
LOGFILE="${LOGDIR}/access.log"
mkdir -p "$LOGDIR"
install -m644 /dev/null "$LOGFILE"
if ! fallocate -l "${LEAK_MB}M" "$LOGFILE" 2>/dev/null; then
dd if=/dev/zero of="$LOGFILE" bs=1M count="${LEAK_MB}" status=none
fi
exec 9>>"$LOGFILE"
rm -f "$LOGFILE"
while true; do
printf '%s GET /api/v1/widgets HTTP/1.1 200 0\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" >&9
# Do not let sleep inherit FD 9 (would show a second writer in lsof +L1).
sleep 45 9<&-
done
Script Mechanics:
fallocate -l 400M "$LOGFILE": Allocates 400 MiB of disk blocks to/var/log/cordoba-app/access.log.exec 9>>"$LOGFILE": Binds Bash File Descriptor9to the log file in append mode (>>) for the shell process.rm -f "$LOGFILE": Unlinks the file from/var/log/cordoba-app/access.log.printf ... >&9: Continues appending log lines directly to FD9.sleep 45 9<&-: Closes FD9specifically for childsleepprocesses to preventsleepfrom inheriting the handle.
Key Takeaway: Simple service restarts (systemctl restart cordoba-hoarder) kill the existing PID (releasing the space temporarily), but the new process immediately re-executes fallocate and rm, re-creating the ghost leak.
Step 4: Auditing Verification Logic
Objective: Read /home/admin/agent/check.sh to ensure the planned fix satisfies the scenario check criteria.
Command Executed:
cat /home/admin/agent/check.sh
Output & Analysis:
#!/bin/bash
# DO NOT MODIFY THIS FILE ("Check My Solution" will fail)
BASELINE_FILE=/var/lib/cordoba-baseline-kb
MIN_RECLAIM_KB=307200
baseline=$(tr -d ' \n\r' <"$BASELINE_FILE" 2>/dev/null)
used=$(df -k / | awk 'NR==2 {print $3}')
if [ -z "$baseline" ] || [ -z "$used" ]; then
echo -n "NO"
exit 0
fi
reclaimed=$((baseline - used))
if [ "$reclaimed" -lt "$MIN_RECLAIM_KB" ]; then
echo -n "NO"
exit 0
fi
echo -n "OK"
exit 0
- Requirement: Current root usage (
df -k /) must be at least 307,200 KiB (300 MiB) lower than the initial baseline.
Step 5: Applying Remediation and Disabling the Service
Objective: Terminate the offending process to release the file handle and disable the service to prevent re-allocation.
Commands Executed:
sudo systemctl stop cordoba-hoarder.service
sudo systemctl disable cordoba-hoarder.service
Output:
Removed '/etc/systemd/system/multi-user.target.wants/cordoba-hoarder.service'.
5. Root Cause Analysis
POSIX File Deletion Architecture
In Linux filesystems (ext4, xfs, btrfs), storage management relies on three distinct structural layers:
┌─────────────────────────────────────────────────────────────┐
│ Directory Structure │
│ Path: /var/log/cordoba-app/access.log (Dentry) │
└──────────────────────────────┬──────────────────────────────┘
│ (Removed by `rm`)
▼
┌─────────────────────────────────────────────────────────────┐
│ Filesystem Metadata │
│ Inode #281405 │
│ ├── Size: 419,430,512 bytes │
│ ├── Link Count (nlink): 0 <-- Decremented by `rm` │
│ └── Data Block Pointers -> [Block 1024, 1025, ...] │
└──────────────┬──────────────────────────────┬───────────────┘
│ │
│ (Referenced by) │ (Allocated in)
▼ ▼
┌─────────────────────────────┐ ┌────────────────────────────┐
│ Process Table │ │ Superblock Bitmaps │
│ PID: 756 (cordoba-h) │ │ Storage Blocks: 419 MB │
│ FD Table: │ │ Status: MARKED AS USED │
│ FD 9 -> Inode #281405 │ │ (Queried by `df`) │
└─────────────────────────────┘ └────────────────────────────┘
- Directory Entries (Dentries): Map human-readable file paths to numerical inode identifiers.
- Inodes: Hold metadata (permissions, owner, block extents, file size) and a hard link counter (
nlink). - Data Blocks: The physical block extents storing file contents.
The Kernel Reference Counting Mechanism
The kernel block allocator evaluates two conditions before releasing storage blocks back to the free pool:
$$\text{Release Storage Blocks} \iff (\text{Hard Link Count } nlink == 0) ;\land; (\text{Open Descriptor Count} == 0)$$
- When
rmwas executed, the dentry was deleted andnlinkwas decremented from1to0. ducalculates space by traversing active dentries; because the dentry was missing,duignored inode#281405.- PID
756maintained an open file descriptor (FD 9) to inode#281405. dfreads allocation bitmaps directly from the superblock, correctly reporting the 419 MB as allocated space until PID756was terminated.
6. Solution and Remediation
Primary Solution: Stopping and Disabling the Unit
Stopping cordoba-hoarder.service terminates PID 756. Upon process exit, the kernel automatically closes all open file descriptors in the process file table, bringing the open handle count for inode #281405 to zero and instantly freeing the 400 MiB of disk space.
sudo systemctl stop cordoba-hoarder.service
sudo systemctl disable cordoba-hoarder.service
Alternative Remediation: Live Truncation via /proc (Zero-Downtime Fix)
In a production scenario where restarting an application daemon is unacceptable due to strict availability SLAs, disk space can be reclaimed live without stopping the process by truncating the inode size to 0 bytes via /proc:
# Method 1: Using truncate
sudo truncate -s 0 /proc/756/fd/9
# Method 2: Shell redirection
sudo bash -c '> /proc/756/fd/9'
- How it works: Accesses the active file descriptor directly through the
/procvirtual filesystem interface and resets the inode payload length to zero. - Risk Consideration: If the application process maintains its write offset internally in memory without seeking to the end of the file, subsequent writes may create a sparse file, leaving virtual offset holes while continuing to write.
7. Verification and Validation
Step 1: Re-assessing Disk Usage Metrics
Command Executed:
df -h / && sudo du -sh /
Output:
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 7.7G 4.7G 2.6G 65% /
du: cannot access '/proc/1261/task/1261/fd/4': No such file or directory
4.7G /
- Result:
dfusage dropped from 5.1 GB to 4.7 GB, matchingduat 4.7 GB. Available space increased from 2.2 GB to 2.6 GB.
Step 2: Executing Validation Script
Command Executed:
/home/admin/agent/check.sh
Output:
OK
8. Alternative Troubleshooting Approaches
When investigating disk space discrepancies, sysadmins can leverage multiple diagnostic pathways depending on available utilities.
Diagnostic Matrix
| Utility | Command Line Example | Advantages | Limitations | Recommended Use Case |
|---|---|---|---|---|
lsof |
sudo lsof +L1 / |
Directly lists unlinked open files with file size and process details. | May be slow on systems with thousands of open sockets/files. | Primary tool for quick df/du discrepancy checks. |
/proc Virtual FS |
`sudo find /proc/[0-9]*/fd -ls 2>/dev/null | grep ‘(deleted)’` | Works on minimal installations where lsof is not installed. |
Requires manual parsing of symlinks. |
fuser |
sudo fuser -v -m / |
Quickly identifies all PIDs accessing a mount point. | Does not explicitly isolate unlinked files or file sizes. | Unmounting busy filesystems (device is busy). |
bpftrace / eBPF |
sudo bpftrace -e 'kprobe:vfs_unlink { printf("%s deleted file\n", comm); }' |
Real-time tracing of deletion events as they happen. | High learning curve; requires kernel header packages. | Catching intermittent log rotators unlinking files in real time. |
9. Linux Command Deep Dive
Command: lsof
Purpose: Lists open files, network sockets, unix domain sockets, pipes, and memory-mapped files across all active system processes.
General Syntax:
lsof [options] [path]
Useful Troubleshooting Options:
| Flag / Option | Description | Practical Use Case |
|---|---|---|
+L1 |
Displays open files with hard link count < 1 (unlinked/deleted files). |
Finding ghost disk leaks (lsof +L1 /). |
-i :<port> |
Displays processes bound to or connected via a specific network port. | Identifying what service is holding port 80 or 443 (lsof -i :80). |
-u <user> |
Filters open files owned by a specific Linux username. | Auditing user activity (lsof -u www-data). |
-p <PID> |
Lists all open file descriptors held by a specific Process ID. | Inspecting process descriptors (lsof -p 756). |
+D <dir> |
Recursively scans a directory hierarchy for open file handles. | Troubleshooting umount: target is busy (lsof +D /var/log). |
Command: fallocate
Purpose: Pre-allocates block storage space to a file directly via filesystem extent metadata without writing zero-bytes to disk.
General Syntax:
fallocate -l <size>[K|M|G|T] <filename>
Comparison of Space Allocation Utilities:
fallocate -l 400M log.txt ---> Modifies extent map instantly (O(1) operation)
dd if=/dev/zero of=log.txt ---> Writes 400 MB of actual zero-bytes to storage (I/O heavy)
fallocate: Fast $O(1)$ filesystem metadata manipulation. Supported on ext4, xfs, btrfs.dd: Physical block I/O streamer. Slower, generates heavy disk write load.
Command: systemctl
Purpose: Controls the systemd system and service manager.
Useful Options for Incident Remediation:
# Check detailed unit state and recent journal entries
systemctl status cordoba-hoarder.service
# Terminate service process group immediately
sudo systemctl stop cordoba-hoarder.service
# Disable unit from starting automatically at boot
sudo systemctl disable cordoba-hoarder.service
# Restart unit while reloading systemd manager configuration
sudo systemctl daemon-reload && sudo systemctl restart cordoba-hoarder.service
10. Underlying Linux Concepts and Architecture
Process File Descriptor Tables & /proc/<PID>/fd/
In Linux, every process maintains a File Descriptor (FD) table within its kernel task structure (task_struct).
/proc/756/fd/
├── 0 -> /dev/null (stdin)
├── 1 -> /dev/null (stdout)
├── 2 -> /dev/null (stderr)
└── 9 -> /var/log/cordoba-app/access.log (deleted)
The /proc/<PID>/fd/ directory contains virtual symbolic links representing every open handle:
- Even after a file is unlinked from its parent directory,
/proc/<PID>/fd/<FD>remains a valid file handle. - Reading from or writing to
/proc/756/fd/9interacts directly with the underlying unlinked inode data blocks.
Bash Redirection & Descriptor Mechanics
The service script contained advanced Bash redirection syntax:
# Binds File Descriptor 9 in append mode to LOGFILE
exec 9>>"$LOGFILE"
# Redirects printf stdout directly into File Descriptor 9
printf '%s ...\n' "..." >&9
# Closes File Descriptor 9 for the child process 'sleep'
sleep 45 9<&-
exec 9>>"$LOGFILE"creates a persistent descriptor in the parent shell.9<&-prevents child process inheritance. If9<&-were omitted,sleepwould inherit FD9, causinglsofto report multiple process writers to the unlinked log file.
11. Real-World Sysadmin and SRE Applications
Production Monitoring & Alerting Strategies
To detect open deleted file leaks before disk capacity is exhausted:
- Prometheus
node_exporterAlert Rules: Monitor the difference betweennode_filesystem_size_bytesand sum of directory sizes, or alert on low available space:
expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
for: 5m
labels:
severity: critical
annotations:
summary: "Low disk space on root volume (potential ghost leak)"
- Automated
lsofHealth Checks: Integrate periodic checks into monitoring scripts to surface unlinked descriptors larger than 100 MB:
sudo lsof +L1 / | awk '$7 > 104857600 { print $0 }'
Safe Log Rotation Best Practices
Improper log deletion is the single most common cause of open deleted file leaks in production.
Incorrect Pattern (Causes Leaks):
# DO NOT DO THIS in cron or scripts:
rm -f /var/log/app/access.log
- Why it fails: Running applications retain open handles to the deleted file. Storage blocks are never freed.
Correct Pattern A: Signal Re-opening (logrotate)
/var/log/app/*.log {
daily
rotate 7
missingok
postrotate
/usr/bin/systemctl reload app.service > /dev/null 2>&1
endscript
}
- Mechanism: Instructs the application via signal (
SIGHUPorsystemctl reload) to close its existing file descriptor and open a new log file path.
Correct Pattern B: In-Place Truncation (copytruncate)
/var/log/app/*.log {
daily
rotate 7
copytruncate
}
- Mechanism: Copies active log content to a backup file and truncates the active log file to 0 bytes in-place (
> access.log). The application never loses its file handle.
12. Further Learning and Experiments
Hands-on Lab Experiment: Simulating and Resolving a Ghost Leak
Objective
Recreate an open deleted file memory leak in a safe test environment and resolve it live via /proc without stopping the application process.
Step 1: Create a Test Script (leak_test.sh)
#!/bin/bash
TESTFILE="/tmp/ghost_leak.test"
# 1. Allocate 100MB test file
fallocate -l 100M "$TESTFILE"
# 2. Open FD 3 in append mode
exec 3>>"$TESTFILE"
# 3. Unlink the file
rm -f "$TESTFILE"
echo "Process PID: $$ is holding unlinked file handle on FD 3"
while true; do
echo "$(date) - writing..." >&3
sleep 5
done
Step 2: Run the Script in Background
chmod +x leak_test.sh
./leak_test.sh &
# Output note PID (e.g., PID 4321)
Step 3: Observe Discrepancy & Identify Descriptor
# Check lsof output for deleted handle
sudo lsof +L1 /tmp
# Verify file size via /proc
ls -lh /proc/$(pgrep -f leak_test.sh)/fd/3
Step 4: Truncate Handle Live Without Killing Process
PID=$(pgrep -f leak_test.sh)
sudo truncate -s 0 /proc/$PID/fd/3
Step 5: Clean Up
kill $(pgrep -f leak_test.sh)
13. Key Takeaways
- Root Cause Identification: Mismatches between
dfandduindicate that active processes hold open file handles to deleted directory paths (nlink == 0). - Primary Diagnostic Tool:
sudo lsof +L1 <mountpoint>instantly isolates PIDs, file descriptor numbers, and leaked byte sizes. - Zero-Downtime Fix: Disk space can be reclaimed immediately without stopping services by executing
truncate -s 0 /proc/<PID>/fd/<FD>. - Permanent Remediation: Terminating or restarting the offending process closes open descriptors, triggering kernel block release.
- Operational Prevention: Configure application loggers to catch
SIGHUPor uselogrotatewithcopytruncateto prevent unlinked descriptor leaks in production.
14. Quick Troubleshooting Reference / Runbook
┌────────────────────────────────────────────────────────────────────────┐
│ GHOST DISK SPACE TROUBLESHOOTING RUNBOOK │
└────────────────────────────────────────────────────────────────────────┘
1. CONFIRM DISCREPANCY
$ df -h /
$ sudo du -sh /
=> If df used space >> du used space, proceed to Step 2.
2. LOCATE UNLINKED OPEN FILE HANDLES
$ sudo lsof +L1 /
=> Note the PID, File Descriptor (FD), and File Size.
3. ALTERNATIVE CHECK (IF LSOF IS MISSING)
$ sudo find /proc/[0-9]*/fd -ls 2>/dev/null | grep '(deleted)'
4. EVALUATE SERVICE IMPACT
$ ps aux | grep <PID>
$ systemctl status <SERVICE_NAME>
5. APPLY REMEDIATION
Option A: Zero-Downtime Truncation (Production Preferred)
$ sudo truncate -s 0 /proc/<PID>/fd/<FD>
Option B: Service Restart / Stop
$ sudo systemctl restart <SERVICE_NAME>
$ sudo systemctl disable <SERVICE_NAME>
6. VERIFY RECOVERY
$ df -h /
$ sudo du -sh /
=> Confirm df and du report matching used space.