1. Introduction
In Linux system administration, open file handles are at the center of many subtle operational incidents. You might encounter a filesystem that refuses to unmount with target is busy, a deleted log file that continues consuming gigabytes of disk space, or a file locked for writing by an active process that cannot be restarted without causing downtime.
A common reflex is to terminate the process holding the file open (kill <PID>). When a process exits, the Linux kernel automatically closes all of its open file descriptors. However, in production environments—or under strict operational constraints—killing a critical process is unacceptable. Instead, an administrator must determine which process holds the file descriptor, how that process relates to the current execution context, and how to close that specific descriptor safely in place.
This article walks through a complete troubleshooting exercise based on the SadServers “Oaxaca: Close an Open File” challenge. We will examine how to trace an open file back to its owning process, inspect kernel file descriptor tables via /proc, close an active file descriptor inside a running shell without terminating it, and understand alternative techniques (such as debugger injection) when the target process is external.
2. Problem Description and Environment
Original Challenge Specification
- Scenario: “Oaxaca”: Close an Open File
- Difficulty / Type: Medium / Fix
- Tags:
bash,unusual-tricky - Problem Statement: The file
/home/admin/somefileis open for writing by some process. Close this file without killing the process. - Privilege Constraint: Root (
sudo) access is not available (False). - Success Criterion: Running
lsof /home/admin/somefilemust return no output, while the owning process remains alive.
Expected vs. Actual Behaviour
- Expected Behaviour: No running process holds an open file descriptor referencing
/home/admin/somefile(lsof /home/admin/somefileexits with empty output), and the target process continues running uninterrupted. - Actual Behaviour: At least one active process holds
/home/admin/somefileopen in write mode, causinglsof /home/admin/somefileto list an active descriptor.
Environment Details
From the terminal session and diagnostic outputs, the following environmental facts were observed:
- User Account:
admin(unprivileged user; nosudoaccess) - Hostname:
ip-10-1-13-98 - Terminal / TTY:
pts/0 - Storage Device: Major/minor device
259,1(characteristic of an NVMe block device partition) - OS Distribution & Kernel Version: Not captured during the session. Because no
uname -aor/etc/os-releasechecks were required to solve the issue, the exact Linux distribution and kernel release remain unconfirmed, though standard GNU/Linux utilities (bash,lsof,ps,/proc) were present.
Why the Constraints Matter
Two constraints shape the entire investigation strategy:
- No
sudoAccess: On Linux, unprivileged users cannot inspect/proc/<PID>/fdor attach debuggers to processes owned by other users. Therefore, the process holding/home/admin/somefileopen must be running under theadminuser (or an accessible sub-context ofadmin). - Do Not Kill the Process: Sending
SIGTERMorSIGKILLto the PID would satisfylsofreturning nothing, but would violate the requirement to keep the process running. We must close the individual file descriptor inside the process’s kernel descriptor table.
3. Initial Investigation and Hypotheses
Before running any state-changing commands, we establish what is known, what is missing, and what plausible mechanisms could explain an open file descriptor in an unprivileged environment.
Known vs. Unknown Information
- Known: The exact filesystem path (
/home/admin/somefile), the access mode (open for writing), the user context (admin), and the validation test (lsof /home/admin/somefile). - Unknown:
- Which executable (
COMMAND) and Process ID (PID) owns the open file handle. - Which file descriptor number (
FD) is assigned to/home/admin/somefile. - How the owning process relates to our interactive terminal session (e.g., whether it is our current
bashshell, a parent shell, a background job, or a separate daemon).
Candidate Hypotheses
| Hypothesis | Plausibility | Supporting Clues | What Would Confirm or Refute It |
|---|---|---|---|
H1: The current interactive bash shell holds the file open. |
High | Challenge is tagged bash and unusual-tricky; shell initialization files (e.g., .bashrc) can open persistent FDs via exec <FD>>file. |
lsof reports bash as the command, and its PID matches our current shell’s $$. |
H2: A background job or subshell spawned under admin holds the file open. |
Medium | Background scripts (script.sh &) inherit or open descriptors and persist after starting. |
lsof reports a PID different from $$, with our current shell (or an init/session manager) as its parent. |
H3: A separate user-level service or daemon owned by admin holds the file open. |
Medium | User services or long-running processes often keep log or state files open for writing. | lsof reports a non-shell binary (or detached bash process) with a different PID and TTY. |
4. Troubleshooting Process
Step 1: Identify the Process and File Descriptor Holding the File Open
Objective
Map the filesystem path /home/admin/somefile to the exact process ID (PID), owning user (USER), and file descriptor number (FD) using a read-only diagnostic command.
Command
lsof /home/admin/somefile
Command Explanation
lsof(List Open Files) queries the kernel’s open file tables via/procand translates open file descriptors back into human-readable paths.- Passing
/home/admin/somefileas a positional argument instructslsofto filter all accessible open descriptors across the system and display only those referencing that specific file’s inode. - This command is completely read-only and matches the challenge’s verification test.
Output and Analysis
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash 813 admin 77w REG 259,1 0 272875 /home/admin/somefile
Key fields in this output:
COMMAND(bash) &PID(813): Abashprocess with PID813holds the file open.USER(admin): The process belongs to our own user account, confirming why root privileges are unnecessary.FD(77w): The descriptor number is77, and the trailingwindicates it is open in write-only mode (O_WRONLY).TYPE(REG): It is a regular file (as opposed to a directoryDIR, pipeFIFO, or socket).DEVICE(259,1),SIZE/OFF(0),NODE(272875): The file resides on block device259,1, is currently0bytes in size, and uses inode272875.
This output immediately narrows the problem down to bash PID 813 and file descriptor 77, ruling out non-shell binaries (Hypothesis H3). However, we still do not know whether PID 813 is our current interactive shell (Hypothesis H1) or a separate bash process (Hypothesis H2).
Decision
Compare PID 813 against our current shell’s PID ($$) and inspect PID 813’s process metadata using ps.
Step 2: Determine the Relationship Between PID 813 and the Current Shell
Objective
Check whether PID 813 is the interactive bash session we are currently typing into or a separate bash process, and inspect its parent PID (PPID), controlling terminal (TTY), and full command string (CMD).
Command
echo $$ && ps -fp 813
Command Explanation
$$is a specialbashparameter that expands to the Process ID of the current shell.&&executes the second command only ifecho $$succeeds.ps -fp 813queries process status for PID813:-fselects full-format listing, exposingUID,PID,PPID,C,STIME,TTY,TIME, andCMD.-p 813filters strictly by Process ID813.
Output and Analysis
813
UID PID PPID C STIME TTY TIME CMD
admin 813 584 0 15:29 pts/0 00:00:00 bash
Key findings from this output:
echo $$printed813, proving that PID813is our current interactive shell (confirming Hypothesis H1 and ruling out Hypothesis H2).PPIDis584(the parent terminal/session process),TTYispts/0, andCMDisbashwith no script arguments.
Because FD 77 exists inside the private file descriptor table of our own bash process, we do not need external process injection tools like gdb. We can instruct bash itself to close FD 77 using a built-in shell redirection.
Decision
Use the bash built-in exec redirection syntax (exec 77>&-) to close file descriptor 77 in the current shell process.
Step 3: Close File Descriptor 77 Inside the Current Shell
Objective
Close file descriptor 77 in PID 813 without spawning a subshell, deleting /home/admin/somefile, or terminating bash.
Command
exec 77>&-
Command Explanation
- When the
execshell built-in is invoked without a command argument,bashapplies any specified redirections permanently to the current shell process itself rather than replacing the process image. 77>&-is the POSIX shell redirection operator that closes file descriptor77(triggering theclose(77)system call in the kernel).- Standard descriptors
0(stdin),1(stdout), and2(stderr) remain untouched, so our interactive terminal session continues working normally.
Output and Analysis
The command completed silently with exit status 0 and returned directly to the admin@ip-10-1-13-98:/$ prompt. No error message was emitted, and the shell remained responsive.
Decision
Inspect /proc/$$/fd directly and re-run lsof /home/admin/somefile and echo $$ to confirm that FD 77 is gone while PID 813 remains intact.
5. Root Cause Analysis
What Happened and Why
During session initialization (before presenting the interactive prompt—such as via a login profile or environment setup script), the bash shell opened /home/admin/somefile on file descriptor 77 for writing (equivalent to running exec 77>/home/admin/somefile).
In Linux, open file descriptors are maintained per process in kernel memory and remain active until one of two events occurs:
- The owning process exits (at which point the kernel closes all open descriptors during process teardown).
- The process explicitly invokes the
close()system call on that specific file descriptor number.
Because file descriptor 77 is outside the standard I/O channels (0, 1, 2), it sat open silently in the background of PID 813 without interfering with interactive keyboard input or terminal output.
Kernel File Descriptor Architecture (Before vs. After)
BEFORE: exec 77>&-
+-----------------------------------+ +-------------------------------+ +------------------------+
| Per-Process FD Table (PID 813) | | System-Wide Open File Table | | Inode Table |
|-----------------------------------| |-------------------------------| |------------------------|
| FD 0 -------------------------->|------>| /dev/pts/0 (O_RDWR) | | |
| FD 1 -------------------------->|------>| /dev/pts/0 (O_RDWR) | | |
| FD 2 -------------------------->|------>| /dev/pts/0 (O_RDWR) | | |
| FD 77 -------------------------->|------>| File Offset: 0, Mode: O_WRONLY|------>| Inode 272875 (somefile)|
| FD 255 -------------------------->|------>| /dev/pts/0 (O_RDWR) | | |
+-----------------------------------+ +-------------------------------+ +------------------------+
AFTER: exec 77>&- (triggers close(77) syscall)
+-----------------------------------+ +-------------------------------+ +------------------------+
| Per-Process FD Table (PID 813) | | System-Wide Open File Table | | Inode Table |
|-----------------------------------| |-------------------------------| |------------------------|
| FD 0 -------------------------->|------>| /dev/pts/0 (O_RDWR) | | |
| FD 1 -------------------------->|------>| /dev/pts/0 (O_RDWR) | | |
| FD 2 -------------------------->|------>| /dev/pts/0 (O_RDWR) | | |
| FD 77 [CLOSED / UNALLOCATED] | | [Entry Freed - Ref Count = 0] | | Inode 272875 (0 opens) |
| FD 255 -------------------------->|------>| /dev/pts/0 (O_RDWR) | | |
+-----------------------------------+ +-------------------------------+ +------------------------+
6. Solution and Remediation
Applied Fix
To release the open file handle inside PID 813 without terminating the shell, we executed:
exec 77>&-
- What Changed: File descriptor
77was removed from PID813’s kernel file descriptor table viaclose(77). - Configuration Files Modified: None. Because this was an in-memory runtime file descriptor state rather than a persistent configuration file, no files on disk were edited or deleted.
- Service Restarts / Reboots: None required (and explicitly avoided, as restarting the shell would have terminated PID
813). - Side Effects & Reversibility: Zero negative side effects. Standard terminal descriptors remained intact. If FD
77ever needed to be reopened to the same file, it could be reversed immediately withexec 77>/home/admin/somefile.
7. Verification and Validation
After executing exec 77>&-, three checks were performed to confirm both that the file descriptor was closed and that the original process survived intact.
1. Direct Kernel Descriptor Inspection (/proc)
ls -l /proc/$$/fd
Actual Output:
total 0
lrwx------ 1 admin admin 64 Nov 12 15:29 0 -> /dev/pts/0
lrwx------ 1 admin admin 64 Nov 12 15:29 1 -> /dev/pts/0
lrwx------ 1 admin admin 64 Nov 12 15:29 2 -> /dev/pts/0
lrwx------ 1 admin admin 64 Nov 12 15:29 255 -> /dev/pts/0
Why It Matters: /proc/813/fd exposes the kernel’s exact per-process descriptor table. Only 0 (stdin), 1 (stdout), 2 (stderr), and 255 (the descriptor bash internally duplicates from the controlling terminal for job control) remain. Descriptor 77 is completely absent.
2. Challenge Validation Test (lsof)
lsof /home/admin/somefile
Actual Output: (Empty — no lines returned)
Why It Matters: Confirms that no process anywhere on the system holds an open descriptor pointing to /home/admin/somefile’s inode.
3. Process Survival Check ($$)
echo $$
Actual Output:
813
Why It Matters: Proves that PID 813 is still the exact same running process and was not killed, restarted, or replaced via exec <command>.
8. Alternative Troubleshooting Approaches
The commands below were not executed during the original session, but represent essential alternative techniques that an experienced sysadmin or SRE would use depending on available tools and whether the target process is an external binary.
1. Discovering Open Files with fuser (Alternative to lsof)
fuser identifies processes using specified files, directories, or sockets and is often faster and more lightweight than lsof.
fuser -v /home/admin/somefile
- Information Provided: Prints the
USER,PID,ACCESStype (ffor open file,Ffor file open for writing,cfor current directory), andCOMMAND. - Advantages: Fast execution; part of the standard
psmiscpackage; even supports-kto signal matching processes (though-kwould violate our constraint not to kill the process). - Limitations: Provides less granular detail than
lsof(for example, it does not display the exact file descriptor number77, which you would then look up in/proc/<PID>/fd).
2. Inspecting /proc Directly Without External Utilities
On minimal containers or embedded Linux systems where neither lsof nor fuser is installed, you can query the kernel’s /proc virtual filesystem directly using standard coreutils:
ls -l /proc/[0-9]*/fd 2>/dev/null | grep '/home/admin/somefile'
- Information Provided: Displays the symlink
/proc/<PID>/fd/<FD> -> /home/admin/somefile(and/proc/<PID>/fdinfo/<FD>can be read to inspect file flags such asO_WRONLY). - Advantages: Zero external dependencies; reads directly from kernel data structures.
- Limitations: Requires filtering out
Permission deniederrors (2>/dev/null) when scanning/procentries owned by other users.
3. Closing a Descriptor in an External Process via gdb (GNU Debugger)
What if PID 813 had turned out to be a separate process—such as a compiled C/Go daemon, a Python worker, or another bash script running in the background where you could not type exec 77>&- interactively?
In that scenario, you can attach gdb to the running process and invoke the close() C library wrapper directly inside its address space:
gdb -p 813 -batch -ex 'call (int)close(77)'
- How It Works:
gdbuses theptracesystem call to attach to PID813, briefly pauses execution, executesclose(77)within the target process’s context, detaches, and resumes the process. - Advantages: Works on virtually any running Linux process (compiled binaries, interpreters, background shells) without restarting the service.
- Limitations & Operational Risks:
- Briefly freezes the target process (
SIGSTOP) while attached—risky for latency-sensitive or real-time services. - Requires
gdbto be installed on the host. - Blocked by default on many hardened Linux distributions when
kernel.yama.ptrace_scopeis set to1or higher (unless running asrootwithCAP_SYS_PTRACE). - Application logic inside the target process might crash or log
EBADF(Bad file descriptor) errors if it later attempts towrite()to descriptor77without checking return codes.
Comparison of Approaches
| Approach | Target Process Type | Requires Extra Tools | Risk Level | When an SRE Uses It |
|---|---|---|---|---|
exec <FD>&- |
Current shell ($$) |
No (Bash built-in) | Very Low | Closing leaked FDs in the active shell or shell scripts. |
**lsof / fuser** |
Any accessible process | lsof or psmisc |
None (Read-only) | Standard discovery of which PID/FD holds a file or port. |
/proc/<PID>/fd |
Any accessible process | No (ls / cat) |
None (Read-only) | Minimal containers or locked-down hosts missing lsof. |
gdb -p <PID> |
Separate external process | gdb + ptrace access |
Medium–High | Emergency release of a deleted/locked file held by a daemon that cannot be restarted. |
9. Linux Command Deep Dive
Command: lsof
Purpose
lsof (List Open Files) reports information about files opened by processes. Because Linux treats regular files, directories, pipes, block devices, and network sockets as file descriptors, lsof is one of the most versatile diagnostic utilities for storage, process, and network troubleshooting.
General Syntax
lsof [options] [names]
Options and Arguments
| Command / Option | Purpose |
|---|---|
lsof /path/to/file |
Show processes holding a specific file open (used in this exercise). |
lsof -p <PID> |
List all files, shared libraries, and network sockets opened by <PID>. |
lsof -u <USER> |
List all open files belonging to processes owned by <USER>. |
lsof +L1 |
List open files with a link count less than 1 (deleted files still held open by processes). |
lsof +D /mount/point |
Recursively search for open files under a directory (useful for umount: target is busy). |
lsof -iTCP -sTCP:LISTEN -P -n |
Show all listening TCP sockets without resolving hostnames (-n) or port names (-P). |
Practical Examples
- Find why a filesystem cannot be unmounted:
lsof +f -- /mnt/data
- Find deleted files still consuming disk space:
lsof +L1
Common Mistakes and Limitations
- Silent permission filtering: When executed without
root,lsofcannot inspect/proc/<PID>/fdfor processes owned by other users and silently omits them. - Hanging on stale NFS mounts: If an NFS server is unresponsive,
lsofcan hang while callingstat()on network paths. Uselsof -bto bypass blocking kernel calls.
Command: ps
Purpose
ps (Process Status) captures a static point-in-time snapshot of running processes by reading /proc/[pid]/stat, /proc/[pid]/status, and /proc/[pid]/cmdline. It is part of the procps-ng suite.
General Syntax
ps [options]
Note: ps supports three distinct option styles: UNIX (-ef, preceded by a single dash), BSD (aux, no dash), and GNU (--sort, preceded by two dashes).
Options and Arguments
| Command / Option | Purpose |
|---|---|
ps -fp <PID> |
Full-format inspection of a single PID (UID, PID, PPID, TTY, CMD)—used in this exercise. |
ps aux |
BSD-style listing of all processes with %CPU, %MEM, VSZ, RSS, and STAT. |
ps -ef |
UNIX-style listing of all processes; ideal when you need PPID to trace parent-child relationships. |
ps -ef --forest |
Display an ASCII process tree showing which parent spawned each child process. |
ps -u <USER> |
List all processes running under <USER>. |
ps -eo pid,ppid,stat,%cpu,%mem,comm --sort=-%cpu |
Custom output columns sorted by descending CPU usage. |
Practical Examples
- Identify the top 5 memory-consuming processes:
ps -eo pid,ppid,user,%mem,rss,comm --sort=-%mem | head -n 6
- Find processes stuck in uninterruptible disk/NFS I/O (
Dstate):
ps -eo pid,stat,wchan:24,comm | grep ' D'
Common Mistakes and Limitations
- Self-matching with
grep: Runningps aux | grep bashfrequently returns thegrepprocess itself. Usepgrep -a bashorps aux | grep '[b]ash'instead. - Truncated command lines: Without
-f,aux, or-w,pstruncatesCMDoutput to your terminal width.
Command: exec (Bash Built-in)
Purpose
exec is a shell built-in (documented in man bash, not a standalone binary in /bin) with two distinct behaviors depending on whether a command name is provided.
General Syntax
exec [-cl] [-a name] [command [arguments ...]] [redirection ...]
Options and Arguments
| Usage Pattern | Purpose |
|---|---|
exec 77>&- |
Close file descriptor 77 in the current shell without exiting (used in this exercise). |
exec 3>/tmp/out.log |
Open /tmp/out.log for writing on FD 3 for the remainder of the shell’s lifetime. |
exec 3</tmp/in.txt |
Open /tmp/in.txt for reading on FD 3 across subsequent read -u 3 calls. |
exec >/var/log/job.log 2>&1 |
Redirect all subsequent stdout and stderr of a script to a log file. |
exec /usr/sbin/nginx -g 'daemon off;' |
Replace the current shell process image with nginx while keeping the same PID. |
Common Mistakes and Limitations
- Accidentally exiting your interactive session: Running
exec lsin an interactive terminal replaces yourbashshell withls; oncelsfinishes, your terminal session closes immediately. Only omit the command (exec <redirection>) when modifying descriptors in an interactive shell. - Subshell scope trap: Running
( exec 77>&- )inside parentheses executes inside a forked subshell, closing FD77only in the temporary subshell while leaving FD77open in the parent shell.
10. Underlying Linux Concepts and Architecture
1. The Three-Layer Kernel File Abstraction
When a process opens a file via the open() or openat() system call, the Linux kernel tracks the state across three distinct data structures:
- Per-Process File Descriptor Table: An array private to each process (
files_structin the kernel task structure), indexed by non-negative integers (0,1,2, …,77). Each entry contains flags (such asFD_CLOEXEC) and a pointer to an entry in the system-wide open file table. - System-Wide Open File Table (File Descriptions): A kernel-global table storing the current file offset (read/write cursor position), status flags (
O_RDONLY,O_WRONLY,O_RDWR,O_APPEND), and a pointer to the file’s inode. - Filesystem Inode Table: Stores the file’s metadata on disk (permissions, ownership, timestamps, link count
i_nlink, size, and pointers to data blocks)—notably, not the filename (which lives in directory entries).
2. File Descriptor Inheritance and FD_CLOEXEC
When a parent process calls fork(), the child process receives a duplicate of the parent’s file descriptor table, pointing to the exact same system-wide open file descriptions. When the child subsequently calls execve() to run a new program, open descriptors remain open unless they were explicitly marked with the FD_CLOEXEC (close-on-exec) flag (O_CLOEXEC). This inheritance mechanism is how shell redirections (cmd > file) pass stdout to child commands—and why careless scripts can leak open file descriptors into long-lived background processes.
3. Why Deleting an Open File (rm) Does Not Free Disk Space
Understanding this three-layer architecture explains one of the most common Linux production incidents. When you run rm /var/log/app.log while a daemon still holds an open file descriptor to it:
rmunlinks the directory entry and decrements the inode’s hard-link count (i_nlink) from1to0.- However, the daemon’s open file descriptor still holds an active reference count (
f_count > 0) on the underlying inode. - The kernel does not release the inode or reclaim its data blocks until both
i_nlink == 0and all open file descriptors referencing that inode are closed.
11. Real-World Sysadmin and SRE Applications
Production Scenarios Where Open File Descriptors Cause Incidents
- Disk Exhaustion from Deleted-But-Open Log Files: An operator deletes a runaway 50 GB log file with
rm,lsconfirms the file is gone, yetdf -hstill reports 100% disk usage because the application never closed its log descriptor. - File Descriptor Exhaustion (
EMFILE/ENFILE): Every Linux process has a soft and hard file descriptor limit (ulimit -n, configured via/etc/security/limits.conforLimitNOFILE=in systemd unit files). If an application leaks descriptors without callingclose(), it eventually fails to accept new network connections or open files withToo many open files. - Blocked Unmounts and Maintenance Windows: Storage migrations or filesystem checks fail with
umount: /data: target is busybecause an idle shell or monitoring agent still has acwdor open file descriptor on that mount point.
Prevention, Monitoring, and Safe Remediation in Production
- Observability & Alerting: Monitor per-process open FD counts against configured limits (e.g., alerting when
/proc/<PID>/fdcount exceeds 80% of/proc/<PID>/limitsMax open files), and monitor disk space discrepancies betweendf(filesystem blocks) anddu(directory traversal). - Log Rotation Best Practices: Configure
logrotatewithpostrotatehooks that signal the application (kill -USR1orsystemctl reload) to close and reopen its log descriptors cleanly. Usecopytruncateonly as a last resort when an application cannot reopen logs, as it carries a small window for data loss during truncation. - Safe Truncation vs. Deletion: If a live log file is filling up a disk and the service cannot be restarted immediately, truncating the file in place (
: > /var/log/app.logortruncate -s 0 /var/log/app.log) frees disk blocks immediately without breaking the application’s open file descriptor—whereasrmhides the filename while keeping the disk full. - Defensive Shell Scripting: In automation scripts, always close custom descriptors (
exec 3>&-) as soon as work completes, or scope redirections to a single block ({ ... } > /tmp/file) rather than using process-wideexecredirections.
12. Further Learning and Experiments
Hands-On Experiments (Safe Test Environment)
Run these experiments in a disposable directory (such as /tmp/fd-lab) to observe kernel file descriptor behavior firsthand:
Experiment 1: Reproduce and Resolve a Deleted-But-Open File
- Open a persistent write descriptor on FD
5and write data to it:
mkdir -p /tmp/fd-lab
exec 5>/tmp/fd-lab/ghost.log
echo "still open in kernel" >&5
- Delete the file from the filesystem while keeping FD
5open:
rm /tmp/fd-lab/ghost.log
- Inspect
/proc/$$/fd/5andlsof +L1:
ls -l /proc/$$/fd/5
cat /proc/$$/fd/5
lsof +L1 | grep ghost.log
Observation: Even though rm deleted the path, /proc/$$/fd/5 shows /tmp/fd-lab/ghost.log (deleted) and cat /proc/$$/fd/5 still reads the underlying inode’s contents!
4. Close FD 5 and verify the kernel releases the inode:
exec 5>&-
ls -l /proc/$$/fd/5 2>&1
rmdir /tmp/fd-lab
Experiment 2: Trace the Exact Kernel Syscalls with strace
Run a non-interactive bash command under strace to see the exact system calls executed when opening and closing custom file descriptors:
strace -e trace=openat,dup2,fcntl,close bash -c 'exec 7>/dev/null; exec 7>&-'
Observation: You will see openat() open /dev/null, fcntl(..., F_DUPFD, 7) (or dup2) duplicate it onto descriptor 7, and close(7) release it.
Topics to Explore Next
- Resource Limits (
ulimit -nandsystemdLimitNOFILE): How per-process and system-wide (fs.file-max) file descriptor limits are enforced and tuned. - Process Tracing and Debugging (
straceandgdb): Howptraceinteracts with running processes and howkernel.yama.ptrace_scoperestricts debugging in production. - Subshells and Special Bash Variables (
$$vs.$BASHPID): Howfork()duplicates file descriptors and why$$retains the parent shell’s PID inside subshells( ... ).
Documentation and References
man 1 bash(Sections: REDIRECTION and SHELL BUILTIN COMMANDS ->exec)man 8 lsof(Sections: OUTPUT, EXAMPLES, and+Llink count filtering)man 1 ps(Sections: STANDARD FORMAT SPECIFIERS and PROCESS STATE CODES)man 5 proc(Entries:/proc/[pid]/fdand/proc/[pid]/fdinfo)man 2 openandman 2 close(Linux Programmer’s Manual — system call behavior andFD_CLOEXEC)
13. Key Takeaways
- Read Before Writing: Always start with non-destructive diagnostic commands (
lsof,ps,/proc) to map symptoms to exact PIDs and file descriptors before modifying system state. - Process Context Determines the Fix: Comparing
$$against the target PID immediately tells you whether a native shell redirection (exec <FD>&-) can close the descriptor cleanly or whether external process intervention (gdbor a service reload) is required. - Open Descriptors Outlive Deleted Paths: Deleting an open file with
rmremoves only its directory entry; disk space and locks persist until every process holding a descriptor to that inode callsclose(). - Keep
0,1,2, and255Intact: When closing leaked descriptors in an interactivebashshell, never close standard I/O (0–2) orbash’s controlling terminal descriptor (255) unless you intend to terminate or detach your session.
14. Quick Troubleshooting Reference / Runbook
Use this checklist when investigating an open or locked file in Linux:
- Identify the owning process, user, and file descriptor number:
lsof /path/to/target_file
# Or, if lsof is unavailable:
ls -l /proc/[0-9]*/fd 2>/dev/null | grep '/path/to/target_file'
- Inspect the owning process and compare with your current shell (
$$):
echo "Current shell PID: $$"
ps -fp <PID>
- Inspect all open descriptors of the target process:
ls -l /proc/<PID>/fd
- Apply the least disruptive remediation:
- If
<PID>equals$$(your currentbashshell):
exec <FD>&-
- If
<PID>is a service that supports graceful log/config reloads:
systemctl reload <service> # or kill -USR1 <PID> (verify signal behavior first)
- If
<PID>is an external process that cannot be restarted (emergency workaround):
gdb -p <PID> -batch -ex 'call (int)close(<FD>)'
- Verify the file handle is closed and the process remains healthy:
lsof /path/to/target_file
ps -fp <PID>