Domain 10Linux & SRECORE

(Oaxaca) Closing an Open File Descriptor in Linux Without Killing the Process

A step-by-step troubleshooting guide to identifying and closing an open file descriptor held by a running process without terminating it or using root privileges.

Domain: Domain 10: Linux Internals, SRE & Troubleshooting•Updated: 2026-11-12
#Linux#File Descriptors#Bash#lsof#Troubleshooting#Procfs

1. Introduction

In Linux system administration, open file handles are at the center of many subtle operational incidents. You might encounter a filesystem that refuses to unmount with target is busy, a deleted log file that continues consuming gigabytes of disk space, or a file locked for writing by an active process that cannot be restarted without causing downtime.

A common reflex is to terminate the process holding the file open (kill <PID>). When a process exits, the Linux kernel automatically closes all of its open file descriptors. However, in production environments—or under strict operational constraints—killing a critical process is unacceptable. Instead, an administrator must determine which process holds the file descriptor, how that process relates to the current execution context, and how to close that specific descriptor safely in place.

This article walks through a complete troubleshooting exercise based on the SadServers “Oaxaca: Close an Open File” challenge. We will examine how to trace an open file back to its owning process, inspect kernel file descriptor tables via /proc, close an active file descriptor inside a running shell without terminating it, and understand alternative techniques (such as debugger injection) when the target process is external.


2. Problem Description and Environment

Original Challenge Specification

  • Scenario: “Oaxaca”: Close an Open File
  • Difficulty / Type: Medium / Fix
  • Tags: bash, unusual-tricky
  • Problem Statement: The file /home/admin/somefile is open for writing by some process. Close this file without killing the process.
  • Privilege Constraint: Root (sudo) access is not available (False).
  • Success Criterion: Running lsof /home/admin/somefile must return no output, while the owning process remains alive.

Expected vs. Actual Behaviour

  • Expected Behaviour: No running process holds an open file descriptor referencing /home/admin/somefile (lsof /home/admin/somefile exits with empty output), and the target process continues running uninterrupted.
  • Actual Behaviour: At least one active process holds /home/admin/somefile open in write mode, causing lsof /home/admin/somefile to list an active descriptor.

Environment Details

From the terminal session and diagnostic outputs, the following environmental facts were observed:

  • User Account: admin (unprivileged user; no sudo access)
  • Hostname: ip-10-1-13-98
  • Terminal / TTY: pts/0
  • Storage Device: Major/minor device 259,1 (characteristic of an NVMe block device partition)
  • OS Distribution & Kernel Version: Not captured during the session. Because no uname -a or /etc/os-release checks were required to solve the issue, the exact Linux distribution and kernel release remain unconfirmed, though standard GNU/Linux utilities (bash, lsof, ps, /proc) were present.

Why the Constraints Matter

Two constraints shape the entire investigation strategy:

  1. No sudo Access: On Linux, unprivileged users cannot inspect /proc/<PID>/fd or attach debuggers to processes owned by other users. Therefore, the process holding /home/admin/somefile open must be running under the admin user (or an accessible sub-context of admin).
  2. Do Not Kill the Process: Sending SIGTERM or SIGKILL to the PID would satisfy lsof returning nothing, but would violate the requirement to keep the process running. We must close the individual file descriptor inside the process’s kernel descriptor table.

3. Initial Investigation and Hypotheses

Before running any state-changing commands, we establish what is known, what is missing, and what plausible mechanisms could explain an open file descriptor in an unprivileged environment.

Known vs. Unknown Information

  • Known: The exact filesystem path (/home/admin/somefile), the access mode (open for writing), the user context (admin), and the validation test (lsof /home/admin/somefile).
  • Unknown:
  • Which executable (COMMAND) and Process ID (PID) owns the open file handle.
  • Which file descriptor number (FD) is assigned to /home/admin/somefile.
  • How the owning process relates to our interactive terminal session (e.g., whether it is our current bash shell, a parent shell, a background job, or a separate daemon).

Candidate Hypotheses

Hypothesis Plausibility Supporting Clues What Would Confirm or Refute It
H1: The current interactive bash shell holds the file open. High Challenge is tagged bash and unusual-tricky; shell initialization files (e.g., .bashrc) can open persistent FDs via exec <FD>>file. lsof reports bash as the command, and its PID matches our current shell’s $$.
H2: A background job or subshell spawned under admin holds the file open. Medium Background scripts (script.sh &) inherit or open descriptors and persist after starting. lsof reports a PID different from $$, with our current shell (or an init/session manager) as its parent.
H3: A separate user-level service or daemon owned by admin holds the file open. Medium User services or long-running processes often keep log or state files open for writing. lsof reports a non-shell binary (or detached bash process) with a different PID and TTY.

4. Troubleshooting Process

Step 1: Identify the Process and File Descriptor Holding the File Open

Objective

Map the filesystem path /home/admin/somefile to the exact process ID (PID), owning user (USER), and file descriptor number (FD) using a read-only diagnostic command.

Command

lsof /home/admin/somefile

Command Explanation

  • lsof (List Open Files) queries the kernel’s open file tables via /proc and translates open file descriptors back into human-readable paths.
  • Passing /home/admin/somefile as a positional argument instructs lsof to filter all accessible open descriptors across the system and display only those referencing that specific file’s inode.
  • This command is completely read-only and matches the challenge’s verification test.

Output and Analysis

COMMAND PID   USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
bash    813 admin   77w   REG  259,1        0 272875 /home/admin/somefile

Key fields in this output:

  • COMMAND (bash) & PID (813): A bash process with PID 813 holds the file open.
  • USER (admin): The process belongs to our own user account, confirming why root privileges are unnecessary.
  • FD (77w): The descriptor number is 77, and the trailing w indicates it is open in write-only mode (O_WRONLY).
  • TYPE (REG): It is a regular file (as opposed to a directory DIR, pipe FIFO, or socket).
  • DEVICE (259,1), SIZE/OFF (0), NODE (272875): The file resides on block device 259,1, is currently 0 bytes in size, and uses inode 272875.

This output immediately narrows the problem down to bash PID 813 and file descriptor 77, ruling out non-shell binaries (Hypothesis H3). However, we still do not know whether PID 813 is our current interactive shell (Hypothesis H1) or a separate bash process (Hypothesis H2).

Decision

Compare PID 813 against our current shell’s PID ($$) and inspect PID 813’s process metadata using ps.


Step 2: Determine the Relationship Between PID 813 and the Current Shell

Objective

Check whether PID 813 is the interactive bash session we are currently typing into or a separate bash process, and inspect its parent PID (PPID), controlling terminal (TTY), and full command string (CMD).

Command

echo $$ && ps -fp 813

Command Explanation

  • $$ is a special bash parameter that expands to the Process ID of the current shell.
  • && executes the second command only if echo $$ succeeds.
  • ps -fp 813 queries process status for PID 813:
  • -f selects full-format listing, exposing UID, PID, PPID, C, STIME, TTY, TIME, and CMD.
  • -p 813 filters strictly by Process ID 813.

Output and Analysis

813
UID          PID    PPID  C STIME TTY          TIME CMD
admin        813     584  0 15:29 pts/0    00:00:00 bash

Key findings from this output:

  • echo $$ printed 813, proving that PID 813 is our current interactive shell (confirming Hypothesis H1 and ruling out Hypothesis H2).
  • PPID is 584 (the parent terminal/session process), TTY is pts/0, and CMD is bash with no script arguments.

Because FD 77 exists inside the private file descriptor table of our own bash process, we do not need external process injection tools like gdb. We can instruct bash itself to close FD 77 using a built-in shell redirection.

Decision

Use the bash built-in exec redirection syntax (exec 77>&-) to close file descriptor 77 in the current shell process.


Step 3: Close File Descriptor 77 Inside the Current Shell

Objective

Close file descriptor 77 in PID 813 without spawning a subshell, deleting /home/admin/somefile, or terminating bash.

Command

exec 77>&-

Command Explanation

  • When the exec shell built-in is invoked without a command argument, bash applies any specified redirections permanently to the current shell process itself rather than replacing the process image.
  • 77>&- is the POSIX shell redirection operator that closes file descriptor 77 (triggering the close(77) system call in the kernel).
  • Standard descriptors 0 (stdin), 1 (stdout), and 2 (stderr) remain untouched, so our interactive terminal session continues working normally.

Output and Analysis

The command completed silently with exit status 0 and returned directly to the admin@ip-10-1-13-98:/$ prompt. No error message was emitted, and the shell remained responsive.

Decision

Inspect /proc/$$/fd directly and re-run lsof /home/admin/somefile and echo $$ to confirm that FD 77 is gone while PID 813 remains intact.


5. Root Cause Analysis

What Happened and Why

During session initialization (before presenting the interactive prompt—such as via a login profile or environment setup script), the bash shell opened /home/admin/somefile on file descriptor 77 for writing (equivalent to running exec 77>/home/admin/somefile).

In Linux, open file descriptors are maintained per process in kernel memory and remain active until one of two events occurs:

  1. The owning process exits (at which point the kernel closes all open descriptors during process teardown).
  2. The process explicitly invokes the close() system call on that specific file descriptor number.

Because file descriptor 77 is outside the standard I/O channels (0, 1, 2), it sat open silently in the background of PID 813 without interfering with interactive keyboard input or terminal output.

Kernel File Descriptor Architecture (Before vs. After)

BEFORE: exec 77>&-
+-----------------------------------+       +-------------------------------+       +------------------------+
| Per-Process FD Table (PID 813)    |       | System-Wide Open File Table   |       | Inode Table            |
|-----------------------------------|       |-------------------------------|       |------------------------|
| FD 0   -------------------------->|------>| /dev/pts/0 (O_RDWR)           |       |                        |
| FD 1   -------------------------->|------>| /dev/pts/0 (O_RDWR)           |       |                        |
| FD 2   -------------------------->|------>| /dev/pts/0 (O_RDWR)           |       |                        |
| FD 77  -------------------------->|------>| File Offset: 0, Mode: O_WRONLY|------>| Inode 272875 (somefile)|
| FD 255 -------------------------->|------>| /dev/pts/0 (O_RDWR)           |       |                        |
+-----------------------------------+       +-------------------------------+       +------------------------+

AFTER: exec 77>&-  (triggers close(77) syscall)
+-----------------------------------+       +-------------------------------+       +------------------------+
| Per-Process FD Table (PID 813)    |       | System-Wide Open File Table   |       | Inode Table            |
|-----------------------------------|       |-------------------------------|       |------------------------|
| FD 0   -------------------------->|------>| /dev/pts/0 (O_RDWR)           |       |                        |
| FD 1   -------------------------->|------>| /dev/pts/0 (O_RDWR)           |       |                        |
| FD 2   -------------------------->|------>| /dev/pts/0 (O_RDWR)           |       |                        |
| FD 77  [CLOSED / UNALLOCATED]     |       | [Entry Freed - Ref Count = 0] |       | Inode 272875 (0 opens) |
| FD 255 -------------------------->|------>| /dev/pts/0 (O_RDWR)           |       |                        |
+-----------------------------------+       +-------------------------------+       +------------------------+

6. Solution and Remediation

Applied Fix

To release the open file handle inside PID 813 without terminating the shell, we executed:

exec 77>&-
  • What Changed: File descriptor 77 was removed from PID 813’s kernel file descriptor table via close(77).
  • Configuration Files Modified: None. Because this was an in-memory runtime file descriptor state rather than a persistent configuration file, no files on disk were edited or deleted.
  • Service Restarts / Reboots: None required (and explicitly avoided, as restarting the shell would have terminated PID 813).
  • Side Effects & Reversibility: Zero negative side effects. Standard terminal descriptors remained intact. If FD 77 ever needed to be reopened to the same file, it could be reversed immediately with exec 77>/home/admin/somefile.

7. Verification and Validation

After executing exec 77>&-, three checks were performed to confirm both that the file descriptor was closed and that the original process survived intact.

1. Direct Kernel Descriptor Inspection (/proc)

ls -l /proc/$$/fd

Actual Output:

total 0
lrwx------ 1 admin admin 64 Nov 12 15:29 0 -> /dev/pts/0
lrwx------ 1 admin admin 64 Nov 12 15:29 1 -> /dev/pts/0
lrwx------ 1 admin admin 64 Nov 12 15:29 2 -> /dev/pts/0
lrwx------ 1 admin admin 64 Nov 12 15:29 255 -> /dev/pts/0

Why It Matters: /proc/813/fd exposes the kernel’s exact per-process descriptor table. Only 0 (stdin), 1 (stdout), 2 (stderr), and 255 (the descriptor bash internally duplicates from the controlling terminal for job control) remain. Descriptor 77 is completely absent.

2. Challenge Validation Test (lsof)

lsof /home/admin/somefile

Actual Output: (Empty — no lines returned)

Why It Matters: Confirms that no process anywhere on the system holds an open descriptor pointing to /home/admin/somefile’s inode.

3. Process Survival Check ($$)

echo $$

Actual Output:

813

Why It Matters: Proves that PID 813 is still the exact same running process and was not killed, restarted, or replaced via exec <command>.


8. Alternative Troubleshooting Approaches

The commands below were not executed during the original session, but represent essential alternative techniques that an experienced sysadmin or SRE would use depending on available tools and whether the target process is an external binary.

1. Discovering Open Files with fuser (Alternative to lsof)

fuser identifies processes using specified files, directories, or sockets and is often faster and more lightweight than lsof.

fuser -v /home/admin/somefile
  • Information Provided: Prints the USER, PID, ACCESS type (f for open file, F for file open for writing, c for current directory), and COMMAND.
  • Advantages: Fast execution; part of the standard psmisc package; even supports -k to signal matching processes (though -k would violate our constraint not to kill the process).
  • Limitations: Provides less granular detail than lsof (for example, it does not display the exact file descriptor number 77, which you would then look up in /proc/<PID>/fd).

2. Inspecting /proc Directly Without External Utilities

On minimal containers or embedded Linux systems where neither lsof nor fuser is installed, you can query the kernel’s /proc virtual filesystem directly using standard coreutils:

ls -l /proc/[0-9]*/fd 2>/dev/null | grep '/home/admin/somefile'
  • Information Provided: Displays the symlink /proc/<PID>/fd/<FD> -> /home/admin/somefile (and /proc/<PID>/fdinfo/<FD> can be read to inspect file flags such as O_WRONLY).
  • Advantages: Zero external dependencies; reads directly from kernel data structures.
  • Limitations: Requires filtering out Permission denied errors (2>/dev/null) when scanning /proc entries owned by other users.

3. Closing a Descriptor in an External Process via gdb (GNU Debugger)

What if PID 813 had turned out to be a separate process—such as a compiled C/Go daemon, a Python worker, or another bash script running in the background where you could not type exec 77>&- interactively?

In that scenario, you can attach gdb to the running process and invoke the close() C library wrapper directly inside its address space:

gdb -p 813 -batch -ex 'call (int)close(77)'
  • How It Works: gdb uses the ptrace system call to attach to PID 813, briefly pauses execution, executes close(77) within the target process’s context, detaches, and resumes the process.
  • Advantages: Works on virtually any running Linux process (compiled binaries, interpreters, background shells) without restarting the service.
  • Limitations & Operational Risks:
  • Briefly freezes the target process (SIGSTOP) while attached—risky for latency-sensitive or real-time services.
  • Requires gdb to be installed on the host.
  • Blocked by default on many hardened Linux distributions when kernel.yama.ptrace_scope is set to 1 or higher (unless running as root with CAP_SYS_PTRACE).
  • Application logic inside the target process might crash or log EBADF (Bad file descriptor) errors if it later attempts to write() to descriptor 77 without checking return codes.

Comparison of Approaches

Approach Target Process Type Requires Extra Tools Risk Level When an SRE Uses It
exec <FD>&- Current shell ($$) No (Bash built-in) Very Low Closing leaked FDs in the active shell or shell scripts.
**lsof / fuser** Any accessible process lsof or psmisc None (Read-only) Standard discovery of which PID/FD holds a file or port.
/proc/<PID>/fd Any accessible process No (ls / cat) None (Read-only) Minimal containers or locked-down hosts missing lsof.
gdb -p <PID> Separate external process gdb + ptrace access Medium–High Emergency release of a deleted/locked file held by a daemon that cannot be restarted.

9. Linux Command Deep Dive

Command: lsof

Purpose

lsof (List Open Files) reports information about files opened by processes. Because Linux treats regular files, directories, pipes, block devices, and network sockets as file descriptors, lsof is one of the most versatile diagnostic utilities for storage, process, and network troubleshooting.

General Syntax

lsof [options] [names]

Options and Arguments

Command / Option Purpose
lsof /path/to/file Show processes holding a specific file open (used in this exercise).
lsof -p <PID> List all files, shared libraries, and network sockets opened by <PID>.
lsof -u <USER> List all open files belonging to processes owned by <USER>.
lsof +L1 List open files with a link count less than 1 (deleted files still held open by processes).
lsof +D /mount/point Recursively search for open files under a directory (useful for umount: target is busy).
lsof -iTCP -sTCP:LISTEN -P -n Show all listening TCP sockets without resolving hostnames (-n) or port names (-P).

Practical Examples

  • Find why a filesystem cannot be unmounted:
lsof +f -- /mnt/data
  • Find deleted files still consuming disk space:
lsof +L1

Common Mistakes and Limitations

  • Silent permission filtering: When executed without root, lsof cannot inspect /proc/<PID>/fd for processes owned by other users and silently omits them.
  • Hanging on stale NFS mounts: If an NFS server is unresponsive, lsof can hang while calling stat() on network paths. Use lsof -b to bypass blocking kernel calls.

Command: ps

Purpose

ps (Process Status) captures a static point-in-time snapshot of running processes by reading /proc/[pid]/stat, /proc/[pid]/status, and /proc/[pid]/cmdline. It is part of the procps-ng suite.

General Syntax

ps [options]

Note: ps supports three distinct option styles: UNIX (-ef, preceded by a single dash), BSD (aux, no dash), and GNU (--sort, preceded by two dashes).

Options and Arguments

Command / Option Purpose
ps -fp <PID> Full-format inspection of a single PID (UID, PID, PPID, TTY, CMD)—used in this exercise.
ps aux BSD-style listing of all processes with %CPU, %MEM, VSZ, RSS, and STAT.
ps -ef UNIX-style listing of all processes; ideal when you need PPID to trace parent-child relationships.
ps -ef --forest Display an ASCII process tree showing which parent spawned each child process.
ps -u <USER> List all processes running under <USER>.
ps -eo pid,ppid,stat,%cpu,%mem,comm --sort=-%cpu Custom output columns sorted by descending CPU usage.

Practical Examples

  • Identify the top 5 memory-consuming processes:
ps -eo pid,ppid,user,%mem,rss,comm --sort=-%mem | head -n 6
  • Find processes stuck in uninterruptible disk/NFS I/O (D state):
ps -eo pid,stat,wchan:24,comm | grep ' D'

Common Mistakes and Limitations

  • Self-matching with grep: Running ps aux | grep bash frequently returns the grep process itself. Use pgrep -a bash or ps aux | grep '[b]ash' instead.
  • Truncated command lines: Without -f, aux, or -w, ps truncates CMD output to your terminal width.

Command: exec (Bash Built-in)

Purpose

exec is a shell built-in (documented in man bash, not a standalone binary in /bin) with two distinct behaviors depending on whether a command name is provided.

General Syntax

exec [-cl] [-a name] [command [arguments ...]] [redirection ...]

Options and Arguments

Usage Pattern Purpose
exec 77>&- Close file descriptor 77 in the current shell without exiting (used in this exercise).
exec 3>/tmp/out.log Open /tmp/out.log for writing on FD 3 for the remainder of the shell’s lifetime.
exec 3</tmp/in.txt Open /tmp/in.txt for reading on FD 3 across subsequent read -u 3 calls.
exec >/var/log/job.log 2>&1 Redirect all subsequent stdout and stderr of a script to a log file.
exec /usr/sbin/nginx -g 'daemon off;' Replace the current shell process image with nginx while keeping the same PID.

Common Mistakes and Limitations

  • Accidentally exiting your interactive session: Running exec ls in an interactive terminal replaces your bash shell with ls; once ls finishes, your terminal session closes immediately. Only omit the command (exec <redirection>) when modifying descriptors in an interactive shell.
  • Subshell scope trap: Running ( exec 77>&- ) inside parentheses executes inside a forked subshell, closing FD 77 only in the temporary subshell while leaving FD 77 open in the parent shell.

10. Underlying Linux Concepts and Architecture

1. The Three-Layer Kernel File Abstraction

When a process opens a file via the open() or openat() system call, the Linux kernel tracks the state across three distinct data structures:

  1. Per-Process File Descriptor Table: An array private to each process (files_struct in the kernel task structure), indexed by non-negative integers (0, 1, 2, …, 77). Each entry contains flags (such as FD_CLOEXEC) and a pointer to an entry in the system-wide open file table.
  2. System-Wide Open File Table (File Descriptions): A kernel-global table storing the current file offset (read/write cursor position), status flags (O_RDONLY, O_WRONLY, O_RDWR, O_APPEND), and a pointer to the file’s inode.
  3. Filesystem Inode Table: Stores the file’s metadata on disk (permissions, ownership, timestamps, link count i_nlink, size, and pointers to data blocks)—notably, not the filename (which lives in directory entries).

2. File Descriptor Inheritance and FD_CLOEXEC

When a parent process calls fork(), the child process receives a duplicate of the parent’s file descriptor table, pointing to the exact same system-wide open file descriptions. When the child subsequently calls execve() to run a new program, open descriptors remain open unless they were explicitly marked with the FD_CLOEXEC (close-on-exec) flag (O_CLOEXEC). This inheritance mechanism is how shell redirections (cmd > file) pass stdout to child commands—and why careless scripts can leak open file descriptors into long-lived background processes.

3. Why Deleting an Open File (rm) Does Not Free Disk Space

Understanding this three-layer architecture explains one of the most common Linux production incidents. When you run rm /var/log/app.log while a daemon still holds an open file descriptor to it:

  • rm unlinks the directory entry and decrements the inode’s hard-link count (i_nlink) from 1 to 0.
  • However, the daemon’s open file descriptor still holds an active reference count (f_count > 0) on the underlying inode.
  • The kernel does not release the inode or reclaim its data blocks until both i_nlink == 0 and all open file descriptors referencing that inode are closed.

11. Real-World Sysadmin and SRE Applications

Production Scenarios Where Open File Descriptors Cause Incidents

  1. Disk Exhaustion from Deleted-But-Open Log Files: An operator deletes a runaway 50 GB log file with rm, ls confirms the file is gone, yet df -h still reports 100% disk usage because the application never closed its log descriptor.
  2. File Descriptor Exhaustion (EMFILE / ENFILE): Every Linux process has a soft and hard file descriptor limit (ulimit -n, configured via /etc/security/limits.conf or LimitNOFILE= in systemd unit files). If an application leaks descriptors without calling close(), it eventually fails to accept new network connections or open files with Too many open files.
  3. Blocked Unmounts and Maintenance Windows: Storage migrations or filesystem checks fail with umount: /data: target is busy because an idle shell or monitoring agent still has a cwd or open file descriptor on that mount point.

Prevention, Monitoring, and Safe Remediation in Production

  • Observability & Alerting: Monitor per-process open FD counts against configured limits (e.g., alerting when /proc/<PID>/fd count exceeds 80% of /proc/<PID>/limits Max open files), and monitor disk space discrepancies between df (filesystem blocks) and du (directory traversal).
  • Log Rotation Best Practices: Configure logrotate with postrotate hooks that signal the application (kill -USR1 or systemctl reload) to close and reopen its log descriptors cleanly. Use copytruncate only as a last resort when an application cannot reopen logs, as it carries a small window for data loss during truncation.
  • Safe Truncation vs. Deletion: If a live log file is filling up a disk and the service cannot be restarted immediately, truncating the file in place (: > /var/log/app.log or truncate -s 0 /var/log/app.log) frees disk blocks immediately without breaking the application’s open file descriptor—whereas rm hides the filename while keeping the disk full.
  • Defensive Shell Scripting: In automation scripts, always close custom descriptors (exec 3>&-) as soon as work completes, or scope redirections to a single block ({ ... } > /tmp/file) rather than using process-wide exec redirections.

12. Further Learning and Experiments

Hands-On Experiments (Safe Test Environment)

Run these experiments in a disposable directory (such as /tmp/fd-lab) to observe kernel file descriptor behavior firsthand:

Experiment 1: Reproduce and Resolve a Deleted-But-Open File

  1. Open a persistent write descriptor on FD 5 and write data to it:
mkdir -p /tmp/fd-lab
exec 5>/tmp/fd-lab/ghost.log
echo "still open in kernel" >&5
  1. Delete the file from the filesystem while keeping FD 5 open:
rm /tmp/fd-lab/ghost.log
  1. Inspect /proc/$$/fd/5 and lsof +L1:
ls -l /proc/$$/fd/5
cat /proc/$$/fd/5
lsof +L1 | grep ghost.log

Observation: Even though rm deleted the path, /proc/$$/fd/5 shows /tmp/fd-lab/ghost.log (deleted) and cat /proc/$$/fd/5 still reads the underlying inode’s contents! 4. Close FD 5 and verify the kernel releases the inode:

exec 5>&-
ls -l /proc/$$/fd/5 2>&1
rmdir /tmp/fd-lab

Experiment 2: Trace the Exact Kernel Syscalls with strace

Run a non-interactive bash command under strace to see the exact system calls executed when opening and closing custom file descriptors:

strace -e trace=openat,dup2,fcntl,close bash -c 'exec 7>/dev/null; exec 7>&-'

Observation: You will see openat() open /dev/null, fcntl(..., F_DUPFD, 7) (or dup2) duplicate it onto descriptor 7, and close(7) release it.

Topics to Explore Next

  • Resource Limits (ulimit -n and systemd LimitNOFILE): How per-process and system-wide (fs.file-max) file descriptor limits are enforced and tuned.
  • Process Tracing and Debugging (strace and gdb): How ptrace interacts with running processes and how kernel.yama.ptrace_scope restricts debugging in production.
  • Subshells and Special Bash Variables ($$ vs. $BASHPID): How fork() duplicates file descriptors and why $$ retains the parent shell’s PID inside subshells ( ... ).

Documentation and References

  • man 1 bash (Sections: REDIRECTION and SHELL BUILTIN COMMANDS -> exec)
  • man 8 lsof (Sections: OUTPUT, EXAMPLES, and +L link count filtering)
  • man 1 ps (Sections: STANDARD FORMAT SPECIFIERS and PROCESS STATE CODES)
  • man 5 proc (Entries: /proc/[pid]/fd and /proc/[pid]/fdinfo)
  • man 2 open and man 2 close (Linux Programmer’s Manual — system call behavior and FD_CLOEXEC)

13. Key Takeaways

  • Read Before Writing: Always start with non-destructive diagnostic commands (lsof, ps, /proc) to map symptoms to exact PIDs and file descriptors before modifying system state.
  • Process Context Determines the Fix: Comparing $$ against the target PID immediately tells you whether a native shell redirection (exec <FD>&-) can close the descriptor cleanly or whether external process intervention (gdb or a service reload) is required.
  • Open Descriptors Outlive Deleted Paths: Deleting an open file with rm removes only its directory entry; disk space and locks persist until every process holding a descriptor to that inode calls close().
  • Keep 0, 1, 2, and 255 Intact: When closing leaked descriptors in an interactive bash shell, never close standard I/O (0–2) or bash’s controlling terminal descriptor (255) unless you intend to terminate or detach your session.

14. Quick Troubleshooting Reference / Runbook

Use this checklist when investigating an open or locked file in Linux:

  1. Identify the owning process, user, and file descriptor number:
lsof /path/to/target_file
# Or, if lsof is unavailable:
ls -l /proc/[0-9]*/fd 2>/dev/null | grep '/path/to/target_file'
  1. Inspect the owning process and compare with your current shell ($$):
echo "Current shell PID: $$"
ps -fp <PID>
  1. Inspect all open descriptors of the target process:
ls -l /proc/<PID>/fd
  1. Apply the least disruptive remediation:
  • If <PID> equals $$ (your current bash shell):
exec <FD>&-
  • If <PID> is a service that supports graceful log/config reloads:
systemctl reload <service>   # or kill -USR1 <PID> (verify signal behavior first)
  • If <PID> is an external process that cannot be restarted (emergency workaround):
gdb -p <PID> -batch -ex 'call (int)close(<FD>)'
  1. Verify the file handle is closed and the process remains healthy:
lsof /path/to/target_file
ps -fp <PID>