Domain 10Linux & SREINTERMEDIATE

(Kihei) Storage Allocation, LVM Volume Creation, and Process Tracing

A step-by-step deep dive into diagnosing a disk allocation crash in Go binaries using strace, and resolving disk space constraints by creating an LVM logical volume across multiple NVMe drives mounted over an application data directory.

Domain: Domain 10: Linux Internals, SRE & Troubleshooting•Updated: 2026-11-08
#linux#sysadmin#sre#lvm#strace#storage#sadservers

1. Introduction

Storage management is a cornerstone of Linux system administration and Site Reliability Engineering (SRE). Out-of-space conditions (ENOSPC) remain one of the most common causes of unexpected service outages in production environments. However, resolving disk space constraints in production is rarely as simple as deleting large files—especially when those files belong to core databases, state stores, or strict application requirements.

In this technical breakdown, we walk through the resolution of the “Kihei” challenge from SadServers. The problem centers on a compiled Go application (/home/admin/kihei) that crashes on launch due to storage exhaustion, under the explicit constraint that an existing 5.1 GB data file (/home/admin/datafile) must not be deleted.

By reading this guide, you will learn:

  • How to trace opaque compiled binary failures to their underlying system call failures using strace and command-line flags.
  • How to inspect block storage topology using lsblk and identify unallocated storage devices.
  • How to build an emergency storage array using the Logical Volume Manager (LVM) toolchain (pvcreate, vgcreate, lvcreate).
  • How to utilize Virtual File System (VFS) mount point overrides to grant dedicated storage capacity to specific application directories without disrupting the root partition.

2. Problem Description and Environment

Scenario Specifications

  • Scenario Name: “Kihei”: Surely Not Another Disk Space Scenario
  • Platform: SadServers
  • Difficulty: Medium
  • Target Host OS: Debian 11 (Bullseye), Linux kernel 5.x, x86_64 architecture
  • Privileges: Standard user admin with full sudo access

Challenge Statement

There is a program at /home/admin/kihei. Make the changes necessary so it runs successfully and outputs Done., without deleting the /home/admin/datafile file.

Observed Failure

When attempting to execute /home/admin/kihei directly from the terminal, the application immediately panics and exits:

admin@ip-10-1-13-199:~$ /home/admin/kihei
panic: exit status 1

goroutine 1 [running]:
main.main()
        ./main.go:64 +0x47d

System Environment Constraints

  1. File Preservation: /home/admin/datafile (size ~5.1 GB) cannot be unlinked or removed from the filesystem.
  2. Privilege Boundary: The kihei binary checks effective user IDs at runtime and explicitly refuses to run under root privileges (UID 0).

3. Initial Investigation and Hypotheses

When facing an application crash without detailed error logging, systematically separate unverified hypotheses from confirmed system evidence.

Initial State Knowledge

  • Known: The binary fails with exit code 1 originating from an internal Go subprocess call (main.go:64).
  • Known: /home/admin/datafile exists and takes up over 5 GB of disk space.
  • Unknown: What external command or system call kihei is invoking that leads to exit status 1.
  • Unknown: The target directory where kihei writes its output.

Diagnostic Hypotheses

Hypothesis Supporting Evidence Evidence Against Diagnostic Step
H1: Block Allocation Failure (ENOSPC) Root disk space is tight (~576 MB free); 5 GB file present on root partition. None yet. Run df -h and trace file write system calls via strace.
H2: Inode Exhaustion Systems with many small files fail to create files despite free block space. None yet. Run df -i to check inode free counts on /.
H3: Permission / Ownership Error Target path owned by root or another user. Binary executed in /home/admin/ where user admin usually has permissions. Check permissions of /home/admin/ subdirectories with ls -la.
H4: Missing Executable Dependency Go os/exec panics if a binary called by exec.Command is missing from $PATH. Binary output specifies exit status 1 rather than executable not found error. Use strace -f to observe execve calls.

4. Troubleshooting Process

Step 1: Binary Identification and Initial Inspection

Objective

Determine file types, ownership, permissions, and basic properties of the components in /home/admin/.

Command

ls -la /home/admin/
file /home/admin/kihei /home/admin/datafile

Command Explanation

  • ls -la: Lists all directory contents using the long format (-l), including hidden dotfiles (-a).
  • file: Examines file headers and magic numbers to determine file format independently of file extension.

Output and Analysis

admin@ip-10-1-13-199:~$ ls -la /home/admin/
total 5245076
drwxr-xr-x 6 admin admin       4096 Dec 14  2025 .
drwxr-xr-x 3 root  root        4096 Feb 17  2024 ..
drwxr-xr-x 2 admin root        4096 Dec 14  2025 agent
drwxr-xr-x 2 admin root        4096 Dec 14  2025 data
-rw-r--r-- 1 root  root  5368709120 Dec 14  2025 datafile
-rwxr-xr-x 1 admin root     2207109 Dec 14  2025 kihei

admin@ip-10-1-13-199:~$ file /home/admin/kihei /home/admin/datafile
/home/admin/kihei:    ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=..., not stripped
/home/admin/datafile: data

Analysis

  • /home/admin/datafile is 5,368,709,120 bytes (~5.0 GiB / 5.37 GB) and owned by root:root.
  • /home/admin/kihei is a statically linked Go executable owned by admin:root with executable permissions (-rwxr-xr-x).
  • Directory /home/admin/data exists and is owned by admin:root.

Step 2: System Call Tracing and Option Discovery

Objective

Identify the exact subprocess or file operation failing inside /home/admin/kihei.

Commands

./kihei -v
strace -f /home/admin/kihei

Command Explanation

  • ./kihei -v: Runs the binary with a common verbose flag to check if the application logs internal operations to standard output.
  • strace: Intercepts and records system calls made by a process.
  • -f: Follows child threads and subprocesses created by fork(), vfork(), or clone(). Crucial for Go applications which use lightweight OS threads and execute child binaries via os/exec.

Output and Analysis

Running ./kihei -v:

admin@ip-10-1-13-32:~$ ./kihei -v
Creating file /home/admin/data/newdatafile with size 1.5GB...
panic: exit status 1

goroutine 1 [running]:
main.main()
        ./main.go:64 +0x47d

Tracing via strace -f /home/admin/kihei (relevant excerpt):

[pid   845] execve("/usr/bin/fallocate", ["fallocate", "-l", "1.5GB", "/home/admin/data/newdatafile"], ...) = 0
...
[pid   845] openat(AT_FDCWD, "/home/admin/data/newdatafile", O_RDWR|O_CREAT, 0666) = 3
[pid   845] fallocate(3, 0, 0, 1500000000) = -1 ENOSPC (No space left on device)
[pid   845] write(2, "fallocate: ", 11) = 11
[pid   845] write(2, "fallocate failed", 16) = 16
[pid   845] write(2, ": No space left on device\n", 26) = 26
[pid   845] exit_group(1) = ?
[pid   845] +++ exited with 1 +++

Analysis

  1. kihei executes /usr/bin/fallocate -l 1.5GB /home/admin/data/newdatafile as a child process.
  2. The fallocate() system call returns -1 ENOSPC (No space left on device).
  3. The child process writes fallocate failed: No space left on device to stderr and exits with status code 1.
  4. Parent process kihei catches exit status 1, unlinks the partial file /home/admin/data/newdatafile, and panics.

Step 3: Storage Audit and Disk Topology Discovery

Objective

Quantify available space on the root filesystem and identify alternative storage devices attached to the system.

Commands

df -h
df -i
du -h /home/admin/datafile
lsblk -l

Command Explanation

  • df -h: Displays available filesystem disk space in human-readable units (GiB, MiB).
  • df -i: Displays inode utilization statistics per mounted filesystem.
  • du -h: Estimates file space utilization on disk for the target path.
  • lsblk -l: Lists all available block devices in a flat list layout (-l), showing device names, sizes, partition types, and mount points.

Output and Analysis

admin@ip-10-1-13-32:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1  7.7G  6.7G  576M  93% /
udev            215M     0  215M   0% /dev
tmpfs            46M  396K   45M   1% /run

admin@ip-10-1-13-32:~$ df -i
Filesystem      Inodes IUsed  IFree IUse% Mounted on
/dev/nvme0n1p1  516096 35263 480833    7% /

admin@ip-10-1-13-32:~$ du -h /home/admin/datafile
5.1G    /home/admin/datafile

admin@ip-10-1-13-32:~$ lsblk -l
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme0n1     259:0    0    8G  0 disk 
nvme2n1     259:1    0    1G  0 disk 
nvme1n1     259:2    0    1G  0 disk 
nvme0n1p1   259:3    0  7.9G  0 part /
nvme0n1p14  259:4    0    3M  0 part 
nvme0n1p15  259:5    0  124M  0 part /boot/efi

Decision & Analysis

  • Root Partition (/dev/nvme0n1p1): Has only 576 MB free space remaining. kihei requires 1.5 GB (1,500,000,000 bytes).
  • Inode Usage: Inodes are 93% free (480,833 free). Inodes are not the bottleneck.
  • Secondary Block Devices: lsblk -l reveals two completely unused, unpartitioned block devices:
  • /dev/nvme1n1 (1.0 GiB)
  • /dev/nvme2n1 (1.0 GiB)

Neither disk alone can fit the required 1.5 GB file. However, combined using LVM (Logical Volume Manager), they provide 2.0 GiB of total storage capacity.


Step 4: LVM Array Provisioning and Filesystem Mounting

Objective

Aggregate /dev/nvme1n1 and /dev/nvme2n1 into a single Volume Group, create a Logical Volume spanning 100% of both drives, format it as ext4, and mount it directly over /home/admin/data.

Commands

sudo pvcreate /dev/nvme1n1 /dev/nvme2n1
sudo vgcreate vg /dev/nvme1n1 /dev/nvme2n1
sudo lvcreate -n lv -l 100%FREE vg
sudo mkfs.ext4 /dev/vg/lv
sudo mount /dev/vg/lv /home/admin/data
sudo chown -R admin: /home/admin/data

Command Explanation

  1. pvcreate: Initializes physical block devices /dev/nvme1n1 and /dev/nvme2n1 as LVM Physical Volumes (PVs) by writing LVM metadata headers.
  2. vgcreate: Combines the specified PVs into a unified storage pool (Volume Group) named vg.
  3. lvcreate: Carves out a Logical Volume (LV) named lv from Volume Group vg. The -l 100%FREE flag allocates all available Physical Extents (PEs) in the group.
  4. mkfs.ext4: Formats the newly created logical block device /dev/vg/lv with an ext4 filesystem.
  5. mount: Attaches /dev/vg/lv to directory /home/admin/data. Any subsequent file operations inside /home/admin/data bypass the root partition and write directly to the 2 GB LVM volume.
  6. chown -R admin:: Recursively changes ownership of /home/admin/data (including the filesystem root directory created by mkfs) to user admin and default group admin.

Output and Analysis

admin@ip-10-1-13-32:~$ sudo pvcreate /dev/nvme1n1 /dev/nvme2n1
  Physical volume "/dev/nvme1n1" successfully created.
  Physical volume "/dev/nvme2n1" successfully created.

admin@ip-10-1-13-32:~$ sudo vgcreate vg /dev/nvme1n1 /dev/nvme2n1
  Volume group "vg" successfully created

admin@ip-10-1-13-32:~$ sudo lvcreate -n lv -l 100%FREE vg
  Logical volume "lv" created.

admin@ip-10-1-13-32:~$ sudo mkfs.ext4 /dev/vg/lv
mke2fs 1.46.2 (28-Feb-2021)
Creating filesystem with 522240 4k blocks and 130560 inodes
Filesystem UUID: e4bb8c46-e037-42fd-8c5f-0aed748b68e4
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

admin@ip-10-1-13-32:~$ sudo mount /dev/vg/lv /home/admin/data
admin@ip-10-1-13-32:~$ sudo chown -R admin: /home/admin/data

Step 5: Privilege Validation and Final Execution

Objective

Verify program execution under correct user privileges.

Commands

sudo ./kihei
./kihei

Output and Analysis

admin@ip-10-1-13-32:~$ sudo ./kihei
Error: This program cannot be run as the 'root' superuser.

admin@ip-10-1-13-32:~$ ./kihei
Done.

Result

  • Running sudo ./kihei fails gracefully due to an explicit internal application check preventing execution as UID 0.
  • Running ./kihei as user admin successfully allocates 1.5 GB in /home/admin/data/newdatafile, completes internal checks, and prints Done..

5. Root Cause Analysis

The application failure was caused by storage allocation failure on the root partition, compounded by directory mount boundaries.

+-----------------------------------------------------------------------------------+
|                                  ROOT FILESYSTEM (/)                              |
|                                     /dev/nvme0n1p1                                |
|                                (Total: 7.7G, Avail: 576M)                         |
|                                                                                   |
|  /home/admin/datafile (5.1GB)  [DO NOT DELETE]                                    |
|                                                                                   |
|  /home/admin/kihei (Executable)                                                   |
|         │                                                                         |
|         ▼ (calls fallocate -l 1.5GB)                                              |
|  /home/admin/data/newdatafile ───► Fails with ENOSPC (Requires 1.5G, only 576M free)
+-----------------------------------------------------------------------------------+

=================================== REMEDIATION =====================================

+-----------------------------------+     +-----------------------------------+
|        Disk 1: /dev/nvme1n1       |     |        Disk 2: /dev/nvme2n1       |
|               (1.0 GiB)           |     |               (1.0 GiB)           |
+-----------------------------------+     +-----------------------------------+
                  │                                         │
                  └────────────────────┬────────────────────┘
                                       ▼
                  +--------------------------------─────────+
                  |            LVM Volume Group             |
                  |                "vg" (2.0 GiB)           |
                  +--------------------------------─────────+
                                       │
                                       ▼
                  +--------------------------------─────────+
                  |           LVM Logical Volume            |
                  |            "/dev/vg/lv" (2.0 GiB)       |
                  +--------------------------------─────────+
                                       │
                                       ▼ [mount /dev/vg/lv /home/admin/data]
+-----------------------------------------------------------------------------------+
|  /home/admin/data/               ► Now backed by 2.0 GiB dedicated LVM volume     |
|  /home/admin/data/newdatafile    ► Successfully allocates 1.5 GiB                 |
+-----------------------------------------------------------------------------------+

Detailed Failure Sequence

  1. /home/admin/kihei launched under user admin.
  2. The Go binary spawned /usr/bin/fallocate -l 1.5GB /home/admin/data/newdatafile.
  3. The kernel VFS routed write operations to /dev/nvme0n1p1 (the root filesystem mount point covering /home/admin/data).
  4. The ext4 filesystem driver checked free block bitmaps, found only 576 MiB of available blocks, and returned the ENOSPC error code.
  5. fallocate exited with status 1.
  6. kihei caught the non-zero child exit code, removed the zero-length target file, and triggered a Go runtime panic.

6. Solution and Remediation

Implemented Fix

Created a 2.0 GiB LVM Logical Volume combining physical drives /dev/nvme1n1 and /dev/nvme2n1, formatted it with ext4, mounted it directly to /home/admin/data, and assigned directory ownership to admin:admin.

Exact Commands Executed

sudo pvcreate /dev/nvme1n1 /dev/nvme2n1
sudo vgcreate vg /dev/nvme1n1 /dev/nvme2n1
sudo lvcreate -n lv -l 100%FREE vg
sudo mkfs.ext4 /dev/vg/lv
sudo mount /dev/vg/lv /home/admin/data
sudo chown -R admin: /home/admin/data

Key Considerations

  • Persistence: For this configuration to survive a system reboot, an entry must be added to /etc/fstab:
/dev/vg/lv    /home/admin/data    ext4    defaults    0  2
  • Permissions: When mkfs.ext4 builds a filesystem on a block device, the root directory of that new filesystem is created owned by root:root with permissions 0755. Failing to execute chown -R admin: causes non-root user applications to fail with EACCES (Permission denied).

7. Verification and Validation

Step 1: Verify Mount Point and Free Space

df -h /home/admin/data

Output

Filesystem      Size  Used Avail Use% Mounted on
/dev/mapper/vg-lv 2.0G   24K  1.9G   1% /home/admin/data

Step 2: Confirm Application Execution

./kihei

Output

Done.

Step 3: Validate File Preservation Constraint

ls -la /home/admin/datafile

Output

-rw-r--r-- 1 root root 5368709120 Dec 14  2025 /home/admin/datafile

All validation conditions were met: /home/admin/kihei completed with Done., /home/admin/datafile retained its original 5.1 GB size, and no existing data was removed.


8. Alternative Troubleshooting Approaches

Approach Method Advantages Disadvantages / Risks Production Suitability
1. LVM Storage Pooling (Applied) Combine unused block devices via LVM and mount over target directory. Zero disruption to root partition; clean architectural separation; preserves all existing files. Requires available unpartitioned block devices. High (Best Practice)
2. Sparse File Conversion Recreate /home/admin/datafile as a sparse file via truncate -s 5368709120. Instantly reclaims ~5.1 GB of real disk blocks without altering apparent file size reported by ls -la. Risky if automated validation scripts test actual file block allocation (du) or checksums. Medium
3. Directory Symlink Format a single disk (e.g., /dev/nvme1n1), mount to /mnt/storage, and symlink /home/admin/data -> /mnt/storage. Simple to set up without using LVM. Single 1 GB disk insufficient for 1.5 GB file requirement; symlinking requires multi-disk concatenation anyway. Low
4. Root Partition Resizing Grow root partition using cloud hypervisor tooling (aws ec2 modify-volume + resize2fs). Solves root storage shortage system-wide. Requires cloud provider control plane access not available inside the guest OS shell. High (Cloud Ops)

9. Linux Command Deep Dive

strace — System Call Tracer

Monitors interactions between user-space processes and the Linux kernel.

Syntax

strace [options] command [args]

Options Reference

Option Purpose Practical Example
-f Follow child threads and processes created by fork/clone. strace -f ./binary
-e trace=<syscall> Filter output to specific system calls (e.g., openat, write, fallocate). strace -e trace=file ./binary
-p <PID> Attach tracer to an already running process by Process ID. strace -p 1234
-o <filename> Write trace output to a file instead of standard error. strace -o trace.log ./binary
-c Count time, calls, and errors for each system call and print a summary table. strace -c ./binary

lsblk — List Block Devices

Queries the sysfs filesystem (/sys/block) to present information about block devices.

Syntax

lsblk [options] [device...]

Options Reference

Option Purpose Practical Example
-l Display output using flat list format instead of default tree format. lsblk -l
-f Display filesystem metadata (UUID, LABEL, FSTYPE, MOUNTPOINT). lsblk -f
-o <columns> Specify custom output columns (e.g., NAME,SIZE,FSTYPE,MOUNTPOINT). lsblk -o NAME,SIZE,TYPE,MOUNTPOINT

LVM Suite (pvcreate, vgcreate, lvcreate)

LVM Hierarchy Syntax

pvcreate <device1> <device2>...
vgcreate <vg_name> <pv_device1> <pv_device2>...
lvcreate -n <lv_name> -l <extents_percentage> <vg_name>

Options & Flags

Command Flag Meaning
lvcreate -n <name> Sets logical volume name (e.g., -n lv).
lvcreate -l 100%FREE Allocates 100% of unallocated Physical Extents in the VG.
lvcreate -L 1.5G Allocates explicit size (1.5 Gigabytes) instead of percentages.
vgdisplay -v Shows verbose metadata including Physical Extent (PE) size (default 4 MiB).

fallocate — Preallocate Space to a File

Manipulates file space directly at the filesystem block-allocation level without writing zero-bytes sequentially.

Syntax

fallocate -l <length> <filename>

Options Reference

Option Purpose Practical Example
-l <size> Specify length of allocation (e.g., 10M, 1.5G). fallocate -l 1.5G test.img
-d / --dig-holes Detect ranges of zeroes and release physical disk blocks (convert to sparse). fallocate -d sparse_file
-p / --punch-hole Deallocate space in a specific range within a file (requires -o and -l). fallocate -p -o 0 -l 100M file

10. Underlying Linux Concepts and Architecture

1. Logical Volume Manager (LVM) Architecture

LVM inserts a flexible virtual mapping layer between physical storage disk sectors and guest operating system filesystems:

+----------------------------------------------------------------+
| Filesystem (ext4, xfs)                                         |
+----------------------------------------------------------------+
                                │
                                ▼
+----------------------------------------------------------------+
| Logical Volume (LV)           e.g., /dev/vg/lv                 |
+----------------------------------------------------------------+
                                │
                                ▼
+----------------------------------------------------------------+
| Volume Group (VG)             e.g., "vg" (Storage Pool)        |
+----------------------------------------------------------------+
                                │
                 ┌──────────────┴──────────────┐
                 ▼                             ▼
+--------------------------------+ +--------------------------------+
| Physical Volume (PV)           | | Physical Volume (PV)           |
| /dev/nvme1n1                   | | /dev/nvme2n1                   |
+--------------------------------+ +--------------------------------+
  • Physical Extents (PE): The basic chunk size (typically 4 MiB) into which Physical Volumes are divided.
  • Logical Extents (LE): Mapped one-to-one to Physical Extents. An LV grows by linking additional LEs from any PV in the parent VG.

2. Virtual File System (VFS) Mount Point Overrides

The Linux kernel Virtual File System (VFS) handles all system calls related to files and directories (open, read, write).

When a block device is mounted onto an existing directory path (such as mounting /dev/vg/lv over /home/admin/data):

  1. The kernel creates a new mount entry in the mount table pointing the path inode to the superblock of the new filesystem.
  2. Any files previously residing in /home/admin/data on the parent filesystem become masked (hidden, but taking up space on the underlying parent filesystem).
  3. All new file IO operations under /home/admin/data are routed directly to the new block device drivers.

3. Sparse Files vs. Dense Files

  • Dense File: Every logical byte from offset 0 to N is backed by physical disk blocks allocated on the storage hardware.
  • Sparse File: Filesystem metadata records a large logical size, but contiguous sectors containing null bytes (0x00) are not allocated physical disk blocks. When read, the filesystem driver synthesizes null bytes on the fly.

11. Real-World Sysadmin and SRE Applications

Production Monitoring and Alerting

In production environments, disk space exhaustion should be caught long before an application panics:

  • Metric Collection: Monitor disk space using Prometheus node_exporter metrics (node_filesystem_free_bytes and node_filesystem_size_bytes).
  • Alerting Rules:
  • Warning: Disk free space < 15% for > 15 minutes.
  • Critical: Disk predict linear fill (predict_linear) reaching 0 bytes within 4 hours.

Preventing Partition Contention

  • Architectural Standard: Never allow user applications or database engines to write variable data directly to the OS root partition (/).
  • Isolation Pattern: Mount dedicated storage volumes for path targets prone to rapid expansion:
  • /var/log (System logs)
  • /var/lib/docker or /var/lib/containers (Container layers)
  • /var/lib/postgresql or /var/lib/mysql (Database state)

12. Further Learning and Experiments

Hands-on Exercise: Building an LVM Array Using Loop Devices

If you want to practice LVM management on any Linux machine without needing physical spare disks, use loopback devices:

# 1. Create two 500MB empty image files
dd if=/dev/zero of=/tmp/disk1.img bs=1M count=500
dd if=/dev/zero of=/tmp/disk2.img bs=1M count=500

# 2. Attach image files as virtual block devices
sudo losetup -fP /tmp/disk1.img
sudo losetup -fP /tmp/disk2.img

# 3. Identify attached loop devices (e.g., /dev/loop0, /dev/loop1)
losetup -a

# 4. Create LVM array
sudo pvcreate /dev/loop0 /dev/loop1
sudo vgcreate test_vg /dev/loop0 /dev/loop1
sudo lvcreate -n test_lv -l 100%FREE test_vg
sudo mkfs.ext4 /dev/test_vg/test_lv

# 5. Cleanup when finished
sudo lvremove -f /dev/test_vg/test_lv
sudo vgremove test_vg
sudo pvremove /dev/loop0 /dev/loop1
sudo losetup -d /dev/loop0 /dev/loop1
rm /tmp/disk1.img /tmp/disk2.img

13. Key Takeaways

  1. Use strace for Opaque Executables: Statically compiled binaries without verbose logging can be reverse-engineered by tracing system call executions (strace -f).
  2. Always Check for Unused Block Devices: When disk space on root is exhausted, run lsblk -l to check for attached, unpartitioned cloud block volumes.
  3. LVM Combines Storage Pool Capacity: LVM allows administrators to bridge multiple physical disks into a single continuous storage volume group.
  4. Target Mount Overrides Protect Root: Mounting a new storage volume directly over an application’s target sub-directory solves local disk capacity issues without reconfiguring the application binaries.
  5. Remember Mount Ownership: Newly formatted ext4 volumes default ownership to root:root. Always run chown to assign permissions to non-root application users.

14. Quick Troubleshooting Reference / Runbook

[Disk Space Failure Emergency Runbook]

1. IDENTIFY SYMPTOMS & ROOT CAUSE
   ├── Check root disk space:     df -h /
   ├── Check inode availability:  df -i /
   └── Trace application failure: strace -f -e trace=file <command>

2. DISCOVER UNALLOCATED STORAGE
   └── List all block devices:    lsblk -l

3. PROVISION LVM VOLUME (IF DISKS AVAILABLE)
   ├── Initialize PVs:            sudo pvcreate /dev/sdX /dev/sdY
   ├── Create Volume Group:       sudo vgcreate <vg_name> /dev/sdX /dev/sdY
   ├── Create Logical Volume:     sudo lvcreate -n <lv_name> -l 100%FREE <vg_name>
   └── Format Filesystem:         sudo mkfs.ext4 /dev/<vg_name>/<lv_name>

4. MOUNT AND SECURE
   ├── Mount to target directory: sudo mount /dev/<vg_name>/<lv_name> /path/to/dir
   ├── Assign user ownership:    sudo chown -R <user>:<group> /path/to/dir
   └── Persist in /etc/fstab:    echo "/dev/<vg_name>/<lv_name> /path/to/dir ext4 defaults 0 2" | sudo tee -a /etc/fstab

5. VERIFY RESOLUTION
   ├── Verify mount path:         df -h /path/to/dir
   └── Execute application:       <command>