1. Introduction
Storage management is a cornerstone of Linux system administration and Site Reliability Engineering (SRE). Out-of-space conditions (ENOSPC) remain one of the most common causes of unexpected service outages in production environments. However, resolving disk space constraints in production is rarely as simple as deleting large files—especially when those files belong to core databases, state stores, or strict application requirements.
In this technical breakdown, we walk through the resolution of the “Kihei” challenge from SadServers. The problem centers on a compiled Go application (/home/admin/kihei) that crashes on launch due to storage exhaustion, under the explicit constraint that an existing 5.1 GB data file (/home/admin/datafile) must not be deleted.
By reading this guide, you will learn:
- How to trace opaque compiled binary failures to their underlying system call failures using
straceand command-line flags. - How to inspect block storage topology using
lsblkand identify unallocated storage devices. - How to build an emergency storage array using the Logical Volume Manager (LVM) toolchain (
pvcreate,vgcreate,lvcreate). - How to utilize Virtual File System (VFS) mount point overrides to grant dedicated storage capacity to specific application directories without disrupting the root partition.
2. Problem Description and Environment
Scenario Specifications
- Scenario Name: “Kihei”: Surely Not Another Disk Space Scenario
- Platform: SadServers
- Difficulty: Medium
- Target Host OS: Debian 11 (Bullseye), Linux kernel 5.x, x86_64 architecture
- Privileges: Standard user
adminwith fullsudoaccess
Challenge Statement
There is a program at /home/admin/kihei. Make the changes necessary so it runs successfully and outputs Done., without deleting the /home/admin/datafile file.
Observed Failure
When attempting to execute /home/admin/kihei directly from the terminal, the application immediately panics and exits:
admin@ip-10-1-13-199:~$ /home/admin/kihei
panic: exit status 1
goroutine 1 [running]:
main.main()
./main.go:64 +0x47d
System Environment Constraints
- File Preservation:
/home/admin/datafile(size ~5.1 GB) cannot be unlinked or removed from the filesystem. - Privilege Boundary: The
kiheibinary checks effective user IDs at runtime and explicitly refuses to run underrootprivileges (UID 0).
3. Initial Investigation and Hypotheses
When facing an application crash without detailed error logging, systematically separate unverified hypotheses from confirmed system evidence.
Initial State Knowledge
- Known: The binary fails with exit code
1originating from an internal Go subprocess call (main.go:64). - Known:
/home/admin/datafileexists and takes up over 5 GB of disk space. - Unknown: What external command or system call
kiheiis invoking that leads toexit status 1. - Unknown: The target directory where
kiheiwrites its output.
Diagnostic Hypotheses
| Hypothesis | Supporting Evidence | Evidence Against | Diagnostic Step |
|---|---|---|---|
H1: Block Allocation Failure (ENOSPC) |
Root disk space is tight (~576 MB free); 5 GB file present on root partition. | None yet. | Run df -h and trace file write system calls via strace. |
| H2: Inode Exhaustion | Systems with many small files fail to create files despite free block space. | None yet. | Run df -i to check inode free counts on /. |
| H3: Permission / Ownership Error | Target path owned by root or another user. |
Binary executed in /home/admin/ where user admin usually has permissions. |
Check permissions of /home/admin/ subdirectories with ls -la. |
| H4: Missing Executable Dependency | Go os/exec panics if a binary called by exec.Command is missing from $PATH. |
Binary output specifies exit status 1 rather than executable not found error. |
Use strace -f to observe execve calls. |
4. Troubleshooting Process
Step 1: Binary Identification and Initial Inspection
Objective
Determine file types, ownership, permissions, and basic properties of the components in /home/admin/.
Command
ls -la /home/admin/
file /home/admin/kihei /home/admin/datafile
Command Explanation
ls -la: Lists all directory contents using the long format (-l), including hidden dotfiles (-a).file: Examines file headers and magic numbers to determine file format independently of file extension.
Output and Analysis
admin@ip-10-1-13-199:~$ ls -la /home/admin/
total 5245076
drwxr-xr-x 6 admin admin 4096 Dec 14 2025 .
drwxr-xr-x 3 root root 4096 Feb 17 2024 ..
drwxr-xr-x 2 admin root 4096 Dec 14 2025 agent
drwxr-xr-x 2 admin root 4096 Dec 14 2025 data
-rw-r--r-- 1 root root 5368709120 Dec 14 2025 datafile
-rwxr-xr-x 1 admin root 2207109 Dec 14 2025 kihei
admin@ip-10-1-13-199:~$ file /home/admin/kihei /home/admin/datafile
/home/admin/kihei: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=..., not stripped
/home/admin/datafile: data
Analysis
/home/admin/datafileis5,368,709,120bytes (~5.0 GiB / 5.37 GB) and owned byroot:root./home/admin/kiheiis a statically linked Go executable owned byadmin:rootwith executable permissions (-rwxr-xr-x).- Directory
/home/admin/dataexists and is owned byadmin:root.
Step 2: System Call Tracing and Option Discovery
Objective
Identify the exact subprocess or file operation failing inside /home/admin/kihei.
Commands
./kihei -v
strace -f /home/admin/kihei
Command Explanation
./kihei -v: Runs the binary with a common verbose flag to check if the application logs internal operations to standard output.strace: Intercepts and records system calls made by a process.-f: Follows child threads and subprocesses created byfork(),vfork(), orclone(). Crucial for Go applications which use lightweight OS threads and execute child binaries viaos/exec.
Output and Analysis
Running ./kihei -v:
admin@ip-10-1-13-32:~$ ./kihei -v
Creating file /home/admin/data/newdatafile with size 1.5GB...
panic: exit status 1
goroutine 1 [running]:
main.main()
./main.go:64 +0x47d
Tracing via strace -f /home/admin/kihei (relevant excerpt):
[pid 845] execve("/usr/bin/fallocate", ["fallocate", "-l", "1.5GB", "/home/admin/data/newdatafile"], ...) = 0
...
[pid 845] openat(AT_FDCWD, "/home/admin/data/newdatafile", O_RDWR|O_CREAT, 0666) = 3
[pid 845] fallocate(3, 0, 0, 1500000000) = -1 ENOSPC (No space left on device)
[pid 845] write(2, "fallocate: ", 11) = 11
[pid 845] write(2, "fallocate failed", 16) = 16
[pid 845] write(2, ": No space left on device\n", 26) = 26
[pid 845] exit_group(1) = ?
[pid 845] +++ exited with 1 +++
Analysis
kiheiexecutes/usr/bin/fallocate -l 1.5GB /home/admin/data/newdatafileas a child process.- The
fallocate()system call returns-1 ENOSPC (No space left on device). - The child process writes
fallocate failed: No space left on devicetostderrand exits with status code1. - Parent process
kiheicatches exit status1, unlinks the partial file/home/admin/data/newdatafile, and panics.
Step 3: Storage Audit and Disk Topology Discovery
Objective
Quantify available space on the root filesystem and identify alternative storage devices attached to the system.
Commands
df -h
df -i
du -h /home/admin/datafile
lsblk -l
Command Explanation
df -h: Displays available filesystem disk space in human-readable units (GiB, MiB).df -i: Displays inode utilization statistics per mounted filesystem.du -h: Estimates file space utilization on disk for the target path.lsblk -l: Lists all available block devices in a flat list layout (-l), showing device names, sizes, partition types, and mount points.
Output and Analysis
admin@ip-10-1-13-32:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 7.7G 6.7G 576M 93% /
udev 215M 0 215M 0% /dev
tmpfs 46M 396K 45M 1% /run
admin@ip-10-1-13-32:~$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/nvme0n1p1 516096 35263 480833 7% /
admin@ip-10-1-13-32:~$ du -h /home/admin/datafile
5.1G /home/admin/datafile
admin@ip-10-1-13-32:~$ lsblk -l
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 8G 0 disk
nvme2n1 259:1 0 1G 0 disk
nvme1n1 259:2 0 1G 0 disk
nvme0n1p1 259:3 0 7.9G 0 part /
nvme0n1p14 259:4 0 3M 0 part
nvme0n1p15 259:5 0 124M 0 part /boot/efi
Decision & Analysis
- Root Partition (
/dev/nvme0n1p1): Has only 576 MB free space remaining.kiheirequires 1.5 GB (1,500,000,000 bytes). - Inode Usage: Inodes are 93% free (
480,833free). Inodes are not the bottleneck. - Secondary Block Devices:
lsblk -lreveals two completely unused, unpartitioned block devices: /dev/nvme1n1(1.0 GiB)/dev/nvme2n1(1.0 GiB)
Neither disk alone can fit the required 1.5 GB file. However, combined using LVM (Logical Volume Manager), they provide 2.0 GiB of total storage capacity.
Step 4: LVM Array Provisioning and Filesystem Mounting
Objective
Aggregate /dev/nvme1n1 and /dev/nvme2n1 into a single Volume Group, create a Logical Volume spanning 100% of both drives, format it as ext4, and mount it directly over /home/admin/data.
Commands
sudo pvcreate /dev/nvme1n1 /dev/nvme2n1
sudo vgcreate vg /dev/nvme1n1 /dev/nvme2n1
sudo lvcreate -n lv -l 100%FREE vg
sudo mkfs.ext4 /dev/vg/lv
sudo mount /dev/vg/lv /home/admin/data
sudo chown -R admin: /home/admin/data
Command Explanation
pvcreate: Initializes physical block devices/dev/nvme1n1and/dev/nvme2n1as LVM Physical Volumes (PVs) by writing LVM metadata headers.vgcreate: Combines the specified PVs into a unified storage pool (Volume Group) namedvg.lvcreate: Carves out a Logical Volume (LV) namedlvfrom Volume Groupvg. The-l 100%FREEflag allocates all available Physical Extents (PEs) in the group.mkfs.ext4: Formats the newly created logical block device/dev/vg/lvwith anext4filesystem.mount: Attaches/dev/vg/lvto directory/home/admin/data. Any subsequent file operations inside/home/admin/databypass the root partition and write directly to the 2 GB LVM volume.chown -R admin:: Recursively changes ownership of/home/admin/data(including the filesystem root directory created bymkfs) to useradminand default groupadmin.
Output and Analysis
admin@ip-10-1-13-32:~$ sudo pvcreate /dev/nvme1n1 /dev/nvme2n1
Physical volume "/dev/nvme1n1" successfully created.
Physical volume "/dev/nvme2n1" successfully created.
admin@ip-10-1-13-32:~$ sudo vgcreate vg /dev/nvme1n1 /dev/nvme2n1
Volume group "vg" successfully created
admin@ip-10-1-13-32:~$ sudo lvcreate -n lv -l 100%FREE vg
Logical volume "lv" created.
admin@ip-10-1-13-32:~$ sudo mkfs.ext4 /dev/vg/lv
mke2fs 1.46.2 (28-Feb-2021)
Creating filesystem with 522240 4k blocks and 130560 inodes
Filesystem UUID: e4bb8c46-e037-42fd-8c5f-0aed748b68e4
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
admin@ip-10-1-13-32:~$ sudo mount /dev/vg/lv /home/admin/data
admin@ip-10-1-13-32:~$ sudo chown -R admin: /home/admin/data
Step 5: Privilege Validation and Final Execution
Objective
Verify program execution under correct user privileges.
Commands
sudo ./kihei
./kihei
Output and Analysis
admin@ip-10-1-13-32:~$ sudo ./kihei
Error: This program cannot be run as the 'root' superuser.
admin@ip-10-1-13-32:~$ ./kihei
Done.
Result
- Running
sudo ./kiheifails gracefully due to an explicit internal application check preventing execution asUID 0. - Running
./kiheias useradminsuccessfully allocates 1.5 GB in/home/admin/data/newdatafile, completes internal checks, and printsDone..
5. Root Cause Analysis
The application failure was caused by storage allocation failure on the root partition, compounded by directory mount boundaries.
+-----------------------------------------------------------------------------------+
| ROOT FILESYSTEM (/) |
| /dev/nvme0n1p1 |
| (Total: 7.7G, Avail: 576M) |
| |
| /home/admin/datafile (5.1GB) [DO NOT DELETE] |
| |
| /home/admin/kihei (Executable) |
| │ |
| ▼ (calls fallocate -l 1.5GB) |
| /home/admin/data/newdatafile ───► Fails with ENOSPC (Requires 1.5G, only 576M free)
+-----------------------------------------------------------------------------------+
=================================== REMEDIATION =====================================
+-----------------------------------+ +-----------------------------------+
| Disk 1: /dev/nvme1n1 | | Disk 2: /dev/nvme2n1 |
| (1.0 GiB) | | (1.0 GiB) |
+-----------------------------------+ +-----------------------------------+
│ │
└────────────────────┬────────────────────┘
▼
+--------------------------------─────────+
| LVM Volume Group |
| "vg" (2.0 GiB) |
+--------------------------------─────────+
│
▼
+--------------------------------─────────+
| LVM Logical Volume |
| "/dev/vg/lv" (2.0 GiB) |
+--------------------------------─────────+
│
▼ [mount /dev/vg/lv /home/admin/data]
+-----------------------------------------------------------------------------------+
| /home/admin/data/ ► Now backed by 2.0 GiB dedicated LVM volume |
| /home/admin/data/newdatafile ► Successfully allocates 1.5 GiB |
+-----------------------------------------------------------------------------------+
Detailed Failure Sequence
/home/admin/kiheilaunched under useradmin.- The Go binary spawned
/usr/bin/fallocate -l 1.5GB /home/admin/data/newdatafile. - The kernel VFS routed write operations to
/dev/nvme0n1p1(the root filesystem mount point covering/home/admin/data). - The ext4 filesystem driver checked free block bitmaps, found only 576 MiB of available blocks, and returned the
ENOSPCerror code. fallocateexited with status1.kiheicaught the non-zero child exit code, removed the zero-length target file, and triggered a Go runtime panic.
6. Solution and Remediation
Implemented Fix
Created a 2.0 GiB LVM Logical Volume combining physical drives /dev/nvme1n1 and /dev/nvme2n1, formatted it with ext4, mounted it directly to /home/admin/data, and assigned directory ownership to admin:admin.
Exact Commands Executed
sudo pvcreate /dev/nvme1n1 /dev/nvme2n1
sudo vgcreate vg /dev/nvme1n1 /dev/nvme2n1
sudo lvcreate -n lv -l 100%FREE vg
sudo mkfs.ext4 /dev/vg/lv
sudo mount /dev/vg/lv /home/admin/data
sudo chown -R admin: /home/admin/data
Key Considerations
- Persistence: For this configuration to survive a system reboot, an entry must be added to
/etc/fstab:
/dev/vg/lv /home/admin/data ext4 defaults 0 2
- Permissions: When
mkfs.ext4builds a filesystem on a block device, the root directory of that new filesystem is created owned byroot:rootwith permissions0755. Failing to executechown -R admin:causes non-root user applications to fail withEACCES (Permission denied).
7. Verification and Validation
Step 1: Verify Mount Point and Free Space
df -h /home/admin/data
Output
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg-lv 2.0G 24K 1.9G 1% /home/admin/data
Step 2: Confirm Application Execution
./kihei
Output
Done.
Step 3: Validate File Preservation Constraint
ls -la /home/admin/datafile
Output
-rw-r--r-- 1 root root 5368709120 Dec 14 2025 /home/admin/datafile
All validation conditions were met: /home/admin/kihei completed with Done., /home/admin/datafile retained its original 5.1 GB size, and no existing data was removed.
8. Alternative Troubleshooting Approaches
| Approach | Method | Advantages | Disadvantages / Risks | Production Suitability |
|---|---|---|---|---|
| 1. LVM Storage Pooling (Applied) | Combine unused block devices via LVM and mount over target directory. | Zero disruption to root partition; clean architectural separation; preserves all existing files. | Requires available unpartitioned block devices. | High (Best Practice) |
| 2. Sparse File Conversion | Recreate /home/admin/datafile as a sparse file via truncate -s 5368709120. |
Instantly reclaims ~5.1 GB of real disk blocks without altering apparent file size reported by ls -la. |
Risky if automated validation scripts test actual file block allocation (du) or checksums. |
Medium |
| 3. Directory Symlink | Format a single disk (e.g., /dev/nvme1n1), mount to /mnt/storage, and symlink /home/admin/data -> /mnt/storage. |
Simple to set up without using LVM. | Single 1 GB disk insufficient for 1.5 GB file requirement; symlinking requires multi-disk concatenation anyway. | Low |
| 4. Root Partition Resizing | Grow root partition using cloud hypervisor tooling (aws ec2 modify-volume + resize2fs). |
Solves root storage shortage system-wide. | Requires cloud provider control plane access not available inside the guest OS shell. | High (Cloud Ops) |
9. Linux Command Deep Dive
strace — System Call Tracer
Monitors interactions between user-space processes and the Linux kernel.
Syntax
strace [options] command [args]
Options Reference
| Option | Purpose | Practical Example |
|---|---|---|
-f |
Follow child threads and processes created by fork/clone. |
strace -f ./binary |
-e trace=<syscall> |
Filter output to specific system calls (e.g., openat, write, fallocate). |
strace -e trace=file ./binary |
-p <PID> |
Attach tracer to an already running process by Process ID. | strace -p 1234 |
-o <filename> |
Write trace output to a file instead of standard error. | strace -o trace.log ./binary |
-c |
Count time, calls, and errors for each system call and print a summary table. | strace -c ./binary |
lsblk — List Block Devices
Queries the sysfs filesystem (/sys/block) to present information about block devices.
Syntax
lsblk [options] [device...]
Options Reference
| Option | Purpose | Practical Example |
|---|---|---|
-l |
Display output using flat list format instead of default tree format. | lsblk -l |
-f |
Display filesystem metadata (UUID, LABEL, FSTYPE, MOUNTPOINT). | lsblk -f |
-o <columns> |
Specify custom output columns (e.g., NAME,SIZE,FSTYPE,MOUNTPOINT). |
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT |
LVM Suite (pvcreate, vgcreate, lvcreate)
LVM Hierarchy Syntax
pvcreate <device1> <device2>...
vgcreate <vg_name> <pv_device1> <pv_device2>...
lvcreate -n <lv_name> -l <extents_percentage> <vg_name>
Options & Flags
| Command | Flag | Meaning |
|---|---|---|
lvcreate |
-n <name> |
Sets logical volume name (e.g., -n lv). |
lvcreate |
-l 100%FREE |
Allocates 100% of unallocated Physical Extents in the VG. |
lvcreate |
-L 1.5G |
Allocates explicit size (1.5 Gigabytes) instead of percentages. |
vgdisplay |
-v |
Shows verbose metadata including Physical Extent (PE) size (default 4 MiB). |
fallocate — Preallocate Space to a File
Manipulates file space directly at the filesystem block-allocation level without writing zero-bytes sequentially.
Syntax
fallocate -l <length> <filename>
Options Reference
| Option | Purpose | Practical Example |
|---|---|---|
-l <size> |
Specify length of allocation (e.g., 10M, 1.5G). |
fallocate -l 1.5G test.img |
-d / --dig-holes |
Detect ranges of zeroes and release physical disk blocks (convert to sparse). | fallocate -d sparse_file |
-p / --punch-hole |
Deallocate space in a specific range within a file (requires -o and -l). |
fallocate -p -o 0 -l 100M file |
10. Underlying Linux Concepts and Architecture
1. Logical Volume Manager (LVM) Architecture
LVM inserts a flexible virtual mapping layer between physical storage disk sectors and guest operating system filesystems:
+----------------------------------------------------------------+
| Filesystem (ext4, xfs) |
+----------------------------------------------------------------+
│
▼
+----------------------------------------------------------------+
| Logical Volume (LV) e.g., /dev/vg/lv |
+----------------------------------------------------------------+
│
▼
+----------------------------------------------------------------+
| Volume Group (VG) e.g., "vg" (Storage Pool) |
+----------------------------------------------------------------+
│
┌──────────────┴──────────────┐
▼ ▼
+--------------------------------+ +--------------------------------+
| Physical Volume (PV) | | Physical Volume (PV) |
| /dev/nvme1n1 | | /dev/nvme2n1 |
+--------------------------------+ +--------------------------------+
- Physical Extents (PE): The basic chunk size (typically 4 MiB) into which Physical Volumes are divided.
- Logical Extents (LE): Mapped one-to-one to Physical Extents. An LV grows by linking additional LEs from any PV in the parent VG.
2. Virtual File System (VFS) Mount Point Overrides
The Linux kernel Virtual File System (VFS) handles all system calls related to files and directories (open, read, write).
When a block device is mounted onto an existing directory path (such as mounting /dev/vg/lv over /home/admin/data):
- The kernel creates a new mount entry in the mount table pointing the path inode to the superblock of the new filesystem.
- Any files previously residing in
/home/admin/dataon the parent filesystem become masked (hidden, but taking up space on the underlying parent filesystem). - All new file IO operations under
/home/admin/dataare routed directly to the new block device drivers.
3. Sparse Files vs. Dense Files
- Dense File: Every logical byte from offset
0toNis backed by physical disk blocks allocated on the storage hardware. - Sparse File: Filesystem metadata records a large logical size, but contiguous sectors containing null bytes (
0x00) are not allocated physical disk blocks. When read, the filesystem driver synthesizes null bytes on the fly.
11. Real-World Sysadmin and SRE Applications
Production Monitoring and Alerting
In production environments, disk space exhaustion should be caught long before an application panics:
- Metric Collection: Monitor disk space using Prometheus
node_exportermetrics (node_filesystem_free_bytesandnode_filesystem_size_bytes). - Alerting Rules:
- Warning: Disk free space
< 15%for > 15 minutes. - Critical: Disk predict linear fill (
predict_linear) reaching 0 bytes within 4 hours.
Preventing Partition Contention
- Architectural Standard: Never allow user applications or database engines to write variable data directly to the OS root partition (
/). - Isolation Pattern: Mount dedicated storage volumes for path targets prone to rapid expansion:
/var/log(System logs)/var/lib/dockeror/var/lib/containers(Container layers)/var/lib/postgresqlor/var/lib/mysql(Database state)
12. Further Learning and Experiments
Hands-on Exercise: Building an LVM Array Using Loop Devices
If you want to practice LVM management on any Linux machine without needing physical spare disks, use loopback devices:
# 1. Create two 500MB empty image files
dd if=/dev/zero of=/tmp/disk1.img bs=1M count=500
dd if=/dev/zero of=/tmp/disk2.img bs=1M count=500
# 2. Attach image files as virtual block devices
sudo losetup -fP /tmp/disk1.img
sudo losetup -fP /tmp/disk2.img
# 3. Identify attached loop devices (e.g., /dev/loop0, /dev/loop1)
losetup -a
# 4. Create LVM array
sudo pvcreate /dev/loop0 /dev/loop1
sudo vgcreate test_vg /dev/loop0 /dev/loop1
sudo lvcreate -n test_lv -l 100%FREE test_vg
sudo mkfs.ext4 /dev/test_vg/test_lv
# 5. Cleanup when finished
sudo lvremove -f /dev/test_vg/test_lv
sudo vgremove test_vg
sudo pvremove /dev/loop0 /dev/loop1
sudo losetup -d /dev/loop0 /dev/loop1
rm /tmp/disk1.img /tmp/disk2.img
13. Key Takeaways
- Use
stracefor Opaque Executables: Statically compiled binaries without verbose logging can be reverse-engineered by tracing system call executions (strace -f). - Always Check for Unused Block Devices: When disk space on root is exhausted, run
lsblk -lto check for attached, unpartitioned cloud block volumes. - LVM Combines Storage Pool Capacity: LVM allows administrators to bridge multiple physical disks into a single continuous storage volume group.
- Target Mount Overrides Protect Root: Mounting a new storage volume directly over an application’s target sub-directory solves local disk capacity issues without reconfiguring the application binaries.
- Remember Mount Ownership: Newly formatted ext4 volumes default ownership to
root:root. Always runchownto assign permissions to non-root application users.
14. Quick Troubleshooting Reference / Runbook
[Disk Space Failure Emergency Runbook]
1. IDENTIFY SYMPTOMS & ROOT CAUSE
├── Check root disk space: df -h /
├── Check inode availability: df -i /
└── Trace application failure: strace -f -e trace=file <command>
2. DISCOVER UNALLOCATED STORAGE
└── List all block devices: lsblk -l
3. PROVISION LVM VOLUME (IF DISKS AVAILABLE)
├── Initialize PVs: sudo pvcreate /dev/sdX /dev/sdY
├── Create Volume Group: sudo vgcreate <vg_name> /dev/sdX /dev/sdY
├── Create Logical Volume: sudo lvcreate -n <lv_name> -l 100%FREE <vg_name>
└── Format Filesystem: sudo mkfs.ext4 /dev/<vg_name>/<lv_name>
4. MOUNT AND SECURE
├── Mount to target directory: sudo mount /dev/<vg_name>/<lv_name> /path/to/dir
├── Assign user ownership: sudo chown -R <user>:<group> /path/to/dir
└── Persist in /etc/fstab: echo "/dev/<vg_name>/<lv_name> /path/to/dir ext4 defaults 0 2" | sudo tee -a /etc/fstab
5. VERIFY RESOLUTION
├── Verify mount path: df -h /path/to/dir
└── Execute application: <command>