1. Overview
In Linux system administration and Site Reliability Engineering (SRE), block storage management is a foundational skill. Storage under Linux is abstracted by the kernel into block devices, which are exposed as specialized device nodes in the /dev/ hierarchy. These nodes include traditional SCSI/SATA drives (/dev/sda, /dev/sdb), NVMe drives (/dev/nvme0n1), and virtualized cloud disks (/dev/vda).
A raw storage device can be used in its entirety or divided into discrete regions known as partitions. To store data persistently in an organized manner, a partition or raw device must be formatted with a filesystem—such as ext4, xfs, or btrfs. Once formatted, the filesystem is mounted onto a directory target within the Linux unified directory tree (e.g., /mnt, /var, /data). This directory serves as the access point for processes to read and write files.
Physical / Cloud Disk --> Partition / LVM --> Filesystem --> Mount Point Directory
(/dev/nvme0n1) (/dev/nvme0n1p1) (ext4/xfs) (/var)
Purpose and Common Use Cases
- Storage Isolation: Separating system critical directories (
/,/var,/home,/tmp) onto distinct filesystems prevents rogue application logging or user downloads from depleting root partition capacity. - Dynamic Volume Expansion: Utilizing Logical Volume Management (LVM) and scalable filesystems allows operations teams to expand storage online in cloud or enterprise server environments without service downtime.
- Optimized I/O Performance: Matching workload requirements with filesystem capabilities (e.g., selecting XFS for enterprise database systems with heavy parallel write workloads) improves throughput and lowers latency.
Filesystem Comparison: ext4 vs XFS
Choosing between ext4 and XFS depends on distribution standards, workload patterns, and maintenance requirements.
| Feature / Trait | ext4 |
XFS |
|---|---|---|
| Default OS Adoption | Default on Debian and Ubuntu distributions. | Default on RHEL, CentOS, Rocky Linux, and AlmaLinux. |
| Workload Suitability | General-purpose web application servers, small-to-medium databases. | Large storage volumes, enterprise databases, heavy parallel I/O. |
| Capacity Growth | Supports online expansion via resize2fs. |
Supports online expansion via xfs_growfs. |
| Capacity Shrinking | Supports offline volume shrinking. | Cannot be shrunk; requires re-creation to reduce size. |
| Grow Command Format | Target specified by device path (e.g., resize2fs /dev/vg/lv). |
Target specified by mount point (e.g., xfs_growfs /mnt/data). |
2. Architecture and Mental Model
To diagnose storage failures effectively, administrators must reason through the Linux storage stack as a series of layered abstractions.
graph TD
A[Physical / Virtual Disk Device<br/>/dev/nvme0n1 or /dev/sda] --> B[Partitioning Layer<br/>MBR or GPT]
B --> C[Physical Volume - PV<br/>pvcreate /dev/nvme0n1p1]
C --> D[Volume Group - VG<br/>vgcreate my_vg /dev/nvme0n1p1]
D --> E[Logical Volume - LV<br/>lvcreate -L 50G -n my_lv my_vg]
E --> F[Filesystem Layer<br/>mkfs.ext4 or mkfs.xfs]
F --> G[Virtual Filesystem VFS & Mount Point<br/>/etc/fstab -> /var/log]
Storage Stack Layer Responsibilities
- Physical/Virtual Disk Layer (
/dev/sdX,/dev/nvmeXn1,/dev/vdX): The raw hardware or cloud block device presented by the hypervisor or storage controller. - Partitioning Layer (MBR / GPT): Disk metadata defining sub-regions on the device.
- Logical Volume Manager (LVM) Layer:
- Physical Volume (PV): Initializes block partitions or whole disks for use by LVM.
- Volume Group (VG): Aggregates multiple PVs into a single pool of storage allocation units.
- Logical Volume (LV): Allocates blocks from a VG to create virtualized block devices.
- Filesystem Layer (
ext4,XFS): On-disk structural metadata managing directories, files, block pointers, permissions, and free space allocations. - VFS and Mount Point Layer (
/etc/fstab, Mount System): Kernel subsystem mapping filesystem paths to physical block devices.
File Access Control and Lifecycle Flow
When a process initiates an I/O operation (e.g., writing to /var/log/syslog):
- Directory Lookup: The kernel VFS resolves
/var/log/syslogto an inode number on the target filesystem. - Inode Inspection: The kernel retrieves the inode metadata to verify write permissions, lock state, and allocated data block addresses.
- Block Translation: The filesystem translates file offset addresses to Logical Volume block offsets.
- Volume Mapping: LVM maps the LV block offsets to specific physical disk sectors across underlying Physical Volumes.
- Hardware Execution: The Linux block layer driver dispatches I/O commands to the physical disk controller.
3. Core Concepts and Terminology
Block Devices
A block device is a hardware or software abstraction that reads and writes data in fixed-size blocks (typically 512 bytes or 4096 bytes). Block devices support random access and buffered I/O via the kernel page cache.
Partition Tables: MBR vs GPT
Partition tables define how physical disk space is split into logical boundaries.
-
MBR (Master Boot Record): Legacy standard created in the DOS era.
-
Limits disk layouts to a maximum of 4 primary partitions, or 3 primary partitions plus 1 extended partition (which can host multiple logical partitions).
-
Maximum addressable partition size is capped at 2 TB.
-
GPT (GUID Partition Table): Modern standard for UEFI and cloud systems.
-
Supports 128 primary partitions by default without requiring extended/logical partitions.
-
Addresses storage up to 9.4 Zettabytes (virtually unlimited for modern applications).
Inodes (Index Nodes)
An inode is a filesystem metadata structure containing file attribute information, including file size, ownership (UID/GID), access permissions, creation/modification timestamps, link count, and pointers to data blocks on disk.
+-------------------------------------------------------------+
| INODE STRUCT |
+-------------------------------------------------------------+
| File Size: 2841 bytes | Ownership: UID 0 / GID 0 |
| Permissions: 0644 (-rw-r--r--) | Access/Modify Timestamps |
| Link Count: 1 | Data Block Pointers -> [...]|
+-------------------------------------------------------------+
- Filename Isolation: Filenames are stored in directory tables, not inside the inode itself.
- Fixed Limits: Filesystems allocate a fixed number of inodes upon creation. A filesystem can run out of available inodes even when significant disk space remains if millions of tiny files exhaust the inode table.
Logical Volume Management (LVM)
LVM decouples filesystems from physical hardware layouts:
- Physical Volume (PV): Raw disk or partition formatted for LVM inclusion.
- Volume Group (VG): Storage pool composed of one or more PVs.
- Logical Volume (LV): Virtual block partition allocated from a VG upon which filesystems reside.
Bind Mounts
A bind mount maps an existing directory tree to another access point in the filesystem hierarchy, making the contents accessible simultaneously from both paths.
Swap Space
Swap space acts as virtual memory overflow when physical memory (RAM) is exhausted. Low-memory conditions push idle memory pages out to a designated swap partition or swap file.
4. Installation and Version Management
Storage drivers and management utilities are standard across Linux distributions. Ensure core administration packages are available using your package manager.
Package Installation Across Distributions
- Debian / Ubuntu:
sudo apt update
sudo apt install -y lvm2 e2fsprogs xfsprogs gdisk parted cloud-utils
- RHEL / Rocky Linux / AlmaLinux:
sudo dnf install -y lvm2 e2fsprogs xfsprogs gdisk parted cloud-utils-growpart
Tool Version Verification
To verify utility installations and check capabilities:
# Verify LVM utility versions
pvs --version
# Check ext4 tools version
tune2fs -V
# Check XFS tools version
xfs_info -V
5. Configuration Files and Directory Structure
Linux storage configuration relies on specific configuration files, system paths, and virtual filesystem locations.
| File / Directory Path | Purpose | Important Details |
|---|---|---|
/etc/fstab |
Persistent filesystem mount table. | Evaluated during system boot; controls device mounting, mount options, and fsck checks. |
/etc/fstab.d/ |
Supplementary directory for modular mount configurations. | Read by systemd mount generators during boot. |
/dev/ |
Device node virtual directory. | Managed by udev; populates paths such as /dev/sda, /dev/nvme0n1, /dev/mapper/. |
/dev/disk/by-uuid/ |
Persistent device symlinks categorized by UUID. | Preferred source for fstab definitions to prevent device name drifting. |
/dev/mapper/ |
Symlink path for LVM logical volumes and encrypted volumes. | Contains device-mapper links (e.g., /dev/mapper/vg01-lv_data). |
/proc/mounts |
Kernel runtime mount table. | Real-time view of currently mounted filesystems maintained by the kernel. |
/sys/block/ |
Kernel sysfs directory exposing block device attributes. | Used to trigger bus rescans (e.g., echo 1 > /sys/block/sdb/device/rescan). |
The /etc/fstab Field Structure
Each line in /etc/fstab contains six distinct fields separated by whitespace:
# Field 1 Field 2 Field 3 Field 4 Field 5 Field 6
# Device Identifier Mount Point FS Type Mount Options Dump Fsck Pass
UUID=a1b2c3d4-e5f6-7890 /mnt/data ext4 defaults,nofail 0 2
/dev/mapper/vg01-lv01 /var/app xfs defaults 0 0
- Device Identifier: Device UUID (
UUID=...), device path (/dev/mapper/...), or filesystem label (LABEL=...). - Mount Point: Directory path where the filesystem will be attached.
- Filesystem Type:
ext4,xfs,btrfs,vfat,swap, orauto. - Mount Options: Comma-separated settings (
defaults,ro,rw,nofail,noexec,nosuid,nodev). - Dump Flag: Backup utility flag (
0disables dump backups; legacy field). - Fsck Order: Filesystem check order at boot (
1for root/,2for secondary filesystems,0to disable boot checks).
6. Service Management and Common Operations
Filesystems do not run as standalone background daemons; they are managed directly via kernel mount operations and systemd mount units.
Common Mount Operations
- Mounting a Device:
sudo mount /dev/sdb1 /mnt/data
- Mounting via UUID:
sudo mount UUID=a1b2c3d4-e5f6-7890 /mnt/data
- Unmounting a Filesystem:
sudo umount /mnt/data
- Testing All
/etc/fstabMounts:
sudo mount -a
Executes mount for all entries in /etc/fstab. Essential after modifying /etc/fstab to verify syntax before rebooting.
- Remounting Root Filesystem Read-Write:
sudo mount -o remount,rw /
Used during recovery when the root filesystem falls back to read-only mode.
7. Common Administrative Tasks and Workflows
Workflow 1: Initializing a New LVM Storage Volume from Scratch
Objective: Configure a newly attached raw disk (/dev/sdb) into an LVM stack with an ext4 filesystem mounted at /data.
- Initialize Physical Volume (PV):
sudo pvcreate /dev/sdb
- Create Volume Group (VG):
sudo vgcreate vg_data /dev/sdb
- Allocate Logical Volume (LV):
sudo lvcreate -l 100%FREE -n lv_data vg_data
- Format with ext4 Filesystem:
sudo mkfs.ext4 /dev/vg_data/lv_data
- Configure Persistent Mount: Retrieve the UUID:
blkid /dev/vg_data/lv_data
Append to /etc/fstab:
UUID=e4f3a210-9b8c-7d6e-5f4a-3b2c1d0e9f8a /data ext4 defaults,nofail 0 2
- Mount and Validate:
sudo mkdir -p /data
sudo mount -a
df -hT /data
Workflow 2: Expanding a Storage Volume in Cloud/Virtual Environments
Expanding virtual cloud disks or SAN LUNs requires updating all 4 layers of the storage hierarchy sequentially.
Step 1: Partition Resize (growpart)
Step 2: LVM Physical Volume Resize (pvresize)
Step 3: LVM Logical Volume Resize (lvextend)
Step 4: Filesystem Resize (resize2fs or xfs_growfs)
# Layer 1: Expand partition boundary to consume new raw cloud disk space
sudo growpart /dev/nvme0n1 1
# Layer 2: Update LVM Physical Volume metadata
sudo pvresize /dev/nvme0n1p1
# Layer 3: Extend Logical Volume to absorb free space in Volume Group
sudo lvextend -l +100%FREE /dev/myvg/mylv
# Layer 4: Expand the filesystem online
# Option A: For ext4 filesystems (takes device path)
sudo resize2fs /dev/myvg/mylv
# Option B: For XFS filesystems (requires MOUNT POINT target)
sudo xfs_growfs /mount/point
Workflow 3: Resolving Duplicate UUID Conflicts on Cloned Volumes
When cloning disk snapshots or SAN volumes, mounting the cloned block device alongside the source device triggers a UUID collision failure.
- Mount Clone Temporarily Disabling UUID Checks (XFS):
sudo mount -o nouuid /dev/nvme2n1p1 /mnt/clone
- Unmount cleanly to reset file journal states:
sudo umount /mnt/clone
- Generate a New Random Filesystem UUID:
- For XFS Filesystems:
sudo xfs_admin -U generate /dev/nvme2n1p1
- For ext2/ext3/ext4 Filesystems:
sudo tune2fs -U random /dev/nvme2n1p1
- Verify Updated Device UUID:
blkid | grep nvme2
- Mount volume persistently:
Update
/etc/fstabwith the new UUID and runsudo mount -a.
8. CLI Command Reference and Cheatsheet
Inspection and Information Gathering
| Command | Purpose | Important Flags/Arguments | Example | Explanation |
|---|---|---|---|---|
df |
Report filesystem disk space usage. | -h (human-readable), -T (show filesystem type), -i (show inode usage). |
df -hT |
Displays total, used, and available space alongside filesystem types across all mounts. |
lsblk |
List block devices in tree format. | -f (output filesystem details and UUIDs). |
lsblk -f |
Displays parent-child mappings between disks, partitions, LVM layers, and mount points. |
blkid |
Locate/print block device attributes. | None / Device target. | blkid /dev/sdb1 |
Prints filesystem type, UUID, and volume label for target block devices. |
du |
Estimate file space usage. | -s (summary), -h (human-readable), -x (stay on one filesystem), -d1 (max depth 1). |
du -xhd1 /var |
Calculates space consumed by subdirectories on the local mount without crossing onto other mounts. |
stat |
Display file or filesystem status. | None / Path. | stat /etc/passwd |
Reports size, block count, ownership, permissions, and specific inode number. |
findmnt |
Find a filesystem mount point. | --verify (verify fstab syntax). |
findmnt --verify |
Parses /etc/fstab for configuration errors before rebooting. |
Partitioning and Formatting
| Command | Purpose | Important Flags/Arguments | Example | Explanation |
|---|---|---|---|---|
fdisk |
Manipulate disk partition tables (MBR/GPT). | -l (list partition layout). |
sudo fdisk -l /dev/sda |
Interactive disk partitioning utility for inspecting or modifying partition tables. |
parted |
Partition manipulation tool for large disks. | print (display table). |
sudo parted /dev/sda print |
Partitioning tool capable of handling GPT structures and large (>2TB) disk volumes. |
sfdisk |
Scriptable partition table editor. | -d (dump partition layout). |
sfdisk -d /dev/sda > backup.txt |
Dumps partition layout to a text file for backup/restoration. |
mkfs.ext4 |
Build an ext4 filesystem. | Target device block. | sudo mkfs.ext4 /dev/sdb1 |
Formats target partition with an ext4 filesystem layout. |
mkfs.xfs |
Build an XFS filesystem. | Target device block. | sudo mkfs.xfs /dev/mapper/vg-lv |
Formats target device with an XFS filesystem layout. |
fsck |
Check and repair a Linux filesystem. | -f (force check). |
sudo fsck -f /dev/sdb1 |
Scans and repairs file system corruption. Must only be run on unmounted filesystems. |
LVM Management
| Command | Purpose | Important Flags/Arguments | Example | Explanation |
|---|---|---|---|---|
pvs / vgs / lvs |
Display summary reports of LVM objects. | None. | sudo lvs |
Prints single-line operational status summaries for Physical Volumes, VGs, and LVs. |
pvcreate |
Initialize disk/partition for LVM usage. | Target device. | sudo pvcreate /dev/sdb |
Writes LVM metadata headers to raw device target. |
vgextend |
Add Physical Volume to Volume Group. | VG name, PV path. | sudo vgextend myvg /dev/sdb |
Expands storage pool capacity by adding a new PV. |
lvextend |
Extend Logical Volume capacity. | -L (size delta), -l (percentage/extents). |
sudo lvextend -l +100%FREE /dev/vg/lv |
Increases LV size using available extents in the Volume Group. |
9. Logs, Monitoring and Diagnostics
Effective storage administration requires inspecting kernel messages, monitoring resource constraints, and identifying process usage patterns.
System Diagnostics and Event Inspection
- Kernel Storage Error Inspection:
sudo dmesg | tail -n 30
Looks for hardware block errors, disk I/O timeouts, or automatic read-only filesystem protection triggers.
- Boot Mount Failures:
sudo journalctl -b | grep -i mount
Filters system logs for missing mount targets, fstab syntax errors, or UUID lookup timeouts.
- Checking Journal Storage Footprint:
sudo journalctl --disk-usage
Investigating Capacity and Inode Exhaustion
Always check both total disk space (df -hT) and inode allocation (df -i) when troubleshooting storage issues.
# Check block space usage
df -hT
# Check inode percentage allocation
df -i
Finding Open Files Held by Unlinked Processes
When a file is deleted while still open by an active process, its directory entry is removed, but its allocated disk blocks are not freed until the process terminates. Standard du commands will not account for this space, leading to discrepancies with df.
# Identify deleted files still held open by processes
sudo lsof +L1
Output interpretation:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
nginx 14201 www-data 4u REG 253,1 524288000 0 13134 /var/log/nginx/access.log (deleted)
A process with PID 14201 holds a 500MB deleted file open. Restarting or terminating PID 14201 releases the space immediately.
10. Troubleshooting Guide and Common Failure Scenarios
Problem 1: Partition Table Corrupted or Missing
Symptoms
fdisk -l /dev/sda shows incorrect drive boundaries, or parted reports unrecognized disk label. Applications cannot access underlying partitions.
Possible Causes
- Accidental overwrite of sector 0 (MBR) or primary GPT headers during manual block operations (
dd). - Abrupt node crash or hardware block failure during partitioning operations.
Investigation and Diagnosis
- Step 1: Perform immediate layout backup:
sudo sfdisk -d /dev/sda > sda-backup.txt
Safeguards current disk structures before running recovery commands. 2. Step 2: Check GPT secondary header integrity:
sudo parted /dev/sda print
Inspect whether parted detects the backup GPT header at the end of the disk volume.
Root Cause Identification
If primary GPT headers are missing but secondary backup headers are intact, restoring the backup header will recover partition layouts.
Resolution
- For GPT Disks:
Launch
gdiskto restore from the backup header:
sudo gdisk /dev/sda
# In gdisk prompt:
# Press 'r' (Recovery & transformation options)
# Press 'b' (Use backup GPT header)
# Press 'w' (Write table to disk and exit)
- For MBR Disks:
Use
testdiskto scan for lost partition boundaries:
sudo apt install -y testdisk
sudo testdisk /dev/sda
Verification
Run lsblk and verify all expected partition slices reappear correctly.
Prevention
Regularly archive partition layouts using sfdisk -d /dev/sda > sda-layout.txt prior to maintenance operations.
Problem 2: Filesystem UUID Conflict After Volume Cloning
Symptoms
A cloned disk volume fails to mount, or mount returns filesystem errors. Running blkid reveals duplicate UUID values across multiple block paths.
Possible Causes
Cloning a disk or cloud snapshot copies raw filesystem superblock structures byte-for-byte, preserving identical UUID identifiers.
Investigation and Diagnosis
Check attached block UUID entries:
blkid | grep -i nvme2
If two distinct devices report matching UUID strings, a UUID collision exists.
Resolution
- Mount clone bypassing UUID verification:
sudo mount -o nouuid /dev/nvme2n1p1 /mnt/clone
sudo umount /mnt/clone
- Generate a fresh UUID:
- For XFS:
sudo xfs_admin -U generate /dev/nvme2n1p1
- For ext4:
sudo tune2fs -U random /dev/nvme2n1p1
- Update Persistent Configuration:
Update
/etc/fstabwith the newly assigned UUID.
Verification
Execute sudo mount -a and ensure both original and cloned mounts co-exist without error.
Problem 3: “No space left on device” (Disk Capacity Full)
Symptoms
Write operations fail with No space left on device errors. df -hT indicates 100% capacity utilization on target mount.
Possible Causes
- Uncontrolled log growth (e.g., application debug logging).
- Accumulated application dumps, cache files, or un-vacuumed systemd journals.
- Deleted files held open by active processes.
Investigation and Diagnosis
- Step 1: Locate full mount target:
df -hT
- Step 2: Inspect directory space consumption:
sudo du -xhd1 /var | sort -h
- Step 3: Check for deleted-but-open files:
sudo lsof +L1
Resolution
- Safe Emergency Truncation (keeps open file handles intact):
sudo > /var/log/runaway_app.log
Truncating via standard output redirection (>) immediately reduces file size to 0 bytes without removing inode references, freeing space instantly even while held open by a active process.
- Vacuum Systemd Journals:
sudo journalctl --vacuum-size=500M
- Remove Stale Packages (Debian/Ubuntu):
sudo apt autoremove --purge
[!WARNING] Never delete entries inside virtual file systems such as
/procor/sys—they are in-memory representations exposed by the kernel that take up zero actual disk space. Avoid blindly removing unknown paths inside/var/libwithout identifying the managing service.
Verification
Run df -hT to confirm capacity drops below target alert thresholds.
Problem 4: Inode Exhaustion (Disk Space Available, Writes Fail)
Symptoms
Applications fail with No space left on device, yet standard capacity checks (df -h) show ample free space.
Possible Causes
Small files (such as PHP session files, micro-caching objects, or queued mail spools) have consumed all available inode structural entries.
Investigation and Diagnosis
- Check inode usage:
df -i
Look for IUse% reporting 100% on the impacted filesystem.
2. Identify directories containing dense file clusters:
sudo find /var -xdev -type f | cut -d/ -f1-4 | sort | uniq -c | sort -n
Resolution
Purge obsolete small files from target directories:
sudo find /var/spool/postfix/maildrop -type f -delete
Verification
Execute df -i to verify available inodes drop back to acceptable operating levels.
Problem 5: Missing Mount Point After Reboot
Symptoms
Services fail at startup because attached data volumes are missing from target directories.
Possible Causes
- Mount target was attached manually using
mountwithout adding a persistent entry to/etc/fstab. - Syntax error, invalid UUID, or mistyped mount path in
/etc/fstab.
Investigation and Diagnosis
- Step 1: Check kernel system logs:
sudo journalctl -b | grep -i mount
- Step 2: Validate
/etc/fstabsyntax:
sudo findmnt --verify
Resolution
Correct errors in /etc/fstab. For non-critical mounts, append the nofail flag to prevent system boot halts if the disk volume fails to attach:
UUID=a1b2c3d4-e5f6-7890 /mnt/data ext4 defaults,nofail 0 2
Verification
Run sudo mount -a to verify all entries mount successfully.
Problem 6: mount: wrong fs type, bad option, bad superblock
Symptoms
Executing mount fails with the error message: mount: wrong fs type, bad option, bad superblock on /dev/sdb1.
Possible Causes
- Mismatch between actual filesystem on disk and the filesystem type specified in
/etc/fstab(e.g., specifyingext4for anxfsvolume). - Damaged filesystem metadata structures or corrupted superblock header blocks.
Investigation and Diagnosis
- Verify filesystem type reported by raw device headers:
sudo blkid /dev/sdb1
sudo lsblk -f /dev/sdb1
- Compare the output with
/etc/fstabdefinitions.
Resolution
- If Configuration Mismatched: Correct the filesystem type column inside
/etc/fstabto matchblkid. - If Superblock Corrupted: Perform filesystem repair on an unmounted volume:
sudo umount /dev/sdb1
sudo fsck -f /dev/sdb1
Problem 7: target is busy Failure on Unmount (umount)
Symptoms
Executing umount /mnt/data fails with umount: /mnt/data: target is busy.
Possible Causes
Active processes hold open file handles or maintain working directory paths inside target mount points.
Investigation and Diagnosis
Find active processes referencing the path:
sudo lsof +f -- /mnt/data
# OR
sudo fuser -vm /mnt/data
Resolution
- Terminate or gracefully stop processes holding references:
sudo fuser -k -9 -m /mnt/data
- Alternative (Lazy Unmount): If background processes cannot be interrupted immediately, detach the filesystem from the directory tree using a lazy unmount:
sudo umount -l /mnt/data
[!NOTE] Lazy unmount (
-l) detaches the filesystem immediately from the file hierarchy, but actual storage resources remain allocated until all active file references are closed.
Problem 8: Filesystem Unexpectedly Remounted Read-Only
Symptoms
Applications fail with Read-only file system errors. mount shows mount options switched to ro.
Possible Causes
Underlying disk I/O errors or filesystem metadata inconsistencies caused the kernel to remount the filesystem as read-only to prevent data loss.
Investigation and Diagnosis
Inspect kernel logs for storage errors:
sudo dmesg | tail -n 40
Look for I/O transport errors, block timeouts, or filesystem journal failures.
Resolution
- Free up space or address physical storage failures.
- Unmount and run filesystem checks:
sudo fsck -f /dev/sdb1
- Remount target read-write:
sudo mount -o remount,rw /
Problem 9: Extended Cloud/Virtual Disk but Filesystem Size Remained Unchanged
Symptoms
Storage capacity was expanded in the cloud hypervisor console, but df -h shows the original volume size.
Possible Causes
Only the underlying physical device layer was resized. Intermediary layers (partitions, LVM PVs, LVM LVs, or the filesystem) were skipped.
Investigation and Diagnosis
Compare device layer sizes across the storage stack:
lsblk
sudo pvs
sudo lvs
df -h
If lsblk shows a larger disk than df -h, one or more resize steps were omitted.
Resolution
Execute the full multi-tier extension workflow:
sudo growpart /dev/nvme0n1 1
sudo pvresize /dev/nvme0n1p1
sudo lvextend -l +100%FREE /dev/myvg/mylv
sudo resize2fs /dev/myvg/mylv # ext4
# OR
sudo xfs_growfs /mount/point # XFS
Problem 10: Block Device Name Instability Across Reboots (/dev/sdb -> /dev/sdc)
Symptoms
After adding a new drive or rebooting, mount points attach to wrong volumes or fail entirely.
Possible Causes
Kernel block device allocation (/dev/sdX) is non-deterministic and can change across boot cycles as devices respond to enumeration in varying order.
Resolution
Never rely on volatile /dev/sdX device paths in /etc/fstab. Use persistent UUID assignments instead:
# Obtain persistent UUID
blkid /dev/sdb1
Update /etc/fstab to use UUID=...:
UUID=b5a987d6-3e2f-4a1b-9876-c5d4e3f2a1b0 /data ext4 defaults,nofail 0 2
11. Security and Best Practices
Recommended Options for /etc/fstab Mounts
Applying mount restrictions prevents security compromise vectors:
nosuid: Disables execution of set-user-identifier (SUID) and set-group-identifier (SGID) bits.nodev: Prevents interpretation of character or block special devices on the mounted filesystem.noexec: Blocks execution of binaries on the target volume (recommended for/tmpor/var/tmp).nofail: Prevents system boot failures if non-essential disks are disconnected.
Security Baseline for Persistent Volume Mounts
UUID=a1b2c3d4-e5f6-7890 /var/app/data ext4 defaults,nosuid,nodev,nofail 0 2
12. Performance and Resource Optimisation
Alignment and Block Sizes
Always ensure partition boundaries align with 1MB/4MB disk sector boundaries (default in modern tools like parted and gdisk). Misaligned partitions cause read-modify-write performance degradation across physical sector boundaries.
Workload Optimizations
- ext4 Optimization: Disable barrier updates on battery-backed hardware RAID controllers:
mount -o defaults,barrier=0 /dev/vg/lv /data
- XFS Parallelism: XFS automatically partitions storage into independent Allocation Groups (AGs), enabling high-throughput parallel execution across multi-core systems. XFS is optimal for high-concurrency database storage engines.
13. Backup, Recovery and Maintenance
Partition Metadata Backups
Always backup raw partition layouts before undertaking disk slice adjustments:
sudo sfdisk -d /dev/sda > sda-partition-layout.txt
To restore a saved layout:
sudo sfdisk /dev/sda < sda-partition-layout.txt
Filesystem Repair Guidelines
Never execute repairs on mounted filesystems—doing so can cause severe filesystem corruption.
# Correct unmount procedure prior to execution
sudo umount /dev/sdb1
sudo fsck -f /dev/sdb1
14. Automation and Integration
Ansible Automation Pattern for LVM Setup
- name: Configure LVM Storage Stack
hosts: servers
tasks:
- name: Create Volume Group
community.general.lvg:
vg: vg_app
pvs: /dev/sdb
- name: Create Logical Volume
community.general.lvol:
vg: vg_app
lv: lv_data
size: 100%FREE
- name: Format ext4 Filesystem
community.general.filesystem:
fstype: ext4
dev: /dev/vg_app/lv_data
- name: Mount Volume Persistently
ansible.posix.mount:
path: /mnt/appdata
src: /dev/vg_app/lv_data
fstype: ext4
opts: defaults,nofail
state: mounted
15. Practical Labs and Exercises
Lab 1: Capacity and Inode Inspection (Beginner)
- Objective: Differentiate block space and inode utilization.
- Instructions:
- Inspect mounted volume usage using
df -hT. - Inspect inode statistics using
df -i. - Locate the single largest directory inside
/varusingdu -xhd1 /var | sort -h.
- Validation: Confirm understanding of how inode exhaustion differs from block storage exhaustion.
Lab 2: Safe Volume Expansion (Intermediate)
- Objective: Perform a 4-layer LVM volume extension on simulated hardware.
- Instructions:
- Initialize an LVM block pool on a loopback device or test virtual disk.
- Format and mount an ext4 filesystem.
- Expand the underlying drive block boundary and run
pvresize,lvextend, andresize2fs.
- Validation: Run
lsblkanddf -hto verify expanded block capacity at every layer.
16. Quick Reference and Cheat Sheet
Important Configuration Files
| Path | Purpose |
|---|---|
/etc/fstab |
Persistent mount settings evaluated at boot. |
/proc/mounts |
Active kernel mount records. |
/dev/disk/by-uuid/ |
Persistent symlinks categorized by device UUID. |
Important Log Locations
| Log Target | Command / Path | Purpose |
|---|---|---|
| Kernel I/O Ring | `dmesg | tail` |
| Mount Service Logs | `journalctl -b | grep -i mount` |
Common Administrative Commands
| Task | Syntax |
|---|---|
| Mount by UUID | mount UUID=<uuid> /target/path |
Test /etc/fstab |
mount -a |
| Remount Root Read-Write | mount -o remount,rw / |
| Find Unlinked Open Files | lsof +L1 |
| Grow ext4 Filesystem | resize2fs /dev/mapper/vg-lv |
| Grow XFS Filesystem | xfs_growfs /mount/point |
Diagnostic Quick Reference Workflow
1. Map Storage Stack --> lsblk -f && df -hT && df -i
2. Find Heavy Space Users --> du -xhd1 /var | sort -h
3. Locate Deleted Files --> lsof +L1
4. Validate fstab Errors --> findmnt --verify && mount -a
17. Further Reading and References
- Filesystem Hierarchy Standard (FHS 3.0):
refspecs.linuxfoundation.org/FHS_3.0 - The Linux Documentation Project - LVM HOWTO:
tldp.org/HOWTO/LVM-HOWTO - Linux Programmer’s Manual - fdisk(8):
man7.org/linux/man-pages/man8/fdisk.8.html - Linux Programmer’s Manual - parted(8):
man7.org/linux/man-pages/man8/parted.8.html - Linux Programmer’s Manual - fstab(5):
man7.org/linux/man-pages/man5/fstab.5.html - Linux Programmer’s Manual - lsblk(8):
man7.org/linux/man-pages/man8/lsblk.8.html - Arch Wiki - Logical Volume Management:
wiki.archlinux.org/title/LVM