Domain 4Running SystemsCORE

Managing Troubleshooting

Comprehensive guide to RHEL boot troubleshooting, root password recovery, initramfs rebuilds, fstab error recovery, and rescue environment navigation.

Domain: Operate running systems•Updated: 2026-08-30
#Troubleshooting#GRUB2#Dracut#Initramfs#SELinux#Root Password Reset#RHCSA#RHEL

Managing Troubleshooting

Theory Questions

1. Identify the order of components in the RHEL 9 boot procedure.

  • Order of Execution:
    1. Firmware (UEFI / BIOS)
    2. Boot Loader (GRUB 2)
    3. Kernel Initialization & Initramfs (dracut)
    4. systemd (PID 1) / target units (default.target)

* **Why:** Firmware initializes basic hardware and loads GRUB 2; GRUB 2 loads the Linux kernel and initramfs into memory; initramfs mounts the root filesystem; `systemd` initializes system units and target states.
* **Gotchas:** Confusing the roles of `initramfs` (early temporary root filesystem in RAM) and `systemd` (first user-space process loaded from disk).
* **Verify:**
```bash
systemd-analyze plot > boot.svg

2. Start the system with a custom kernel argument from the GRUB 2 boot prompt.

  • Command(s) / Procedure:
  1. Highlight the desired kernel entry at the GRUB 2 boot menu and press e to edit.
  2. Locate the line starting with linux.
  3. Append the custom argument (e.g., systemd.debug_shell=1 or systemd.unit=emergency.target) to the end of the line.
  4. Press Ctrl+X or F10 to boot.
  • Why: Modifying kernel arguments temporarily in GRUB allows administrators to alter runtime kernel parameters or override target initialization for immediate troubleshooting without modifying configuration files on disk.
  • Gotchas: Edits made directly in the GRUB menu editor are non-persistent runtime changes and revert upon the next reboot.
  • Verify:
cat /proc/cmdline

3. Check the configuration file for initramfs/dracut options when LVM volumes fail to initialize.

  • Path / Command:
/etc/dracut.conf
/etc/dracut.conf.d/*.conf
  • Why: dracut builds the temporary initramfs image used during boot. Modules like lvm must be included in dracut.conf or /etc/dracut.conf.d/ (e.g., add_dracutmodules+=" lvm ") to ensure LVM drivers populate the initramfs.
  • Gotchas: Modifying dracut.conf files without regenerating the initramfs image (dracut -f). Changes do not take effect until the initramfs image file is recompiled.
  • Verify:
lsinitrd | grep -i lvm

  • Command(s) / Procedure:
  1. Interrupt GRUB 2 boot menu and press e on the default entry.
  2. Append rd.break to the end of the linux line.
  3. Press Ctrl+X to boot into the dracut emergency prompt.
  4. Execute:
mount -o remount,rw /sysroot
chroot /sysroot
echo "newpassword" | passwd --stdin root
touch /.autorelabel
exit
exit
  • Why: rd.break halts execution just before the kernel passes control from initramfs to systemd, allowing direct root access to /sysroot without authenticating.
  • Gotchas: Forgetting touch /.autorelabel. Without this flag, SELinux prevents logins due to invalid security contexts on /etc/shadow.
  • Verify: Log in as root with the new password after reboot.

5. Display detailed boot messages by removing the appropriate GRUB 2 boot options.

  • Command(s):
sed -i 's/rhgb quiet//' /etc/default/grub
grub2-mkconfig -o /boot/grub2/grub.cfg
  • Why: rhgb (Red Hat Graphical Boot) displays the splash screen, and quiet suppresses kernel console logging. Removing both exposes real-time text logs during boot.
  • Gotchas: Editing /boot/grub2/grub.cfg directly instead of modifying /etc/default/grub and running grub2-mkconfig.
  • Verify:
cat /proc/cmdline

6. Enter the most minimal troubleshooting mode (fewest services loaded).

  • Command(s): Append systemd.unit=emergency.target to the linux command line in GRUB 2.
  • Why: emergency.target loads the absolute minimal environment (mounts root filesystem read-only, starts no network or extra services, launches a single emergency shell).
  • Gotchas: Confusing emergency.target with rescue.target. rescue.target mounts all local filesystems and starts basic service targets, whereas emergency.target mounts only root read-only.
  • Verify:
systemctl list-jobs

7. Identify situations that require a rescue disk.

  • Situations:
  1. Corrupted bootloader (GRUB 2 stage 1 / stage 2 files missing or damaged).
  2. Corrupted /boot filesystem or corrupted main kernel/initramfs files.
  3. Invalid /etc/fstab entries on essential filesystems where local system repair tools fail or drop into non-interactive states.
  4. Disk drive corruption affecting fundamental system binaries required for initial shell launch.
  • Why: When local bootloader binaries or core system kernels are inaccessible or damaged, the host cannot initiate memory execution internally and requires external boot media.
  • Gotchas: Attempting to run rd.break when the kernel or GRUB 2 itself is missing or corrupted.
  • Verify: Boot host using RHEL Installation ISO set to “Troubleshooting -> Rescue a Red Hat Enterprise Linux system”.

8. Remount a read-only root file system as read-write in troubleshooting mode.

  • Command(s):
mount -o remount,rw /
# OR inside rd.break environment:
mount -o remount,rw /sysroot
  • Why: Emergency targets and dracut break points mount root storage in read-only (ro) mode to protect filesystem data integrity. Remounting with rw permits file modification.
  • Gotchas: Trying to edit /etc/shadow or /etc/fstab while the root filesystem remains mounted read-only (Read-only file system error).
  • Verify:
mount | grep -E " / | /sysroot "

9. Troubleshoot a server that shows only a blinking cursor at boot (no GRUB 2 menu).

  • Command(s) / Procedure:
  1. Boot server using RHEL Rescue Media.
  2. Select 1 (Continue) to mount the installation under /sysroot.
  3. Execute:
chroot /sysroot
grub2-install /dev/vda   # Match target block device
grub2-mkconfig -o /boot/grub2/grub.cfg
exit
reboot
  • Why: A blinking cursor indicates the system BIOS/UEFI passed control to the disk, but the Master Boot Record (MBR) or EFI system partition lacks valid GRUB 2 stage 1 bootloader binaries.
  • Gotchas: Installing GRUB to a partition (e.g., /dev/vda1) instead of the base block device (e.g., /dev/vda).
  • Verify: Reboot the server and confirm the GRUB 2 boot selection menu appears.

10. Restart the system normally after resetting the root password from init=/bin/bash.

  • Command(s):
touch /.autorelabel
exec /sbin/init
# OR
mount -o remount,ro /
reboot -f
  • Why: When invoking init=/bin/bash, PID 1 is standard bash rather than systemd. systemctl reboot will fail because the systemd D-Bus daemon is not running. Calling exec /sbin/init replaces bash with systemd to perform a graceful reboot.
  • Gotchas: Executing reboot directly under init=/bin/bash without remounting or calling init, which can cause unwritten filesystem buffers to corrupt.
  • Verify: System boots normally into default multi-user target.

11. Enter the GRUB boot menu editor mode.

  • Command(s): Press e while highlighting a kernel entry on the GRUB 2 menu screen.
  • Why: Opens the interactive inline configuration editor for the selected boot entry.
  • Gotchas: Pressing c instead of e. Pressing c opens the interactive GRUB command-line shell interface rather than the menu item script editor.
  • Verify: Screen displays editable kernel configuration lines starting with set, linux, and initrd.

12. Diagnose why the boot procedure stops and asks for the root password.

  • Command(s):
journalctl -xb
  • Why: When systemd encounters a critical failure during startup—such as an unmountable filesystem in /etc/fstab lacking the nofail option or filesystem corruption—it drops to a maintenance prompt requiring the root password.
  • Gotchas: Skipping journalctl -xb logs and guessing the failure cause. The journal explicitly highlights failing units in red text.
  • Verify:
systemctl --failed

13. Enter troubleshooting mode without knowing the root password.

  • Command(s) / Procedure:
  1. Edit GRUB menu entry by pressing e.
  2. Append rd.break to the linux line.
  3. Press Ctrl+X to boot.
  4. System enters dracut shell before root password authentication is enforced.
  • Why: rd.break stops early boot execution inside initramfs before the root account checks or systemd-sulogin maintenance prompts run.
  • Gotchas: Using systemd.unit=rescue.target without knowing the root password; rescue.target prompts for the root password before opening the shell.
  • Verify: Prompt displays switch_root:/# without requesting password credentials.

14. Take the first troubleshooting step when the server shows only a blinking cursor.

  • Procedure: Boot the host using external RHEL installation media (ISO/USB) and navigate to the Rescue Environment.
  • Why: If no GRUB menu appears, internal disk bootloader files are inaccessible, preventing on-disk recovery tools from executing without external media.
  • Gotchas: Attempting keyboard shortcuts (like Ctrl+Alt+Del) expecting local recovery options when no bootloader is present.
  • Verify: System successfully boots into the rescue installer menu interface.

15. Find which units are available in a specific troubleshooting environment.

  • Command(s):
systemctl list-units --all
# OR
systemctl list-unit-files
  • Why: Lists all loaded and available units within the current target state context.
  • Gotchas: Attempting to query unit status in a rd.break shell before systemd has initialized. systemctl requires systemd as PID 1.
  • Verify:
systemctl list-units --type=service

16. Start the system normally after changing the root password from init=/bin/bash.

  • Command(s):
touch /.autorelabel
exec /sbin/init
  • Why: Replaces the current bash process with systemd (PID 1), triggering normal system startup and SELinux relabeling routines.
  • Gotchas: Forgetting touch /.autorelabel, causing SELinux to block authentication on the rebooted system.
  • Verify: System completes full startup sequence to login prompt.

17. Ensure correct SELinux context labels after resetting the root password.

  • Command(s):
touch /.autorelabel
  • Why: Creating /.autorelabel in the root directory signals the SELinux kernel subsystem to perform an automatic filesystem-wide context relabeling during the subsequent boot cycle.
  • Gotchas: Skipping this step when SELinux is in Enforcing mode. Updating /etc/shadow outside of SELinux enforcement causes the file to retain invalid context labels, blocking subsequent root logins.
  • Verify:
ls -Z /etc/shadow

(Should output shadow_t type context)


18. Make a read-only root file system writable in troubleshooting mode.

  • Command(s):
mount -o remount,rw /
  • Why: Remounts the active root filesystem with read-write permissions without unmounting active mount structures.
  • Gotchas: Running mount -w /, which attempts to perform a new mount operation rather than remounting the existing partition.
  • Verify:
touch /testfile && rm -f /testfile

19. Save GRUB 2 boot loader changes permanently.

  • Command(s):
# 1. Apply configuration changes to /etc/default/grub:
vi /etc/default/grub

# 2. Regenerate main GRUB config file:
grub2-mkconfig -o /boot/grub2/grub.cfg
  • Why: Editing /etc/default/grub alone does not modify boot behavior until grub2-mkconfig compiles the settings into /boot/grub2/grub.cfg.
  • Gotchas: Hand-editing /boot/grub2/grub.cfg directly. Manual edits get overwritten during system kernel upgrades.
  • Verify:
grep "GRUB_CMDLINE_LINUX" /etc/default/grub

20. Enter the most minimal troubleshooting mode without knowing the root password.

  • Command(s) / Procedure:
  1. Interrupt GRUB boot menu with e.
  2. Append rd.break to the linux line.
  3. Boot with Ctrl+X.
  • Why: rd.break drops execution into an unauthenticated root shell within initramfs before systemd maintenance authentication prompts are initialized.
  • Gotchas: Appending init=/bin/bash without ensuring the root filesystem can be remounted read-write cleanly, or using rescue.target which requires password entry.
  • Verify: Terminal displays root shell prompt switch_root:/#.

Lab Tasks

Troubleshooting Practice

Task 1: Restart your server and change the root password from the appropriate troubleshooting mode.

  • Command(s):
# 1. Reboot server and press 'e' at GRUB menu.
# 2. Append 'rd.break' to the line starting with 'linux'. Press Ctrl+X.

# 3. Inside dracut shell, execute:
mount -o remount,rw /sysroot
chroot /sysroot
echo "redhat123" | passwd --stdin root
touch /.autorelabel
exit
exit
  • Why:

  • rd.break: Interrupts boot sequence prior to mounting real root filesystems and loading systemd.

  • mount -o remount,rw /sysroot: Grants write access to system root storage under /sysroot.

  • chroot /sysroot: Switches environment context to root filesystem root path.

  • touch /.autorelabel: Ensures SELinux relabels /etc/shadow correctly upon next boot.

  • Gotchas: Omitting touch /.autorelabel. SELinux will deny access to /etc/shadow, preventing login with the new password.

  • Verify: Log into the system using username root and password redhat123.


Task 2: In /etc/fstab, change one of the device names so the file system cannot be mounted on the next reboot. Restart, then diagnose and fix the issue.

  • Command(s):
# Step 1: Simulate breakage (Edit /etc/fstab and introduce typo into a non-root mount line)
# Example: Change UUID or change device /dev/vg_data/lvgroup to /dev/vg_data/invalid_name

# Step 2: Reboot system to trigger failure
reboot

# Step 3: (Emergency Shell Recovery) Enter root password at prompt
# Step 4: Inspect failure logs:
journalctl -xb | grep -i "failed"

# Step 5: Remount root filesystem as read-write:
mount -o remount,rw /

# Step 6: Fix typo in /etc/fstab using vi:
vi /etc/fstab

# Step 7: Reload systemd units and test mounts:
systemctl daemon-reload
mount -a

# Step 8: Exit emergency shell to resume boot:
exit
  • Why:

  • journalctl -xb: Pinpoints the exact unmountable line in /etc/fstab.

  • mount -o remount,rw /: Permits saving corrections to /etc/fstab.

  • mount -a: Validates /etc/fstab syntax before exiting the emergency shell.

  • Gotchas: Attempting to edit /etc/fstab in emergency mode without remounting / as read-write.

  • Verify:

findmnt --verify

(Returns SUCCESS with no errors)


Task 3 & 4: Use a rescue disk to bring your server up in full troubleshooting mode from the rescue environment. Re-create the initramfs.

  • Command(s):
# Step 1: Attach RHEL ISO media and boot server into Rescue Media.
# Step 2: Select "Troubleshooting" -> "Rescue a Red Hat Enterprise Linux system".
# Step 3: Select option '1' (Continue) to automatically mount target system under /sysroot.

# Step 4: Access target environment shell:
chroot /sysroot

# Step 5: Backup existing initramfs image:
cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak

# Step 6: Regenerate initramfs image for current running kernel:
dracut -f /boot/initramfs-$(uname -r).img $(uname -r)

# Step 7: Exit chroot and reboot:
exit
reboot
  • Why:

  • Rescue Media option 1: Automatically locates local Linux installations and mounts them under /sysroot.

  • chroot /sysroot: Changes working environment to the installed target system.

  • dracut -f: Forces clean regeneration of the initramfs binary image.

  • Gotchas: Running dracut outside the chroot environment. Doing so generates an initramfs image based on the rescue ISO environment rather than the host system configuration.

  • Verify Post-Reboot:

ls -l /boot/initramfs-$(uname -r).img

(Confirms file modification timestamp matches the re-creation execution time)