Managing Troubleshooting
Theory Questions
1. Identify the order of components in the RHEL 9 boot procedure.
- Order of Execution:
1. Firmware (UEFI / BIOS) 2. Boot Loader (GRUB 2) 3. Kernel Initialization & Initramfs (dracut) 4. systemd (PID 1) / target units (default.target)
* **Why:** Firmware initializes basic hardware and loads GRUB 2; GRUB 2 loads the Linux kernel and initramfs into memory; initramfs mounts the root filesystem; `systemd` initializes system units and target states.
* **Gotchas:** Confusing the roles of `initramfs` (early temporary root filesystem in RAM) and `systemd` (first user-space process loaded from disk).
* **Verify:**
```bash
systemd-analyze plot > boot.svg
2. Start the system with a custom kernel argument from the GRUB 2 boot prompt.
- Command(s) / Procedure:
- Highlight the desired kernel entry at the GRUB 2 boot menu and press
eto edit. - Locate the line starting with
linux. - Append the custom argument (e.g.,
systemd.debug_shell=1orsystemd.unit=emergency.target) to the end of the line. - Press
Ctrl+XorF10to boot.
- Why: Modifying kernel arguments temporarily in GRUB allows administrators to alter runtime kernel parameters or override target initialization for immediate troubleshooting without modifying configuration files on disk.
- Gotchas: Edits made directly in the GRUB menu editor are non-persistent runtime changes and revert upon the next reboot.
- Verify:
cat /proc/cmdline
3. Check the configuration file for initramfs/dracut options when LVM volumes fail to initialize.
- Path / Command:
/etc/dracut.conf
/etc/dracut.conf.d/*.conf
- Why:
dracutbuilds the temporaryinitramfsimage used during boot. Modules likelvmmust be included indracut.confor/etc/dracut.conf.d/(e.g.,add_dracutmodules+=" lvm ") to ensure LVM drivers populate the initramfs. - Gotchas: Modifying
dracut.conffiles without regenerating theinitramfsimage (dracut -f). Changes do not take effect until the initramfs image file is recompiled. - Verify:
lsinitrd | grep -i lvm
4. Reset the root password using the recommended kernel argument.
- Command(s) / Procedure:
- Interrupt GRUB 2 boot menu and press
eon the default entry. - Append
rd.breakto the end of thelinuxline. - Press
Ctrl+Xto boot into the dracut emergency prompt. - Execute:
mount -o remount,rw /sysroot
chroot /sysroot
echo "newpassword" | passwd --stdin root
touch /.autorelabel
exit
exit
- Why:
rd.breakhalts execution just before the kernel passes control from initramfs to systemd, allowing direct root access to/sysrootwithout authenticating. - Gotchas: Forgetting
touch /.autorelabel. Without this flag, SELinux prevents logins due to invalid security contexts on/etc/shadow. - Verify: Log in as
rootwith the new password after reboot.
5. Display detailed boot messages by removing the appropriate GRUB 2 boot options.
- Command(s):
sed -i 's/rhgb quiet//' /etc/default/grub
grub2-mkconfig -o /boot/grub2/grub.cfg
- Why:
rhgb(Red Hat Graphical Boot) displays the splash screen, andquietsuppresses kernel console logging. Removing both exposes real-time text logs during boot. - Gotchas: Editing
/boot/grub2/grub.cfgdirectly instead of modifying/etc/default/gruband runninggrub2-mkconfig. - Verify:
cat /proc/cmdline
6. Enter the most minimal troubleshooting mode (fewest services loaded).
- Command(s):
Append
systemd.unit=emergency.targetto thelinuxcommand line in GRUB 2. - Why:
emergency.targetloads the absolute minimal environment (mounts root filesystem read-only, starts no network or extra services, launches a single emergency shell). - Gotchas: Confusing
emergency.targetwithrescue.target.rescue.targetmounts all local filesystems and starts basic service targets, whereasemergency.targetmounts only root read-only. - Verify:
systemctl list-jobs
7. Identify situations that require a rescue disk.
- Situations:
- Corrupted bootloader (GRUB 2 stage 1 / stage 2 files missing or damaged).
- Corrupted
/bootfilesystem or corrupted main kernel/initramfs files. - Invalid
/etc/fstabentries on essential filesystems where local system repair tools fail or drop into non-interactive states. - Disk drive corruption affecting fundamental system binaries required for initial shell launch.
- Why: When local bootloader binaries or core system kernels are inaccessible or damaged, the host cannot initiate memory execution internally and requires external boot media.
- Gotchas: Attempting to run
rd.breakwhen the kernel or GRUB 2 itself is missing or corrupted. - Verify: Boot host using RHEL Installation ISO set to “Troubleshooting -> Rescue a Red Hat Enterprise Linux system”.
8. Remount a read-only root file system as read-write in troubleshooting mode.
- Command(s):
mount -o remount,rw /
# OR inside rd.break environment:
mount -o remount,rw /sysroot
- Why: Emergency targets and dracut break points mount root storage in read-only (
ro) mode to protect filesystem data integrity. Remounting withrwpermits file modification. - Gotchas: Trying to edit
/etc/shadowor/etc/fstabwhile the root filesystem remains mounted read-only (Read-only file systemerror). - Verify:
mount | grep -E " / | /sysroot "
9. Troubleshoot a server that shows only a blinking cursor at boot (no GRUB 2 menu).
- Command(s) / Procedure:
- Boot server using RHEL Rescue Media.
- Select
1(Continue) to mount the installation under/sysroot. - Execute:
chroot /sysroot
grub2-install /dev/vda # Match target block device
grub2-mkconfig -o /boot/grub2/grub.cfg
exit
reboot
- Why: A blinking cursor indicates the system BIOS/UEFI passed control to the disk, but the Master Boot Record (MBR) or EFI system partition lacks valid GRUB 2 stage 1 bootloader binaries.
- Gotchas: Installing GRUB to a partition (e.g.,
/dev/vda1) instead of the base block device (e.g.,/dev/vda). - Verify: Reboot the server and confirm the GRUB 2 boot selection menu appears.
10. Restart the system normally after resetting the root password from init=/bin/bash.
- Command(s):
touch /.autorelabel
exec /sbin/init
# OR
mount -o remount,ro /
reboot -f
- Why: When invoking
init=/bin/bash, PID 1 is standardbashrather thansystemd.systemctl rebootwill fail because the systemd D-Bus daemon is not running. Callingexec /sbin/initreplaces bash with systemd to perform a graceful reboot. - Gotchas: Executing
rebootdirectly underinit=/bin/bashwithout remounting or calling init, which can cause unwritten filesystem buffers to corrupt. - Verify: System boots normally into default multi-user target.
11. Enter the GRUB boot menu editor mode.
- Command(s):
Press
ewhile highlighting a kernel entry on the GRUB 2 menu screen. - Why: Opens the interactive inline configuration editor for the selected boot entry.
- Gotchas: Pressing
cinstead ofe. Pressingcopens the interactive GRUB command-line shell interface rather than the menu item script editor. - Verify: Screen displays editable kernel configuration lines starting with
set,linux, andinitrd.
12. Diagnose why the boot procedure stops and asks for the root password.
- Command(s):
journalctl -xb
- Why: When
systemdencounters a critical failure during startup—such as an unmountable filesystem in/etc/fstablacking thenofailoption or filesystem corruption—it drops to a maintenance prompt requiring the root password. - Gotchas: Skipping
journalctl -xblogs and guessing the failure cause. The journal explicitly highlights failing units in red text. - Verify:
systemctl --failed
13. Enter troubleshooting mode without knowing the root password.
- Command(s) / Procedure:
- Edit GRUB menu entry by pressing
e. - Append
rd.breakto thelinuxline. - Press
Ctrl+Xto boot. - System enters dracut shell before root password authentication is enforced.
- Why:
rd.breakstops early boot execution inside initramfs before the root account checks orsystemd-suloginmaintenance prompts run. - Gotchas: Using
systemd.unit=rescue.targetwithout knowing the root password;rescue.targetprompts for the root password before opening the shell. - Verify: Prompt displays
switch_root:/#without requesting password credentials.
14. Take the first troubleshooting step when the server shows only a blinking cursor.
- Procedure: Boot the host using external RHEL installation media (ISO/USB) and navigate to the Rescue Environment.
- Why: If no GRUB menu appears, internal disk bootloader files are inaccessible, preventing on-disk recovery tools from executing without external media.
- Gotchas: Attempting keyboard shortcuts (like
Ctrl+Alt+Del) expecting local recovery options when no bootloader is present. - Verify: System successfully boots into the rescue installer menu interface.
15. Find which units are available in a specific troubleshooting environment.
- Command(s):
systemctl list-units --all
# OR
systemctl list-unit-files
- Why: Lists all loaded and available units within the current target state context.
- Gotchas: Attempting to query unit status in a
rd.breakshell before systemd has initialized.systemctlrequiressystemdas PID 1. - Verify:
systemctl list-units --type=service
16. Start the system normally after changing the root password from init=/bin/bash.
- Command(s):
touch /.autorelabel
exec /sbin/init
- Why: Replaces the current
bashprocess withsystemd(PID 1), triggering normal system startup and SELinux relabeling routines. - Gotchas: Forgetting
touch /.autorelabel, causing SELinux to block authentication on the rebooted system. - Verify: System completes full startup sequence to login prompt.
17. Ensure correct SELinux context labels after resetting the root password.
- Command(s):
touch /.autorelabel
- Why: Creating
/.autorelabelin the root directory signals the SELinux kernel subsystem to perform an automatic filesystem-wide context relabeling during the subsequent boot cycle. - Gotchas: Skipping this step when SELinux is in
Enforcingmode. Updating/etc/shadowoutside of SELinux enforcement causes the file to retain invalid context labels, blocking subsequent root logins. - Verify:
ls -Z /etc/shadow
(Should output shadow_t type context)
18. Make a read-only root file system writable in troubleshooting mode.
- Command(s):
mount -o remount,rw /
- Why: Remounts the active root filesystem with read-write permissions without unmounting active mount structures.
- Gotchas: Running
mount -w /, which attempts to perform a new mount operation rather than remounting the existing partition. - Verify:
touch /testfile && rm -f /testfile
19. Save GRUB 2 boot loader changes permanently.
- Command(s):
# 1. Apply configuration changes to /etc/default/grub:
vi /etc/default/grub
# 2. Regenerate main GRUB config file:
grub2-mkconfig -o /boot/grub2/grub.cfg
- Why: Editing
/etc/default/grubalone does not modify boot behavior untilgrub2-mkconfigcompiles the settings into/boot/grub2/grub.cfg. - Gotchas: Hand-editing
/boot/grub2/grub.cfgdirectly. Manual edits get overwritten during system kernel upgrades. - Verify:
grep "GRUB_CMDLINE_LINUX" /etc/default/grub
20. Enter the most minimal troubleshooting mode without knowing the root password.
- Command(s) / Procedure:
- Interrupt GRUB boot menu with
e. - Append
rd.breakto thelinuxline. - Boot with
Ctrl+X.
- Why:
rd.breakdrops execution into an unauthenticated root shell within initramfs before systemd maintenance authentication prompts are initialized. - Gotchas: Appending
init=/bin/bashwithout ensuring the root filesystem can be remounted read-write cleanly, or usingrescue.targetwhich requires password entry. - Verify: Terminal displays root shell prompt
switch_root:/#.
Lab Tasks
Troubleshooting Practice
Task 1: Restart your server and change the root password from the appropriate troubleshooting mode.
- Command(s):
# 1. Reboot server and press 'e' at GRUB menu.
# 2. Append 'rd.break' to the line starting with 'linux'. Press Ctrl+X.
# 3. Inside dracut shell, execute:
mount -o remount,rw /sysroot
chroot /sysroot
echo "redhat123" | passwd --stdin root
touch /.autorelabel
exit
exit
-
Why:
-
rd.break: Interrupts boot sequence prior to mounting real root filesystems and loading systemd. -
mount -o remount,rw /sysroot: Grants write access to system root storage under/sysroot. -
chroot /sysroot: Switches environment context to root filesystem root path. -
touch /.autorelabel: Ensures SELinux relabels/etc/shadowcorrectly upon next boot. -
Gotchas: Omitting
touch /.autorelabel. SELinux will deny access to/etc/shadow, preventing login with the new password. -
Verify: Log into the system using username
rootand passwordredhat123.
Task 2: In /etc/fstab, change one of the device names so the file system cannot be mounted on the next reboot. Restart, then diagnose and fix the issue.
- Command(s):
# Step 1: Simulate breakage (Edit /etc/fstab and introduce typo into a non-root mount line)
# Example: Change UUID or change device /dev/vg_data/lvgroup to /dev/vg_data/invalid_name
# Step 2: Reboot system to trigger failure
reboot
# Step 3: (Emergency Shell Recovery) Enter root password at prompt
# Step 4: Inspect failure logs:
journalctl -xb | grep -i "failed"
# Step 5: Remount root filesystem as read-write:
mount -o remount,rw /
# Step 6: Fix typo in /etc/fstab using vi:
vi /etc/fstab
# Step 7: Reload systemd units and test mounts:
systemctl daemon-reload
mount -a
# Step 8: Exit emergency shell to resume boot:
exit
-
Why:
-
journalctl -xb: Pinpoints the exact unmountable line in/etc/fstab. -
mount -o remount,rw /: Permits saving corrections to/etc/fstab. -
mount -a: Validates/etc/fstabsyntax before exiting the emergency shell. -
Gotchas: Attempting to edit
/etc/fstabin emergency mode without remounting/as read-write. -
Verify:
findmnt --verify
(Returns SUCCESS with no errors)
Task 3 & 4: Use a rescue disk to bring your server up in full troubleshooting mode from the rescue environment. Re-create the initramfs.
- Command(s):
# Step 1: Attach RHEL ISO media and boot server into Rescue Media.
# Step 2: Select "Troubleshooting" -> "Rescue a Red Hat Enterprise Linux system".
# Step 3: Select option '1' (Continue) to automatically mount target system under /sysroot.
# Step 4: Access target environment shell:
chroot /sysroot
# Step 5: Backup existing initramfs image:
cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
# Step 6: Regenerate initramfs image for current running kernel:
dracut -f /boot/initramfs-$(uname -r).img $(uname -r)
# Step 7: Exit chroot and reboot:
exit
reboot
-
Why:
-
Rescue Media option
1: Automatically locates local Linux installations and mounts them under/sysroot. -
chroot /sysroot: Changes working environment to the installed target system. -
dracut -f: Forces clean regeneration of theinitramfsbinary image. -
Gotchas: Running
dracutoutside thechrootenvironment. Doing so generates aninitramfsimage based on the rescue ISO environment rather than the host system configuration. -
Verify Post-Reboot:
ls -l /boot/initramfs-$(uname -r).img
(Confirms file modification timestamp matches the re-creation execution time)