Domain 4Running SystemsCORE

Configuring Logging

Comprehensive guide to RHEL logging, covering systemd-journald persistence, rsyslogd configuration rules, log rotation management with logrotate, and log analysis utilities.

Domain: Operate running systems•Updated: 2026-08-30
#Logging#Journald#Rsyslog#Logrotate#RHEL#RHCSA

Configuring Logging

Theory Questions

1. Evaluate statements about systemd-journald, identifying which are true or false.

  • Command(s):
    journalctl --header
  • Statements Evaluation:
    • True: systemd-journald stores logs in an indexed, compressed binary format rather than plain text.
    • True: It captures stdout/stderr from systemd services, kernel messages, syslog events, and early boot logs natively.
    • False: systemd-journald natively writes log entries into standard plain-text files like /var/log/messages. (Correction: rsyslogd reads entries from the journal via imjournal and writes them to standard plain-text log files.)
  • Why: The binary journal architecture optimizes log querying and indexing while enabling secure, tamper-evident logging.
  • Gotchas: Assuming standard text tools like grep or cat can directly parse binary journal files; journalctl must be used.
  • Verify:
    file /run/log/journal/*/*

  • Path / Command:
    /var/log/secure
  • Why: On RHEL systems, rsyslogd routes authentication, security authorization events (such as sshd, sudo, su, and PAM authentication failures/successes) to /var/log/secure.
  • Gotchas: Confusing /var/log/secure (Red Hat / CentOS / Fedora standard) with /var/log/auth.log (Debian / Ubuntu standard).
  • Verify:
    tail -n 20 /var/log/secure

  • Path / Command:
    /var/log/audit/audit.log
  • Why: The auditd daemon logs all Access Vector Cache (AVC) denials and security-relevant SELinux audit events to /var/log/audit/audit.log.
  • Gotchas: Expecting SELinux denial messages to appear exclusively in /var/log/messages without setroubleshoot-server installed.
  • Verify:
    ausearch -m avc -ts recent

4. Identify the directory used to store the systemd journal persistently.

  • Path / Command:
    /var/log/journal/
  • Why: When journal persistence is enabled, systemd-journald writes binary log files into a subfolder named after the system’s Machine ID under /var/log/journal/<machine-id>/.
  • Gotchas: Mistaking /run/log/journal/ for persistent storage; /run is mounted on tmpfs (volatile RAM) and clears upon reboot.
  • Verify:
    ls -ld /var/log/journal/

5. Make the systemd journal persistent across reboots.

  • Command(s):
    mkdir -p /var/log/journal
    systemd-tmpfiles --create --prefix /var/log/journal
    systemctl restart systemd-journald
    Alternative (Explicit configuration directive):
    sed -i 's/^#\?Storage=.*/Storage=persistent/' /etc/systemd/journald.conf
    systemctl restart systemd-journald
  • Why: Creating /var/log/journal signals systemd-journald to switch from volatile memory storage to persistent disk storage. Applying systemd-tmpfiles assigns mandatory ACLs and ownership (root:systemd-journal).
  • Gotchas: Creating the folder manually without restarting systemd-journald or failing to set proper ownership/permissions (2755).
  • Verify:
    journalctl --header | grep "File path"

6. Activate a persistent journal configuration immediately.

  • Command(s):
    systemctl restart systemd-journald
    Alternatively, send SIGUSR1 to force log flushing:
    journalctl --flush
  • Why: Restarting systemd-journald causes it to reload /etc/systemd/journald.conf and switch output handlers immediately. Executing journalctl --flush forces the daemon to flush volatile logs from /run/log/journal/ to /var/log/journal/.
  • Gotchas: Running systemctl reload systemd-journald (reload is not supported by systemd-journald; a restart or --flush is required).
  • Verify:
    systemctl status systemd-journald

7. Identify the rsyslogd configuration file.

  • Path / Command:
    /etc/rsyslog.conf
    /etc/rsyslog.d/*.conf
  • Why: /etc/rsyslog.conf is the main configuration file for rsyslogd. Drop-in custom modular rules should be placed in /etc/rsyslog.d/*.conf.
  • Gotchas: Modifying the main /etc/rsyslog.conf file directly for custom application rules instead of using modular drop-ins in /etc/rsyslog.d/.
  • Verify:
    rsyslogd -N1

8. Identify rsyslog destination types, including module references.

  • Command(s) / Configuration Examples:
    # 1. Standard Files (omfile module)
    *.info /var/log/messages
    
    # 2. Remote Syslog Server via UDP/TCP (omfwd module)
    *.* @192.168.1.50:514    # UDP (@)
    *.* @@192.168.1.50:514   # TCP (@@)
    
    # 3. Logged-in Users / Terminal Broadcast (omusrmsg module)
    *.emerg :omusrmsg:*
    
    # 4. Named Pipes / FIFOs (ompipe module)
    *.info |/dev/xconsole
  • Why: rsyslog utilizes modular output modules (omfile, omfwd, omusrmsg, ompipe, omdb) to target local files, network destinations, terminal sessions, databases, and named pipes.
  • Gotchas: Using a single @ for TCP or double @@ for UDP when configuring remote logging.
  • Verify:
    grep -E "omfile|omfwd|omusrmsg" /etc/rsyslog.conf

9. Choose the appropriate rsyslog facility for a custom application (e.g., Apache).

  • Facility Names:
    local0, local1, local2, local3, local4, local5, local6, local7
  • Why: POSIX syslog reserves the local0 through local7 facilities specifically for custom applications, local software services, and site-specific uses without conflicting with system facilities (auth, cron, daemon, kern, mail).
  • Gotchas: Attempting to assign custom service logs to system-reserved facilities such as kern or auth.
  • Verify:
    logger -p local0.notice "Custom App Test Event"

10. Configure maximum log file size using logrotate.

  • Command(s) / Configuration Directive: Add the size or maxsize directive inside /etc/logrotate.d/<app_name> or /etc/logrotate.conf:
    /var/log/custom-app.log {
        size 50M
        rotate 5
        create 0640 root root
    }
  • Why: The size 50M directive instructs logrotate to trigger log rotation whenever the target log file reaches or exceeds 50 Megabytes, regardless of the calendar rotation schedule (daily/weekly).
  • Gotchas: Confusing size (rotates when size threshold is breached) with maxsize (rotates at schedule time or earlier if max size is breached).
  • Verify:
    logrotate -d /etc/logrotate.d/<app_name>

11. Identify the file used to configure rsyslogd.

  • Path / Command:
    /etc/rsyslog.conf
  • Why: Primary system logging configuration for rsyslogd is read from /etc/rsyslog.conf and included drop-in configs matching /etc/rsyslog.d/*.conf.
  • Gotchas: Editing /etc/sysconfig/rsyslog (which defines daemon launch flags) instead of /etc/rsyslog.conf (which defines log routing rules).
  • Verify:
    rsyslogd -N1

12. Identify the log file for authentication messages.

  • Path / Command:
    /var/log/secure
  • Why: Authentication events (user logons, sudo executions, SSH sessions, PAM authentications) are processed by rsyslog and recorded into /var/log/secure.
  • Gotchas: Looking for authentication logs in /var/log/auth.log on RHEL systems.
  • Verify:
    grep "Failed password" /var/log/secure

13. Determine the default log rotation schedule.

  • Command(s):
    grep -E "^(weekly|daily|monthly|rotate|create)" /etc/logrotate.conf
  • Why: Default RHEL installations configure logrotate with:
    • Schedule: weekly
    • Retained copies: rotate 4 (keeps 4 weeks of backlogs)
    • File handling: create (generates a new empty log file after rotating)
  • Gotchas: Assuming logs are rotated daily by default; the default out-of-the-box system global schedule is weekly.
  • Verify:
    cat /etc/logrotate.conf | head -n 20

14. Log a message from the command line to a specific facility and priority.

  • Command(s):
    logger -p local0.info "Custom application event logged via CLI"
  • Why: The logger command allows scripts and command-line users to send messages directly to the syslog engine with custom facility and priority specification (-p <facility>.<priority>).
  • Gotchas: Omitting the -p parameter, which defaults the log entry to user.notice.
  • Verify:
    journalctl -t logger -n 5

15. Write an rsyslog rule to direct messages of a given priority to a specific file.

  • Command(s):
    echo 'local0.info /var/log/custom-app.log' > /etc/rsyslog.d/custom-app.conf
    systemctl restart rsyslog
  • Why: Syntax follows <facility>.<priority> <destination>. Specifying local0.info routes all messages sent to local0 facility at priority info and higher to /var/log/custom-app.log.
  • Gotchas: Inverting facility and priority (e.g., writing info.local0), or forgetting to restart rsyslog service to activate rules.
  • Verify:
    logger -p local0.info "Testing custom rsyslog destination rule"
    tail -n 1 /var/log/custom-app.log

16. Allow the journal to grow beyond its default size restrictions.

  • Command(s):
    mkdir -p /etc/systemd/journald.conf.d
    cat << 'EOF' > /etc/systemd/journald.conf.d/maxsize.conf
    [Journal]
    SystemMaxUse=1G
    SystemKeepFree=2G
    EOF
    systemctl restart systemd-journald
  • Why: Setting SystemMaxUse=1G in journal configuration allows disk storage consumption to expand up to 1 Gigabyte (overriding default limits of 10% of volume size or 4GB cap).
  • Gotchas: Editing /etc/systemd/journald.conf directly instead of placing custom override files in /etc/systemd/journald.conf.d/.
  • Verify:
    journalctl --disk-usage

17. Check the systemd journal for boot messages with explanations.

  • Command(s):
    journalctl -xb
  • Why: The -b option restricts journal output to the current system boot, and -x (--catalog) attaches explanations and help documentation from the system catalog to error log entries.
  • Gotchas: Using -b -1 when trying to inspect current boot logs instead of past boot logs.
  • Verify:
    journalctl -xb | head -n 30

18. View all journald messages for a specific PID within a time range.

  • Command(s):
    journalctl _PID=1234 --since "2026-08-01 00:00:00" --until "2026-08-28 23:59:59"
  • Why: Filtering by matches like _PID=1234 combined with --since and --until date/time ranges isolates exact operational windows for precise process auditing.
  • Gotchas: Incorrect timestamp syntax; standard format is "YYYY-MM-DD HH:MM:SS".
  • Verify:
    journalctl _PID=1 --since "1 hour ago"

19. View all messages logged for a specific service (e.g., sshd).

  • Command(s):
    journalctl -u sshd.service
  • Why: The -u (--unit) option filters journal output to show logs associated specifically with the named systemd service unit.
  • Gotchas: Omitting .service suffix when the service name shares an identifier with an executable or timer unit.
  • Verify:
    journalctl -u sshd.service -n 10

20. Make the systemd journal persistent.

  • Command(s):
    mkdir -p /var/log/journal
    systemd-tmpfiles --create --prefix /var/log/journal
    systemctl restart systemd-journald
  • Why: Ensuring /var/log/journal exists with proper permissions (2755 root:systemd-journal) causes systemd-journald to store logs permanently on disk across system reboots.
  • Gotchas: Not running systemd-tmpfiles or failing to restart systemd-journald to apply storage changes.
  • Verify:
    ls -la /var/log/journal

Lab Tasks

Task 1: Configure persistent systemd journal

  • Command(s):
    mkdir -p /var/log/journal
    systemd-tmpfiles --create --prefix /var/log/journal
    systemctl restart systemd-journald
    Alternative Method via journald.conf:
    mkdir -p /etc/systemd/journald.conf.d
    echo -e "[Journal]

Storage=persistent” > /etc/systemd/journald.conf.d/persistent.conf mkdir -p /var/log/journal systemd-tmpfiles –create –prefix /var/log/journal systemctl restart systemd-journald

* **Why:**
* `mkdir -p`: Creates the target persistent storage directory path on `/var/log/journal`.
* `systemd-tmpfiles`: Configures correct permissions (`2755`) and group ownership (`systemd-journal`).
* `systemctl restart`: Forces the journal daemon to re-evaluate storage configuration and begin writing binary log files to disk.
* **Gotchas:** Skipping `systemd-tmpfiles` resulting in permission errors, or forgetting to restart `systemd-journald`.
* **Verify:**
```bash
journalctl --header | grep -i "File path"

Task 2: Configure rsyslog info priority to /var/log/messages.info

  • Command(s):
    echo '*.info /var/log/messages.info' > /etc/rsyslog.d/messages-info.conf
    systemctl restart rsyslog
  • Why:
    • *.info: Captures all log facilities (*) at priority info and higher (info, notice, warning, err, crit, alert, emerg).
    • /etc/rsyslog.d/: Uses modular drop-in configuration directory for clean administration.
    • systemctl restart: Directs rsyslogd to re-read rules and open the target log destination.
  • Gotchas: Reversing syntax to info.* instead of *.info, or omitting the leading / for absolute file path.
  • Verify:
    logger -p user.info "RHCSA test message"
    grep "RHCSA test message" /var/log/messages.info

Task 3: Configure logrotate to keep ten old versions of log files

  • Command(s):
    cat << 'EOF' > /etc/logrotate.d/messages-info
    /var/log/messages.info {
        rotate 10
        weekly
        missingok
        notifempty
        create 0640 root root
    }
    EOF
    Or to update global default settings across all non-overridden logs:
    sed -i 's/^rotate .*/rotate 10/' /etc/logrotate.conf
  • Why:
    • rotate 10: Instructs logrotate to retain 10 rotated historical log archives before purging the oldest.
    • Creating a drop-in file under /etc/logrotate.d/ targets specific application logs cleanly without polluting global system defaults.
  • Gotchas: Changing global /etc/logrotate.conf when the requirement specifies targeting a distinct custom log file.
  • Verify:
    logrotate -d /etc/logrotate.d/messages-info