Configuring Logging
Theory Questions
1. Evaluate statements about systemd-journald, identifying which are true or false.
- Command(s):
journalctl --header - Statements Evaluation:
- True:
systemd-journaldstores logs in an indexed, compressed binary format rather than plain text. - True: It captures
stdout/stderrfrom systemd services, kernel messages, syslog events, and early boot logs natively. - False:
systemd-journaldnatively writes log entries into standard plain-text files like/var/log/messages. (Correction:rsyslogdreads entries from the journal viaimjournaland writes them to standard plain-text log files.)
- True:
- Why: The binary journal architecture optimizes log querying and indexing while enabling secure, tamper-evident logging.
- Gotchas: Assuming standard text tools like
greporcatcan directly parse binary journal files;journalctlmust be used. - Verify:
file /run/log/journal/*/*
2. Identify the log file for authentication-related messages.
- Path / Command:
/var/log/secure - Why: On RHEL systems,
rsyslogdroutes authentication, security authorization events (such assshd,sudo,su, and PAM authentication failures/successes) to/var/log/secure. - Gotchas: Confusing
/var/log/secure(Red Hat / CentOS / Fedora standard) with/var/log/auth.log(Debian / Ubuntu standard). - Verify:
tail -n 20 /var/log/secure
3. Identify the log file for SELinux-related events.
- Path / Command:
/var/log/audit/audit.log - Why: The
auditddaemon logs all Access Vector Cache (AVC) denials and security-relevant SELinux audit events to/var/log/audit/audit.log. - Gotchas: Expecting SELinux denial messages to appear exclusively in
/var/log/messageswithoutsetroubleshoot-serverinstalled. - Verify:
ausearch -m avc -ts recent
4. Identify the directory used to store the systemd journal persistently.
- Path / Command:
/var/log/journal/ - Why: When journal persistence is enabled,
systemd-journaldwrites binary log files into a subfolder named after the system’s Machine ID under/var/log/journal/<machine-id>/. - Gotchas: Mistaking
/run/log/journal/for persistent storage;/runis mounted ontmpfs(volatile RAM) and clears upon reboot. - Verify:
ls -ld /var/log/journal/
5. Make the systemd journal persistent across reboots.
- Command(s):
Alternative (Explicit configuration directive):mkdir -p /var/log/journal systemd-tmpfiles --create --prefix /var/log/journal systemctl restart systemd-journaldsed -i 's/^#\?Storage=.*/Storage=persistent/' /etc/systemd/journald.conf systemctl restart systemd-journald - Why: Creating
/var/log/journalsignalssystemd-journaldto switch from volatile memory storage to persistent disk storage. Applyingsystemd-tmpfilesassigns mandatory ACLs and ownership (root:systemd-journal). - Gotchas: Creating the folder manually without restarting
systemd-journaldor failing to set proper ownership/permissions (2755). - Verify:
journalctl --header | grep "File path"
6. Activate a persistent journal configuration immediately.
- Command(s):
Alternatively, sendsystemctl restart systemd-journaldSIGUSR1to force log flushing:journalctl --flush - Why: Restarting
systemd-journaldcauses it to reload/etc/systemd/journald.confand switch output handlers immediately. Executingjournalctl --flushforces the daemon to flush volatile logs from/run/log/journal/to/var/log/journal/. - Gotchas: Running
systemctl reload systemd-journald(reload is not supported bysystemd-journald; arestartor--flushis required). - Verify:
systemctl status systemd-journald
7. Identify the rsyslogd configuration file.
- Path / Command:
/etc/rsyslog.conf /etc/rsyslog.d/*.conf - Why:
/etc/rsyslog.confis the main configuration file forrsyslogd. Drop-in custom modular rules should be placed in/etc/rsyslog.d/*.conf. - Gotchas: Modifying the main
/etc/rsyslog.conffile directly for custom application rules instead of using modular drop-ins in/etc/rsyslog.d/. - Verify:
rsyslogd -N1
8. Identify rsyslog destination types, including module references.
- Command(s) / Configuration Examples:
# 1. Standard Files (omfile module) *.info /var/log/messages # 2. Remote Syslog Server via UDP/TCP (omfwd module) *.* @192.168.1.50:514 # UDP (@) *.* @@192.168.1.50:514 # TCP (@@) # 3. Logged-in Users / Terminal Broadcast (omusrmsg module) *.emerg :omusrmsg:* # 4. Named Pipes / FIFOs (ompipe module) *.info |/dev/xconsole - Why:
rsyslogutilizes modular output modules (omfile,omfwd,omusrmsg,ompipe,omdb) to target local files, network destinations, terminal sessions, databases, and named pipes. - Gotchas: Using a single
@for TCP or double@@for UDP when configuring remote logging. - Verify:
grep -E "omfile|omfwd|omusrmsg" /etc/rsyslog.conf
9. Choose the appropriate rsyslog facility for a custom application (e.g., Apache).
- Facility Names:
local0, local1, local2, local3, local4, local5, local6, local7 - Why: POSIX syslog reserves the
local0throughlocal7facilities specifically for custom applications, local software services, and site-specific uses without conflicting with system facilities (auth,cron,daemon,kern,mail). - Gotchas: Attempting to assign custom service logs to system-reserved facilities such as
kernorauth. - Verify:
logger -p local0.notice "Custom App Test Event"
10. Configure maximum log file size using logrotate.
- Command(s) / Configuration Directive:
Add the
sizeormaxsizedirective inside/etc/logrotate.d/<app_name>or/etc/logrotate.conf:/var/log/custom-app.log { size 50M rotate 5 create 0640 root root } - Why: The
size 50Mdirective instructslogrotateto trigger log rotation whenever the target log file reaches or exceeds 50 Megabytes, regardless of the calendar rotation schedule (daily/weekly). - Gotchas: Confusing
size(rotates when size threshold is breached) withmaxsize(rotates at schedule time or earlier if max size is breached). - Verify:
logrotate -d /etc/logrotate.d/<app_name>
11. Identify the file used to configure rsyslogd.
- Path / Command:
/etc/rsyslog.conf - Why: Primary system logging configuration for
rsyslogdis read from/etc/rsyslog.confand included drop-in configs matching/etc/rsyslog.d/*.conf. - Gotchas: Editing
/etc/sysconfig/rsyslog(which defines daemon launch flags) instead of/etc/rsyslog.conf(which defines log routing rules). - Verify:
rsyslogd -N1
12. Identify the log file for authentication messages.
- Path / Command:
/var/log/secure - Why: Authentication events (user logons, sudo executions, SSH sessions, PAM authentications) are processed by
rsyslogand recorded into/var/log/secure. - Gotchas: Looking for authentication logs in
/var/log/auth.logon RHEL systems. - Verify:
grep "Failed password" /var/log/secure
13. Determine the default log rotation schedule.
- Command(s):
grep -E "^(weekly|daily|monthly|rotate|create)" /etc/logrotate.conf - Why: Default RHEL installations configure
logrotatewith:- Schedule:
weekly - Retained copies:
rotate 4(keeps 4 weeks of backlogs) - File handling:
create(generates a new empty log file after rotating)
- Schedule:
- Gotchas: Assuming logs are rotated daily by default; the default out-of-the-box system global schedule is weekly.
- Verify:
cat /etc/logrotate.conf | head -n 20
14. Log a message from the command line to a specific facility and priority.
- Command(s):
logger -p local0.info "Custom application event logged via CLI" - Why: The
loggercommand allows scripts and command-line users to send messages directly to the syslog engine with custom facility and priority specification (-p <facility>.<priority>). - Gotchas: Omitting the
-pparameter, which defaults the log entry touser.notice. - Verify:
journalctl -t logger -n 5
15. Write an rsyslog rule to direct messages of a given priority to a specific file.
- Command(s):
echo 'local0.info /var/log/custom-app.log' > /etc/rsyslog.d/custom-app.conf systemctl restart rsyslog - Why: Syntax follows
<facility>.<priority> <destination>. Specifyinglocal0.inforoutes all messages sent tolocal0facility at priorityinfoand higher to/var/log/custom-app.log. - Gotchas: Inverting facility and priority (e.g., writing
info.local0), or forgetting to restartrsyslogservice to activate rules. - Verify:
logger -p local0.info "Testing custom rsyslog destination rule" tail -n 1 /var/log/custom-app.log
16. Allow the journal to grow beyond its default size restrictions.
- Command(s):
mkdir -p /etc/systemd/journald.conf.d cat << 'EOF' > /etc/systemd/journald.conf.d/maxsize.conf [Journal] SystemMaxUse=1G SystemKeepFree=2G EOF systemctl restart systemd-journald - Why: Setting
SystemMaxUse=1Gin journal configuration allows disk storage consumption to expand up to 1 Gigabyte (overriding default limits of 10% of volume size or 4GB cap). - Gotchas: Editing
/etc/systemd/journald.confdirectly instead of placing custom override files in/etc/systemd/journald.conf.d/. - Verify:
journalctl --disk-usage
17. Check the systemd journal for boot messages with explanations.
- Command(s):
journalctl -xb - Why: The
-boption restricts journal output to the current system boot, and-x(--catalog) attaches explanations and help documentation from the system catalog to error log entries. - Gotchas: Using
-b -1when trying to inspect current boot logs instead of past boot logs. - Verify:
journalctl -xb | head -n 30
18. View all journald messages for a specific PID within a time range.
- Command(s):
journalctl _PID=1234 --since "2026-08-01 00:00:00" --until "2026-08-28 23:59:59" - Why: Filtering by matches like
_PID=1234combined with--sinceand--untildate/time ranges isolates exact operational windows for precise process auditing. - Gotchas: Incorrect timestamp syntax; standard format is
"YYYY-MM-DD HH:MM:SS". - Verify:
journalctl _PID=1 --since "1 hour ago"
19. View all messages logged for a specific service (e.g., sshd).
- Command(s):
journalctl -u sshd.service - Why: The
-u(--unit) option filters journal output to show logs associated specifically with the named systemd service unit. - Gotchas: Omitting
.servicesuffix when the service name shares an identifier with an executable or timer unit. - Verify:
journalctl -u sshd.service -n 10
20. Make the systemd journal persistent.
- Command(s):
mkdir -p /var/log/journal systemd-tmpfiles --create --prefix /var/log/journal systemctl restart systemd-journald - Why: Ensuring
/var/log/journalexists with proper permissions (2755 root:systemd-journal) causessystemd-journaldto store logs permanently on disk across system reboots. - Gotchas: Not running
systemd-tmpfilesor failing to restartsystemd-journaldto apply storage changes. - Verify:
ls -la /var/log/journal
Lab Tasks
Task 1: Configure persistent systemd journal
- Command(s):
Alternative Method via journald.conf:mkdir -p /var/log/journal systemd-tmpfiles --create --prefix /var/log/journal systemctl restart systemd-journaldmkdir -p /etc/systemd/journald.conf.d echo -e "[Journal]
Storage=persistent” > /etc/systemd/journald.conf.d/persistent.conf mkdir -p /var/log/journal systemd-tmpfiles –create –prefix /var/log/journal systemctl restart systemd-journald
* **Why:**
* `mkdir -p`: Creates the target persistent storage directory path on `/var/log/journal`.
* `systemd-tmpfiles`: Configures correct permissions (`2755`) and group ownership (`systemd-journal`).
* `systemctl restart`: Forces the journal daemon to re-evaluate storage configuration and begin writing binary log files to disk.
* **Gotchas:** Skipping `systemd-tmpfiles` resulting in permission errors, or forgetting to restart `systemd-journald`.
* **Verify:**
```bash
journalctl --header | grep -i "File path"
Task 2: Configure rsyslog info priority to /var/log/messages.info
- Command(s):
echo '*.info /var/log/messages.info' > /etc/rsyslog.d/messages-info.conf systemctl restart rsyslog - Why:
*.info: Captures all log facilities (*) at priorityinfoand higher (info,notice,warning,err,crit,alert,emerg)./etc/rsyslog.d/: Uses modular drop-in configuration directory for clean administration.systemctl restart: Directsrsyslogdto re-read rules and open the target log destination.
- Gotchas: Reversing syntax to
info.*instead of*.info, or omitting the leading/for absolute file path. - Verify:
logger -p user.info "RHCSA test message" grep "RHCSA test message" /var/log/messages.info
Task 3: Configure logrotate to keep ten old versions of log files
- Command(s):
Or to update global default settings across all non-overridden logs:cat << 'EOF' > /etc/logrotate.d/messages-info /var/log/messages.info { rotate 10 weekly missingok notifempty create 0640 root root } EOFsed -i 's/^rotate .*/rotate 10/' /etc/logrotate.conf - Why:
rotate 10: Instructslogrotateto retain 10 rotated historical log archives before purging the oldest.- Creating a drop-in file under
/etc/logrotate.d/targets specific application logs cleanly without polluting global system defaults.
- Gotchas: Changing global
/etc/logrotate.confwhen the requirement specifies targeting a distinct custom log file. - Verify:
logrotate -d /etc/logrotate.d/messages-info