The logging stack.
persistent journald, rsyslog rules, and logrotate.
An audit is in three days. journald is volatile and loses logs on reboot, a custom app facility needs its own file, and retention has to be provable. The full RHEL 9 logging stack.
The compliance audit is in three days and logadmin1 has two gaps that will not survive it. journald is volatile, so every reboot wipes the system logs, and the auditors want history that outlives a restart, capped at 500MB so it cannot eat the disk. And the app logs, everything on facility local5, need their own file at /var/log/app_local5.log owned 0640 root:logadmin. Then three specific journal queries the auditors will make you run live, and proof that /var/log/messages keeps at least four weeks.
The thing to hold in your head is that RHEL runs two logging systems side by side. journald collects everything and can be told to keep it. rsyslog writes the familiar text files under /var/log and routes by facility. logrotate decides how long any of it sticks around. Knowing which of the three owns which job is the entire ticket.
The ticket
id logadmin1 getent group logadmin || sudo groupadd logadmin sudo usermod -aG logadmin logadmin1 systemctl status rsyslog systemd-journald --no-pager | head
Concept review. Two logging systems, working together.
man rsyslog.conf man journald.conf man journalctl man logrotate
journald collects everything, the kernel, service output, and the syslog API, into a structured binary journal. It forwards to rsyslog, which writes the human-readable text files under /var/log. Both run by default, and they are complementary, not competitors. rsyslog routes by facility and priority.
facility.priority action authpriv.* /var/log/secure local5.* /var/log/app_local5.log *.info;mail.none /var/log/messages # info+ from all except mail # facilities: kern, cron, authpriv, local0..local7 (local is yours) # priorities low to high: debug info notice warning err crit alert emerg
-b / -b -1 current boot / previous boot -k kernel only, like dmesg -u UNIT by systemd unit -p err priority err and above --since '1 hour ago' time based -n 50 last 50 lines, -f follow live
Make journald persistent
By default journald writes to /run, which is wiped on reboot. Setting storage to persistent and giving it a size cap fixes both the loss and the risk of it eating the disk. Then restart journald so the change takes effect, which is the step people forget.
sudo mkdir -p /var/log/journal sudo cp /etc/systemd/journald.conf /etc/systemd/journald.conf.bak sudo sed -i 's/^#Storage=.*/Storage=persistent/' /etc/systemd/journald.conf sudo sed -i 's/^#SystemMaxUse=.*/SystemMaxUse=500M/' /etc/systemd/journald.conf grep -E 'Storage|SystemMaxUse' /etc/systemd/journald.conf sudo systemctl restart systemd-journald journalctl --disk-usage ls /var/log/journal/
A custom rsyslog rule
Write a drop-in, then pre-create the log file with the right owner and mode, because rsyslog would otherwise create it 0644 root:root. Restart rsyslog and send a test message.
sudo tee /etc/rsyslog.d/50-app-local5.conf << 'EOF' local5.* /var/log/app_local5.log EOF sudo touch /var/log/app_local5.log sudo chown root:logadmin /var/log/app_local5.log sudo chmod 0640 /var/log/app_local5.log sudo systemctl restart rsyslog logger -p local5.info "test from $(whoami)" sudo cat /var/log/app_local5.log
Three audit-style journalctl queries
journalctl -k -b # kernel only, this boot journalctl -u sshd --since "1 hour ago" # one unit, last hour journalctl -p err --since "24 hours ago" # err and above, last day # more the auditors like: journalctl -u sshd -p err -b # sshd errors this boot journalctl --list-boots # enumerate boots for -b -N
logrotate for retention
logrotate handles keeping and compressing old logs. Inspect what already rotates /var/log/messages, then add a rule for the new file. The postrotate SIGHUP is important: rsyslog holds the old file open, so after logrotate moves it, rsyslog must reopen its handles or it keeps writing to the moved file.
cat /etc/logrotate.d/rsyslog # controls /var/log/messages
sudo logrotate -d /etc/logrotate.conf # dry run, shows what would happen
sudo tee /etc/logrotate.d/app_local5 << 'EOF'
/var/log/app_local5.log {
weekly
rotate 4
missingok
notifempty
compress
delaycompress
create 0640 root logadmin
postrotate
/usr/bin/systemctl kill -s HUP rsyslog.service > /dev/null 2>&1 || true
endscript
}
EOF
sudo logrotate -d /etc/logrotate.d/app_local5 # validateExam questions
Write the command first.
Q1. Show all journal entries from the previous boot.
Q2. Configure journald to store logs persistently and verify after a restart.
Q3. Route local3 info-and-above to /var/log/app_local3.log owned 0640 root:logadmin, rotating weekly with 8 weeks of compressed history. Verify with logger.
Answers.
# A1 journalctl -b -1 # A2 sudo mkdir -p /var/log/journal sudo sed -i 's/^#Storage=.*/Storage=persistent/' /etc/systemd/journald.conf sudo systemctl restart systemd-journald; journalctl --list-boots # A3: rsyslog drop-in with 'local3.info /var/log/app_local3.log', # pre-create 0640 root:logadmin, restart rsyslog, add a logrotate # rule with rotate 8 weekly compress, then: logger -p local3.info "test"
Final checklist: confirm everything works
If every check passes, the ticket is done.
# 1. journal persistent and capped at 500M ls /var/log/journal/; journalctl --disk-usage # 2. local5 rule active, file 0640 root:logadmin logger -p local5.info test; sudo ls -l /var/log/app_local5.log # 3. logrotate rule validates and messages rotates weekly x4 sudo logrotate -d /etc/logrotate.d/app_local5 # 4. all three audit queries documented # 5. all three exam questions written from scratch # 6. tracker entry checked off
Reply to logadmin1: journal is persistent at /var/log/journal/ capped at 500M, local5.* routes to /var/log/app_local5.log (0640 root:logadmin) rotated weekly by four with a SIGHUP postrotate, and /var/log/messages already rotates weekly by four. Audit queries documented.