Configure a Centralized Log Management Server

Putting incoming logs on their own logical volumes is a deliberate isolation choice. If remote hosts flood this server, the worst case is that /var/log/remote fills up; the root filesystem, and therefore the ability to log in and fix the problem, stays healthy. This is the same reasoning behind giving /var or /home separate volumes in production.

The mechanics are the standard LVM stack: a physical volume initialises the disk for LVM, a volume group pools that space, and logical volumes are carved out of the pool. Because both volumes come from one vg_logs, they draw from a shared pool, which is what lets Task 10 hand more space to lv_logs later without touching the disk. That only works if you leave free extents in the group, so a 5 GiB disk holding a 3 GiB and a 512 MiB volume keeps room to grow.

Mount by UUID rather than device path. Device names like /dev/sdb can shift if disks are added or reordered, but a filesystem UUID is stable, so the fstab entry stays correct across reboots.

By default rsyslog only handles local messages. To accept logs from other machines you load a network input module and tell it which port to listen on. imudp is the UDP receiver and imtcp is the TCP receiver; syslog traditionally uses port 514 for both. UDP is fast and lossy, fine for high-volume telemetry where a dropped line does not matter; TCP is reliable and ordered, better when you cannot afford to lose events. This server accepts both.

A template defines an output path or message format, and here it uses the %HOSTNAME% property to build a per-sender filename. The rule *.* ?RemoteLogs says route messages of every facility and priority through that template, so each remote host lands in its own file under /var/log/remote. Module-load directives have to appear before the rules that depend on them.

Two things are easy to forget. The firewall must allow 514 on both protocols with —permanent so it survives a reboot, and the receiving directory needs the correct SELinux label, since a confined rsyslog cannot write to a freshly mounted volume that still carries the wrong context.

Log files grow without bound unless something trims them, and that something is logrotate. It runs on a schedule (a daily systemd timer on RHEL 9) and, for each managed path, decides whether to rotate based on the directives you give it.

The directives in this config each do one job. daily sets the cadence and rotate 30 keeps thirty old copies, so together they retain a month. compress gzips the rotated files to save space. missingok stops logrotate complaining if a matching file does not exist yet. notifempty skips rotation when the file is empty, which avoids a pile of useless empty archives.

The postrotate … endscript block runs a command after rotation. Here it restarts rsyslog so it reopens its file handles and writes to the fresh file rather than the now-renamed old one. That block must be closed with endscript or the whole config fails to parse. Test with logrotate —debug, which simulates a run and reports what it would do without actually rotating anything.

A good alert script does three things: it makes a decision, it records that decision somewhere durable, and it signals the result through its exit code so other tools can react. This one does all three.

THRESHOLD=${1:-80} is parameter expansion with a default: use the first argument if given, otherwise 80. That makes the script usable both interactively and from a timer that passes a stricter number. The exit code is the important contract: the script returns 1 if any filesystem is over the limit and 0 if all are fine, so a monitoring system can treat a nonzero exit as a real alert.

Parsing df output is where these scripts quietly break. Plain df wraps long device names onto a second line, which throws off a fixed NR==2 field grab. df -P forces the POSIX single-line format, so the use-percentage is always on the second line in the fifth column. Stripping the % with tr then gives a clean integer for the numeric comparison.

Counting matches in a file is a one-line job with grep, but two flags and one gotcha decide whether it is correct. -i makes the match case-insensitive, so Error, ERROR, and error all count; without it you silently miss most real-world log lines. -c prints the count instead of the matching lines.

The gotcha is grep’s exit code. When it finds no matches it still prints 0, but it also exits with status 1. A reflex like grep -ci pattern file || echo 0 therefore fires the fallback on a zero-match file and prints a second zero, turning a clean count into a corrupted two-line value. The fix is simply not to add the fallback: capture grep -c directly, since it already reports zero correctly.

The loop also guards against an empty match. When a glob like /var/log/remote/*.log matches nothing, the shell leaves the literal pattern in place, so [ -f “$f” ] || continue skips that non-file and keeps the script from reporting a phantom host.

This task uses both schedulers on purpose, because cron and systemd timers answer the question differently. cron is calendar-based and terse: */30 * * * * means every thirty minutes, 0 0 * * * means midnight. It is perfect for simple, time-of-day jobs.

A systemd timer is a unit that activates another unit on a schedule, and it comes in two flavours. OnCalendar is wall-clock based like cron. OnBootSec and OnUnitActiveSec, used here, are monotonic: the first fires a fixed time after boot, the second a fixed interval after the last run. So this timer runs five minutes after boot and then every hour, regardless of the absolute time.

The key habit is what you enable. The timer drives a paired Type=oneshot service, so you enable —now the timer, not the service. Enabling the service would just run it once. And after writing any unit file you run daemon-reload so systemd reparses it before you enable it. Timers install into timers.target.

Service hardening on a server is mostly about reducing what is running to only what is needed. Every active service is a potential attack surface and a consumer of resources, so a logging server has no business running a print spooler (cups) or a desktop discovery daemon (avahi-daemon).

The distinction that trips people up is start/stop versus enable/disable. start and stop act on the running system right now; enable and disable control whether the unit comes up at boot. A service you only stop will quietly return after the next reboot, which is exactly the kind of thing the post-reboot check catches. disable —now does both at once: stop it and keep it from starting again. (mask goes further, making a unit impossible to start at all, but disable is enough here.)

After changing service states, systemctl list-units —state=failed surfaces anything that died or refused to start. Capturing it to a file is good operational hygiene: it is a snapshot you can compare against later.

Standard permissions cannot express what this task needs: two accounts in the same group that must have different access to the same directory. That is precisely the job of POSIX ACLs, which add named entries beyond the single owner, group, and other.

The service accounts come first. logadmin1 and logadmin2 exist to own access, not to log in, so they get -s /sbin/nologin, a shell that refuses interactive sessions. Their group logadmins owns the directory, and the base mode 750 gives the owning group read and execute while shutting out everyone else.

On top of that, named ACLs split the two users: u:logadmin1:rwx can write, u:logadmin2:r-x can only read. The -d (default) ACL is the part that handles the future: it is a template applied to files created later, so new archives automatically grant the logadmins group read and write without anyone re-running setfacl. Read the result with getfacl, and remember the mask caps the effective rights of every named entry.

Running a container is easy; making it come back after a reboot is the real task. The approach is to let podman generate a systemd unit that manages the container’s lifecycle.

Two flags on the podman run line carry weight on RHEL. :Z on the volume mount tells podman to relabel the host directory with a private SELinux category so the confined container can actually read it; without it, SELinux denies access even though the bind mount exists. -p 9090:80 publishes the container’s port 80 on the host’s 9090.

podman generate systemd —new writes a unit that creates a fresh container on every start instead of pinning to one container ID, so it keeps working after a reboot or a recreation. The catch: because the unit will create its own container named logviewer, you must remove the one you started by hand first, or enabling the service collides on the name. On newer RHEL, generate systemd is being replaced by Quadlet, where you describe the container in a .container file under /etc/containers/systemd and systemd builds the service for you; the generate approach still works and is what most current exam material uses.

Growing storage while the system stays online is one of the strongest arguments for LVM, and it is a two-layer operation. The logical volume is the container; the filesystem is what lives inside it. Making more room means enlarging both, in order.

lvextend -L +1G hands another gigabyte of free extents from the volume group to lv_logs. At that instant the block device is bigger, but the filesystem still believes it is the old size, so the extra space is unusable until you grow the filesystem into it.

xfs_growfs does that second step, and three details matter. It takes the mount point, not the device path. It works fully online, with the filesystem mounted and in use, so there is no downtime. And XFS only ever grows: it cannot be shrunk, which is a deliberate design tradeoff, so plan sizes with that in mind. You can collapse both steps into lvextend -r, which resizes the filesystem automatically, but doing them separately makes each layer visible.

These are the small persistent settings that define a server’s identity and timekeeping. hostnamectl set-hostname writes the name to /etc/hostname so it survives reboot, unlike the transient hostname command. Adding the name to /etc/hosts lets the machine resolve itself without DNS.

Accurate time is not optional on a log server: correlating events across hosts depends on synchronised clocks, and timestamps that drift make an audit trail useless. chrony is the RHEL 9 time daemon. You point it at a source with a server line, where iburst makes the first sync happen quickly. Comment out the shipped pool and server lines first so chrony is not also averaging in defaults. Timezone is a separate concern from synchronisation: timedatectl set-timezone repoints /etc/localtime, changing how time is displayed, not how it is kept.

Finally, systemctl set-default multi-user.target sets the boot target. multi-user.target is the text, networked, no-GUI mode appropriate for a headless server; graphical.target would pull in a desktop this machine does not need.

The reboot is the real exam, not a formality. Nearly every failure on a hands-on test traces to one thing: a setting that is correct in memory right now but was never written somewhere that survives a restart.

Sort each task into two buckets as you verify. On-disk state persists by nature: the fstab entries, ACLs stored in the filesystem, the logrotate and rsyslog configs, the unit files, the SELinux labels you fixed with restorecon. Runtime state does not: a service or timer you started but never enabled, a firewall rule added without —permanent, a hostname set with the wrong command.

The verbs that catch most of it are enable (start at every boot) versus start (start now), —permanent on firewall changes, and daemon-reload so systemd sees new units before you enable them. Reboot, then walk the checklist line by line. Anything missing was runtime-only, and now you know exactly where to go back and make it stick.