Root recovery.
rd.break, chroot, passwd, and the relabel.

2:47 AM call. The root password was changed during a DR drill and not written down, there is no sudo user, and root SSH is off. Recover it from the console with rd.break, without a reinstall.

The phone goes at 2:47 AM. During the DR drill someone changed root’s password and never wrote it down, there is no second admin with sudo, and root SSH was turned off two tickets ago, so there is no working login left. The box is up and serving traffic and completely locked to you. ops1 wants it back without a reinstall, and the clock is running the whole time.

This is a memorize-it-cold procedure: five minutes once you know it, a very long night if you do not. Every step has a reason, and the one people skip, the SELinux relabel, is exactly what locks you out a second time even after the password is reset. Console access only, and snapshot the VM before you touch anything.

The ticket

This lab reboots the machine and needs console access, not SSH. Snapshot the VM before you start so you have a way back if anything goes sideways.

# take a hypervisor snapshot named pre-rd-break, then work from the console

Concept review. The recovery flow, and why each step.

man 7 systemd.special    # emergency.target, rescue.target

rd.break stops the boot after the initramfs but before it switches to the real root. At that point the real root is mounted read-only at /sysroot. You remount it writable, chroot into it so passwd can find the real /etc/shadow, change the password, and then trigger a relabel because the freshly written shadow file ends up with the wrong SELinux context.

1  reboot, hold a key at GRUB
2  press e to edit the boot entry
3  on the linux line, add: rd.break
4  Ctrl-X to boot
5  at switch_root:/# you are in the initramfs, root not switched yet
6  mount -o remount,rw /sysroot
7  chroot /sysroot
8  passwd root
9  touch /.autorelabel     <- the step people forget
10 exit    (leaves chroot)
11 exit    (leaves initramfs, boot continues and relabels)

Break into the initramfs at GRUB

Reboot from the console. When GRUB appears you have a few seconds. Highlight the default entry, press e, find the line that starts with linux, and add rd.break at the end of it, then boot the edited entry.

# from the console:
sudo systemctl reboot

# at GRUB: press e, find the linux line, append a space then rd.break
# then Ctrl-X to boot

# you land at:
# switch_root:/#

Remount and chroot

The real root is mounted read-only at /sysroot. Remount it read-write, then chroot so the rest of the commands act on the real system.

mount | grep sysroot          # shows /sysroot mounted ro
mount -o remount,rw /sysroot
chroot /sysroot              # now effectively root in the real system

Reset the password and relabel

Set the new password, then create /.autorelabel. That file tells the next boot to relabel every file to match current policy, which fixes the context on the shadow file you just changed.

passwd root                  # enter the new password twice
grep root /etc/shadow | head -1    # hash is now a fresh $6$ entry

touch /.autorelabel          # critical, or you cannot log in after boot

Exit, reboot, verify

Two exits: the first leaves the chroot, the second leaves the initramfs shell so the boot continues. The relabel adds a few minutes and then the system reboots itself once more and comes up clean.

exit    # leave chroot
exit    # leave initramfs, boot continues

# after it relabels and reboots, log in as root with the new password
getenforce    # Enforcing
sestatus

The step everyone forgets

If you skip touch /.autorelabel, the system boots but SELinux denies access to the relabeled-wrong shadow file, and you cannot log in even with the correct password. If that happens, redo rd.break and fix just that one file.

# inside the chroot, the targeted fix if you forgot the autorelabel:
restorecon -v /etc/shadow

Exam questions

Write the answer first.

Q1. What kernel parameter drops you into a debug shell before the real root mounts?

Q2. After resetting root with rd.break, what step is critical for a normal boot, and why?

Q3. Walk through every command, in order, to recover root on RHEL 9 with rd.break, assuming console access and no other admin.

Answers.

# A1
rd.break    # stops in initramfs, real root is ro at /sysroot
# A2
touch /.autorelabel    # passwd left /etc/shadow with the wrong SELinux
# context; the relabel fixes it so SELinux can read it at next boot
# A3
mount -o remount,rw /sysroot
chroot /sysroot
passwd root
touch /.autorelabel
exit
exit

Final checklist: confirm everything works

If every check passes, the ticket is done.

# 1. snapshot taken before the lab
# 2. entered the initramfs shell via rd.break
# 3. remounted /sysroot rw and chrooted
# 4. passwd root succeeded and /.autorelabel created
# 5. system relabeled, rebooted, login with the new password works
getenforce
# 6. all three exam answers written from scratch
# 7. tracker entry checked off

Reply to ops1: root recovered via rd.break, chroot, passwd, and autorelabel. SELinux back to enforcing, total downtime about six minutes including the relabel. Adding the procedure to the runbook.