Runaway processes.
ps, signals, nice, renice, and job control.
Load average hit 18 on a four-core box from two runaway processes. Find the PIDs, kill them gracefully before forcing, renice a backup so it stops competing with production, and snapshot the top consumers.
Monitoring is screaming: load average 18.42 on a four-core box, which means work is stacking up nine deep for every core it has. top points at two processes owned by dbadmin1, a find / that got stuck and a dd that ran away, and dbadmin1 is not answering Slack. ops1 wants them gone, but gone carefully: graceful first, force only if they ignore it, because whether they die clean is itself a clue for the postmortem.
The instinct under pressure is kill -9 on everything, and that is the mistake. A graceful signal gives a process the chance to flush its buffers and release its locks. Force is what you escalate to when it refuses, not what you open with. The rest of the ticket is reprioritizing a backup so it stops elbowing production, and grabbing a top-consumers snapshot for the incident.
The ticket
Set up the chaos yourself so you can practice fixing it. These run as the scenario users in the background.
id alice dbadmin1 ops1 sudo -u dbadmin1 bash -c 'find / -type f 2>/dev/null > /dev/null &' sudo -u dbadmin1 bash -c 'dd if=/dev/zero of=/dev/null &' sudo -u alice bash -c 'yes > /dev/null &'
Concept review. ps, signals, and priority.
man ps man 7 signal man nice renice
ps aux every process, with %CPU and %MEM ps -ef full command line and parent PID ps -u dbadmin1 one user's processes ps -eo pid,user,%cpu,%mem,comm --sort=-%cpu custom columns, sorted pgrep -u dbadmin1 -a PIDs by user, with command line
SIGTERM 15 the default kill, asks a process to exit cleanly SIGKILL 9 uncatchable, the kernel destroys it, no cleanup SIGSTOP 19 pause, uncatchable SIGCONT resume a stopped process SIGHUP 1 many daemons reload their config on this
Niceness runs from -20 to +19, where lower means more CPU and higher means nicer to everyone else. Any user can raise their own niceness, but only root can lower it below 0. Set it at start with nice, change it on a running process with renice.
Find the runaway processes
Sort by CPU, then narrow to the one user. pgrep is cleaner than grepping ps, which matches the word anywhere on the line including arguments.
ps -u dbadmin1 -o pid,%cpu,stat,command --sort=-%cpu # PID %CPU STAT COMMAND # 4820 99.7 R dd if=/dev/zero of=/dev/null # 4791 97.3 R find / -type f pgrep -u dbadmin1 -a # confirm the owner and the exact PIDs # 4791 find / -type f # 4820 dd if=/dev/zero of=/dev/null
Graceful termination, then escalate
Send SIGTERM first, wait, then send SIGKILL only to whatever ignored it.
# graceful, SIGTERM is the default sudo pkill -TERM -u dbadmin1 dd sudo pkill -TERM -u dbadmin1 find sleep 5 pgrep -u dbadmin1 -a # still alive? force it sudo pkill -KILL -u dbadmin1 dd pgrep -u dbadmin1 -a || echo "all clear"
kill PID targets one numeric PID, pkill PATTERN matches by name with filters like -u user, and killall NAME matches an exact command name. kill -0 PID sends no signal at all, it just checks whether the process still exists.Renice a running process
Niceness is a live kernel attribute, so you can change it without restarting anything.
pgrep -u alice -a yes # find the PID, say 4221 ps -o pid,ni,comm -p 4221 # current niceness sudo renice -n 15 -p 4221 # much lower priority ps -o pid,ni,comm -p 4221 # or renice every process owned by a user sudo renice -n 10 -u alice
Start a process low, and make it survive logout
Start it niced from the beginning, and combine nohup with & so closing your session does not kill it.
# niced and detached: run as alice, ignore SIGHUP, start at +15, background it sudo -u alice nohup nice -n 15 /usr/local/bin/nightly_backup.sh > /tmp/backup.log 2>&1 & pgrep -u alice -a nightly_backup ps -o pid,ni,user,comm -C nightly_backup.sh # NI should be 15
Snapshot the top consumers
echo '=== top 5 by CPU ===' ps -eo pid,user,%cpu,%mem,comm --sort=-%cpu | head -6 echo '=== top 5 by memory ===' ps -eo pid,user,%cpu,%mem,comm --sort=-%mem | head -6 # or a one-shot non-interactive top dump top -b -n 1 | head -20
Job control and pausing a process
The shell manages background jobs, and you can stop and resume a process with signals.
sleep 300 & sleep 600 & jobs bg %1 # resume job 1 in the background kill %2 # kill job 2 by job number # pause and resume by signal sleep 600 & PID=$! kill -STOP $PID # state goes to T, stopped kill -CONT $PID # resumes kill $PID # clean up
Exam questions
Write the command first.
Q1. Show all processes owned by alice with PID, niceness, and command.
Q2. PID 8421 is pegged at 100% and unresponsive. Send a graceful signal, wait 5 seconds, then force-kill if still running, and verify.
Q3. Start /usr/local/bin/nightly_backup.sh as alice at niceness +15 so it survives an SSH disconnect, then lower it to +19 without restarting.
Answers.
# A1 ps -u alice -o pid,ni,comm # A2 kill 8421 # SIGTERM by default sleep 5 kill -0 8421 2>/dev/null && sudo kill -9 8421 ps -p 8421 || echo "gone" # A3 sudo -u alice nohup nice -n 15 /usr/local/bin/nightly_backup.sh > /tmp/b.log 2>&1 & pgrep -u alice -a nightly_backup # say 9001 sudo renice -n 19 -p 9001
Final checklist: confirm everything works
If every check passes, the ticket is done.
# 1. runaway PIDs identified and confirmed as dbadmin1's pgrep -u dbadmin1 -a || echo 'clear' # 2. SIGTERM sent first, SIGKILL only after a timeout # 3. alice's backup reniced to +15 live, low-priority start documented # 4. top-5 CPU and memory snapshot captured # 5. all three exam commands written from scratch # 6. tracker entry checked off
Reply to ops1: killed dbadmin1’s runaway find and dd, SIGTERM first then SIGKILL on the dd after it ignored the term. Load is back to normal. alice’s backup reniced to +15 live, and future runs wrapped with nice -n 15. Top-5 CPU and memory snapshot attached.