Home / Learn Linux / Lesson 17

Signals and jobs: kill, bg, fg and nohup

Stop, pause and resume programs, and keep them running after you log out.

Lesson

17 of 30

Time

About 35 minutes

On this page

1. Signals: messages to a process
2. kill: send a signal by PID
3. Killing by name: pkill, killall and pidof
4. Jobs: foreground and background
5. Keep it running after you log out
6. Priority: nice and renice
7. Common mistakes
8. Practice
9. Cheat sheet

Signals: messages to a process

You talk to running processes by sending them signals: small numbered messages like "please stop", "stop now" or "reload your settings". A program can choose how to react to most of them. kill -l lists them all, but you'll only use a handful:

The difference between TERM and KILL is the most important thing on this page. TERM lets a program save its work, close its files and remove its temporary data. KILL pulls the plug, so a database killed with -9 may have to repair itself on the next start. Always try TERM first.

kill: send a signal by PID

Despite the name, kill just sends a signal, and TERM is the default:

$ sleep 600 &
[1] 189512
$ ps -o pid,cmd -p $!
    PID CMD
 189512 sleep 600
$ kill $!
$ ps -o pid,cmd -p $!
    PID CMD
[1]+  Terminated                 sleep 600

Bash reported that the job was Terminated, and ps no longer finds it.

Here's why TERM matters. This little program catches TERM and cleans up before it exits, the way a well-written service does (lesson 29 explains trap):

$ bash -c 'trap "echo cleaning up; rm -f /tmp/work.lock; exit" TERM; touch /tmp/work.lock; while true; do sleep 1; done' &
[1] 189567
$ sleep 1; ls /tmp/work.lock
/tmp/work.lock
$ kill $!
$ sleep 1; ls /tmp/work.lock
cleaning up
ls: cannot access '/tmp/work.lock': No such file or directory
[1]+  Done                       bash -c 'trap "echo cleaning up; rm -f /tmp/work.lock; exit" TERM; touch /tmp/work.lock; while true; do sleep 1; done'

It got the message and removed its lock file. Now the same thing with KILL:

$ bash -c 'trap "echo cleaning up; rm -f /tmp/work.lock; exit" TERM; touch /tmp/work.lock; while true; do sleep 1; done' &
[1] 189628
$ sleep 1; ls /tmp/work.lock
/tmp/work.lock
$ kill -9 $!
$ sleep 1; ls /tmp/work.lock
[1]+  Killed                     bash -c 'trap "echo cleaning up; rm -f /tmp/work.lock; exit" TERM; touch /tmp/work.lock; while true; do sleep 1; done'
/tmp/work.lock

No cleanup, and the lock file was left behind. Next time that program starts, it might refuse to run because it thinks another copy is still going. That's the real cost of kill -9.

Use kill -9 only when a process ignores TERM for a few seconds. And remember you can only signal your own processes. Anyone else's needs sudo.

Try it

1. Start sleep 900 &, find its PID with pgrep, and stop it with TERM.
2. What does kill -l 15 print?

$ sleep 900 &
[1] 189689
$ pgrep -a sleep
189689 sleep 900
$ kill $(pgrep sleep)
$ kill -l 15
TERM
[1]+  Terminated                 sleep 900

Killing by name: pkill, killall and pidof

Looking up PIDs gets tedious. These work with names instead:

$ sleep 700 &
[1] 189744
$ sleep 800 &
[2] 189745
$ pidof sleep
189745 189744
$ killall sleep
[1]-  Terminated                 sleep 700
$ pidof sleep; echo $?
[2]+  Terminated                 sleep 800
1

Careful with pkill: its pattern matches anywhere in the name, so pkill ssh would also hit sshd, the SSH server, and you'd lock yourself out of a remote server. Check what matches with pgrep -a pattern first, the same habit as ls before rm.

Jobs: foreground and background

A command normally runs in the foreground: your terminal waits until it finishes. Add & to run it in the background, and you get your prompt back straight away. Bash calls each of these a job, and numbers them:

$ sleep 500 &
[1] 189801
$ sleep 600 &
[2] 189802
$ jobs
[1]-  Running                    sleep 500 &
[2]+  Running                    sleep 600 &
$ kill %1 %2

%1 and %2 refer to jobs by number, and work anywhere a PID would, like the kill %1 %2 that cleaned them up.

Ctrl+Z, bg and fg

Ctrl+Z pauses the foreground job and gives you the prompt back. Then bg lets it carry on in the background, and fg brings it back to the foreground:

$ sleep 300
^Z
[1]+  Stopped                    sleep 300
$ jobs
[1]+  Stopped                    sleep 300
$ bg %1
[1]+ sleep 300 &
$ jobs
[1]+  Running                    sleep 300 &
$ kill %1

That's the rescue for a common mistake: you started something long without &, and you need your terminal back. Press Ctrl+Z, then bg. You saw the same T state in ps in lesson 16; it means stopped.

Try it

1. Start sleep 400 in the foreground, pause it, then let it continue in the background, and check with jobs.
2. Bring it back to the foreground and stop it with Ctrl+C.

$ sleep 400
^Z
[1]+  Stopped                    sleep 400
$ bg
[1]+ sleep 400 &
$ jobs
[1]+  Running                    sleep 400 &
$ fg
sleep 400
^C
$ jobs

Keep it running after you log out

Background jobs belong to your shell. When you close the terminal or your SSH connection drops, the shell sends them SIGHUP and they usually die. To keep something running:

• nohup command & starts it immune to SIGHUP, and saves its output in nohup.out.
• disown %1 tells Bash to forget a job that's already running, so it isn't sent SIGHUP.

$ mkdir ~/nohup-lab
$ cd ~/nohup-lab
$ nohup bash -c 'for i in 1 2 3; do echo "step $i"; sleep 1; done' &
[1] 189963
$ sleep 4
nohup: ignoring input and appending output to 'nohup.out'
[1]+  Done                       nohup bash -c 'for i in 1 2 3; do echo "step $i"; sleep 1; done'
$ cat nohup.out
step 1
step 2
step 3

For anything you'll come back to, use tmux instead. It's a terminal that keeps running on the server, so you can detach, log out, log back in later and reattach to exactly where you left off:

$ tmux new -s work

Inside tmux, run your long job, then press Ctrl+B followed by D to detach. Log out if you like. Later:

$ tmux ls
$ tmux attach -t work

And for something that must always run, like a web app, neither of these is right. That's what services are for (lesson 26): systemd starts them at boot and restarts them when they crash.

Priority: nice and renice

When several processes want the CPU, the nice value decides who gets more. It runs from -20 (greedy) to 19 (very polite), and 0 is normal. Start a heavy job politely with nice, so it doesn't slow down everything else:

$ nice -n 5 sleep 200 &
[1] 190021
$ ps -o pid,ni,cmd -p $!
    PID  NI CMD
 190021   5 sleep 200
$ renice -n 15 -p $!
190021 (process ID) old priority 5, new priority 15
$ renice -n 5 -p $!
renice: failed to set priority for 190021 (process ID): Permission denied
$ ps -o pid,ni,cmd -p $!
    PID  NI CMD
 190021  15 sleep 200
$ kill %1

Anyone can make their own processes nicer. But going back the other way, even to where it was, counts as asking for more CPU, and that needs sudo. So does any negative number.

Common mistakes

• **Reaching for kill -9 first.** It skips cleanup. TERM, wait a few seconds, then KILL.
• **pkill matching too much.** Check with pgrep -a first. pkill ssh can kill your own connection.
• Long jobs started over SSH without tmux or nohup. If the connection drops, the job dies. Start important work inside tmux.
• Forgetting a job is paused. A Ctrl+Z'd job does nothing until you bg or fg it. jobs shows Stopped.
• **kill on someone else's process without sudo.** You'll get Operation not permitted.

Practice

1. Start three sleep commands in the background with different times, list them with their PIDs, then kill the second one by job number.

$ sleep 301 &
[1] 190078
$ sleep 302 &
[2] 190079
$ sleep 303 &
[3] 190080
$ jobs -l
[1]  190078 Running                    sleep 301 &
[2]- 190079 Running                    sleep 302 &
[3]+ 190080 Running                    sleep 303 &
$ kill %2
$ jobs
[1]   Running                    sleep 301 &
[2]-  Terminated                 sleep 302
[3]+  Running                    sleep 303 &

1. Start yes > /dev/null in the foreground, pause it, check its state with ps, then kill it.

$ yes > /dev/null
^Z
[1]+  Stopped                    yes > /dev/null
$ ps -o pid,stat,cmd -C yes
    PID STAT CMD
 190133 T    yes
$ kill %1

[1]+  Stopped                    yes > /dev/null
$ jobs
[1]+  Terminated                 yes > /dev/null

1. Start sleep 1000 as a background job, then use pkill to stop every sleep process you own, checking first what will match.

$ sleep 1000 &
[1] 190187
$ pgrep -a -u ubuntu sleep
190187 sleep 1000
$ pkill -u ubuntu sleep
$ pgrep -a -u ubuntu sleep; echo $?
[1]+  Terminated                 sleep 1000
1

1. Run a 3-second loop with nohup in the background in ~/nohup-lab, appending to nohup.out, and read the result.

$ cd ~/nohup-lab
$ nohup bash -c 'echo "run 2 started"; sleep 3; echo "run 2 done"' &
[1] 190243
$ sleep 4
nohup: ignoring input and appending output to 'nohup.out'
sleep 4
[1]+  Done                       nohup bash -c 'echo "run 2 started"; sleep 3; echo "run 2 done"'
$ tail -n 2 nohup.out
run 2 started
run 2 done

1. What's the difference between kill 1234, kill -15 1234 and kill -TERM 1234?

Cheat sheet

← Processes: ps, top and htopNetwork basics: ip, static IPs and troubleshooting →