Processes: ps, top and htop

See what is running, who started it and what it is costing the machine.

Lesson: 16 of 30
Time: About 40 minutes

What a process is

A program is a file on disk, like /usr/bin/bash. A process is that program running: loaded into memory, using CPU time, with files open. The same program can run as many processes at once. Every terminal you open runs its own bash.

Linux tracks every process with a few key facts:

	Means
PID	Process ID, a unique number
PPID	The parent's PID: every process is started by another one
User	Whose rights it runs with
State	Running, sleeping, stopped, or a zombie
Command	The program and its arguments

The user part ties back to lesson 14. A process can only do what its user is allowed to do. A web server running as www-data can't read /etc/shadow even if it's hacked, which is why services run as their own users rather than as root.

Your shell is a process too, and $$ holds its PID:

$ echo $$
190296

ps: a snapshot of processes

ps on its own shows just the processes in your terminal:

$ ps
    PID TTY          TIME CMD
 190348 pts/0    00:00:00 bash
 190355 pts/0    00:00:00 ps

To see everything, two styles are common. They come from two older Unix families, and both still work:

$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.1  0.4  27972 18404 ?        Ss   11:06   0:05 /usr/
root           2  0.0  0.0      0     0 ?        S    11:06   0:00 [kthr
root           3  0.0  0.0      0     0 ?        S    11:06   0:00 [pool
root           4  0.0  0.0      0     0 ?        I<   11:06   0:00 [kwor
root           5  0.0  0.0      0     0 ?        I<   11:06   0:00 [kwor
...
$ ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 11:06 ?        00:00:05 /usr/lib/systemd/sys
root           2       0  0 11:06 ?        00:00:00 [kthreadd]
root           3       2  0 11:06 ?        00:00:00 [pool_workqueue_rele
root           4       2  0 11:06 ?        00:00:00 [kworker/R-rcu_gp]
root           5       2  0 11:06 ?        00:00:00 [kworker/R-sync_wq]
...

The columns in ps aux:

Column	Means
`USER`, `PID`	Owner and process ID
`%CPU`, `%MEM`	Share of the CPU and of the memory it's using
`VSZ`, `RSS`	Virtual memory size, and the memory it's actually holding (KiB)
`TTY`	Which terminal it's attached to (`?` means none, like a service)
`STAT`	State: `R` running, `S` sleeping, `D` waiting on the disk, `T` stopped, `Z` zombie
`START`, `TIME`	When it started, and how much CPU time it has used
`COMMAND`	What it is. Names in `[brackets]` are kernel threads

Most of the time everything is S, sleeping, waiting for something to do. That's normal. A process stuck in D for a long time usually means a slow or failing disk.

Picking the columns and the order

-o chooses columns, --sort orders them (a - means biggest first), and head keeps the top. This answers "what's eating my memory?":

$ ps -eo pid,user,%mem,rss,comm --sort=-%mem | head -n 6
    PID USER     %MEM   RSS COMMAND
 177240 root      0.9 39588 fwupd
    951 root      0.8 34652 unattended-upgr
    859 root      0.7 28796 networkd-dispat
 145176 root      0.6 27852 packagekitd
    869 root      0.6 27536 udisksd

And this, "what's eating my CPU?". To give it something to find, start yes, which prints y forever and keeps one CPU busy (lesson 17 explains the & and kill %1). Give it a few seconds before running ps, because ps averages %CPU over a process's whole life, so anything started a moment ago looks idle:

$ yes > /dev/null &
[1] 190568
$ ps -eo pid,user,%cpu,comm --sort=-%cpu | head -n 4
    PID USER     %CPU COMMAND
 190568 ubuntu   99.6 yes
 190546 ubuntu    0.3 python3
 177240 root      0.1 fwupd
$ kill %1

Other useful selections: ps -u ubuntu shows one user's processes, ps -p 1 one process by PID, and ps -o pid,ppid,cmd with your shell's PID shows its parent:

$ ps -o pid,ppid,user,cmd -p $$
    PID    PPID USER     CMD
 190617  190602 ubuntu   bash -i

Finding processes by name: pgrep

ps aux | grep name works, but it also finds the grep itself. pgrep is cleaner: it prints the PIDs of processes whose name matches, -a adds the full command, -u limits it to one user:

$ pgrep -a sshd
895 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
1249 sshd-session: ubuntu [priv]
1251 [sshd] <defunct>
1299 sshd-session: ubuntu@notty
$ pgrep -c -u root
95

Try it

How many processes are running in total?
Which three processes use the most memory?
What's the PID of cron?

Answer

$ ps -e --no-headers | wc -l
109
$ ps -eo pid,%mem,comm --sort=-%mem | head -n 4
    PID %MEM COMMAND
 177240  0.9 fwupd
    951  0.8 unattended-upgr
    859  0.7 networkd-dispat
$ pgrep cron
848

pstree: who started what

Every process has a parent, all the way back to PID 1. pstree draws the family tree, and -p adds PIDs:

$ pstree -p
systemd(1)─┬─ModemManager(1014)─┬─{ModemManager}(1036)
           │                    ├─{ModemManager}(1038)
           │                    └─{ModemManager}(1044)
           ├─agetty(905)
           ├─agetty(909)
           ├─chronyd-starter(846)───chronyd(934)───chronyd(946)
           ├─cron(848)
           ├─dbus-daemon(850)
           ├─fwupd(177240)─┬─{fwupd}(177244)
           │               ├─{fwupd}(177245)
           │               └─{fwupd}(177246)
           ├─multipathd(460)─┬─{multipathd}(466)
...

PID 1 is systemd, the first process the kernel starts. It starts every service, and it adopts any process whose parent dies. Lesson 26 is all about it.

To see one process's ancestors, give pstree its PID and -s. Here's cron, the scheduler from lesson 24, started directly by systemd:

$ pstree -s -p $(pgrep -o cron)
systemd(1)───cron(848)

top: live view

ps is a snapshot. top refreshes every few seconds, sorted by CPU, until you press q. Here's one screen of it (-b -n 1 prints a single update instead of taking over the terminal, which is handy in scripts):

$ top -b -n 1
top - 12:20:27 up  1:14,  1 user,  load average: 0.10, 0.11, 0.06
Tasks: 108 total,   1 running, 106 sleeping,   0 stopped,   1 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,
MiB Mem :   3895.8 total,   2602.9 free,    475.5 used,   1007.7 buff/ca
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   3420.3 avail M

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
      1 root      20   0   27972  18404  10556 S   0.0   0.5   0:05.81
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00
      3 root      20   0       0      0      0 S   0.0   0.0   0:00.00
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00
      7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00
...

The header is a whole health check in five lines:

Line	Tells you
1	Time, uptime, users, and the load average for the last 1, 5 and 15 minutes
2	How many processes, and how many are running, sleeping, stopped or zombies
3	Where CPU time goes: `us` your programs, `sy` the kernel, `id` idle, `wa` waiting for disk
4, 5	Memory and swap. Look at `avail Mem`, as with `free` in lesson 1

Reading the load average

The load average is roughly how many processes wanted the CPU (or were waiting on the disk) on average. Compare it with your number of CPUs (nproc):

Below the CPU count: the machine has room to spare.
Around the CPU count: fully busy.
Well above it: things are queueing and slowing down.

Read the three numbers together. 4.0, 1.0, 0.5 on a 2-CPU machine means it just got busy. 4.0, 4.0, 4.0 means it's been overloaded for a quarter of an hour.

Keys inside top

Key	Does
`P` / `M`	Sort by CPU / memory
`1`	Show each CPU separately
`u`	Show only one user's processes
`c`	Show full commands
`k`	Kill a process (asks for the PID; lesson 17)
`h`	Help
`q`	Quit

htop: top, but friendlier

You installed htop in lesson 1. It shows the same information with colour bars for each CPU and for memory, and you can scroll and use the mouse. Run htop, then:

Key	Does
F3 or `/`	Search for a process
F4 or `\`	Filter, showing only matching processes
F5 or `t`	Tree view, like `pstree`
F6	Choose the sort column
F9	Send a signal (kill)
F10 or `q`	Quit

Use top when you're on a server where htop isn't installed, and htop everywhere else.

Everything about a process: /proc

Remember /proc from lesson 4? Every process has a directory there named after its PID, full of live details. status is the readable summary:

$ grep -E "^(Name|State|PPid|Uid|VmRSS|Threads)" /proc/$$/status
Name:   bash
State:  S (sleeping)
PPid:   191082
Uid:    1000    1000    1000    1000
VmRSS:      6264 kB
Threads:        1
$ ls /proc/1 | head -n 12
attr
autogroup
auxv
cgroup
clear_refs
cmdline
comm
coredump_filter
cwd
environ
exe
fd

Tools like ps and top are reading these files for you.

Watch it happen

Start something that keeps a CPU busy, so you have something to find. yes prints y forever, and sending it to /dev/null keeps your screen clean. The & runs it in the background (lesson 17), and $! is the PID of the last background job:

$ yes > /dev/null &
[1] 191211
$ ps -o pid,stat,%cpu,cmd -p $!
    PID STAT %CPU CMD
 191211 R    99.6 yes
$ top -b -n 1 | head -n 8 | tail -n 2
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 191211 ubuntu    20   0   17364   7112   4912 R 100.0   0.2   0:03.21 yes
$ kill %1

yes shows up at the top of top with nearly 100% of one CPU. kill %1 stops it, and lesson 17 explains that line.

Common mistakes

**Panicking at high VSZ.** Virtual size includes memory a process could use but isn't holding. Look at RSS for real use.
**Reading %CPU above 100% as an error.** In top, it's per CPU, so a busy program on a 4-CPU machine can show 400%.
**Trusting ps aux | grep x counts.** The grep itself shows up. Use pgrep -c x.
**Judging the load average without nproc.** A load of 3 is overloaded on 2 CPUs and quiet on 16.
Worrying about zombies. A zombie (Z) has already finished and is only waiting for its parent to collect its exit code. It uses no CPU or memory. Many of them point to a bug in the parent program.

Practice

Show the PID, parent PID and command of every process belonging to ubuntu.

Answer

$ ps -u ubuntu -o pid,ppid,cmd
    PID    PPID CMD
   1134       1 /usr/lib/systemd/systemd --user
   1136    1134 (sd-pam)
   1299    1249 sshd-session: ubuntu@notty
 191247    1299 python3 /tmp/session.py /tmp/session-cmds
 191262  191247 bash -i
 191269  191262 ps -u ubuntu -o pid,ppid,cmd

Find the PID of the SSH server (sshd) and show how long it has been running.

Answer

$ ps -o pid,etime,cmd -p $(pgrep -o sshd)
    PID     ELAPSED CMD
    895    01:14:24 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 start

-o with pgrep gives the oldest match, which is the main server process. etime is the elapsed time since it started.

Start two sleep 300 & processes, then count how many sleep processes are running.

Answer

$ sleep 300 &
[1] 191376
$ sleep 300 &
[2] 191377
$ pgrep -c sleep
2

Show only the first five lines of top, once.

Answer

$ top -b -n 1 | head -n 5
top - 12:20:31 up  1:14,  1 user,  load average: 0.17, 0.13, 0.07
Tasks: 109 total,   1 running, 107 sleeping,   0 stopped,   1 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3895.8 total,   2606.4 free,    472.0 used,   1007.7 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   3423.8 avail Mem

Show every process the SSH server has started, with PIDs. (Your own connection is in there.)

Answer

$ pstree -p $(pgrep -o sshd)
sshd(895)

What's the load average right now, and is it high for this machine?

Answer

$ uptime
 12:20:32 up  1:14,  1 user,  load average: 0.17, 0.13, 0.07
$ nproc
2

Compare the first load number with the CPU count. Below it means the machine has room.

Cheat sheet

Command	What it does
`ps`	Processes in this terminal
`ps aux`, `ps -ef`	Every process
`ps -eo pid,user,%cpu,%mem,comm --sort=-%cpu`	Pick columns, busiest first
`ps -u user`, `ps -p PID`	One user's, one process
`pgrep -a name`, `pgrep -c name`	Find by name, count
`pstree -p`, `pstree -s PID`	Process tree, ancestors of PID
`top`, `top -b -n 1`	Live view, one snapshot
`P`, `M`, `1`, `k`, `q` (in top)	Sort CPU, sort memory, per CPU, kill, quit
`htop`	Friendlier live view (F5 tree, F9 kill)
`uptime`, `nproc`	Load average, CPU count
`/proc/PID/status`	Everything about one process
`$$`, `$!`	This shell's PID, the last background job's PID

← A shared team folder Signals and jobs: kill, bg, fg and nohup →

Processes: ps, top and htop

On this page

What a process is

ps: a snapshot of processes

Picking the columns and the order

Finding processes by name: pgrep

Try it

pstree: who started what

top: live view

Reading the load average

Keys inside top

htop: top, but friendlier

Everything about a process: /proc

Watch it happen

Common mistakes

Practice

Cheat sheet