Home / Learn Linux / Lesson 8

Pipes and redirection

Send output into files and into other commands, and keep errors separate from results.

Lesson

8 of 30

Time

About 40 minutes

On this page

1. Three streams
2. Send output to a file: > and >>
3. Errors: 2> and 2>&1
4. Feed a file in: <, here-documents and here-strings
5. Pipes: |
6. tee: save and see
7. Common mistakes
8. Practice
9. Cheat sheet

Three streams

Every command you run gets three channels, called streams:

Results and errors both end up on your screen, so they look the same, but they're separate. That's what makes everything in this lesson possible: you can send results to a file and still see the errors, or throw the errors away and keep the results.

Set up a practice directory with a small fake web server log. Each line is an IP address, a page and a status code (200 means OK, 404 means not found, 500 means a server error):

$ mkdir ~/pipe-lab
$ cd ~/pipe-lab
$ printf "10.0.0.5 /home 200\n10.0.0.7 /login 200\n10.0.0.5 /cart 404\n10.0.0.9 /home 200\n10.0.0.5 /home 200\n10.0.0.7 /pay 500\n10.0.0.2 /home 200\n10.0.0.9 /cart 200\n" > access.log
$ cat access.log
10.0.0.5 /home 200
10.0.0.7 /login 200
10.0.0.5 /cart 404
10.0.0.9 /home 200
10.0.0.5 /home 200
10.0.0.7 /pay 500
10.0.0.2 /home 200
10.0.0.9 /cart 200

Run everything below from ~/pipe-lab.

Send output to a file: > and >>

> sends a command's output into a file instead of the screen. If the file exists, it's emptied first. >> adds to the end instead:

$ echo "first line" > notes.txt
$ echo "second line" >> notes.txt
$ cat notes.txt
first line
second line
$ echo "replaced" > notes.txt
$ cat notes.txt
replaced

That last > wiped both earlier lines without asking. Bash can protect you: set -o noclobber makes > refuse to overwrite an existing file. When you really mean it, >| overwrites anyway:

$ set -o noclobber
$ echo "oops" > notes.txt
bash: notes.txt: cannot overwrite existing file
$ echo "on purpose" >| notes.txt
$ set +o noclobber

set +o noclobber turns it off again. Add set -o noclobber to ~/.bashrc if you want it all the time.

Try it

1. Save the output of ls -l /etc to etc-list.txt, then add the date to the end of the same file.
2. How many lines does etc-list.txt have? Use wc -l (word count, -l counts lines).

$ ls -l /etc > etc-list.txt
$ date >> etc-list.txt
$ wc -l etc-list.txt
197 etc-list.txt

Errors: 2> and 2>&1

> only catches stdout. Errors still go to the screen:

$ ls /etc/hostname /nope > out.txt
ls: cannot access '/nope': No such file or directory
$ cat out.txt
/etc/hostname

The error appeared on the screen, and only the good result went into the file. To catch errors, put the stream's number in front: 2>:

$ ls /etc/hostname /nope 2> errors.txt
/etc/hostname
$ cat errors.txt
ls: cannot access '/nope': No such file or directory

You'll use these combinations all the time:

2>&1 means "send stream 2 wherever stream 1 is going right now". The order matters: > all.txt 2>&1 puts both in the file, but 2>&1 > all.txt sends errors to the screen, because when 2>&1 ran, stream 1 was still the screen.

2> /dev/null is the one you'll type most. find (lesson 9) prints a Permission denied line for every directory you can't read, and this hides them:

$ ls /root /etc/hostname
/etc/hostname
ls: cannot open directory '/root': Permission denied
$ ls /root /etc/hostname 2> /dev/null
/etc/hostname

Try it

1. Run ls /etc/hostname /nope and save both the result and the error in both.txt.
2. Run the same command, but keep only the error on screen and throw away the result.

$ ls /etc/hostname /nope &> both.txt
$ cat both.txt
ls: cannot access '/nope': No such file or directory
/etc/hostname
$ ls /etc/hostname /nope > /dev/null
ls: cannot access '/nope': No such file or directory

Feed a file in: <, here-documents and here-strings

< connects a file to a command's stdin, as if you'd typed the file's contents:

$ wc -l < access.log
8
$ wc -l access.log
8 access.log

With <, wc doesn't know the file's name, so it prints just the number, which is handy in scripts.

A here-document feeds several lines you type straight into a command. It's the classic way to write a small file without opening an editor. Everything up to the line EOF is the input. The word EOF is only a marker, and any word works:

$ cat > hosts.txt << EOF
> web1 10.0.0.5
> web2 10.0.0.7
> db1 10.0.0.9
> EOF
$ cat hosts.txt
web1 10.0.0.5
web2 10.0.0.7
db1 10.0.0.9

The > at the start of those lines is Bash asking for more input. You type the lines; Bash shows >.

A here-string, <<<, feeds a single line:

$ wc -w <<< "how many words are here"
5

Pipes: |

A pipe connects one command's stdout straight to the next command's stdin. No temporary files, and you can chain as many as you like. This is the idea that makes the Linux command line so powerful: lots of small tools, each doing one thing well, joined together.

$ ls /usr/bin | wc -l
1124

That counts the programs in /usr/bin. Pipe anything long into less to read it at your own pace:

$ ls -l /usr/bin | less
$ apt --help | less

Only stdout goes through a pipe. Errors still go to the screen, unless you send them along with 2>&1 | (or |&, which means the same thing).

The tools you'll pipe into

Watch a pipeline build up step by step. Which pages were visited, and how often?

$ cut -d " " -f 2 access.log
/home
/login
/cart
/home
/home
/pay
/home
/cart
$ cut -d " " -f 2 access.log | sort
/cart
/cart
/home
/home
/home
/home
/login
/pay
$ cut -d " " -f 2 access.log | sort | uniq -c
      2 /cart
      4 /home
      1 /login
      1 /pay
$ cut -d " " -f 2 access.log | sort | uniq -c | sort -rn
      4 /home
      2 /cart
      1 /pay
      1 /login

Read it left to right: take the second field (the page), sort so that identical pages sit together, count each run of identical lines, then sort by that count, biggest first. That's a real question answered in one line, and it works just the same on a log with ten million lines.

Try it

1. Which IP address made the most requests?
2. How many requests didn't return 200?
3. Print the pages in uppercase.

$ cut -d " " -f 1 access.log | sort | uniq -c | sort -rn | head -1
      3 10.0.0.5
$ cut -d " " -f 3 access.log | sort | uniq -c
      6 200
      1 404
      1 500
$ cut -d " " -f 2 access.log | tr a-z A-Z
/HOME
/LOGIN
/CART
/HOME
/HOME
/PAY
/HOME
/CART

tee: save and see

tee copies its input to a file and to the screen at the same time, like a T-junction in a pipe. Use it when you want to watch output and keep a copy:

$ cut -d " " -f 3 access.log | sort | uniq -c | tee status-report.txt
      6 200
      1 404
      1 500
$ cat status-report.txt
      6 200
      1 404
      1 500

tee -a adds to the file instead of replacing it.

Writing to root's files with sudo tee

This one catches everyone. You want to add a line to a file that only root may change, so you try sudo:

$ sudo echo "10.0.0.5 web1" >> /etc/hosts
bash: /etc/hosts: Permission denied

It fails because the >> isn't part of the sudo command. Your own shell, running as ubuntu, opens /etc/hosts for writing before sudo even starts. sudo tee fixes it, because then the program writing the file is the one running as root:

$ echo "10.0.0.5 web1" | sudo tee -a /etc/hosts
10.0.0.5 web1
$ tail -n 1 /etc/hosts
10.0.0.5 web1

Add > /dev/null at the end if you don't want tee to echo the line back.

Try it

1. Save the list of IP addresses (just the first field of access.log, sorted, without duplicates) to ips.txt and show it on screen at the same time.
2. Add the line # checked to /etc/hosts with sudo tee, then remove it again with sudo sed -i '$d' /etc/hosts (sed deletes the last line; lesson 12 explains how).

$ cut -d " " -f 1 access.log | sort -u | tee ips.txt
10.0.0.2
10.0.0.5
10.0.0.7
10.0.0.9
$ echo "# checked" | sudo tee -a /etc/hosts > /dev/null
$ tail -n 1 /etc/hosts
# checked
$ sudo sed -i '$d' /etc/hosts
$ tail -n 1 /etc/hosts
127.0.1.1 linux

Common mistakes

• Reading and writing the same file. sort access.log > access.log empties the file, because > truncates it before sort reads it. Write to a new file, then mv it over the old one.
• **sudo with >.** The redirection runs as you, not as root. Use | sudo tee.
• **2>&1 in the wrong place.** It goes after > file: > file 2>&1.
• Expecting errors to go through a pipe. They don't. Use 2>&1 | or |&.
• **uniq without sort.** uniq only merges lines that are next to each other, so unsorted input still has duplicates.
• **> when you meant >>.** One character, and your file is gone. Try set -o noclobber.

Practice

1. Count how many users on the system have /bin/bash as their shell. (/etc/passwd has one user per line, fields separated by :, and the shell is field 7.)

$ cut -d : -f 7 /etc/passwd | sort | uniq -c | sort -rn
     27 /usr/sbin/nologin
      4 /bin/bash
      2 /bin/false
      1 /bin/sync

1. Save the 5 biggest files in /usr/bin (by size) to big.txt, then show the file.

$ ls -lS /usr/bin | head -n 6 | tail -n 5 > big.txt
ls: Broken pipe
$ cat big.txt
-rwxr-xr-x    1 root root    18698056 Sep  4 07:31 perf
-rwxr-xr-x  116 root root    10643488 Sep  3 08:01 coreutils
-rwxr-xr-x    1 root root     8788176 Mar 15  2026 bpftrace
-rwxr-xr-x    1 root root     7503184 Aug 20 10:41 python3.14
-rwxr-xr-x    1 root root     4724640 Mar 15  2026 bpftrace-aotrt

1. Write a three-line file called todo.txt with a here-document, then count its words.

$ cat > todo.txt << EOF
> buy milk
> fix the server
> call mum
> EOF
$ wc -w todo.txt
7 todo.txt

1. Run ls /etc /nope /root, saving the listing to ok.txt and the errors to bad.txt. How many errors were there?

$ ls /etc /nope /root > ok.txt 2> bad.txt
$ wc -l < bad.txt
2
$ cat bad.txt
ls: cannot access '/nope': No such file or directory
ls: cannot open directory '/root': Permission denied

1. Print the different status codes in access.log, each once, in numeric order.

$ cut -d " " -f 3 access.log | sort -nu
200
404
500

1. Sort access.log by status code, highest first, and save it to sorted.log without touching the original.

$ sort -k 3 -rn access.log > sorted.log
$ head -n 3 sorted.log
10.0.0.7 /pay 500
10.0.0.5 /cart 404
10.0.0.9 /home 200

Cheat sheet

← Creating, copying, moving and deleting filesFinding files and text: find, locate and grep →