Moving files over the network: scp, rsync and wget

Copy files between machines, sync folders so only changes move, and download from the web.

Lesson: 20 of 30
Time: About 40 minutes

Set up

You'll copy files between your own user and Ravi's account over SSH, using the key and the lab shortcut from lesson 19. To SSH it's no different from copying to a server across the world. Make something to copy:

$ mkdir -p ~/xfer-lab/site/{css,img}
$ cd ~/xfer-lab
$ echo "<h1>Hello</h1>" > site/index.html
$ echo "body { color: navy; }" > site/css/main.css
$ head -c 200000 /dev/urandom > site/img/photo.bin
$ echo "remember the milk" > notes.txt
$ tree
.
├── notes.txt
└── site
    ├── css
    │   └── main.css
    ├── img
    │   └── photo.bin
    └── index.html

4 directories, 4 files

/dev/urandom produces endless random bytes, and head -c 200000 keeps 200 KB of them, which makes a stand-in for a photo.

scp: copy over SSH

scp works like cp, except either side can be user@host:path:

Command	Copies
`scp file user@host:`	A file into the remote home directory
`scp file user@host:/path/`	Into a remote directory
`scp user@host:/path/file .`	From the server to here
`scp -r dir user@host:`	A whole directory
`scp -P 2222 ...`	Using another port (a capital `P`, unlike `ssh -p`)

The colon is what makes it remote. Without it, scp file ravi@localhost makes a local copy called ravi@localhost.

$ cd ~/xfer-lab
$ scp notes.txt ravi@localhost:
notes.txt                             100%   18   160.2KB/s   00:00
$ scp ravi@localhost:/etc/hostname hostname-copy
hostname                              100%    6    16.0KB/s   00:00
$ cat hostname-copy
linux
$ scp -r site lab:site-copy
photo.bin                             100%  195KB 188.2MB/s   00:00
index.html                            100%   15   115.4KB/s   00:00
main.css                              100%   22   189.1KB/s   00:00
$ ssh lab 'ls -R site-copy'
site-copy:
css
img
index.html

site-copy/css:
main.css

site-copy/img:
photo.bin

The lab name from ~/.ssh/config works here too, so lab:site-copy means "site-copy in Ravi's home directory".

Modern OpenSSH runs scp over SFTP behind the scenes, but the commands are the same. For browsing a server interactively, sftp user@host gives you a little shell with ls, cd, get and put.

Try it

Copy /etc/os-release into Ravi's home directory as os.txt, then print it over SSH.
Copy Ravi's notes.txt back to your current directory as notes-back.txt.

Answer

$ cd ~/xfer-lab
$ scp /etc/os-release lab:os.txt
os-release                            100%  410     3.1MB/s   00:00
$ ssh lab 'head -n 2 os.txt'
PRETTY_NAME="Ubuntu 26.04.1 LTS"
NAME="Ubuntu"
$ scp lab:notes.txt notes-back.txt
notes.txt                             100%   18    61.4KB/s   00:00
$ cat notes-back.txt
remember the milk

rsync: copy only what changed

scp copies everything, every time. rsync compares both sides and sends only the differences, so the second run of a 50 GB backup takes seconds. It's the tool for backups, deployments and syncing folders, locally or over SSH.

The options you'll nearly always use are -a (archive: keep permissions, times and links, and go into subdirectories) and -v (verbose). Add -h for human-readable sizes:

$ cd ~/xfer-lab
$ rsync -avh site/ backup/
sending incremental file list
created directory backup
./
index.html
css/
css/main.css
img/
img/photo.bin

sent 200.42K bytes  received 121 bytes  133.69K bytes/sec
total size is 200.04K  speedup is 1.00

Run it again, and nothing is sent, because nothing changed:

$ cd ~/xfer-lab
$ rsync -avh site/ backup/
sending incremental file list

sent 191 bytes  received 14 bytes  410.00 bytes/sec
total size is 200.04K  speedup is 975.79

Change one file and only that file goes across:

$ cd ~/xfer-lab
$ echo "<p>New line</p>" >> site/index.html
$ rsync -avh site/ backup/
sending incremental file list
index.html

sent 272 bytes  received 37 bytes  618.00 bytes/sec
total size is 200.05K  speedup is 647.42

The trailing slash

This is the one rsync rule everyone gets wrong once:

rsync -a site/ backup/ copies the contents of site into backup.
rsync -a site backup/ copies the directory site into backup, giving backup/site/.

$ cd ~/xfer-lab
$ rsync -a site backup2/
$ ls backup2
site
$ rsync -a site/ backup3/
$ ls backup3
css  img  index.html

Only the source's slash matters. The destination's doesn't.

Mirroring: --delete, and testing with -n

By default rsync never deletes anything at the destination, so files you removed from the source pile up in the backup. --delete makes the destination an exact mirror. Because that deletes things, always try it with -n (dry run) first, which shows what would happen without doing it:

$ cd ~/xfer-lab
$ rm site/css/main.css
$ rsync -avn --delete site/ backup/
sending incremental file list
deleting css/main.css
css/

sent 176 bytes  received 40 bytes  432.00 bytes/sec
total size is 200,031  speedup is 926.07 (DRY RUN)
$ rsync -av --delete site/ backup/
sending incremental file list
deleting css/main.css
css/

sent 176 bytes  received 40 bytes  432.00 bytes/sec
total size is 200,031  speedup is 926.07
$ ls backup/css

deleting css/main.css in the dry run told you exactly what the real run would do.

rsync over SSH

Put user@host: on either side, just like scp. -z compresses the data on the way, which helps on slow connections, and --progress shows each file:

$ cd ~/xfer-lab
$ rsync -avz --progress site/ lab:backup/
sending incremental file list
created directory backup
./
index.html
             31 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=3/5)
css/
img/
img/photo.bin
        200,000 100%  190.73MB/s    0:00:00 (xfr#2, to-chk=0/5)

sent 200,348 bytes  received 102 bytes  400,900.00 bytes/sec
total size is 200,031  speedup is 1.00
$ ssh lab 'ls -R backup'
backup:
css
img
index.html

backup/css:

backup/img:
photo.bin

To use another SSH port or key, give rsync the SSH command: rsync -av -e "ssh -p 2222" src/ user@host:dest/. Or put the details in ~/.ssh/config, and rsync picks them up like scp did.

--exclude leaves things out, using patterns like tar:

$ cd ~/xfer-lab
$ rsync -av --exclude="*.bin" site/ lab:no-images/
sending incremental file list
./
index.html
css/
img/

sent 198 bytes  received 56 bytes  508.00 bytes/sec
total size is 31  speedup is 0.12

Try it

Sync ~/xfer-lab/site/ to backup4/ locally, then run it again and see how much was sent the second time.
Do a dry run of mirroring site/ to Ravi's backup/ with --delete. Would anything be deleted?

Answer

$ cd ~/xfer-lab
$ rsync -ah --stats site/ backup4/ | grep "Total transferred"
Total transferred file size: 200.03K bytes
$ rsync -ah --stats site/ backup4/ | grep "Total transferred"
Total transferred file size: 0 bytes
$ rsync -avn --delete site/ lab:backup/
sending incremental file list

sent 177 bytes  received 21 bytes  396.00 bytes/sec
total size is 200,031  speedup is 1,010.26 (DRY RUN)

--stats prints a summary, and the second run transferred 0 bytes of file data. The dry run deletes nothing, because Ravi's copy already matches.

Downloading: wget and curl

wget downloads files from the web. Give it a URL and it saves the file under its own name, showing progress as it goes. This one is the list of every Ubuntu LTS release, which Ubuntu's own upgrade tool reads:

$ mkdir ~/dl-lab
$ cd ~/dl-lab
$ wget -q https://changelogs.ubuntu.com/meta-release-lts
$ ls -l
total 8
-rw-rw-r-- 1 ubuntu ubuntu 7125 Sep 29 11:03 meta-release-lts
$ head -n 4 meta-release-lts
Dist: dapper
Name: Dapper Drake
Version: 6.06 LTS
Date: Thu, 01 Jun 2006 9:00:00 UTC

-q (quiet) hides the progress output. Other options worth knowing:

Option	Does
`-O name`	Save under another name (`-O -` prints it instead)
`-c`	Continue a download that was interrupted
`-P dir`	Save into a directory
`--limit-rate=1m`	Don't use more than 1 MB/s
`-i list.txt`	Download every URL listed in a file

curl (from lesson 18) can download too, with different defaults: it prints to the screen unless you say otherwise. curl -O URL saves under the remote name, curl -o name URL under one you choose, and -L follows redirects, which most download links use:

$ cd ~/dl-lab
$ curl -sL -o lts.txt https://changelogs.ubuntu.com/meta-release-lts
$ ls -l
total 16
-rw-rw-r-- 1 ubuntu ubuntu 7125 Oct 21 15:48 lts.txt
-rw-rw-r-- 1 ubuntu ubuntu 7125 Sep 29 11:03 meta-release-lts

Use wget for downloading files, especially big ones you might need to resume, and curl for talking to web APIs and testing websites.

Did it arrive intact? Checksums

A checksum is a short fingerprint of a file's contents. Change one byte and the fingerprint is completely different. sha256sum makes one:

$ cd ~/dl-lab
$ sha256sum meta-release-lts lts.txt
cae6a7742c82da9836798a9ce209ef457af46ef9d26bc9fbe3a9944e50f014f8  meta-release-lts
cae6a7742c82da9836798a9ce209ef457af46ef9d26bc9fbe3a9944e50f014f8  lts.txt

Two different downloads of the same file, the same fingerprint: they're identical. Software projects publish the checksums of their downloads, so you can check yours wasn't damaged or tampered with. Ubuntu does it for its images:

$ cd ~/dl-lab
$ wget -q https://cloud-images.ubuntu.com/releases/resolute/release/SHA256SUMS
$ grep -c . SHA256SUMS
74
$ head -n 2 SHA256SUMS
014088e5aeeaf72a90a8f67ad92b764600f583f7ce1268563c4f79c4674e81bf *ubuntu-26.04-server-cloudimg-amd64v3.tar.gz
02d908acd0e358ce81de52e98fe19e84fbf6c6a31678296edd58007530df88f9 *ubuntu-26.04-server-cloudimg-amd64v3-root.tar.xz

sha256sum -c checks files against a list like that. Make a list for your own files, break one, and check again:

$ cd ~/xfer-lab
$ sha256sum site/index.html site/img/photo.bin > sums.txt
$ sha256sum -c sums.txt
site/index.html: OK
site/img/photo.bin: OK
$ echo "tampered" >> site/index.html
$ sha256sum -c sums.txt
site/index.html: FAILED
site/img/photo.bin: OK
sha256sum: WARNING: 1 computed checksum did NOT match

Comparing two files: cmp and diff

cmp tells you whether two files are identical and where the first difference is. It works on any file. diff shows exactly which lines differ in text files:

$ cd ~/xfer-lab
$ cmp site/img/photo.bin backup/img/photo.bin && echo identical
identical
$ cmp site/index.html backup/index.html
cmp: EOF on ‘backup/index.html’ after byte 31, line 2
$ diff backup/index.html site/index.html
2a3
> tampered

In diff's output, > marks lines only in the second file and < lines only in the first. 2a3 means "after line 2 of the first file, add line 3 of the second". diff -u gives the "unified" format you'll see in code reviews, with + and - in front of lines.

Try it

Download https://releases.ubuntu.com/26.04/SHA256SUMS with wget and save it as ubuntu-isos.txt. How many downloads does it list?
Check whether backup/ and backup4/ hold the same index.html.

Answer

$ cd ~/dl-lab
$ wget -q -O ubuntu-isos.txt https://releases.ubuntu.com/26.04/SHA256SUMS
$ wc -l ubuntu-isos.txt
6 ubuntu-isos.txt
$ cd ~/xfer-lab
$ cmp backup/index.html backup4/index.html && echo same
same

Common mistakes

Forgetting the colon. scp file ravi@host makes a local file called ravi@host. It's ravi@host:.
The rsync trailing slash. site/ copies what's inside, site copies the directory itself.
**--delete without -n first.** A mirror can wipe out the wrong side if you swap source and destination. Dry run, read, then run.
**scp -p for the port.** For scp it's a capital -P. Lowercase -p preserves times.
Trusting a download without checking. If a checksum is published, compare it.
**curl printing a binary file to your terminal.** Use -o name or -O.

Practice

Sync ~/xfer-lab/site/ to Ravi's account as deploy/ over SSH, compressed, then show what's there.

Answer

$ cd ~/xfer-lab
$ rsync -az site/ lab:deploy/
$ ssh lab 'find deploy -type f'
deploy/img/photo.bin
deploy/index.html

Delete site/img/photo.bin locally, then make Ravi's deploy/ match exactly, with a dry run first.

Answer

$ cd ~/xfer-lab
$ rm site/img/photo.bin
$ rsync -azn --delete -v site/ lab:deploy/
sending incremental file list
deleting img/photo.bin
img/

sent 149 bytes  received 41 bytes  380.00 bytes/sec
total size is 40  speedup is 0.21 (DRY RUN)
$ rsync -az --delete site/ lab:deploy/
$ ssh lab 'find deploy -type f'
deploy/index.html

Copy Ravi's whole deploy/ directory back to ~/xfer-lab/restored with scp.

Answer

$ cd ~/xfer-lab
$ scp -r lab:deploy restored
index.html                            100%   40   160.2KB/s   00:00
$ tree restored
restored
├── css
├── img
└── index.html

3 directories, 1 file

Make a checksum list of every file in ~/xfer-lab/restored, then check it.

Answer

$ cd ~/xfer-lab
$ find restored -type f -exec sha256sum {} + > restored.sums
$ sha256sum -c restored.sums
restored/index.html: OK

Show the differences between backup/index.html and site/index.html in unified format.

Answer

$ cd ~/xfer-lab
$ diff -u backup/index.html site/index.html
--- backup/index.html   2026-10-21 15:48:21.188949839 +0000
+++ site/index.html     2026-10-21 15:48:41.175495232 +0000
@@ -1,2 +1,3 @@
 <h1>Hello</h1>
 <p>New line</p>
+tampered

Cheat sheet

Command	What it does
`scp file user@host:path`	Copy to a server
`scp user@host:path .`	Copy from a server
`scp -r dir host:`, `scp -P 2222`	A directory, another port
`rsync -avh src/ dest/`	Sync contents (only changes are copied)
`rsync -avz src/ user@host:dest/`	Sync over SSH, compressed
`src/` vs `src`	Contents vs the directory itself
`rsync -avn --delete src/ dest/`	Dry run of an exact mirror
`--exclude="*.log"`, `--progress`, `--stats`	Leave out, show files, summary
`-e "ssh -p 2222"`	rsync over another SSH port
`wget URL`, `wget -c`, `wget -O name`	Download, resume, save as
`curl -L -o name URL`, `curl -O URL`	Download with curl
`sha256sum file`, `sha256sum -c list`	Make, check fingerprints
`cmp a b`, `diff a b`, `diff -u a b`	Identical?, which lines differ, unified format

← SSH: connecting, keys and a safer server Backups between two machines over SSH →

Moving files over the network: scp, rsync and wget

On this page

Set up

scp: copy over SSH

Try it

rsync: copy only what changed

The trailing slash

Mirroring: --delete, and testing with -n

rsync over SSH

Try it

Downloading: wget and curl

Did it arrive intact? Checksums

Comparing two files: cmp and diff

Try it

Common mistakes

Practice

Cheat sheet