wget --mirror: Mirror a Site or Directory
wget --mirror is a shortcut that turns on recursion, infinite depth, timestamping, and FTP listing preservation.
When you need a complete, re-runnable copy of a directory or site, --mirror bundles the right flags so repeat runs only fetch what changed.
What it does
wget --mirror (-m) is equivalent to -r -N -l inf --no-remove-listing. It recurses with no depth limit, uses timestamping so re-runs skip unchanged files, and keeps FTP directory listings. Add -np to confine it to a subtree.
Common usage
wget --mirror -np -nH --cut-dirs=1 \
https://example.com/artifacts/
# mirror and convert links for offline browsing
wget --mirror -k -p https://docs.example.com/Options
| Flag | What it does |
|---|---|
| --mirror / -m | Shortcut for -r -N -l inf --no-remove-listing |
| -np | Stay within the starting directory |
| -k / --convert-links | Rewrite links for local browsing |
| -p / --page-requisites | Also fetch images/CSS needed to render pages |
| -N | Timestamping (only fetch newer files), part of --mirror |
In CI
Combine --mirror with -np so a scheduled job mirrors only the directory you care about. Because --mirror enables -N, repeated pipeline runs against a persisted cache re-download only changed files, which saves bandwidth and time.
Common errors in CI
A --mirror run that pulls the whole site means -np was missing. Last-modified header missing -- time-stamps turned off appears when the server omits Last-Modified, so -N cannot compare and re-downloads everything. robots.txt can also block the crawl; add -e robots=off where appropriate.