wget -r and -np: Recursive Downloads
wget -r follows links and downloads recursively; -np stops it from ascending above the starting directory.
To pull a directory of build artifacts off an index page, wget -r with -np grabs the tree without wandering into the rest of the site.
What it does
wget -r (--recursive) follows links from the start URL and downloads them, up to --level deep (default 5). -np (--no-parent) prevents wget from ascending to parent directories, keeping the crawl inside the directory you pointed at. --accept and --reject filter by file extension.
Common usage
wget -r -np -nH --cut-dirs=2 \
https://example.com/builds/v1.2.0/
# only .tar.gz files, one level deep
wget -r -np -l1 -A "*.tar.gz" https://example.com/builds/Options
| Flag | What it does |
|---|---|
| -r / --recursive | Download recursively |
| -np / --no-parent | Do not ascend to the parent directory |
| -l <n> / --level=<n> | Maximum recursion depth (default 5) |
| -A <list> / -R <list> | Accept / reject by extension or pattern |
| -nH | Do not create a host-named directory |
| --cut-dirs=<n> | Drop n leading path components from local paths |
In CI
Always add -np for directory pulls so wget does not crawl the whole site. Use -A to restrict to the artifact extension and -l to cap depth; an unbounded recursive download can pull far more than intended and blow out the job time.
Common errors in CI
A recursive download that fetches almost nothing usually means robots.txt disallowed the crawl; add -e robots=off if appropriate. Pulling the entire site means -np was omitted. If only index.html arrives, the server returns a listing page wget cannot parse for links; check that directory listing is enabled.