wget -e robots=off: Ignore robots.txt
wget -e robots=off sets the robots option inline so a recursive download ignores robots.txt and Robots meta tags.
By default recursive wget obeys robots.txt, which can silently stop a crawl. -e robots=off overrides that for sites you own or are allowed to fetch.
What it does
wget -e <command> executes a wgetrc-style command inline. -e robots=off disables wget’s default obedience to robots.txt and Robots nofollow meta tags during recursive retrieval. It only affects recursive and spider modes, not single-file downloads.
Common usage
wget -r -np -e robots=off https://example.com/files/
# combine with mirror
wget --mirror -np -e robots=off https://example.com/artifacts/Options
| Flag | What it does |
|---|---|
| -e robots=off | Ignore robots.txt and Robots meta tags |
| -e <cmd> | Run any wgetrc command inline |
| -r | Recursive retrieval (where robots applies) |
| --mirror | Mirror mode (also subject to robots by default) |
In CI
Use -e robots=off only against endpoints you control or are permitted to crawl. It is the usual fix when a recursive or mirror job fetches only robots.txt and then stops. Do not use it to bypass restrictions on third-party sites.
Common errors in CI
A recursive download that gets robots.txt and nothing else is the classic symptom; the site disallows the path and wget obeyed. Adding -e robots=off resumes the crawl. If it still fetches nothing, the issue is link parsing or -np scope, not robots.