wget --spider: Check a URL Without Downloading
wget --spider verifies that a URL is reachable without saving the body, acting like a link checker.
Before a deploy reads from a URL, you can gate on whether it exists. --spider checks reachability and returns a non-zero exit code if not.
What it does
wget --spider behaves like a web spider: it checks that pages or files are there without downloading them. It is handy for verifying a URL or running a link check. The exit status is non-zero when the URL is broken, which makes it scriptable as a gate.
Common usage
wget --spider https://example.com/releases/app-1.2.0.tar.gz
# quiet health gate
wget -q --spider https://example.com/health \
&& echo up || echo down
# link-check a set of URLs recursively
wget --spider -r -nv -np https://docs.example.com/Options
| Flag | What it does |
|---|---|
| --spider | Check the URL but do not download |
| -q | Quiet, for use in a conditional gate |
| -r | Spider recursively to check many links |
| --tries=<n> | Retry transient failures during the check |
In CI
Use wget -q --spider URL && ... as a readiness or reachability gate before a deploy. Remember --spider sends a HEAD (falling back to GET); some servers do not support HEAD and may answer 405, so a failed spider check is not always a missing resource.
Common errors in CI
Remote file does not exist -- broken link! is the broken-URL message. ERROR 404: Not Found from a spider check means the resource is missing. ERROR 405: Method Not Allowed means the server rejects HEAD; the file may still exist, so verify with a real GET before failing the gate.