iwantcoding.com
🔥 Daily 👥 Rooms 🏆 Top Log in Sign up

wget

By · Reviewed & last updated · Part of Linux & Bash Tutorial

wget is the venerable batch downloader: resumable, recursive, robots-aware, and capable of cloning a site for offline reading. curl is what you reach for in scripts (better protocol coverage, cleaner UX); wget is what you reach for when you need to download a tree, follow links, or hand off a job and walk away.

wget recipes: resume, mirror, rate-limit, retries

EXAMPLE
# 1) Plain download — saves to current dir under the original filename
wget https://example.com/data/big.csv.gz

# 2) Save to a specific path
wget -O /tmp/data.csv.gz https://example.com/data/big.csv.gz

# 3) Resume an interrupted download (-c continues from byte offset)
wget -c https://example.com/data/big.csv.gz

# 4) Rate limit so you do not saturate the link
wget --limit-rate=2M https://example.com/data/big.csv.gz

# 5) Retry with exponential backoff (default 20 tries, 1s starting delay)
wget --tries=10 --waitretry=10 --retry-connrefused \
     https://example.com/data/big.csv.gz

# 6) Quiet but keep the progress bar in scripts
wget -q --show-progress https://example.com/data/big.csv.gz

# 7) Download a list of URLs from a file (one per line)
wget -i urls.txt -P /tmp/downloads

# 8) Mirror a small static site for offline reading
wget --mirror --convert-links --adjust-extension \
     --page-requisites --no-parent \
     https://example.com/docs/

# 9) Authenticated download (basic auth, header, cookie)
wget --user=alice --password=hunter2 https://example.com/private/report.pdf
wget --header='Authorization: Bearer $TOKEN' https://api.example.com/export
wget --load-cookies cookies.txt --save-cookies cookies.txt --keep-session-cookies \
     https://example.com/dashboard

# 10) POST / form data — wget can do it, but curl is usually cleaner
wget --post-data='name=alice&email=a@x.test' https://example.com/api

# 11) Custom user agent (some sites refuse the wget default)
wget --user-agent='Mozilla/5.0 (offline-archive)' https://example.com/page

# 12) Background download (good with --no-verbose and a log)
wget -b --no-verbose -o wget.log https://example.com/big.iso
tail -f wget.log

# 13) Verify with sha256
echo '<hash>  big.csv.gz' | sha256sum -c -

# 14) Cron pattern — fetch nightly, fail loudly if it does not return 200
# 0 2 * * *  wget -q --tries=3 --timeout=60 -O /var/data/feed-$(date +\%F).json \
#               https://supplier.example/feed.json || \
#               curl -X POST -d 'feed download failed' $ALERT_WEBHOOK

# 15) When to prefer curl
#   - scripting (curl -fsSL has cleaner exit codes and headers)
#   - APIs that need fine-grained HTTP control
#   - protocols wget does not speak well (SMTP, SCP, MQTT)

Why it matters

For nightly jobs, always pair wget with `--tries`, `--timeout`, and an explicit `-O` output path. The default behaviour (write to current dir, retry forever) leaves you with mystery files and silently-hanging cron jobs when the upstream changes its URL or hangs the TCP connection.

Tip: Tweak the snippet with Try it Yourself », then sit the quiz at the bottom of the page.

Example

Example
wget -O file.tar.gz https://example.com/file.tar.gz
Try it Yourself »

Discussion

Loading…