If you followed our Magic Mirror tutorial, you’d see the wget command was used to download some files.
This tutorial explains how to use wget, a command-line tool for downloading (getting) files from the web, be it via HTTP, HTTPS, FTP, or FTPS.
Use it to download files from the internet from the Linux shell, call it from Bash scripts – it’s simple and versatile and doesn’t require user interference once started so that it can run in the background.
Syntax
wget [OPTIONS]... [ URLS ]...
Note that:
- OPTIONS is a list of options that can be passed to the application from the below table, separated by a space
- URLS is a list of URLs to download, separated by a space
Here are the wget options, straight from the docs:
| Common Options | |
|---|---|
| -o logfile | Log all messages to logfile. The messages are normally reported to standard error. |
| -a logfile | Append to logfile. This is the same as -o, only it appends to logfile instead of overwriting the old log file. If logfile does not exist, a new file is created. |
| -q | Turn off Wget’s output. |
| -i file | Read URLs from a local or external file. If this function is used, no URLs need be present on the command line. |
| -t number | Set number of retries to number. |
| -c | Continue getting a partially-downloaded file. |
| -T seconds | Set the network timeout to seconds seconds. |
| -w seconds | Wait the specified number of seconds between the retrievals. |
| –user=user | Set the HTTP or FTP authentication username. |
| –password=password | Set the HTTP or FTP authentication password. |
| –post-data=string | Make a POST request instead of GET and send data. string should be in the format “key1=value1&key2=value2” |
| -r | Turn on recursive retrieving. |
| -l depth | Specify recursion maximum depth level depth. The default maximum depth is 5. |
| -m | Turn on options suitable for mirroring. This option turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings. |
| -p | This option causes Wget to download all the files that are necessary to properly display a given HTML page. This includes such things as inlined images, sounds, and referenced stylesheets. |
Examples
Download a File from an HTTPS Server
Download a single file, basic usage:
wget https://www.example.com/file.zip
Continue Downloading a File
If a download only partially completed, continue/resume downloading it with the -c option:
wget -c https://www.example.com/file.zip
Download From a List of Files, Appending to Log
If you have a text file containing a list of URLs to download, you can pass it directly to wget and write a log of the results for later inspection:
wget -a log.txt -i url-list.txt
You could also use -o to write out the log file, and it will overwrite rather than append an existing log file if it’s already there.
Download a File, Retry 5 Times, Quietly
Retry downloading a file and don’t’ print progress to the terminal:
wget -t 5 -q https://www.example.com/file.zip
Download From a List of Files, Waiting 6 Seconds Between Each Download, with a 12 Second Timeout
Wait between downloads to reduce server load, and abort if the server fails to respond within 12 seconds:
wget -w 6 -T 12 -i url-list.txt
Download a File from an FTPS Server which Requires a Username and Password
Download from an FTPS server with the username bob and the password boat:
wget --user=bob --password=boat ftps://ftp.example.com/file.zip
Download a File with a POST Request
Make an HTTP POST request instead of the default GET request, and send data. A blank string can be sent with –post-data:
wget --post-data="postcode=2000&country=Australia" https://www.example.com/file.zip
In this example, we’re sending two pieces of POST data – postcode and country.
If making POST requests, cURL can be more versatile.
Download Directory Recursively via FTP with a Depth Limit
Downloading recursively will download the contents of a folder and the contents of the folders in that folder. A depth limit of 3 is defined in this example – meaning that if a folder is nested within 3 other folders, it won’t be downloaded:
wget -r -l 3 ftps://ftp.example.com/path/to/folder
Downloading a Whole Directory including ALL Contents via FTP
wget -m ftps://ftp.example.com/path/to/folder
Cloning a Whole Web Page Using Wget
If you want to try and grab a whole webpage – including all images, styles, and scripts, you can use
wget -p https://www.example.com/page.html
Your success will vary – some modern web pages don’t really work well when ripped out of their native habitat.
