What is Wget?
Wget lets you download any files from the internet and even mirror entire websites for offline viewing. Wget is a command line utility. The only requirement is that it must be available over HTTP, HTTPS or FTP protocols; otherwise, Wget can’t be downloaded using Wget. Some examples of what Wget can be used for:
- Download all the mp3 files on a site
- Multiple files download
- Mirror a whole website
- Ability to resume downloads
- Download files behind a login page
Installing Wget:
1. On macOS Sierra
We need to install Wget to use it on our systems. Most Linux systems come with Wget pre-installed. However, for mac users, Wget installation is needed. The easiest way is to use Homebrew. Here are the steps to install it:
Step 1: To install Homebrew open Terminal on your macOS and type the following command:
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Step 2: After Homebrew is installed, type the below mentioned command to install Wget on Terminal.
brew install wget
Step 3: Now after the installation is completed you can use the following command to check if Wget is running or not on Terminal:
wget -V
2. For Windows
Installing Wget on windows is a lengthier process than installing on macOS. It can be easily done by following below mentioned steps:
- Visit Cygwin and download the respective file and run it.
- When it asks for download source, choose the option “Install from Internet” and then click “Next”.
- Select the Root directory and Click “Next”.
- Click on “Direct Connection” under Select your internet connection box and click “Next”.
- Now you have to select a mirror link to download Cygwin, choose any of these. If the download fails please select different mirror link.
- In the next window, write wget in the search bar and then under web tab select the version and click “Next”.
- Now after the above process just click next on the next window that shows up.
- Once the installation is completed, you will be asked to create a desktop shortcut icon and/or Icon to start menu. Select according to your preference and click “Finish”.
That’s all you need to do to install Wget on your Windows PC. To run it you need to launch Cygwin Terminal and use Wget from there.
Using Wget:
Now after the above process wget is installed in your systems. Let’s check out some cool Wget commands to download files/folders and even whole website from the internet.
1. Use Wget to Download Files
Use this command to download files off the internet.
wget <url>
2. Download a File and Save it Locally Under Different Name
Use this command to download a file from the internet and save it under a different name and extension other that the predefined name.
wget -o <filename.extension> <download_url>
3. Download Only Specific File Types
There are many types of files on a website. Use the following command to download only a specific type of file that you need.
wget -r -A pdf <download_url>
wget -r -A jpg, jpeg, png, bmp <download_url>
4. Download Files from Multiple URL’s with Wget
- Firstly, create a text file with URL on each line from which you want to download the files, as shown below.
- Now key in the below mentioned command to use the above mentioned URL’s to download the file you need using Wget.
wget -i <file_name.extension>
5. Download Files with Specifying Download Speed Limit
By normally using Wget it will generally try to acquire maximum bandwidth possible. To limit the download speed use the below mentioned command.
wget --limit-rate=<speed>k <download_url>
In the example above 200k will limit bandwidth speed to 200kB, not 200kb.
6. Use Wget as a Web Spider to Check Download URL
You can use the below mentioned command in Wget to check if the download URL exists or it doesn’t.
wget --spider <download_url>
If the given URL is correct, it will display something like this:
If the given URL is incorrect, it will display:
You can use spider option for:
- Checking before scheduling a download
- Monitoring if a website is available or not
- Checking the list of your bookmarked URL if they still exist or not
7. Download Restricted HTTP & FTP Downloads with Username & Password
You can download files from password restricted HTTP and FTP sites using the following commands:
wget --http-user=username --http-password=password <download_url>
wget --ftp-user=username --ftp-password=password <download_url>
8. Reject a Certain Type of File from being Downloaded
Reject a certain type of file from being downloaded from a website using the following command:
wget --reject=gif <download_url>
9. Download Files from FTP Servers
Use Wget to download files from FTP servers, Anonymous FTP downloads using:
wget <ftp_URL>
10. Continue Incomplete Downloads
This is a really cool feature of Wget. For instance, if you were downloading a file using wget and have to shut down your PC/Mac. You can continue the download from where it stopped using the below mentioned command.
wget -c <download_url>
Wget will start the downloading from where it stopped itself after this command. This is very helpful when you start downloading a huge file and the download is interrupted.
If the file on the server is changed you won’t be able to use the above command. You have to start the download again.
There are some exceptions, you can read about it in the Manual.
11. Download Entire Website
Yes, you can download the entire contents of a website using the command mentioned below. Do remember if the website is too big this command will take a lot of time to execute.
wget --mirror -p <url> -P ./LOCAL-DIR WEBSITE-URL
Here your download will be stored in your current directory under the folder named “LOCAL-DIR” with the same name as the website.
Also Read: 16 Useful MacOS Terminal Commands You Probably Do Not Know
