11 Cool Wget Commands to Download Anything from Internet

668

What is Wget?

Wget lets you download any files from the internet and even mirror entire websites for offline viewing. Wget is a command line utility. The only requirement is that it must be available over HTTP, HTTPS or FTP protocols; otherwise, Wget can’t be downloaded using Wget. Some examples of what Wget can be used for:

  • Download all the mp3 files on a site
  • Multiple files download
  • Mirror a whole website
  • Ability to resume downloads
  • Download files behind a login page

Installing Wget:

1. On macOS Sierra

We need to install Wget to use it on our systems. Most Linux systems come with Wget pre-installed. However, for mac users, Wget installation is needed. The easiest way is to use Homebrew. Here are the steps to install it:

Step 1: To install Homebrew open Terminal on your macOS and type the following command:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Installing Homebrew

Step 2: After Homebrew is installed, type the below mentioned command to install Wget on Terminal.

brew install wget

Installing wget

Step 3: Now after the installation is completed you can use the following command to check if Wget is running or not on Terminal:

wget -V

Checking wget running successfully

2. For Windows

Installing Wget on windows is a lengthier process than installing on macOS. It can be easily done by following below mentioned steps:

  • Visit Cygwin and download the respective file and run it.
  • When it asks for download source, choose the option “Install from Internet” and then click “Next”.

Cygwin wget installation toolkit

  • Select the Root directory and Click “Next”.

Setting the directory

  • Click on “Direct Connection” under Select your internet connection box and click “Next”.

Direct connection to download

  • Now you have to select a mirror link to download Cygwin, choose any of these. If the download fails please select different mirror link.

select mirrors

  • In the next window, write wget in the search bar and then under web tab select the version and click “Next”.

selecting wget version

  • Now after the above process just click next on the next window that shows up.

Resolving dependencies before downloading

  • Once the installation is completed, you will be asked to create a desktop shortcut icon and/or Icon to start menu. Select according to your preference and click “Finish”.

Finish Installation

That’s all you need to do to install Wget on your Windows PC. To run it you need to launch Cygwin Terminal and use Wget from there.

Using Wget:

Now after the above process wget is installed in your systems. Let’s check out some cool Wget commands to download files/folders and even whole website from the internet.

1. Use Wget to Download Files

Downloading a file using wget

Use this command to download files off the internet.

wget <url>

2. Download a File and Save it Locally Under Different Name

download and save file with any name

Use this command to download a file from the internet and save it under a different name and extension other that the predefined name.

wget -o <filename.extension> <download_url>

3. Download Only Specific File Types

Download specific files

There are many types of files on a website. Use the following command to download only a specific type of file that you need.

wget -r -A pdf <download_url>
wget -r -A jpg, jpeg, png, bmp <download_url>

4. Download Files from Multiple URL’s with Wget

  • Firstly, create a text file with URL on each line from which you want to download the files, as shown below.

untitled text document

  • Now key in the below mentioned command to use the above mentioned URL’s to download the file you need using Wget.

Download from multiple url

wget -i <file_name.extension>

5. Download Files with Specifying Download Speed Limit

download limit specifer

By normally using Wget it will generally try to acquire maximum bandwidth possible. To limit the download speed use the below mentioned command.

wget --limit-rate=<speed>k <download_url>

In the example above 200k will limit bandwidth speed to 200kB, not 200kb.

6. Use Wget as a Web Spider to Check Download URL

Check URL avaliability

You can use the below mentioned command in Wget to check if the download URL exists or it doesn’t.

wget --spider <download_url>

If the given URL is correct, it will display something like this:

URL Correct

If the given URL is incorrect, it will display:

URL Incorrect

You can use spider option for:

  • Checking before scheduling a download
  • Monitoring if a website is available or not
  • Checking the list of your bookmarked URL if they still exist or not

7. Download Restricted HTTP & FTP Downloads with Username & Password

download http

download ftp

You can download files from password restricted HTTP and FTP sites using the following commands:

wget --http-user=username --http-password=password <download_url>
wget --ftp-user=username --ftp-password=password <download_url>

8. Reject a Certain Type of File from being Downloaded

reject a certain file type

Reject a certain type of file from being downloaded from a website using the following command:

wget --reject=gif <download_url>

9. Download Files from FTP Servers

download files from ftp servers

Use Wget to download files from FTP servers, Anonymous FTP downloads using:

wget <ftp_URL>

10. Continue Incomplete Downloads

Continue incomplete downloads

This is a really cool feature of Wget. For instance, if you were downloading a file using wget and have to shut down your PC/Mac. You can continue the download from where it stopped using the below mentioned command.

wget -c <download_url>

Wget will start the downloading from where it stopped itself after this command. This is very helpful when you start downloading a huge file and the download is interrupted.

If the file on the server is changed you won’t be able to use the above command. You have to start the download again.

There are some exceptions, you can read about it in the Manual.

11. Download Entire Website

Mirror entire website

Yes, you can download the entire contents of a website using the command mentioned below. Do remember if the website is too big this command will take a lot of time to execute.

wget --mirror -p <url> -P ./LOCAL-DIR WEBSITE-URL

Here your download will be stored in your current directory under the folder named “LOCAL-DIR” with the same name as the website.

 

LEAVE A REPLY

Please enter your comment!
Please enter your name here