Last updated on October 10, 2020 by Dan Nanni
wget
is a Linux command-line utility for retrieving files from the web, via HTTP, HTTPS and FTP protocols. When you are using wget
to download a file at a particular HTTP URL, wget
sends an appropriate HTTP request to a destination web server.
To view default HTTP request header being sent by wget
, you can use -d
option.
$ wget -d http://www.google.com/
---request begin--- GET / HTTP/1.0 User-Agent: Wget/1.12 (linux-gnu) Accept: */* Host: www.google.com Connection: Keep-Alive ---request end---
Sometimes you may want to customize the default HTTP request header used by wget
. For example, you may want to customize User-Agent
field as some websites rely on User-Agent
string to block robots like wget
to retrieve their content. You may want to add an additional Accept-Encoding
field in order to test encoding schemes of your web server. In some other cases, you may need to set Host
field properly to be able to access a web server running on name-based virtual hosting.
wget
allows you to send an HTTP request with custom HTTP headers. To supply custom HTTP headers, use --header
option. You can use --header
option as many time as you want in a single run.
$ wget -d --header="User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11" --header="Referer: http://xmodulo.com/" --header="Accept-Encoding: compress, gzip" http://www.google.com/
---request begin--- GET / HTTP/1.0 User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 Accept: */* Host: www.google.com Connection: Keep-Alive Referer: http://xmodulo.com/ Accept-Encoding: compress, gzip ---request end---
If you would like to permanently set the default HTTP request header you want to use with wget
, you can use ~/.wgetrc
configuration file. You can specify as many header fields as you want in ~/.wgetrc
.
$ vi ~/.wgetrc
header = User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 header = Referer: http://xmodulo.com/ header = Accept-Encoding: compress, gzip
Once you have configured ~/.wgetrc
, you no longer need to use --header
option with wget
.
There are also other command-line tools that can send HTTP requests with custom HTTP headers. For example, curl
provides a similar interface to set custom HTTP headers, and HTTPie is more user-friendly to perform the same task.
This website is made possible by minimal ads and your gracious donation via PayPal (Credit Card) or Bitcoin (BTC Wallet: 1M161JGAkz3oaHNvTiPFjNYkeABox8rb4g
).
Xmodulo © 2020 ‒ About ‒ Powered by DigitalOcean