How to use custom HTTP headers with wget

Last updated on October 10, 2020 by Dan Nanni

wget is a Linux command-line utility for retrieving files from the web, via HTTP, HTTPS and FTP protocols. When you are using wget to download a file at a particular HTTP URL, wget sends an appropriate HTTP request to a destination web server.

To view default HTTP request header being sent by wget, you can use -d option.

$ wget -d http://www.google.com/

---request begin---
GET / HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: */*
Host: www.google.com
Connection: Keep-Alive

---request end---

Sometimes you may want to customize the default HTTP request header used by wget. For example, you may want to customize User-Agent field as some websites rely on User-Agent string to block robots like wget to retrieve their content. You may want to add an additional Accept-Encoding field in order to test encoding schemes of your web server. In some other cases, you may need to set Host field properly to be able to access a web server running on name-based virtual hosting.

wget allows you to send an HTTP request with custom HTTP headers. To supply custom HTTP headers, use --header option. You can use --header option as many time as you want in a single run.

$ wget -d --header="User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11" --header="Referer: http://xmodulo.com/" --header="Accept-Encoding: compress, gzip" http://www.google.com/

---request begin---
GET / HTTP/1.0
User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11
Accept: */*
Host: www.google.com
Connection: Keep-Alive
Referer: http://xmodulo.com/
Accept-Encoding: compress, gzip

---request end---

If you would like to permanently set the default HTTP request header you want to use with wget, you can use ~/.wgetrc configuration file. You can specify as many header fields as you want in ~/.wgetrc.

$ vi ~/.wgetrc

header = User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11
header = Referer: http://xmodulo.com/
header = Accept-Encoding: compress, gzip

Once you have configured ~/.wgetrc, you no longer need to use --header option with wget.

There are also other command-line tools that can send HTTP requests with custom HTTP headers. For example, curl provides a similar interface to set custom HTTP headers, and HTTPie is more user-friendly to perform the same task.

Support Xmodulo

This website is made possible by minimal ads and your gracious donation via PayPal (Credit Card) or Bitcoin (BTC Wallet: 1M161JGAkz3oaHNvTiPFjNYkeABox8rb4g).