urllib.request, Request, urlopen, gzip

Fig.1: using Python to request compressed file

#!/usr/bin/env python3
from urllib.request import Request, urlopen
import gzip

req = Request('http://www.debian.org')
req.add_header('Accept-Encoding', 'gzip')
response = urlopen(req)
content = gzip.decompress(response.read())
decomp_req = content.splitlines()
for line in decomp_req:
    print(line.decode('utf-8'))

If file compression is available, Python can be used to request a compressed file. In this case, the Accept-Encoding header specifying gzip was used. Once the compressed file is received, it is opened, using urlopen(), then decompressed, using decompress(). The file is then arranged into separate lines using splitlines(). Finally, a for loop is used to print() each line— line by line—and decode() bytes to utf-8.