Home ›

Use compressed data directly - from ZIP files or gzip http response

Christian Harms's picture
by Christian Harms on Mon, 10/17/2011 - 14:02
in

Did you use compressing while using resources from your scripts and projects? Many data files are compressed to save disc space and compressed requests over the network saves bandwith.

This article gives some hints to use the “battery included” power of python for the handling of compressed files or use HTTP with gzip compression:

  1. reading log files as GZIP or uncompressed
  2. using files directly from ZIP- or RAR compression archive
  3. use gzip compressing while fetching web sites



reading GZip compressed log files

The most common use case for reading compressed files are log files. My access_log files are archived in an extra directory and can be easy used for creating statistics. This is done normally with zgrep/zless and other command line tools. But opening a gzip-compressed file with python is build-in and can be transparent for the file handling.

  1. import gzip, sys, os
  2.  
  3. if len(sys.argv)<2 or not os.path.isfile(sys.argv[1]):
  4.     print "%s <filename> - print file content" % sys.argv[0]
  5.     sys.exit()
  6.  
  7. try:
  8.     fp = gzip.GzipFile(sys.argv[1])
  9.     line = fp.readline()
  10. except IOError:
  11.     fp = open(sys.argv[1])
  12.     line = fp.readline()
  13.  
  14. while line:
  15.     print line
  16.     line = fp.readline()
  17. fp.close()

The tricky part is the IOError line. If you open a plain text file with the GzipFile class it fails while reading the first line (not while calling the constructor).

You can use the bz2-module with the BZ2File constructor if you have bzip2-compressed files.

Reading a file directly from zip archive

If you need a british word list your linux can help (if the british dictionary is installed). If not you can get it from several sources. I choosed the zip-compressed file from pyxidium.co.uk and included it in a small python script.

Zip-files are a container format. You can put many files in it and have to choose which file you want to decompress from the ZIP-file. In the example I will fetch the ZIP achive into memory and unzip the file en-GB-wlist.txt directly.

  1. import zipfile, os, StringIO, urllib
  2.  
  3. def openWordFile():
  4.     #reading english word dictionary
  5.     pathToBritishWords = "/usr/share/dict/british-english"
  6.     uriToBritshWords = "http://en-gb.pyxidium.co.uk/dictionary/en-GB-wlist.zip"
  7.  
  8.     if os.path.isfile(pathToBritishWords):
  9.         fp = file(pathToBritishWords)
  10.     else:
  11.         #fetch from uri
  12.         data = urllib.urlopen(uriToBritshWords).read()
  13.         #get an ZipFile object based on fetched data
  14.         zf = zipfile.ZipFile(StringIO.StringIO(data))
  15.         #read one file directly from ZipFile object
  16.         fp = zf.open("en-GB-wlist.txt")
  17.     return fp
  18.  
  19. #read all lines
  20. words = openWordFile().readlines()
  21.  
  22. print "read %d words" % len(words)

If you want to read directly from a RAR file you have to install the rarfile module.

  1. sudo easy_install rarfile

The module use the command line utility rar/unrar, but the usage is the same like the zipfile module.
  1. import rarfile
  2. rf = rarfile.RarFile("test.rar")
  3. fp = rf.open(“compressed.txt”)

Use gzip compression while fetching web pages

The speed of fetching web pages has many parameters. To save the important parameter band width you should fetch http resources compressed. Every modern browser support this feature (see test on browserscope.org) and every webserver should be able to compress the text content.

Your HTTP client must send the HTTP header “Accept-Encoding” to offer the possibility for compressed content. And you have to check the response header if the server sent compressed content. A web server can ignore this request header!

  1. import urllib2, zlib, gzip, StringIO, sys
  2.  
  3. uri = "http://web.de/index.html"
  4. req = urllib2.Request(uri, headers={"Accept-Encoding":"gzip, deflate"})
  5. res = urllib2.urlopen(req)
  6. if res.getcode()==200:
  7.     if res.headers.getheader("Content-Encoding").find("gzip")!=-1:
  8.         # the urllib2 file-object dont support tell/seek, repacking in StringIO
  9.         fp = gzip.GzipFile(fileobj = StringIO.StringIO(res.read()))
  10.         data = fp.read()
  11.     elif res.headers.getheader("Content-Encoding").find("deflate")!=-1:
  12.         data = zlib.decompress(res.read())
  13.     else:
  14.         data = res.read()
  15. else:
  16.     print "Error <%s> while fetching ..." % res.msg
  17.     sys.exit(-1)
  18.  
  19. print "read %s bytes (compression: %s), decompressed to %d bytes" % (
  20.     res.headers.getheader("Content-Length"),
  21.     res.headers.getheader("Content-Encoding"),
  22.     len(data))

As a developer I did not found the automatic support for gzip-enabled HTTP requests for HTTP clients in different libraries. And python dont offer the support build-in too. Copy/paste this lines in your next project or convert it in your favorite language and your HTTP request layer will become faster.

conclusion

One disadvantage: your software will consume some percent more cpu to decompress the data on-the-fly and will be slower on your local machine. Python use the c-binding to the zlib and is fast as any other component and in a network environment you can messure the benefit.

Comments

Savraj's picture
Funny
Savraj (not verified) - Mon, 10/17/2011 - 13:35

I was recently impressed by the built-in gzip handling in Python -- it's pretty cool, funny to see a story about it now. :)

Anonymous's picture
You should use context
Anonymous (not verified) - Mon, 10/17/2011 - 14:38

You should use context managers with file objects. ie:

  1. with open('foo') as f:
  2.     # File objects are iterable, so you can also loop over them as follows:
  3.     for line in f:
  4.         do_something(line)

That way the context managers will handle the cleaning up for you (which you haven't done in your examples! It's important to close files to free up file descriptors once you're done with them).

hdaz's picture
core commands
hdaz (not verified) - Tue, 01/31/2012 - 13:01

hmm why would you not just use linux core commands ??
zcat
zdiff
zegre
zcmp
zfgrep
zgrep
zless
zmore

Christian's picture
usage in python
Christian (not verified) - Wed, 02/01/2012 - 19:23

This blogpost describes the build-in support for python. Because if you want to use compressed support in the google app engine there is no support for linux command line tools.