SWHarden.com

The personal website of Scott W Harden

Compress and Store Files in Python

While writing code for my graduate research thesis I came across the need to lightly compress a huge and complex variable (a massive 3D data array) and store it in a text file for later retrieval. I decided to use the zlib compression library because it’s open source and works pretty much on every platform. I ran into a snag for a while though, because whenever I loaded data from a text file it wouldn’t properly decompress. I fixed this problem by adding the “rb” to the open line, forcing python to read the text file as binary data rather than ascii data. Below is my code, written in two functions to save/load compressed string data to/from files in Python.

import zlib  
  
def saveIt(data,fname):  
    data=str(data)  
    data=zlib.compress(data)  
    f=open(fname,'wb')  
    f.write(data)  
    f.close()  
    return  
  
def openIt(fname,evaluate=True):  
    f=open(fname,'rb')  
    data=f.read()  
    f.close()  
    data=zlib.decompress(data)  
    if evaluate: data=eval(data)  
    return data  

Oh yeah, don’t forget the evaluate option in the openIt function. If set to True (default), the returned variable will be an evaluated object. For example, [[1,2],[3,4]] will be returned as an actual 2D list, not just a string. How convenient is that?