Speed up file caching in Django

I recently had to find a bug in a site running Django. The site made heavy use of Django's file based caching, somewhere in the region of 100,00 files. The problem turned out to be that the current Django file caching code was not really intended to handle this many files.

The main culprit turned out to be a function in the file django/core/cache/backends/filebased.py called _get_num_entries which actually walks through the cache directory structure counting files. This function works well for a small amount files but becomes inefficient after a point.

The solution is to alter the function like so:

def _get_num_entries(self):
        count = 0
        if max_entries == 0: return count # Don't count files if max_entries is set to 0
        for _,_,files in os.walk(self._dir):
            count += len(files)
        return count
    _num_entries = property(_get_num_entries)

Now I set max_entries to 0 in the settings file and Django stops counting the files everytime it creates a new cache file.

I also created a ticket here.

© Copyright 2009