Django ORM Memory Leaks in Debug Mode
August 15, 2012
TL;DR: Django has tremendous memory leaks for any long running task with many SQL operations with debug on.
Solution: Turn debug off.
Story:
So I just spent a long time digging this up. We have a management command that runs a ton of SQL statements and queries/inserts all kinds of objects into the database. I was attempting to run it locally but it kept getting slower and slower over time and the memory was just linearly expanding. I naively first did a really simple profile with heapy. This told me that I had a whole lot of strings taking up a whole lot of memory. My first inclination was that maybe python was doing something dumb by not interning strings properly or by not garbage collecting properly. I added a few intern() functions on what I thought might be candidates for duplication, and i explicitly garbage collected throughout the script. It had zero effect.
Written by
Cory Zue
Chief Accelerator
Product Updates