Django ORM Memory Leaks in Debug Mode
TL;DR: Django has tremendous memory leaks for any long running task with many SQL operations with debug on.
Solution: Turn debug off.
Story:
So I just spent a long time digging this up. We have a management command that runs a ton of SQL statements and queries/inserts all kinds of objects into the database. I was attempting to run it locally but it kept getting slower and slower over time and the memory was just linearly expanding. I naively first did a really simple profile with heapy. This told me that I had a whole lot of strings taking up a whole lot of memory. My first inclination was that maybe python was doing something dumb by not interning strings properly or by not garbage collecting properly. I added a few intern() functions on what I thought might be candidates for duplication, and i explicitly garbage collected throughout the script. It had zero effect.
Share
Tags
Similar Articles
Mitigating the Impact of COVID-19 on Global Vaccination Efforts
Disruptions in routine immunization due to COVID-19 can lead to secondary health crises, especially in children. That's why we've expanded on the platform used to provide more than 1 million COVID-19 vaccines to include routine immunization, as well.
Product Updates
October 11, 2021
CommCare Precision Tasking Framework: A Prioritized Tasking Solution for Supervisors and Health Workers
Dimagi has created a new Prioritized Tasking Framework that aims to help CHWs focus on service delivery, enable data-driven decision making and provide patients access to better care.
Product Updates
September 6, 2021
Championing New Integrations for Increased Interoperability and Impact
Health information systems often need to be able to communicate with one another and share data back and forth in a secure, reliable way. Dimagi has partnered with PATH and Digital Square to create a turnkey integration with CommCare and Fast Healthcare Interoperability Resources (FHIR).
Product Updates
August 19, 2021