Possible Memory Leak / Bloat

(Clione Hamilton Levi) #1

I’ve written an application using PushType and have encountered a memory leak. I’ve been scouring my own code and stripping down my pages to find out just what could be causing the leak.

PushType uses Dragonfly to manage assets, and I’ve heard that there are memory problems with Dragonfly related to ImageMagick. I’ve turned off all identify and convert ImageMagick code I’ve been using and made sure that all of my images are stored in s3 storage and retrieved via #remote_url. However, this does not seem to have changed my memory situation.

I’ve also tried other things such as garbage collection (via GC.start) after every request with a after_action callback without success. I’ve tried refreshing particular pages over and over again to see if I could narrow down what areas are causing problems, but I’ve seen no differences. As far as I can tell the only thing present on every page are calls to PushType to, for example, build the header navigation.

Has anyone else had memory problems with applications using PushType? Or Dragonfly/ImageMagick? If anyone has any advice at all, I’d really appreciate it.

(Aaron Russell) #2

Hi there. I’m not aware of any specific “memory leaks” but I’m not too hot on the more low-level side of things so it’s possible. No one’s ever taken a deeper look from a performance/optimisation point of view. All I can say is that we’ve got PushType running a number of clients’ sites and we haven’t had any significant problems.

Dragonfly, memory leak or not, can be resource intensive if we don’t optimise the Rails setup a little. Particularly on Heroku, which sometimes can give us out of memory errors. On all our sites we’ll activate Rails’ cache_store to use Redis, and also setup asset_host with S3 and cloudfront. Doing this usually sorts things out for us.

What kind of environment are you running you site on and getting these problems? What else is going on in the app other than PushType?

(Clione Hamilton Levi) #3

Thank you so much for your prompt reply.

The app uses dragonfly to manage files, the pdf-reader and docx gems to extract text from pdfs and word documents, and the elasticsearch gem to index everything and make it searchable. Indexing and any dragonfly image manipulation is performed on a worker process managed with the resque gem. The site is hosted on heroku, which is providing the postgresql database, redis store, and elasticsearch server via add-ons. I am using puma as the app server.

For styling and other graphical flourishes, the app is using bootstrap, font-awesome, select2, and owl-carousel2. Pagination is handled via kaminari.

I currently have Scout and TuneMyGC setup in an attempt to find out what’s going on, though I haven’t had much success with them. I also setup cache_store with redis and asset_host with S3 earlier today, but that didn’t seem to change the situation either.

According to heroku, the app’s memory usage steadily climbs until it reaches 200% capacity, upon which I assume heroku restarts it and the memory usage drops to its initial value. This seems to happen whether or not people are accessing the site, though I believe memory use grows faster when people are accessing the site. Click here for a screenshot of the app’s memory usage according to heroku’s metrics view.

(Aaron Russell) #4

Issues like this can be a pain to diagnose - you have my sympathies. It seems like there’s quite a lot going on in your app - not a simple clean PushType site - so at this point until we can conclusively point the finger at PushType I can’t really do much to help.

I have had issues like this before with Rails apps on Heroku. A couple of things I can suggest trying:

  1. Try swapping puma for other app servers. I’ve actually found the standalone phusion passenger (which works well on Heroku) to be one of the best behaved options when others are giving me problems.
  2. Setup newrelic and watch for long requests (scout might already give you this info, I’m not sure). I’ve had apps where one or two requests where hammering the database in ways we didn’t pick up in testing and causing all sorts of problems.

Let me know how it goes. If you do find something that clearly points the finger at PushType then definitely let me know and we’ll try and sort it out.

(Priit Pärna) #5

Hey, my site heavily built on push_type and with dragonfly + google storage, it has some memory leaks issues as well. Likely connected to the assets management.