Possible memory leak in rails-autoscale-web #65
Comments
|
Thanks for opening the issue and with so much detail! I assume your development environment does not have the Rails Autoscale add-on installed? What I think is happening is the request middleware is collecting request queue time in memory, but since it's never reporting (if the add-on isn't installed), that in-memory store is never flushed. I'm still working out the best way to fix it, but I'm betting if you installed the free version of Rails Autoscale in development environment, you won't see the memory leak. |
|
Hi @adamlogic. Sure. Happy to help. The add-on has been a huge help for us. I do have the Rails Autoscale add-on installed; however, I have the "Autoscaling" toggled off for my web and worker dynos so it wouldn't scale the infrastructure at night because it's a dev env. With auto scaling toggled off, would it behave the same as if the add-on wasn't installed? |
No, it shouldn't matter whether it's toggled on/off. Just having it installed disproves my theory above. 🤔 |
We do have a safeguard in place for this scenario, so this doesn't appear to be the issue. @carrabi Could you check your logs to see if any exceptions are being raised by the gems? The easiest way to do this is to tail & grep your logs, then restart your app to see the logs during boot: Feel free to email me these logs privately. Adam at railsautoscale.com. Thanks! |
|
Yep. Will do |
|
Hello! Just wanted to report that we're seeing the same web memory consumption increase after switching to Let me know if that's any other information I can give to assist! |
|
@jnebeker Thanks for the report! Any chance you have a before & after screenshot of Heroku memory metrics? Also, if you're able to search your logs for "rails-autoscale" (not "RailsAutoscale"), are you seeing any exceptions/backtraces? |
|
@adamlogic Reporting back from our offline conversation (I'll recap below). Great catch on running puma in "cluster mode" with a single process. Once I switched puma into "single mode" I no longer experienced the memory leak. I also, let these changes sit for 24 hours. As expected, switching to "single mode" resulted in a drop in overall memory consumption and an increase in response time (no thread concurrency). Overall my app performance remained unchanged and I plan on rolling this out to my production environment after further monitoring. What wasn't expected is how much the memory would drop. It was roughly 50%. This article does a good job explaining how puma operates at a high level. https://medium.com/@CGA1123/reducing-rails-memory-usage-by-15-56090b6294bd
What still remains unanswered is running puma in "cluster mode". I still believe there is an issue there. For those of you who what to try and run puma in single mode, you'll need to comment out the following lines in
One gotcha that I ran into. If you set |
Sure thing @adamlogic, below are some screenshots of Heroku memory metrics in our telemetry tool as well as a screenshot of Heroku memory metrics. Each of the dotted lines in the telemetry screenshots is a dyno restart from a new release. Screenshots
Web dyno memory usage 24 hours before (left) and after(right) we switched from 
Web dyno memory usage 24 hours before (left) and after(right) we switched back to 
Heroku web memory metrics over the past 72 hours, the I searched our logs for Net::ReadTimeoutOpenSSL::SSL::SSLErrorNet::OpenTimeoutOne of these looked like a |
|
I'm still not sure what's going on, or why I'm not seeing the same issues in our own apps using
That's normal. Web dynos report both web and worker metrics, so that we still get worker metrics if workers are scaled down to zero. Sometimes Redis encounters some transient errors when fetching Sidekiq metrics, so nothing to be concerned about unless you're seeing LOTS of them. |
That makes sense! We're not seeing too many of those Redis errors, only a handful. 👍 |
|
I wanted to provide an update here, but unfortunately I don't have any progress to report. I'm not seeing this memory issue on the other Rails Autoscale customer apps I've looked at, and I haven't found any suspicious areas of the code. I'll keep this issue open so others can provide more info if they experience the same thing. For anyone posting here with this issue, please include the following:
In the meantime, if you do see this issue, you are fine to continue using |
|
I've recently upgrade our app from rails Here are my stats for the last 3 days. You can ignore the But as you can see the memory usage jumps quite a bit, especially on the 7 day screenshot. I've not yet migrated over to the new My current
Here is my memory usage over the last 7 days
|
|
Stairstep increases in memory like that are bloat, not a leak. In other words, you have an endpoint that's loading a bunch of objects into memory. I can't explain why the Rails/Ruby upgrades would have exacerbated that issue, but there's nothing here that a) shows a memory leak, or b) shows that it's related to Judoscale. I'd recommend using an APM tool like Scout or Sentry to see if they provide insight into memory bloat. |
|
Okay that makes sense, thanks for the insight. |
I recently migrated from
rails_autoscale_agenttorails-autoscale-webwithrails-autoscale-sidekiq. I've been testing the update in my development environment to ensure things are working properly. Over the last two days I've noticed a jump in web dyno memory consumption. The screenshot below shows the span when I added the gem and rolled back to the old version. After the rollback, memory consumption returned to normal. Note - I didn't experience the memory jump on my worker dynos which indicates an issue withrails-autoscale-webgem.I also found this issue which references a memory leak in the
rails_autoscale_agentwhich has been closed. I tried using heapy to get the heap dump but I was getting errors running it. I'm also not sure how heapy would work in Heroku because its a shared virtualized dyno#39
The text was updated successfully, but these errors were encountered: