Wednesday, June 19, 2013

Sidekiq thread dump on Heroku

We have occasionally been seeing jobs getting stuck in Sidekiq. A worker thread has started the job but then gotten stuck executing it. The Sidekiq documentation suggests that you send the Sidekiq process the TTIN signal which will make Sidekiq show backtraces for all worker threads. But since we are running our app on Heroku there is no way of logging in to a certain dyno or send its processes a signal from the command line. The solution around this that we came up with was to send the signal from witin a Sidekiq job. So we created a simple class that could be run as a Sidekiq job.
class SendTTIN
  include Sidekiq::Worker

  def perform
    pid = Process.pid
    `kill -s ttin #{pid}`
  end
end
Open a console on Heroku and enqueue the job with Sidekiq.
irb(main):001:0> SendTTIN.perform_async()
Then check the logs for the backtraces of all Sidekiq threads.

6 comments:

aledalgrande said...
This comment has been removed by the author.
aledalgrande said...

ACE!!! Thank you!

freegenie said...

But if all the workers are stuck this one would never run, right?

Mikael Amborn said...

freegenie: Yes that's correct.

Josh said...

I think it should be "include" as lowercase, right?

Mikael Amborn said...

Josh: thank you, fixed.