Setting up monit's environment to start and stop delayed_job
- January 08, 2011
- Tim Harrison
Update March 18, 2012:
Using Rails 3?
I'm using Rails 3.1.3. To get DJ working I've had to stick with delayed_job 2.1.4. The 3.0.1 build results in "undefined method `before_fork' for nil:NilClass". If you google around enough you'll find the reasons why. If you're like me and have other things to do, just stick with 2.1.4 for now.
Thanks jonathan dean.
I use Monit to keep delayed_job running. Getting monit to start and stop delayed_job is a giant pain, mostly because of environment problems. In this post I explain how I make it work.
Monit
First off, install monit. It’s pretty easy. But remember to set startup=1 in /etc/default/monit. Also, I recommend adding your email as an alert recipient in /etc/monit/monitrc.
To tell monit how to start and stop delayed_job, I create /etc/monit.d/check_process_delayed_job with the contents below. Check monit docs for details around my alert and restart settings.
check process myapp_delayed_job with pidfile /path/to/myapp/tmp/pids/myapp_delayed_job.pid
start program = "/bin/bash /path/to/myapp/script/monit_delayed_job.sh start"
as uid wei and gid wei
stop program = "/bin/bash /path/to/myapp/script/monit_delayed_job.sh stop"
as uid wei and gid wei
if cpu > 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if totalmem > 200.0 MB for 5 cycles then restart
if 3 restarts within 5 cycles then timeout
group background_tasks
monit_delayed_job.sh
Notice that it’s starting and stopping delayed_job through an sh script. My rails_root/script/monit_delayed_job.sh is below. It does two important things: loading /etc/profile, and logging. But the greatest of these is logging!
All the output of monit_delayed_job.sh is logged to log/monit_delayed_job.log, so you can see if you have any environmental errors. Otherwise monit will swallow the errors and you’ll go mad. We don’t want that.
BTW, you can see I’m using bundle exec. You can easily change the last line of the sh script to start delayed_job however you like.
#!/usr/bin/env bash
if [ $# -lt 1 ] ; then
echo "Usage: " $0 " <start | stop> "
exit 1
fi
action=$1
script_location=$(cd ${0%/*} && pwd -P)
cd $script_location/..
rails_root=`pwd`
if [ -f "/etc/profile" ]; then
. /etc/profile
fi
logfile=$rails_root/log/monit_delayed_job.log
echo "-----------------------------------------------" >> $logfile 2>&1
echo "Running bundle exec ./script/delayed_job $action" >> $logfile 2>&1
echo `date` >> $logfile 2>&1
echo `env` >> $logfile 2>&1
bundle exec ./script/delayed_job $action >> $logfile 2>&1
/etc/profile
In /etc/profile somewhere I set PATH and RAILS_ENV. Obviously, you should change these accordingly. Keeping the path and environment settings in /etc/profile allows me to setup hosts differently.
export PATH="/usr/local/sbin:/usr/sbin:/sbin:/usr/local/ruby-enterprise/bin:$PATH"
export RAILS_ENV=production
Capistrano
I use capistrano to deploy. My delayed_job.rb recipe is below. It’s pretty straight forward.
Capistrano::Configuration.instance(:must_exist).load do
namespace :delayed_job do
desc "Start delayed_job daemon"
task :start do
alert_user("Starting delayed job using #{use_monit_for_delayed_job ? 'monit' : 'script/worker'}")
if use_monit_for_delayed_job
sudo "monit start #{monit_service_name}"
else
run "cd #{current_path} && RAILS_ENV=#{rails_env} script/worker start"
end
end
desc "Stop delayed_job daemon"
task :stop do
alert_user("Stopping delayed job using #{use_monit_for_delayed_job ? 'monit' : 'script/worker'}")
if use_monit_for_delayed_job
sudo "monit stop #{monit_service_name}"
else
run "cd #{current_path} && RAILS_ENV=#{rails_env} script/worker stop"
end
end
desc "Restart delayed_job daemon"
task :restart do
alert_user("Restarting delayed job using #{use_monit_for_delayed_job ? 'monit' : 'script/worker'}")
if use_monit_for_delayed_job
sudo "monit restart #{monit_service_name}"
else
run "cd #{current_path} && RAILS_ENV=#{rails_env} script/worker restart"
end
end
end
end
To enable monit in the recipe above, I set these two things in my deploy.rb:
namespace :delayed_job do
set :use_monit_for_delayed_job, true
set :monit_service_name, 'myapp_delayed_job'
end
Logrotate
Rotating logs is fun. Do it. Here’s my /etc/logrotate.d/myapp file.
/path/to/myapp/log/*.log {
weekly
missingok
rotate 100
compress
copytruncate
delaycompress
sharedscripts
}
Victory
Simple enough, eh? I hope this helps you get it working.