inanzzz | Handling beanstalk and RabbitMQ workers of symfony applications with supervisor in capistrano deployment process

In this example we are going to deal with supervisor when deploying our application with capistrano. Supervisor will control beanstalk and RabbitMQ console command so that the workers are always up to date and never down. Follow example below to see how it can be done.

Application

Supervisor structure

Setup environment specific supervisor configuration files.

# Structure
app
   config
        supervisor
             production.conf
             staging.conf

The commands mentioned in config files below would normally be run manually as php bin/console job:processor country_create --limit={5 or any number} --env={prod or stag} for beanstalk and php bin/console rabbitmq:consumer -m {5 or any number} country_create --env={prod or stag} for RabbitMQ under symfony application. The critical parts in the config below are -DFOREGROUND flag and startsecs=0 option. They are needed to prevent supervisor crashing with errors. Such as # INFO gave up: ... entered FATAL state, too many start retries too quickly or Exited too quickly (process log may have details). For more info, read startsecs and Nondaemonizing of Subprocesses pages.

# app/config/supervisor/production.conf
[program:football-prod-country-create]
command=php bin/console job:processor country_create --limit=5 --env=prod -DFOREGROUND # Beanstalk
#command=php bin/console rabbitmq:consumer -m 5 country_create --env=prod -DFOREGROUND # RabbitMQ
directory=/srv/www/football/current
autostart=true
autorestart=true
startretries=5
startsecs=0
user=deployer # Our deployment user
numprocs=1
process_name=%(program_name)s_%(process_num)02d
stderr_logfile=/srv/www/football/current/var/logs/%(program_name)s_stderr.log
stderr_logfile_maxbytes=10MB
stdout_logfile=/srv/www/football/current/var/logs/%(program_name)s_stdout.log
stdout_logfile_maxbytes=10MB

# app/config/supervisor/staging.conf
[program:football-stag-country-create]
command=php bin/console job:processor country_create --limit=5 --env=stag -DFOREGROUND # Beanstalk
#command=php bin/console rabbitmq:consumer -m 5 country_create --env=stag -DFOREGROUND # RabbitMQ
directory=/srv/www/football/current
autostart=true
autorestart=true
startretries=5
startsecs=0
user=deployer # Our deployment user
numprocs=1
process_name=%(program_name)s_%(process_num)02d
stderr_logfile=/srv/www/football/current/var/logs/%(program_name)s_stderr.log
stderr_logfile_maxbytes=10MB
stdout_logfile=/srv/www/football/current/var/logs/%(program_name)s_stdout.log
stdout_logfile_maxbytes=10MB

Deployment files

deploy.rb

# deploy/deploy.rb

set :application, "football"
set :supervisor_srv_conf_path, "/etc/supervisor/conf.d"
set :supervisor_app_conf, "#{fetch(:app_config_path)}/supervisor/"+fetch(:stage).to_s+".conf"
after "deploy:cleanup", "application:supervisor:setup"

namespace :application do
    namespace :supervisor do
        desc "Sets up supervisor"
        task :setup do
            on roles (:app) do
                puts "-" * 6
                puts "Setting up supervisor"
                ask :answer, "Would you like to restart supervisor? [y/N]"
                if fetch(:answer) == "y"
                    source = "#{release_path}/#{fetch(:supervisor_app_conf)}"
                    destination = "#{fetch(:supervisor_srv_conf_path)}/#{fetch(:application)}.conf"
                    execute :cp, "#{source} #{destination}"
                    execute :supervisorctl, "reread"
                    execute :supervisorctl, "update"
                    execute :supervisorctl, "reload"
                end
                puts "-" * 6
            end
        end
    end
end

Supervisor remote server permissions

In order for "deployer" user to interact with supervisor in deployment process, we need to grant him some permissions.

conf.d

This is where our application configuration files will be put by "deployer" user so do the following to grant him W permissions.

Current status.

root@prod:~# ls -l /etc/supervisor/
drwxr-xr-x 2 root root 4096 Jul 30  2013 conf.d
-rw-r--r-- 1 root root 1245 May  4 20:54 supervisord.conf

root@prod:~# ls -l /etc/supervisor/conf.d/
total 0

Create dedicated linux group named as "supervisor" and add "deployer" user to it.

root@prod:~# groupadd supervisor
root@prod:~# usermod -a -G supervisor deployer

root@prod:~# groups deployer
deployer : deployer supervisor

Let "supervisor" group own /etc/supervisor/conf.d/ directory.

root@prod:~# chgrp -R supervisor /etc/supervisor/conf.d

root@prod:~# ls -l /etc/supervisor
drwxr-xr-x 2 root supervisor 4096 Jul 30  2013 conf.d
-rw-r--r-- 1 root root       1245 May  4 20:54 supervisord.conf

Grant 775 permissions to the folder so that "deployer" can write to it.

root@prod:~# chmod 775 /etc/supervisor/conf.d

root@prod:~# ls -l /etc/supervisor
drwxrwxr-x 2 root supervisor 4096 Jul 30  2013 conf.d
-rw-r--r-- 1 root root       1245 May  4 20:54 supervisord.conf

supervisord.conf

Update config file to match section below.

# Replace this part
[unix_http_server]
file=/var/run/supervisor.sock
chmod=0700

# With this part
[unix_http_server]
file=/var/run/supervisor.sock
chmod=0770
chown=root:supervisor

Restart

To reflect to changes we need to logout and login again.

root@prod:~# logout
vagrant@prod:~# su -l root

root@prod:~# service supervisor restart
Restarting supervisor: supervisord.

Switch to "deployer" user and see if "deployer" user can reload supervisorctl.

root@prod:~# su -l deployer
deployer@prod:~$ supervisorctl reload
Restarted supervisord

If we didn't carry out steps above, we would get error below instead.

deployer@prod:~$ supervisorctl reload
error: , [Errno 13] Permission denied: file: /usr/lib/python2.7/socket.py line: 224

Test

This is the output of capistrano for supervisor.

deployer@deploy:/projects/football$ bundle exec cap production deploy
Setting up supervisor
00:18 application:supervisor:setup
      01 cp /srv/www/football/releases/20170501121638/app/config/supervisor/production.conf /etc/supervisor/conf.d/football.conf
    ✔ 01 deployer@192.168.99.50 0.009s
      02 supervisorctl reread
      02 football-prod-country-create: available
    ✔ 02 deployer@192.168.99.50 0.107s
      03 supervisorctl update
      03 football-prod-country-create: added process group
    ✔ 03 deployer@192.168.99.50 0.088s
      04 supervisorctl reload
      04 Restarted supervisord
    ✔ 04 deployer@192.168.99.50 0.141s

And this is what file permission looks like.

deployer@prod:~$ ls -l /etc/supervisor/conf.d/
-rw-rw-r-- 1 deployer deployer 508 May  5 20:03 football.conf

Our configuration file settings defined to run only one worker for given tube so lets confirm it.

deployer@prod:~$ supervisorctl 
football-prod-country-create:football-prod-country-create_00 RUNNING    pid 4906, uptime 0:03:18 # Beanstalk
#football-prod-country-create:football-prod-country-create_00 RUNNING    pid 15664, uptime 0:03:18 # RabbitMQ
supervisor>

GUI shows the same result.

As you can see below, the beanstalk tube has been created and ready for incoming jobs as well as RabbitMQ queue and exchange.