Monitoring processes with Supervisor

Supervisors is a system that monitors and controls process state on UNIX-like operating systems. It watches processes and restart them in the case of failure. Also it makes sure that the processes start on system boot. For more information read Supervisor: A Process Control System.

When you have long running scripts (e.g. RabbitMQ, Beanstalkd etc.), you want to make sure that they really are always running. However, we also know that for any given reason at any time, scripts and systems fail. In such cases, the scripts should restart automatically without human interaction so we can use supervisor process watcher to do it for us.

Installation

$ sudo apt-get install supervisor

--- Version
$ supervisord -v
3.0b2

Commands

For more details, please visit Running Supervisor and Running Supervisorctl pages.

$ sudo service supervisor {start|stop|restart|force-reload|status|force-stop}

$ sudo supervisorctl # Lists all processes and lets you use internal commands

Log

The location to the main log file is /var/log/supervisor/supervisord.log. This will be different to your custom service/script log files.

Configuration

Location to the configuration file is /etc/supervisor/supervisord.conf. If you print its content, you'll see section below at the end of the file.

[include]
files = /etc/supervisor/conf.d/*.conf

This means, any files with extension *.conf under /etc/supervisor/conf.d/ directory will be included and run by supervisord. When you create a configuration file for your service/script, you will put it there. E.g. /etc/supervisor/conf.d/hello-world.conf

Configuration Options

For more details, please visit Configuration page.

• [program:*]: The name to the service/script. e.g. [program:symfony-prod-cache-clear]


• command: This command to run service/script. e.g. command=/usr/bin/php /var/www/html/symfony app/console cache:clear --env=prod -DFOREGROUND. Important: Always add -DFOREGROUND flag to you command to make workers more reliable. For more info, read Nondaemonizing of Subprocesses page.


• directory: The directory where supervisord will "cd" into before running the service/script. e.g. directory=/var/www/html/symfony


• autostart: Set it to "true" if you want supervisord to start service/script on system boot. e.g. autostart=true


• autorestart: Set it to "true" if you want supervisord to start service/script if it stops unexpectedly. e.g. autorestart=true


• startretries: The number of retries to do before giving up trying to run service/script. e.g. startretries=5


• user: The user who runs the service/script. e.g. user=deploy


• numprocs: Set how many instances of service/script you want to run. e.g. numprocs=5


• startsecs: The total number of seconds which the program needs to stay running after a startup to consider the start successful. Always use as startsecs=0. This is a very important option! For more info, read startsecs page.


• process_name The process name of the service/script. e.g. process_name=%(program_name)s_%(process_num)02d


• stderr_logfile: The file where the errors will be written. e.g. stderr_logfile /home/logs/app/symfony/cache-clear_stderr.log


• stderr_logfile_maxbytes: The maximum number of bytes that may be consumed by stderr_logfile before it is rotated. e.g. stderr_logfile_maxbytes=10MB


• stdout_logfile: The file where the regular outputs will be written. e.g. stdout_logfile /home/logs/app/symfony/cache-clear_stdout.log


• stdout_logfile_maxbytes: The maximum number of bytes that may be consumed by stdout_logfile before it is rotated. e.g. stdout_logfile_maxbytes=10MB

You must always use -DFOREGROUND flag and startsecs=0 option. They are needed to prevent supervisor crashing with errors. Such as # INFO gave up: ... entered FATAL state, too many start retries too quickly or Exited too quickly (process log may have details).

Loading configuration files

If you created a new config file for your service/script, you must use commands below to activate them by notifying supervisor.

$ sudo supervisorctl reread
$ sudo supervisorctl update

Listing processes and internal commands

For more information, please visit Running Supervisorctl page.

$ sudo supervisorctl
supervisor>

Use help flag to list all available commands.

supervisor> help

default commands (type help ):
=====================================
add    clear  fg        open  quit    remove  restart   start   stop  update
avail  exit   maintail  pid   reload  reread  shutdown  status  tail  version

Web Interface

Open sudo nano /etc/supervisor/supervisord.conf to add lines below into it.

[inet_http_server]
port = 9001
username = admin
password = admin

Restart supervisor.

$ sudo service supervisor restart

Visit http://192.168.50.30:9001/ to access web interface.