PHP-FPM Restarts Using systemd

Recently, we were working with one of our clients to diagnose high load on their web servers. We traced the cause of the load to an opcache_reset() call used after code deploys as a way of preventing the PHP OpCache from overfilling as new code was deployed to the servers. The issue was due to (and resolved by) bug #72590, but troubleshooting the problem prompted us to look at ways to non-intrusively restart PHP-FPM. While it seems like graceful PHP-FPM restarts that don't abort requests still in process should be something obvious and trivial, it took a surprising amount of digging and some timely help from systemd's socket activation to accomplish.

The Stack

The system is a relatively standard high performance multi-node configuration:

• HAProxy
• Varnish
• Apache (2.4.x, running mod_mpm_event)
• PHP-FPM (5.6.x)
• CentOS 7

The Problem

Whenever we ran a PHP code deploy, we would run opcache_reset() to clear out files from the PHP OpCache to avoid it from filling up. While this worked-around the issue of OpCache overfilling, it ended up causing abnormally high load on the web servers. The root cause of this was that ACCEL_LOG_ERROR was aborting the process before it could trigger kill(), so the child is never terminated and the OpCache is never cleared (see bug #72590).

We needed a fix for the immediate pain, however, and adding a restart of PHP-FPM into the deployment scripts seemed to be the most straightforward solution. We considered other methods - taking the web node out of rotation at the load balancer, leveraging Varnish's ability to serve stale content while backends are unavailable - but for various reasons they were all too cumbersome or unworkable. We kept coming back to the fact that we needed to reload the PHP-FPM daemon. Even when using the supposedly-graceful USR2 restart, PHP-FPM would abort any currently-running requests, returning a 50x error to the client.

The Solution

At the same time we were searching for a way to do graceful PHP-FPM restarts, some unrelated searching brought me to a post by Kim Ausloos - php-fpm ondemand with systemd. While the article didn't address the situation we were facing - it is more focused on low-traffic hosting than high availability - it did provide the seeds of a solution. I was interested in working with systemd socket activation for other reasons, but it also provides us with an easy way to run two PHP-FPM master processes in parallel. In short, it allows us to have an active daemon serving user requests and a passive daemon that can be reloaded while not receiving user requests and then put into service.

The Configuration Details

PHP-FPM Configuration

First we needed to set up the PHP-FPM daemons. Using systemd socket activation is a significant departure from more traditional init systems. Services and sockets in systemd are managed using unit files. We'll detail here how to create the socket and service unit files needed to run multiple PHP-FPM daemons under systemd.

If you use Puppet to manage your PHP-FPM configuration, there is a pull request open to the popular thias/php Puppet module that will handle all of this configuration for you.

First we need to create two unit files for the sockets. Paths vary depending on the distribution, but the syntax of the files is simple and consistent:

$ vim /usr/lib/systemd/system/php-fpm-www-systemd1.socket
[Socket]
ListenStream=127.0.0.1:9001
[Install]
WantedBy=sockets.target

$ vim /usr/lib/systemd/system/php-fpm-www-systemd2.socket
[Socket]
ListenStream=127.0.0.1:9002
[Install]
WantedBy=sockets.target

The only customization needed is setting the ListenStream to use the desired port. In this case we chose ports 9001 and 9002.

Then we need to create the corresponding service unit files:

$ vim /usr/lib/systemd/system/php-fpm-www-systemd1.service
[Service] 
User=apache
Group=apache
Environment="FPM_SOCKETS=/var/run/php-fpm/www-systemd1.socket=3"
RuntimeDirectory=php-fpm
RuntimeDirectoryMode=0755
ExecStart=/sbin/php-fpm --fpm-config=/etc/php-fpm.d/www-systemd1.conf

$ vim /usr/lib/systemd/system/php-fpm-www-systemd2.service
[Service]
User=apache
Group=apache
Environment="FPM_SOCKETS=/var/run/php-fpm/www-systemd2.socket=3"
RuntimeDirectory=php-fpm
RuntimeDirectoryMode=0755
ExecStart=/sbin/php-fpm --fpm-config=/etc/php-fpm.d/www-systemd2.conf

There are a couple significant customizations in the service file:

• Environment needs to be set to the path of the corresponding .socket file.
• RuntimeDirectory must be set to the name of the directory you want the PHP-FPM daemons to use for their PID files (for example: "php-fpm" for "/var/run/php-fpm") as specified in the "/etc/php-fpm.d/*.conf" files. This will create that directory with the proper ownership (apache in this example) as the PHP-FPM daemons do not run as root and they need write access to this directory to write their PID files.

Lastly, we need to create the PHP-FPM config files:

$ vim /etc/php-fpm.d/www-systemd1.conf
[global]
pid = /var/run/php-fpm/www-systemd1.pid
error_log = syslog
daemonize = no
[www-systemd1]
listen = /var/run/php-fpm/www-systemd1.socket
listen.backlog = -1
listen.allowed_clients = 127.0.0.1
user = apache
group = apache
pm = dynamic
pm.max_children = 25
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 15
pm.max_requests = 2000
ping.response = pong

$ vim /etc/php-fpm.d/www-systemd2.conf
[global]
pid = /var/run/php-fpm/www-systemd2.pid
error_log = syslog
daemonize = no
[www-systemd2]
listen = /var/run/php-fpm/www-systemd2.socket
listen.backlog = -1
listen.allowed_clients = 127.0.0.1
user = apache
group = apache
pm = dynamic
pm.max_children = 25
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 15
pm.max_requests = 2000
ping.response = pong

Notes on the PHP-FPM daemon configuration files:

• These differ from other examples because we are running in a high performance/high availability environment where we'd rather have a few extra threads sitting idle than wait for the ondemand process manager to spin up a new one.
• The two files are nearly identical (although they don't have to be). The differences are the naming ([www-systemd1] vs [www-systemd2]) and the paths to the sockets in the listen directive.
• The [global] section is slightly different than a traditional PHP-FPM master configuration. The "daemonize = no" is because systemd handles the backgrounding/forking of the processes, so we don't want PHP-FPM to do it. Also the pid needs to align with the RuntimeDirectory of the service unit file.

Now that we have the PHP-FPM configuration in place, we need to activate the sockets. First we need to tell systemd to reload its unit files so that it can enable the newly-created ones:

sudo systemctl --system daemon-reload

Then we enable and start the new sockets:

sudo systemctl start php-fpm-www-systemd1.socket
sudo systemctl enable php-fpm-www-systemd1.socket
sudo systemctl start php-fpm-www-systemd2.socket
sudo systemctl enable php-fpm-www-systemd2.socket

Note that we don't need to start the services. Any request sent to either of those ports will be passed to the corresponding PHP-FPM master daemon. If the corresponding PHP-FPM master daemon is not running, it starts one. A nice feature, but essentially tangential to our desired behavior.

A check using systemctl now shows the sockets are active and listening on localhost ports 9001 and 9002:

$ systemctl status php-fpm-www-systemd1.socket
● www-systemd1.socket
   Loaded: loaded (/usr/lib/systemd/system/php-fpm-www-systemd1.socket; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2016-09-19 16:31:59 CDT; 2 weeks 3 days ago
   Listen: 127.0.0.1:9001 (Stream)

$ systemctl status php-fpm-www-systemd2.socket
● www-systemd2.socket
   Loaded: loaded (/usr/lib/systemd/system/php-fpm-www-systemd2.socket; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2016-09-19 16:31:51 CDT; 2 weeks 3 days ago
   Listen: 127.0.0.1:9002 (Stream)

Web server configuration

With PHP-FPM configured and running, a standard Apache FCGI SetHandler configuration for PHP can be used to route PHP requests to the PHP-FPM daemons:

<FilesMatch "\.php$">
  Require all granted
  SetHandler proxy:balancer://localhost
</FilesMatch>
Include "php-pools.d/active.pool"

We're using a balancer proxy configuration here because it allows us to use the BalancerMember status flag to switch active daemons by putting one into "drain" mode, thereby allowing it to gracefully complete any active requests while preventing new requests from being initiated to that daemon/port.

We have two files in the "php-pools.d" directory:

$ vim php-pools.d/9001.pool
<Proxy "balancer://localhost">
BalancerMember "fcgi://localhost:9001" retry=1
BalancerMember "fcgi://localhost:9002" retry=1 status=N
</Proxy>

$ vim php-pools.d/9002.pool
<Proxy "balancer://localhost">
BalancerMember "fcgi://localhost:9001" retry=1 status=N
BalancerMember "fcgi://localhost:9002" retry=1
</Proxy>

php-pools.d/9001.pool sets the pool listening on port 9001 as active and sets the pool listening on port 9002 into "drain" mode to ensure it completes all active requests but will not be dispatched any new requests. php-pools.d/9002.pool does the natural opposite.

We also have a symlink "php-pools.d/active.pool" that points to one of the two pool files. This symlink allows us to quickly switch the active PHP-FPM daemon by re-setting the symlink and telling apache to reload.

Deployment script

Since we use Jenkins for our code deployments, we've added the following script to our deploy scripts, immediately after the git checkout:

$ vim restart_php.sh
#!/bin/sh

# Provides rolling restarts of PHP-FPM when using systemd socket activation.
#
# Assumes there is a /etc/httpd/php-pools.d/ containing two .pool files
# with Apache balancer configurations:
#
# /etc/httpd/php-pools.d/9001.pool:
#         <Proxy "balancer://localhost">
#           BalancerMember "fcgi://localhost:9001" retry=1
#           BalancerMember "fcgi://localhost:9002" retry=1 status=N
#         </Proxy>

# Set the pool directory
POOL_DIR="/etc/httpd/php-pools.d"
# Pool file names
POOL1="9001.pool"
POOL2="9002.pool"
# Pool service names
POOL1_SRV="php-fpm-www-systemd1.service"
POOL2_SRV="php-fpm-www-systemd2.service"

# Get the currently-active pool
ACTIVE=`readlink -f ${POOL_DIR}/active.pool | awk -F'/' {'print $5'}`

case $ACTIVE in
  "$POOL1")
    echo "${ACTIVE} is active, switching to ${POOL_DIR}/${POOL2} ...";
    # Bounce the dormant pool to make sure it's up-to-date
    systemctl restart $POOL2_SRV || { echo "Error reloading dormant pool."; exit 1; }
    cd $POOL_DIR;
    # Set active.pool to be the other pool
    ln -sf $POOL2 active.pool || { echo "Error linking active.pool."; exit 1; }
    # HUP httpd to update the balancer config
    systemctl reload httpd.service || { echo "Error reloading httpd."; exit 1; }
    ;;
  "$POOL2")
    echo "${ACTIVE} is active, switching to ${POOL_DIR}/${POOL1} ...";
    # Bounce the dormant pool to make sure it's up-to-date
    systemctl restart $POOL1_SRV || { echo "Error reloading dormant pool."; exit 1; }
    cd $POOL_DIR;
    # Set active.pool to be the other pool
    ln -sf $POOL1 active.pool || { echo "Error linking active.pool."; exit 1; }
    # HUP httpd to update the balancer config
    systemctl reload httpd.service || { echo "Error reloading httpd."; exit 1; }
    ;;
  *)
    echo "ACTIVE is: '${ACTIVE}', setting ${POOL1} as active ..."
    cd $POOL_DIR;
    ln -sf $POOL1 active.pool || { echo "Error linking active.pool."; exit 1; }
    # HUP httpd to update the balancer config
    systemctl reload httpd.service || { echo "Error reloading httpd."; exit 1; }
    ;;
esac

FINAL=`readlink -f ${POOL_DIR}/active.pool`
echo "Done. ${FINAL} is active."

Testing

We tested restarting PHP-FPM under load using Locust. We ran 100 simultaneous simulated users calling both fast-returning date() calls as well as a long-running, I/O-intensive find/grep. Throughput varied from 300-500 requests per second. We found we could run the restart script at will without dropping any active sessions. While under load, two restarts close enough together (less than a second or two apart) could cause a PHP-FPM master restart to happen on the dormant daemon before it's finished processing the requests from its previous term as the active daemon and cause termination of active requests. The odds of this happening in a real-world deployment, however, are slim.

Conclusion

The key to this configuration is the ability under systemd to run two separate PHP-FPM master daemons. Because the httpd reload is graceful, switching the active pool in the balancer configuration will not interrupt current transactions. The balancer configuration, through its use of the "drain" mode, allows the current transactions to complete on the now-dormant PHP-FPM daemon, but will not send any new requests to it. While this configuration does leave a full set of inactive PHP-FPM master and worker threads up at all times, the additional overhead is small enough and the benefits derived are significant enough to justify the cost.