Pro Puppet (27 page)

Both back-end Puppet master virtual hosts are now online and responding to requests. You can check the status of the Ruby processes Passenger has started using the
passenger-status
command. The
passenger-status
command indicates that the Puppet master process IDs started by Passenger when Puppet agent requests are routed to the back-end worker virtual hosts (see
Listing 4-20
).

You can see the two Passenger processes servicing the front-end. With that, we’ve configured a simple and very scalable Puppet master implementation. To scale it further, all you now need to do is follow a subset of these steps to add additional back-end workers to the configuration and into the pool.

We also chose to configure the front-end and back-end virtual hosts all on the same system, as we can see through the use of 127.0.0.1 in each of the back-end configuration files and the Proxy section of the front-end virtual host. The choice to run all of the worker processes on the same host has greatly simplified the signing of SSL certificates when connecting new Puppet agent nodes. As mentioned previously in this chapter, the serial number and certificate revocation lists must be kept in sync across Puppet master systems that issue new client certificates. In the next section, you’ll see how to manage back-end worker processes on separate systems.

Thus far in this chapter, you’ve configured the Puppet master service in a stand-alone Apache virtual host. Scaling the Puppet master system horizontally, you configured a number of Apache virtual hosts working together behind a reverse proxy load balancer. In both configurations, all of the Puppet master worker virtual hosts are running on the same host.

With multiple Puppet master workers running on the same system, all workers write Puppet certificates to the same file system location. For this reason, you don’t need to worry which worker accepts the certificate signing requests. But if you’d like to scale even further than what we’ve already done, and spread our Puppet master workers onto multiple systems, then management of certificates becomes an issue that you need to address.

There are a couple of ways you can address this issue:

• Synchronize the Puppet CA directory across all of the worker systems
  
• Make one worker system the active Puppet CA service and a second worker system the hot standby Puppet CA service
  

We’re going to show you how to use a hot (active) standby CA model to keep your certificate data synchronized. This architecture allows you to keep all Puppet CA data in one place, thereby minimizing the effort needed to maintain the Puppet master infrastructure (see
Figure 4-2
).

To do this, you will configure a second system to periodically synchronize the Puppet CA files. If the active Puppet CA system falls offline, the front-end load balancer will automatically redirect certificate requests to the hot standby. With the CA kept in sync, the hot standby will be ready to serve certificate-signing requests for new hosts.

The hot standby model requires the front-end Apache load balancer to redirect all certificate requests from all Puppet agent nodes to a specific set of Puppet master workers. We’ll demonstrate how to do this and see how to test the new configuration. Finally, we’ll show how to take the primary Puppet CA offline for maintenance and back online again, including handling if Puppet agents have submitted certificate requests to the hot standby.

Figure 4-2.
Puppet agent HTTPS load balancing

The first step to take when scaling the Puppet master infrastructure across multiple worker systems is to add two additional work virtual hosts for the Puppet CA service. While developing and testing, we’re going to configure these using the 127.0.0.1 local host address. In a production configuration, the addresses of each Puppet master and Puppet CA worker should be on different hosts.

First create two new virtual hosts using the existing configurations of the two we created earlier. The virtual host workers listening on port 18142 and 18143 will be specifically for the active and hot standby Puppet CA service:

Substitute all instances of the string “18140” with the string “18142” and write the output to the new configuration file named
42_puppetmaster_worker_18142.conf
. This configuration file is the virtual host for the Puppet CA.

Next, copy the entire contents of the original Rack configuration from
Listing 4-7
without modification. This configuration will use the default SSL directory:

Repeat the process of searching and replacing the port number for the standby Puppet CA worker. The virtual host listening on port 18142 will become the active Puppet CA back-end worker. The virtual host listening on port 18143 will become the hot standby Puppet CA back-end worker.

In order to simulate the two different Puppet CA workers living on two different systems, you’ll need to duplicate the existing CA directory into a new location. This configuration prevents the two Puppet CA systems from sharing the same CA directory and certificate revocation list:

The rsync command duplicates the existing Puppet CA directory into a new directory at
/var/lib/puppet/ssl/ca.standby/
:

We’re editing the file to add a single line immediately above the existing “ARGV” line. The new line contains
ARGV << "--cadir" << "/var/lib/puppet/ssl/ca.standby"
. A complete listing of the standby CA’s Rack configuration is provided in
Listing 4-21
.

Using this configuration, the Active Puppet CA worker will continue to use
/var/lib/puppet/ssl/ca/
while the standby worker will use
/var/lib/puppet/ssl/ca.standby/
.

We should now restart the Apache service to make sure the changes are valid but at this point the front-end HTTPS reverse proxy has not yet been configured to route any requests to either of these two Puppet CA workers.

We now need to configure the front-end load balancer to redirect all certificate related requests to the new port 18142 worker. We also configure the load balancer to fall back to the hot standby running at 18143 if the primary Puppet CA worker is offline (see
Listing 4-22
).

As we can see in
Listing 4-22
, a new Proxy section configures the load balancer to first connect to
http://127.0.0.1:18142
, and then connect to
http://127.0.0.1:18143
when a request is sent to the balancer named
puppetmasterca
. The option
status=+H
tells the front end that the second member is a hot standby.

With the back-end Puppet CA workers configured, the load balancer must now be configured to route certificate requests, and only certificate requests, to the two member workers. This configuration listing goes in the main Apache front-end virtual host block, as shown in
Listing 4-23
.

Here, we configured the load balancer to handle requests matching a pattern indicating they are certificate-related. We configured the load balancer to direct these requests to the group of workers named
balancer://puppetmasterca,
which were defined in
Listing 4-22
. Using this group of workers guarantees that the load balancer will send the request to the worker on 18142 if it is online, and 18143 if 18142 is down, and return HTTP status 503 Temporarily Unavailable if neither is available.

The
ProxyPassMatch
directive configures a regular expression to match against the request URI of the Puppet agent. In this case, we have configured the URI containing certificate in the second path element as a match. This ensures that certificate requests are directed appropriately, regardless of the environment or the Puppet agent name.

After configuring the two back-end Puppet CA worker virtual hosts on ports 18142 and 18143, you need to restart Apache:

Let’s test the new configuration, with a new system named
mock.example.com
:

Once the new Puppet agent creates a certificate-signing request and submits it to the load balancer, we can check the Apache logs to make sure that CA requests are being routed properly to the worker listening on port 18142.

In
Listing 4-24
, you can see a number of HTTP 404 status results on the second and third line of the logs. Apache is returning status 404 “Not Found” because the Puppet node
mock.example.com
is a new node and no signed certificates or certificate requests exist for this system. Until we sign the new certificate request using the
puppet cert --sign
command, the Puppet CA worker will continue to return 404 “Not Found” status codes to the Puppet agent on
mock.example.com
.

To make sure the Puppet agent is routed to the correct worker system, you need to sign the new certificate request: