Jump to content

Portal:Cloud VPS/Admin/Runbooks/PuppetCertificateAboutToExpire

From Wikitech
The procedures in this runbook require project admin permissions to complete.

This happens when a Puppet certificate is about to expire. It might be either a normal host certificate or a CA certificate.

Error / Incident

This usually comes in the form of an alert in alertmanager.

The alert will include the CN for the certificate that's expiring.

Debugging

The alert will tell you also how much is left for the certificate to expire in the summary, for example:

 summary: Puppet CA certificate Puppet CA: clouddb-services-puppetmaster-01.clouddb-services.eqiad.wmflabs is about to expire in 27d 3h 0m 43s

Common issues

Host certificate about to expire

If a host certificate is about to expire, run the wmcs.vps.refresh_puppet_certs cookbook to renew it. (This usually means that the host is quite old, so consider replacing it with one that uses a newer OS version!)

If the cookbook fails with "The certificate retrieved from the master does not match the agent's private key", try cleaning up the cert manually from the puppet server:

# puppetserver ca clean --certname {FQDN}

Then from the puppet client delete the old cert and re-run Puppet to generate a new one:

# rm /var/lib/puppet/ssl/{FQDN}.pem
# run-puppet-agent

CA certificate about to expire

Sometimes it's the Puppet CA certificate itself that's about to expire. This is happening if the certificate CN starts with Puppet CA:.

To renew that certificate, follow Help:Standalone_puppetmaster#Renewing_puppetmaster_CA_certificate.


Old incidents

  • T354714 - [clouddb-service-puppetmaster-2] Renew puppet CA certificates