Fundraising/techops/procedures/users-ssl client certs yearly renewal
SSL Client Certificate Yearly Renewal Process
Every year, we need to renew all of the SSL client certificates for our users. These are the steps and tools used to do so.
Expiration list
Every day, the ssl_user_admin tool runs with the expirecheck option. This will output any certificates that will expire within the next 14 days and email that to fr-tech ops. The output of this command or email can be used by a script to generate the necessary tickets and notifications.
Notification and ticket scripting
Since we have many of these to run every year, we have automated the process of creating the necessary tickets and email notification. This is done by running the the script as follows: template_ssl_cert_expiration.pl sample_certs_expiring
Phab ticket generation
In the output of the script, there is a link that can be clicked/copy-pasted to generate a phab task in the appropriate fr-tech-ops workboard. This can be used to track the progress of the renewal.
User notification email
Additionally, the script outputs a mailto link that can be clicked and will generate a template email for contacting the user. Generally this has been used to spawn a gmail message that can be checked for sanity and then sent on to the user. This informs them that their certificate is up for renewal and asks for a time that we could send them a new certificate. It clearly states that a meeting isn't needed and that the ask is just to ensure they have time then to install the certificate after we generate it.
Certificate renewal
Standard renewal with no changes
If you want to renew a certificate keeping all of the certificate details as they are (ie. name, location, email, etc) you can just use: ssl_user_admin renew username
This will revoke and renew the certificate using the configuration from the existing certificate. This is the easiest way to renew a certificate.
Renewal with changes
If you want to renew a certificate but need to update any details (ie. name, location, email, etc) you will need to do a two step process.
First you will revoke the existing certificate: ssl_user_admin revoke username
Then you will create a new certificate specifying all the details as if it were a first time creation: ssl_user_admin create username
At this time, there is no function to seed the new certificate creation with default data.
Password notification
After you renew the certificate, you will get the standard user.pw file in your home directory. That will need to be sent to the user via SMS the same as a standard certificate creation.
CRL update and probable git dance
The CRL will need to be updated to remove the access for the old certificate. We don't want to do this until we verify that the user has loaded the updated certificate on their host. This can cause some issues around renew time as we may have people installing certificates out of order but the CRL is linear.
Example:
- Users A, B, C are all notified of a renewal.
- User A gets back and their certificate is renewed.
- User B gets back and their certificate is renewed.
- User A applies certificate
- User C gets back and their certificate is renewed.
- User C applies certificate
- User B applies certificate
If we merge the CRL changes after User C applies their certificate, it will include the revocation of User B's cert and they will lose access to resources. As there may be days before a user may apply the certificate, the chance of it happening is likely. This is suboptimal.
To get around this, I have generally held a branch with CRL updates and created a new diff for each revocation tagged with the Phab task as the Bug:. I then regularly track back through the expirecheck emails to ensure that all users prior have applied their certificate before I commit each change to the main git branch. Additionally, I can check the expire date on the user, since it is clear to push the CRL once their expire date has been hit even if they haven't applied the certificate yet.
It is best to use the email output since as soon as a certificate is updated, their entry is removed from the next day's email.
An alternative would be to wait to apply the CRL until all certificates have been updated and applied at the end of the month. I have tended not to go this route as it seems unwise to have the frpm on disk file be so far out of date with what is deployed to the other hosts.