Puppet request window

What is it?

Patches submitted to Puppet request windows will be looked at by Site Reliability Engineers (SRE) twice a week. The window is similar to MediaWiki backport windows in structure. The patches will be reviewed and, if approved, merged. The goal is to encourage more people to write patches for operations/puppet.git, and have a Service Level Agreement (SLA) of sorts for patches to be looked at and/or merged.

The time slot of the Puppet request window is typically twice weekly, but this can shift (in advance) to accommodate other deployments. Two SREs are typically signed up per window. During (or before) the window, those (and other) SRE team members should review listed patches and provide feedback.

How to get a patch in a Puppet request window

• Add it to the designated Puppet request window on the Deployments page.
  • This is intentionally 'fast and loose'. As long as a patch is on the listing before the Puppet request window, SRE will do what we can to review and merge these. Due to the repetition of the Puppet request windows, any patches submitted too close to the window (and unable to be properly reviewed) may be pushed to the next Puppet request window.
• Be present during the Puppet request window for real-time conversations regarding the patch and testing/deployment/reverting. (This will take place on libera.chat IRC, in #wikimedia-operations ^connect.)

What kind of patches can go through Puppet request windows?

Puppet request window patches should be:

• Trivial in complexity.
• Easy to verify and test for people who are not in the SRE team.
• (Preferred) At least one Code-Review +1 from someone familiar with the area of code the patch is changing.

Patches that have potentially far reaching impact (ssh, varnish, apache) will likely be rejected from Puppet request windows. Substantial changes to the Apache configuration for MediaWiki application servers are not eligible for the request window, due to the potentially far reaching impact / unavailability. These need extensive testing and should be scheduled with SRE outside Puppet request windows. This guideline is still evolving.

Patches to private Puppet can be merged in the Puppet request window, as long as they meet the criteria above. There are no code review tools for the private repository; the SRE running the window will make your change on the puppetmaster in a text editor. You can give them the diff in any convenient way, as long as it's clear what you need -- including instructions for generating any passwords, etc. (Please still get a "looks good to me" from someone familiar with the code, even though they can't click a +1 button.)

The SRE doing the request window has final discretion on which patches they merge, since they are ultimately responsible for the stability of the cluster. Also, do not use Puppet request windows as a way to speed up work if you're already collaborating with any SREs on a specific project. If patches are lagging behind there, there is a specific reason and you should refer to the person you're working with, or escalate this.

Examples (from Giuseppe):

Good patches for Puppet request windows:

• https://gerrit.wikimedia.org/r/230382 - This was -2'd by me because I was already on a better solution. But, it is limited in scope, easy to understand, and works on something already existing.
• https://gerrit.wikimedia.org/r/207140
• https://gerrit.wikimedia.org/r/226901

Changes that cannot go through Puppet request windows:

• https://gerrit.wikimedia.org/r/221827 - Lots of code changes to review, will need some SRE to follow the deployment with the author.
• https://gerrit.wikimedia.org/r/229727 - Large patch introducing new functions, part of work being followed by multiple SREs already

The ideal Puppet request window patch has...

1. The author / someone involved with the patch around on IRC during the Puppet request window
2. A +1 on the patch from someone.
3. Puppet Compiler has been run on the patch and it has given a go ahead
4. Rebases cleanly to master
5. For patches that can be tested on the Beta Cluster, they should be tested by being cherry-picked to the beta cluster puppetmaster.

The ideal Puppet request window patch does *not* have...

1. Any sudo / access rights changes
2. Any outstanding -1s / unaddressed concerns

Who is going to do it?

The SRE team allocates two members for Puppet request windows. These members are designated in advance of their assigned weeks. Currently, the same SREs handle the request windows each week, given the low rate of submitted patches in recent months. If other SREs take over, either on a temporary or on a rotating basis, they must update the Deployments page to list them for those allotted Puppet request windows.