Deploying a software update fleet-wide using debdeploy and debmonitor

Each update needs to be deployed using a spec file. To generate one log into one of the cluster management hosts (cumin1002.eqiad.wmnet, cumin2002.codfw.wmnet as your regular user and run:

 generate-debdeploy-spec

It will walk you through the necessary steps. For the 'update type' pick 'tool'. Note that if you deploy an update to a server, this usually means that the Debian package update will restart the service, so make sure to coordinate them with proper depools. For updating libraries see the next section.

For each distro, enter the version to update to. If an update is only available for a specific distro, simply leave the version for the others empty. debdeploy will check the installed distro release during deployment and only update system which have a target version specified.

elukey@cumin1002:~$ generate-debdeploy-spec
Please enter the name of source package (e.g. openssl). Leave blank or type 'quit' to abort
>prometheus-memcached-exporter
You can enter an optional comment, e.g. a reference to a security advisory or a CVE ID mapping
>
tool           -> The updated packages is an enduser tool, can be
                  rolled-out immediately.
daemon-direct  -> Daemons which are restarted during update, but which
                  do no affect existing users.
daemon-disrupt -> Daemons which are restarted during update, where the
                  users notice an impact. The update procedure is almost
                  identical, but displays additional warnings
library        -> After a library is updated, programs may need to be
                  restarted to fully effect the change. In addition
                  to libs, some applications may also fall under this rule,
                  e.g. when updating QEMU, you might need to restart VMs.
Please enter the update type:
>daemon-direct
Please enter the version which fixed in in jessie. Leave blank if no fix is available/required for a given distro.
>0.4.1+git20181010.2fa99eb-1
Please enter the version which fixed in in trusty. Leave blank if no fix is available/required for a given distro.
>
Please enter the version which fixed in in stretch. Leave blank if no fix is available/required for a given distro.
>0.4.1+git20181010.2fa99eb-1
Please enter a name under which the YAML file should be created
>2018-10-11-prometheus-memcached-exporter.yaml

Once you have generated the spec file you can deploy it with debdeploy. Debdeploy primarily operates on Cumin aliases, you can see the available aliases in /etc/cumin/aliases.yaml.

To e.g. deploy the update to all restbase hosts you can run:

 sudo debdeploy deploy -u $SPECFILE -s restbase

You can also deploy towards complete data centers by using the aliases eqiad, codfw, etc. There's also all.

For addressing hosts not matched with existing Cumin aliases you can also target a specific Puppetdb query using -Q, e.g. to only deploy to host db1234:

 sudo debdeploy deploy -u $SPECFILE -Q "{db1234*}"

Note that debdeploy won't run apt update, so if you uploaded the new version very recently, it won't see any updateable packages and not apply anyy changes. You can either use cumin to run apt update on all affected hosts, or wait up to 30 minutes for it to be run automatically along with the Puppet agent.

Example of execution:

elukey@cumin2002:~$ sudo debdeploy deploy -u 2018-10-11-prometheus-memcached-exporter.yaml -s memcached-codfw
Rolling out prometheus-memcached-exporter:
Daemon update without user impact

These hosts are already up-to-date:
  mc[2020-2021,2026,2030,2035-2036].codfw.wmnet (6 hosts)

prometheus-memcached-exporter was updated: 0.3.0+ds1-1 -> 0.4.1+git20181010.2fa99eb-1
  mc[2019,2022-2025,2027-2029,2031-2034].codfw.wmnet (12 hosts)

To track which servers still need a given update you can use https://debmonitor.wikimedia.org/

Deploying a library update fleet-wide using debdeploy

Rolling out a library update is a little different compared to a standalone tool. The notable difference is that applications using a library need to be restarted to pick up the update (as the dynamic loader keep the old copy of a shared library in memory as long as applications are using it). debdeploy has detection for library restarts built in.

When deploying an update for a library select library as the update type. It will prompt for a list of library base names, these are usually prefilled from Puppet already, so you can simply confirm.

Instead of deploying an update with deploy, for a library you can use the deploy_restartcheck command of debdeploy. It will print a list of all services which have still run a copy of the old library version (you can also re-run the restart check without the deployment step by running the query_restart command.

A lot of service restarts are automated in our environment via the profile::auto_restarts::service define. It runs daily (and splayed across the day). In order to keep the output of the restart command useful, these auto-restarted services are automatically filtered out. If you pass the --printall parameter you can have debdeploy print the unfiltered list (if you e.g. want to debug an issue with a failing auto restart or similar).

In addition there are also some services which cannot be restarted. They are tracked in puppet.git in hieradata/common/profile/debdeploy/client.yaml. Examples are agetty, systemd pid 1 or dbus. If any of these would have an impactful security issue, instead of restarting the service we'd need to issue reboots instead.

Running the remaining service restarts is not in scope for debdeploy. At this point we usually have cookbooks for all the services which need to be restarted.