Server Admin Log/Archive 85

2024-09-30

• 22:44 jhathaway@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1002.eqiad.wmnet with OS bookworm
• 22:00 zabe@deploy2002: Finished scap sync-world: Backport for s3: Reduce revision-slots cache expiry to 60 seconds (T183490) (duration: 06m 54s)
• 21:55 zabe@deploy2002: zabe: Continuing with sync
• 21:55 zabe@deploy2002: zabe: Backport for s3: Reduce revision-slots cache expiry to 60 seconds (T183490) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 21:54 jhathaway@cumin1002: START - Cookbook sre.hosts.reimage for host sretest1002.eqiad.wmnet with OS bookworm
• 21:54 jhathaway@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1002.eqiad.wmnet with OS bookworm
• 21:53 zabe@deploy2002: Started scap sync-world: Backport for s3: Reduce revision-slots cache expiry to 60 seconds (T183490)
• 21:47 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 21:47 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 21:31 zabe@deploy2002: Finished scap sync-world: Backport for Make revision-slots expiry configurable (T183490) (duration: 07m 42s)
• 21:26 zabe@deploy2002: zabe: Continuing with sync
• 21:25 zabe@deploy2002: zabe: Backport for Make revision-slots expiry configurable (T183490) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 21:23 zabe@deploy2002: Started scap sync-world: Backport for Make revision-slots expiry configurable (T183490)
• 21:06 jhathaway@cumin1002: START - Cookbook sre.hosts.reimage for host sretest1002.eqiad.wmnet with OS bookworm
• 20:56 jhathaway@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1002.eqiad.wmnet with OS bookworm
• 20:56 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site ulsfo [reason: repool ulsfo as cr3-ulsfo was replaced, T375345]
• 20:56 sukhe@cumin1002: START - Cookbook sre.dns.admin DNS admin: pool site ulsfo [reason: repool ulsfo as cr3-ulsfo was replaced, T375345]
• 20:45 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 20:35 ebernhardson@deploy2002: Finished scap sync-world: Backport for ClosedWikiProvider: Support canAlwaysAutocreate option (T374987) (duration: 10m 40s)
• 20:34 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 20:30 ebernhardson@deploy2002: ebernhardson: Continuing with sync
• 20:27 ebernhardson@deploy2002: ebernhardson: Backport for ClosedWikiProvider: Support canAlwaysAutocreate option (T374987) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:24 ebernhardson@deploy2002: Started scap sync-world: Backport for ClosedWikiProvider: Support canAlwaysAutocreate option (T374987)
• 20:22 ebernhardson@deploy2002: Finished scap sync-world: Backport for Change votewiki language back to English. (T302443) (duration: 06m 41s)
• 20:18 ebernhardson@deploy2002: ebernhardson, ahonc: Continuing with sync
• 20:18 ebernhardson@deploy2002: ebernhardson, ahonc: Backport for Change votewiki language back to English. (T302443) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:16 ebernhardson@deploy2002: Started scap sync-world: Backport for Change votewiki language back to English. (T302443)
• 20:13 ebernhardson@deploy2002: Finished scap sync-world: Backport for cirrus: Remove unused Regex pool counter (T369808) (duration: 07m 34s)
• 20:08 ebernhardson@deploy2002: ebernhardson: Continuing with sync
• 20:08 ebernhardson@deploy2002: ebernhardson: Backport for cirrus: Remove unused Regex pool counter (T369808) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:05 ebernhardson@deploy2002: Started scap sync-world: Backport for cirrus: Remove unused Regex pool counter (T369808)
• 19:56 jhathaway@cumin1002: START - Cookbook sre.hosts.reimage for host sretest1002.eqiad.wmnet with OS bookworm
• 19:56 jhathaway@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1002.eqiad.wmnet with OS bookworm
• 19:08 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs[4008-4010].ulsfo.wmnet
• 19:08 sukhe@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs[4008-4010].ulsfo.wmnet
• 18:41 jhathaway@cumin1002: START - Cookbook sre.hosts.reimage for host sretest1002.eqiad.wmnet with OS bookworm
• 18:41 jhathaway@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1002.eqiad.wmnet with OS bookworm
• 18:04 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on lvs[4008-4010].ulsfo.wmnet with reason: site is depooled, cr3-ulsfo is being replaced
• 18:04 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on lvs[4008-4010].ulsfo.wmnet with reason: site is depooled, cr3-ulsfo is being replaced
• 17:47 ebernhardson@deploy2002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:47 ebernhardson@deploy2002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:44 ebernhardson@deploy2002: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:44 ebernhardson@deploy2002: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:25 ebernhardson@deploy2002: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:24 ebernhardson@deploy2002: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:22 jhathaway@cumin1002: START - Cookbook sre.hosts.reimage for host sretest1002.eqiad.wmnet with OS bookworm
• 17:22 jhathaway@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1002.eqiad.wmnet with OS bookworm
• 17:12 ebernhardson@deploy2002: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:11 ebernhardson@deploy2002: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:53 swfrench-wmf: built and published new php8.1 images - T376036
• 16:51 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 16:41 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 16:27 swfrench-wmf: imported php8.1_8.1.30-1+wmf11u1 into component/php81 - T376036
• 16:05 jhathaway@cumin1002: START - Cookbook sre.hosts.reimage for host sretest1002.eqiad.wmnet with OS bookworm
• 15:56 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logging-hd1005.eqiad.wmnet with OS bookworm
• 15:56 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 15:55 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 15:52 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logging-hd1004.eqiad.wmnet with OS bookworm
• 15:52 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 15:51 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 15:49 ayounsi@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1002.eqiad.wmnet with OS bookworm
• 15:45 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc-misc1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 15:42 jclark@cumin1002: START - Cookbook sre.hosts.provision for host mc-misc1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 15:41 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc-misc1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 15:41 jclark@cumin1002: START - Cookbook sre.hosts.provision for host mc-misc1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 15:41 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:40 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc-misc1002
• 15:39 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host mc-misc1002
• 15:39 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc-misc1001
• 15:39 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host mc-misc1001
• 15:39 jclark@cumin1002: START - Cookbook sre.dns.netbox
• 15:35 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logging-hd1005.eqiad.wmnet with reason: host reimage
• 15:32 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 15:32 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logging-hd1004.eqiad.wmnet with reason: host reimage
• 15:31 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 15:29 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:28 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt mc-misc - jclark@cumin1002"
• 15:28 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt mc-misc - jclark@cumin1002"
• 15:27 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc-misc1002
• 15:27 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host mc-misc1002
• 15:27 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc-misc1001
• 15:27 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on logging-hd1005.eqiad.wmnet with reason: host reimage
• 15:26 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host mc-misc1001
• 15:26 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on logging-hd1004.eqiad.wmnet with reason: host reimage
• 15:25 jclark@cumin1002: START - Cookbook sre.dns.netbox
• 15:12 volans: upgraded spicerack to 8.14.0 on cumin1002
• 15:11 marostegui@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dbproxy2005.codfw.wmnet with OS bookworm
• 15:08 mforns@deploy2002: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
• 15:08 mforns@deploy2002: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
• 14:55 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host logging-hd1005.eqiad.wmnet with OS bookworm
• 14:55 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host logging-hd1004.eqiad.wmnet with OS bookworm
• 14:51 swfrench@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=shellbox-video,name=codfw [reason: Pooling shellbox-video in codfw before (re)pooling eqiad on Wednesday - T370962]
• 14:45 elukey: force a run of upload_puppet_facts.service on puppetserver1001 to pick up new facts/hosts
• 14:41 _joe_: updating conftool to 3.3.0
• 14:39 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host logging-hd2005
• 14:39 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host logging-hd2005
• 14:38 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host logging-hd2004
• 14:38 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host logging-hd2004
• 14:36 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host logging-hd1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:36 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host logging-hd1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:36 jclark@cumin1002: START - Cookbook sre.hosts.provision for host logging-hd1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:36 jclark@cumin1002: START - Cookbook sre.hosts.provision for host logging-hd1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:35 _joe_: running requestctl --debug upgrade-schema
• 14:35 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:35 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt logging-hd1005 - jclark@cumin1002"
• 14:35 ayounsi@cumin1002: START - Cookbook sre.hosts.reimage for host sretest1002.eqiad.wmnet with OS bookworm
• 14:34 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt logging-hd1005 - jclark@cumin1002"
• 14:31 jclark@cumin1002: START - Cookbook sre.dns.netbox
• 14:31 _joe_: uploaded conftool 3.3.0 to apt
• 14:01 volans: upgraded spicerack to 8.14.0 on cumin2002
• 13:59 gmodena@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 13:59 gmodena@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 13:52 elukey@deploy2002: Finished scap sync-world: Allow udp traffic to irc1003 (duration: 02m 04s)
• 13:52 gmodena@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich-next: apply
• 13:52 gmodena@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich-next: apply
• 13:51 elukey@deploy2002: Started scap sync-world: Allow udp traffic to irc1003
• 12:55 stevemunene@cumin1002: END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker1177.eqiad.wmnet
• 12:54 stevemunene@cumin1002: START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1177.eqiad.wmnet
• 12:53 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=99) for hosts an-worker1177.eqiad.wmnet
• 12:48 stevemunene@cumin1002: START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1177.eqiad.wmnet
• 12:47 stevemunene@cumin1002: END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker1176.eqiad.wmnet
• 12:46 stevemunene@cumin1002: START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1176.eqiad.wmnet
• 12:43 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=99) for hosts an-worker1176.eqiad.wmnet
• 11:56 Dreamy_Jazz: Running `foreachwiki extensions/CheckUser/maintenance/purgeOldData.php` on a tmux session
• 11:38 dreamyjazz@deploy2002: Finished scap sync-world: Backport for Don't implement CheckUserQueryInterface in purgeOldData.php (T376022) (duration: 06m 35s)
• 11:37 volans: uploaded spicerack_8.14.0 to apt.wikimedia.org bullseye-wikimedia
• 11:34 dreamyjazz@deploy2002: dreamyjazz: Continuing with sync
• 11:34 dreamyjazz@deploy2002: dreamyjazz: Backport for Don't implement CheckUserQueryInterface in purgeOldData.php (T376022) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 11:32 dreamyjazz@deploy2002: Started scap sync-world: Backport for Don't implement CheckUserQueryInterface in purgeOldData.php (T376022)
• 11:31 zabe@deploy2002: Finished scap sync-world: Backport for Pin wgRevisionSlotsCacheExpiry to default (T183490), Move closed wikis to group0 except aawiki (duration: 07m 30s)
• 11:27 zabe@deploy2002: zabe: Continuing with sync
• 11:26 zabe@deploy2002: zabe: Backport for Pin wgRevisionSlotsCacheExpiry to default (T183490), Move closed wikis to group0 except aawiki synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 11:24 zabe@deploy2002: Started scap sync-world: Backport for Pin wgRevisionSlotsCacheExpiry to default (T183490), Move closed wikis to group0 except aawiki
• 11:18 jnuche@deploy2002: Installing scap version "4.104.0" for 212 hosts
• 11:16 dreamyjazz@deploy2002: scap failed: <CalledProcessError> Command '['docker', 'run', '--rm', '--attach', 'stdin', '--attach', 'stdout', '--attach', 'stderr', '--user', 'www-data', '--mount', 'type=bind,source=/srv/mediawiki-staging,target=/srv/mediawiki-staging', '--mount', 'type=bind,source=/tmp,target=/tmp', '--workdir', '/srv/mediawiki-staging', '--entrypoint', '/bin/bash', '--network', 'none', 'docker-registry.wiki
• 11:16 dreamyjazz@deploy2002: Started scap sync-world: Testing deployment using no-op deployment change
• 11:05 stran@deploy2002: helmfile [codfw] DONE helmfile.d/services/ipoid: apply
• 11:04 stran@deploy2002: helmfile [codfw] START helmfile.d/services/ipoid: apply
• 11:03 stran@deploy2002: helmfile [eqiad] DONE helmfile.d/services/ipoid: apply
• 11:03 zabe@deploy2002: scap failed: <CalledProcessError> Command '['docker', 'run', '--rm', '--attach', 'stdin', '--attach', 'stdout', '--attach', 'stderr', '--user', 'www-data', '--mount', 'type=bind,source=/srv/mediawiki-staging,target=/srv/mediawiki-staging', '--mount', 'type=bind,source=/tmp,target=/tmp', '--workdir', '/srv/mediawiki-staging', '--entrypoint', '/bin/bash', '--network', 'none', 'docker-registry.wikimedia.
• 11:03 zabe@deploy2002: Started scap sync-world: Backport for Pin wgRevisionSlotsCacheExpiry to default (T183490), Move closed wikis to group0 except aawiki
• 11:03 stran@deploy2002: helmfile [eqiad] START helmfile.d/services/ipoid: apply
• 11:03 zabe@deploy2002: scap failed: <CalledProcessError> Command '['docker', 'run', '--rm', '--attach', 'stdin', '--attach', 'stdout', '--attach', 'stderr', '--user', 'www-data', '--mount', 'type=bind,source=/srv/mediawiki-staging,target=/srv/mediawiki-staging', '--mount', 'type=bind,source=/tmp,target=/tmp', '--workdir', '/srv/mediawiki-staging', '--entrypoint', '/bin/bash', '--network', 'none', 'docker-registry.wikimedia.
• 11:03 zabe@deploy2002: Started scap sync-world: Backport for Pin wgRevisionSlotsCacheExpiry to default (T183490), Move closed wikis to group0 except aawiki
• 11:01 stran@deploy2002: helmfile [staging] DONE helmfile.d/services/ipoid: apply
• 11:00 stran@deploy2002: helmfile [staging] START helmfile.d/services/ipoid: apply
• 11:00 stran@deploy2002: helmfile [eqiad] DONE helmfile.d/services/ipoid: apply
• 11:00 stran@deploy2002: helmfile [eqiad] START helmfile.d/services/ipoid: apply
• 10:59 stran@deploy2002: helmfile [staging] DONE helmfile.d/services/ipoid: apply
• 10:58 stran@deploy2002: helmfile [staging] START helmfile.d/services/ipoid: apply
• 10:57 ladsgroup@cumin1002: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw
• 10:45 ladsgroup@cumin1002: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw
• 10:45 elukey@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host irc1003.wikimedia.org
• 10:45 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host irc1003.wikimedia.org with OS bookworm
• 10:43 ladsgroup@cumin1002: END (FAIL) - Cookbook sre.switchdc.databases.finalize (exit_code=99) for the switch from eqiad to codfw
• 10:43 ladsgroup@cumin1002: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw
• 10:36 moritzm: upload ircstream 0.12.0~dev+wmf1 to apt.wikimedia.org T376014
• 10:32 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on irc1003.wikimedia.org with reason: host reimage
• 10:27 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on irc1003.wikimedia.org with reason: host reimage
• 10:19 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:18 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:15 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host irc1003.wikimedia.org with OS bookworm
• 10:13 Dreamy_Jazz: Restarted MediaModeration scanning script, starting it up again using mwscript-k8s - https://wikitech.wikimedia.org/wiki/MediaModeration
• 10:11 Dreamy_Jazz: Started time limited MediaModeration scan on enwiki - https://wikitech.wikimedia.org/wiki/MediaModeration
• 10:05 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM irc1003.wikimedia.org - elukey@cumin1002"
• 10:05 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM irc1003.wikimedia.org - elukey@cumin1002"
• 10:04 elukey@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) irc1003.wikimedia.org on all recursors
• 10:04 elukey@cumin1002: START - Cookbook sre.dns.wipe-cache irc1003.wikimedia.org on all recursors
• 10:04 elukey@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:04 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM irc1003.wikimedia.org - elukey@cumin1002"
• 10:04 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM irc1003.wikimedia.org - elukey@cumin1002"
• 09:45 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:44 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:42 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:42 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:06 ladsgroup@deploy2002: Finished scap sync-world: Backport for Remove metawiki dark mode exceptions (duration: 22m 27s)
• 09:00 gmodena@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 09:00 gmodena@deploy2002: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply
• 08:59 gmodena@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 08:59 gmodena@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply
• 08:57 gmodena@deploy2002: helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 08:57 gmodena@deploy2002: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply
• 08:56 ladsgroup@deploy2002: ebrahim, ladsgroup: Continuing with sync
• 08:56 ladsgroup@deploy2002: ebrahim, ladsgroup: Backport for Remove metawiki dark mode exceptions synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 08:44 ladsgroup@deploy2002: Started scap sync-world: Backport for Remove metawiki dark mode exceptions
• 08:37 gmodena@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 08:37 gmodena@deploy2002: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply
• 08:26 gmodena@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 08:26 gmodena@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply
• 08:23 gmodena@deploy2002: helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 08:23 gmodena@deploy2002: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply
• 08:12 elukey@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host aux-k8s-ctrl1003.eqiad.wmnet
• 08:12 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-ctrl1003.eqiad.wmnet with OS bullseye
• 08:01 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-ctrl1003.eqiad.wmnet with reason: host reimage
• 07:57 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-ctrl1003.eqiad.wmnet with reason: host reimage
• 07:47 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 07:46 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 07:45 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host aux-k8s-ctrl1003.eqiad.wmnet with OS bullseye
• 07:33 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM aux-k8s-ctrl1003.eqiad.wmnet - elukey@cumin1002"
• 07:33 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM aux-k8s-ctrl1003.eqiad.wmnet - elukey@cumin1002"
• 07:33 elukey@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aux-k8s-ctrl1003.eqiad.wmnet on all recursors
• 07:33 elukey@cumin1002: START - Cookbook sre.dns.wipe-cache aux-k8s-ctrl1003.eqiad.wmnet on all recursors
• 07:33 elukey@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 07:33 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-ctrl1003.eqiad.wmnet - elukey@cumin1002"
• 07:33 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-ctrl1003.eqiad.wmnet - elukey@cumin1002"
• 07:25 elukey@cumin1002: START - Cookbook sre.dns.netbox
• 07:25 elukey@cumin1002: START - Cookbook sre.ganeti.makevm for new host aux-k8s-ctrl1003.eqiad.wmnet

2024-09-29

• 22:57 eileen: config revision changed from 88cf5ff4 to a2ba4a8d

2024-09-28

• 22:13 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site eqsin [reason: repool eqsin take two, cr2-eqsin config restored, T375961]
• 22:13 sukhe@cumin1002: START - Cookbook sre.dns.admin DNS admin: pool site eqsin [reason: repool eqsin take two, cr2-eqsin config restored, T375961]
• 22:13 sukhe@cumin1002: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: pool site eqsin [reason: repool eqsin take two, cr2-eqsin config restored, no task ID specified]
• 22:13 sukhe@cumin1002: START - Cookbook sre.dns.admin DNS admin: pool site eqsin [reason: repool eqsin take two, cr2-eqsin config restored, no task ID specified]
• 21:17 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site eqsin [reason: testing failed, depooling again, no task ID specified]
• 21:16 sukhe@cumin1002: START - Cookbook sre.dns.admin DNS admin: depool site eqsin [reason: testing failed, depooling again, no task ID specified]
• 21:15 sukhe: sudo cumin 'A:lvs-eqsin' 'systemctl restart pybal'
• 21:12 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site eqsin [reason: testing pooling after cr2-eqsin was down and site was depooled, no task ID specified]
• 21:12 sukhe@cumin1002: START - Cookbook sre.dns.admin DNS admin: pool site eqsin [reason: testing pooling after cr2-eqsin was down and site was depooled, no task ID specified]
• 20:33 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site eqsin [reason: cr2-eqsin down, no task ID specified]
• 20:30 sukhe@cumin1002: START - Cookbook sre.dns.admin DNS admin: depool site eqsin [reason: cr2-eqsin down, no task ID specified]
• 18:58 ejegg: disabled CiviCRM contact dedupe job

2024-09-27

• 23:18 Dreamy_Jazz: Starting time limited scan on enwiki to catch-up with monthly limit rate - https://wikitech.wikimedia.org/wiki/MediaModeration
• 16:26 dduvall@deploy2002: Finished scap sync-world: Backport for Define $wmgLBFactoryConfigCallback in offline mode (duration: 22m 18s)
• 16:18 dduvall@deploy2002: dduvall: Continuing with sync
• 16:17 dduvall@deploy2002: dduvall: Backport for Define $wmgLBFactoryConfigCallback in offline mode synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 16:04 dduvall@deploy2002: Started scap sync-world: Backport for Define $wmgLBFactoryConfigCallback in offline mode
• 13:52 jayme: imported cricrl 1.31.1-2 to bookworm-wikimedia - T362408
• 13:32 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 13:32 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 13:23 akosiaris@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mw[1393-1397].eqiad.wmnet
• 13:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1393-1397].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - akosiaris@cumin1002"
• 13:21 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1393-1397].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - akosiaris@cumin1002"
• 13:08 jynus@cumin1002: dbctl commit (dc=all): 'repool es1022 with 100% load', diff saved to https://phabricator.wikimedia.org/P69434 and previous config saved to /var/cache/conftool/dbconfig/20240927-130855-jynus.json
• 13:08 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 12:53 jynus@cumin1002: dbctl commit (dc=all): 'repool es1022 with 1% load', diff saved to https://phabricator.wikimedia.org/P69433 and previous config saved to /var/cache/conftool/dbconfig/20240927-125317-jynus.json
• 12:48 akosiaris@cumin1002: START - Cookbook sre.hosts.decommission for hosts mw[1393-1397].eqiad.wmnet
• 12:47 akosiaris@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mw[1374-1393].eqiad.wmnet
• 12:47 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:47 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1374-1393].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - akosiaris@cumin1002"
• 12:45 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1374-1393].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - akosiaris@cumin1002"
• 12:42 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 12:25 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1304.eqiad.wmnet
• 12:25 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1304.eqiad.wmnet
• 12:25 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1303.eqiad.wmnet
• 12:25 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1303.eqiad.wmnet
• 12:25 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1302.eqiad.wmnet
• 12:25 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1302.eqiad.wmnet
• 12:25 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1301.eqiad.wmnet
• 12:25 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1301.eqiad.wmnet
• 12:25 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1300.eqiad.wmnet
• 12:25 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1300.eqiad.wmnet
• 12:25 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1299.eqiad.wmnet
• 12:25 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1299.eqiad.wmnet
• 12:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1298.eqiad.wmnet
• 12:24 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1298.eqiad.wmnet
• 12:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1297.eqiad.wmnet
• 12:24 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1297.eqiad.wmnet
• 12:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1296.eqiad.wmnet
• 12:24 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1296.eqiad.wmnet
• 12:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1295.eqiad.wmnet
• 12:24 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1295.eqiad.wmnet
• 12:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1294.eqiad.wmnet
• 12:24 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1294.eqiad.wmnet
• 12:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1293.eqiad.wmnet
• 12:24 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1293.eqiad.wmnet
• 12:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1292.eqiad.wmnet
• 12:24 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1292.eqiad.wmnet
• 12:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1291.eqiad.wmnet
• 12:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1291.eqiad.wmnet
• 12:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1290.eqiad.wmnet
• 12:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1290.eqiad.wmnet
• 12:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1289.eqiad.wmnet
• 12:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1289.eqiad.wmnet
• 12:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1288.eqiad.wmnet
• 12:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1288.eqiad.wmnet
• 12:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1287.eqiad.wmnet
• 12:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1287.eqiad.wmnet
• 12:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1286.eqiad.wmnet
• 12:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1286.eqiad.wmnet
• 12:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1285.eqiad.wmnet
• 12:22 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1285.eqiad.wmnet
• 12:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1284.eqiad.wmnet
• 12:22 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1284.eqiad.wmnet
• 12:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1283.eqiad.wmnet
• 12:22 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1283.eqiad.wmnet
• 12:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1282.eqiad.wmnet
• 12:22 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1282.eqiad.wmnet
• 12:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1281.eqiad.wmnet
• 12:22 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1281.eqiad.wmnet
• 12:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1280.eqiad.wmnet
• 12:22 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1280.eqiad.wmnet
• 12:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1279.eqiad.wmnet
• 12:21 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1279.eqiad.wmnet
• 12:21 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1278.eqiad.wmnet
• 12:21 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1278.eqiad.wmnet
• 12:21 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1277.eqiad.wmnet
• 12:21 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1277.eqiad.wmnet
• 12:21 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1276.eqiad.wmnet
• 12:21 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1276.eqiad.wmnet
• 12:21 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1275.eqiad.wmnet
• 12:21 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1275.eqiad.wmnet
• 12:21 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1274.eqiad.wmnet
• 12:21 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1274.eqiad.wmnet
• 12:21 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1273.eqiad.wmnet
• 12:21 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1273.eqiad.wmnet
• 12:21 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1272.eqiad.wmnet
• 12:21 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1272.eqiad.wmnet
• 12:20 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1271.eqiad.wmnet
• 12:20 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1271.eqiad.wmnet
• 12:20 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1270.eqiad.wmnet
• 12:20 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1270.eqiad.wmnet
• 12:20 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1269.eqiad.wmnet
• 12:20 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1269.eqiad.wmnet
• 12:20 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1268.eqiad.wmnet
• 12:20 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1268.eqiad.wmnet
• 12:20 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1267.eqiad.wmnet
• 12:20 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1267.eqiad.wmnet
• 12:19 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1266.eqiad.wmnet
• 12:19 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1266.eqiad.wmnet
• 12:19 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1265.eqiad.wmnet
• 12:19 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1265.eqiad.wmnet
• 12:19 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1264.eqiad.wmnet
• 12:19 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1264.eqiad.wmnet
• 12:19 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1263.eqiad.wmnet
• 12:19 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1263.eqiad.wmnet
• 12:19 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1262.eqiad.wmnet
• 12:19 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1262.eqiad.wmnet
• 12:19 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet
• 12:19 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet
• 12:19 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1260.eqiad.wmnet
• 12:19 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1260.eqiad.wmnet
• 12:19 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1259.eqiad.wmnet
• 12:18 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1259.eqiad.wmnet
• 12:18 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1258.eqiad.wmnet
• 12:18 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1258.eqiad.wmnet
• 12:18 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1257.eqiad.wmnet
• 12:18 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1257.eqiad.wmnet
• 12:18 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1256.eqiad.wmnet
• 12:18 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1256.eqiad.wmnet
• 12:18 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1255.eqiad.wmnet
• 12:18 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1255.eqiad.wmnet
• 12:18 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1254.eqiad.wmnet
• 12:18 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1254.eqiad.wmnet
• 12:18 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
• 12:18 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
• 12:18 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
• 12:17 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
• 12:17 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet
• 12:17 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet
• 12:17 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
• 12:17 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
• 12:17 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1249.eqiad.wmnet
• 12:17 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1249.eqiad.wmnet
• 12:16 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1248.eqiad.wmnet
• 12:16 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1248.eqiad.wmnet
• 12:16 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1247.eqiad.wmnet
• 12:16 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1247.eqiad.wmnet
• 12:16 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1246.eqiad.wmnet
• 12:16 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1246.eqiad.wmnet
• 12:16 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1245.eqiad.wmnet
• 12:16 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1245.eqiad.wmnet
• 12:16 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1244.eqiad.wmnet
• 12:16 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1244.eqiad.wmnet
• 12:16 akosiaris: T369744 pool wikikube-worker1280-1304. They are now fully in production
• 12:15 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1243.eqiad.wmnet
• 12:15 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1243.eqiad.wmnet
• 12:15 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1242.eqiad.wmnet
• 12:15 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1242.eqiad.wmnet
• 12:15 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1241.eqiad.wmnet
• 12:15 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1241.eqiad.wmnet
• 12:15 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1240.eqiad.wmnet
• 12:15 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1240.eqiad.wmnet
• 12:15 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1304.eqiad.wmnet
• 12:15 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1304.eqiad.wmnet
• 12:15 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1303.eqiad.wmnet
• 12:15 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1303.eqiad.wmnet
• 12:14 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1302.eqiad.wmnet
• 12:14 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1302.eqiad.wmnet
• 12:14 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1301.eqiad.wmnet
• 12:14 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1301.eqiad.wmnet
• 12:14 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1300.eqiad.wmnet
• 12:14 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1300.eqiad.wmnet
• 12:14 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1299.eqiad.wmnet
• 12:14 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1299.eqiad.wmnet
• 12:14 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1298.eqiad.wmnet
• 12:14 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1298.eqiad.wmnet
• 12:14 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1297.eqiad.wmnet
• 12:14 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1297.eqiad.wmnet
• 12:14 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1296.eqiad.wmnet
• 12:14 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1296.eqiad.wmnet
• 12:14 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1295.eqiad.wmnet
• 12:13 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1295.eqiad.wmnet
• 12:13 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1294.eqiad.wmnet
• 12:13 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1294.eqiad.wmnet
• 12:13 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1293.eqiad.wmnet
• 12:13 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1293.eqiad.wmnet
• 12:13 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1292.eqiad.wmnet
• 12:13 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1292.eqiad.wmnet
• 12:13 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1291.eqiad.wmnet
• 12:13 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1291.eqiad.wmnet
• 12:13 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1290.eqiad.wmnet
• 12:13 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1290.eqiad.wmnet
• 12:13 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1289.eqiad.wmnet
• 12:13 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1289.eqiad.wmnet
• 12:13 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1288.eqiad.wmnet
• 12:12 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1288.eqiad.wmnet
• 12:12 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1287.eqiad.wmnet
• 12:12 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1287.eqiad.wmnet
• 12:12 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1286.eqiad.wmnet
• 12:12 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1286.eqiad.wmnet
• 12:12 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1285.eqiad.wmnet
• 12:12 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1285.eqiad.wmnet
• 12:12 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1284.eqiad.wmnet
• 12:12 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1284.eqiad.wmnet
• 12:12 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1283.eqiad.wmnet
• 12:12 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1283.eqiad.wmnet
• 12:12 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1282.eqiad.wmnet
• 12:12 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1282.eqiad.wmnet
• 12:12 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1281.eqiad.wmnet
• 12:11 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1281.eqiad.wmnet
• 12:11 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1280.eqiad.wmnet
• 12:11 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1280.eqiad.wmnet
• 12:11 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1279.eqiad.wmnet
• 12:11 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1279.eqiad.wmnet
• 12:11 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1278.eqiad.wmnet
• 12:11 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1278.eqiad.wmnet
• 12:11 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1277.eqiad.wmnet
• 12:11 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1277.eqiad.wmnet
• 12:11 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1276.eqiad.wmnet
• 12:11 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1276.eqiad.wmnet
• 12:11 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1275.eqiad.wmnet
• 12:11 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1275.eqiad.wmnet
• 12:11 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1274.eqiad.wmnet
• 12:10 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1274.eqiad.wmnet
• 12:10 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1273.eqiad.wmnet
• 12:10 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1273.eqiad.wmnet
• 12:10 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1272.eqiad.wmnet
• 12:10 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1272.eqiad.wmnet
• 12:10 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1271.eqiad.wmnet
• 12:09 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1271.eqiad.wmnet
• 12:09 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1270.eqiad.wmnet
• 12:09 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1270.eqiad.wmnet
• 12:09 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1269.eqiad.wmnet
• 12:09 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1269.eqiad.wmnet
• 12:09 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1268.eqiad.wmnet
• 12:09 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1268.eqiad.wmnet
• 12:09 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1267.eqiad.wmnet
• 12:09 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1267.eqiad.wmnet
• 12:09 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1266.eqiad.wmnet
• 12:09 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1266.eqiad.wmnet
• 12:09 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1265.eqiad.wmnet
• 12:09 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1265.eqiad.wmnet
• 12:08 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1264.eqiad.wmnet
• 12:08 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1264.eqiad.wmnet
• 12:08 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1263.eqiad.wmnet
• 12:08 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1263.eqiad.wmnet
• 12:08 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1262.eqiad.wmnet
• 12:08 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1262.eqiad.wmnet
• 12:08 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1261.eqiad.wmnet
• 12:08 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1261.eqiad.wmnet
• 12:08 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1260.eqiad.wmnet
• 12:08 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1260.eqiad.wmnet
• 12:08 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1259.eqiad.wmnet
• 12:08 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1259.eqiad.wmnet
• 12:08 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1258.eqiad.wmnet
• 12:08 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1258.eqiad.wmnet
• 12:08 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1257.eqiad.wmnet
• 12:07 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1257.eqiad.wmnet
• 12:07 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1256.eqiad.wmnet
• 12:07 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1256.eqiad.wmnet
• 12:07 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1255.eqiad.wmnet
• 12:07 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1255.eqiad.wmnet
• 12:07 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1254.eqiad.wmnet
• 12:07 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1254.eqiad.wmnet
• 12:07 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1253.eqiad.wmnet
• 12:07 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1253.eqiad.wmnet
• 12:07 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1252.eqiad.wmnet
• 12:07 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1252.eqiad.wmnet
• 12:07 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1251.eqiad.wmnet
• 12:07 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1251.eqiad.wmnet
• 12:07 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1250.eqiad.wmnet
• 12:07 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1250.eqiad.wmnet
• 12:06 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1249.eqiad.wmnet
• 12:06 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1249.eqiad.wmnet
• 12:06 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1248.eqiad.wmnet
• 12:06 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1248.eqiad.wmnet
• 12:06 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1247.eqiad.wmnet
• 12:06 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1247.eqiad.wmnet
• 12:06 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1246.eqiad.wmnet
• 12:06 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1246.eqiad.wmnet
• 12:06 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1245.eqiad.wmnet
• 12:06 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1245.eqiad.wmnet
• 12:06 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1244.eqiad.wmnet
• 12:06 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1244.eqiad.wmnet
• 12:06 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1243.eqiad.wmnet
• 12:06 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1243.eqiad.wmnet
• 12:05 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1242.eqiad.wmnet
• 12:05 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1242.eqiad.wmnet
• 12:05 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1241.eqiad.wmnet
• 12:05 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1241.eqiad.wmnet
• 12:05 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker1240.eqiad.wmnet
• 12:05 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker1240.eqiad.wmnet
• 11:23 akosiaris@cumin1002: START - Cookbook sre.hosts.decommission for hosts mw[1374-1393].eqiad.wmnet
• 11:21 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw[1352-1357,1360-1363,1367-1371,1399,1405,1408-1409].eqiad.wmnet
• 11:21 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:21 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1352-1357,1360-1363,1367-1371,1399,1405,1408-1409].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - akosiaris@cumin1002"
• 11:20 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[1352-1357,1360-1363,1367-1371,1399,1405,1408-1409].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - akosiaris@cumin1002"
• 11:15 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 11:06 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
• 10:12 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: sync
• 10:11 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: sync
• 10:03 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:02 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 08:26 akosiaris@cumin1002: START - Cookbook sre.hosts.decommission for hosts mw[1352-1357,1360-1363,1367-1371,1399,1405,1408-1409].eqiad.wmnet
• 08:13 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
• 07:50 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 07:50 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 07:46 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1409.eqiad.wmnet
• 07:45 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1409.eqiad.wmnet
• 07:45 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1408.eqiad.wmnet
• 07:45 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1408.eqiad.wmnet
• 07:44 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1405.eqiad.wmnet
• 07:44 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1405.eqiad.wmnet
• 07:44 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1399.eqiad.wmnet
• 07:43 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1399.eqiad.wmnet
• 07:43 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1397.eqiad.wmnet
• 07:42 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1397.eqiad.wmnet
• 07:42 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1396.eqiad.wmnet
• 07:42 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1396.eqiad.wmnet
• 07:42 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1395.eqiad.wmnet
• 07:41 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1395.eqiad.wmnet
• 07:41 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1394.eqiad.wmnet
• 07:41 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1394.eqiad.wmnet
• 07:40 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1393.eqiad.wmnet
• 07:40 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1393.eqiad.wmnet
• 07:40 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1392.eqiad.wmnet
• 07:39 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1392.eqiad.wmnet
• 07:39 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1391.eqiad.wmnet
• 07:38 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1391.eqiad.wmnet
• 07:38 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1390.eqiad.wmnet
• 07:38 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1390.eqiad.wmnet
• 07:38 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1389.eqiad.wmnet
• 07:37 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1389.eqiad.wmnet
• 07:37 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1388.eqiad.wmnet
• 07:36 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1388.eqiad.wmnet
• 07:36 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1387.eqiad.wmnet
• 07:36 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1387.eqiad.wmnet
• 07:36 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1386.eqiad.wmnet
• 07:35 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1386.eqiad.wmnet
• 07:35 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1385.eqiad.wmnet
• 07:34 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1385.eqiad.wmnet
• 07:34 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1384.eqiad.wmnet
• 07:34 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1384.eqiad.wmnet
• 07:34 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1383.eqiad.wmnet
• 07:30 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1383.eqiad.wmnet
• 07:30 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1382.eqiad.wmnet
• 07:30 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1382.eqiad.wmnet
• 07:30 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1381.eqiad.wmnet
• 07:29 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1381.eqiad.wmnet
• 07:29 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1380.eqiad.wmnet
• 07:28 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1380.eqiad.wmnet
• 07:28 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1379.eqiad.wmnet
• 07:28 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1379.eqiad.wmnet
• 07:28 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1378.eqiad.wmnet
• 07:27 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1378.eqiad.wmnet
• 07:27 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1377.eqiad.wmnet
• 07:26 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1377.eqiad.wmnet
• 07:26 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1376.eqiad.wmnet
• 07:26 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1376.eqiad.wmnet
• 07:26 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1375.eqiad.wmnet
• 07:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1375.eqiad.wmnet
• 07:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1374.eqiad.wmnet
• 07:22 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1374.eqiad.wmnet
• 07:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1371.eqiad.wmnet
• 07:21 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1371.eqiad.wmnet
• 07:21 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1370.eqiad.wmnet
• 07:21 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1370.eqiad.wmnet
• 07:20 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1369.eqiad.wmnet
• 07:20 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1369.eqiad.wmnet
• 07:20 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1368.eqiad.wmnet
• 07:19 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1368.eqiad.wmnet
• 07:19 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1367.eqiad.wmnet
• 07:19 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1367.eqiad.wmnet
• 07:18 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1363.eqiad.wmnet
• 07:18 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1363.eqiad.wmnet
• 07:18 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1362.eqiad.wmnet
• 07:17 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1362.eqiad.wmnet
• 07:17 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1361.eqiad.wmnet
• 07:16 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1361.eqiad.wmnet
• 07:16 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1360.eqiad.wmnet
• 07:16 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1360.eqiad.wmnet
• 07:16 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1357.eqiad.wmnet
• 07:15 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1357.eqiad.wmnet
• 07:15 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1356.eqiad.wmnet
• 07:14 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1356.eqiad.wmnet
• 07:14 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1355.eqiad.wmnet
• 07:14 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1355.eqiad.wmnet
• 07:14 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1354.eqiad.wmnet
• 07:13 akosiaris: T375842 depooling mw1386.eqiad.wmnet mw1387.eqiad.wmnet mw1388.eqiad.wmnet mw1389.eqiad.wmnet mw1390.eqiad.wmnet mw1391.eqiad.wmnet mw1392.eqiad.wmnet mw1393.eqiad.wmnet mw1394.eqiad.wmnet mw1395.eqiad.wmnet mw1396.eqiad.wmnet mw1397.eqiad.wmnet mw1399.eqiad.wmnet mw1405.eqiad.wmnet mw1408.eqiad.wmnet mw1409.eqiad.wmnet T
• 07:13 akosiaris: T375842 depooling mw1371.eqiad.wmnet mw1374.eqiad.wmnet mw1375.eqiad.wmnet mw1376.eqiad.wmnet mw1377.eqiad.wmnet mw1378.eqiad.wmnet mw1379.eqiad.wmnet mw1380.eqiad.wmnet mw1381.eqiad.wmnet mw1382.eqiad.wmnet mw1383.eqiad.wmnet mw1384.eqiad.wmnet mw1385.eqiad.wmnet
• 07:13 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1354.eqiad.wmnet
• 07:13 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1353.eqiad.wmnet
• 07:13 akosiaris: T375842 depooling mw1352.eqiad.wmnet mw1353.eqiad.wmnet mw1354.eqiad.wmnet mw1355.eqiad.wmnet mw1356.eqiad.wmnet mw1357.eqiad.wmnet mw1360.eqiad.wmnet mw1361.eqiad.wmnet mw1362.eqiad.wmnet mw1363.eqiad.wmnet mw1367.eqiad.wmnet mw1368.eqiad.wmnet mw1369.eqiad.wmnet mw1370.eqiad.wmnet
• 07:12 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1353.eqiad.wmnet
• 07:12 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw1352.eqiad.wmnet
• 07:12 akosiaris: depooling mw1352.eqiad.wmnet mw1353.eqiad.wmnet mw1354.eqiad.wmnet mw1355.eqiad.wmnet mw1356.eqiad.wmnet mw1357.eqiad.wmnet mw1360.eqiad.wmnet mw1361.eqiad.wmnet mw1362.eqiad.wmnet mw1363.eqiad.wmnet mw1367.eqiad.wmnet mw1368.eqiad.wmnet mw1369.eqiad.wmnet mw1370.eqiad.wmnet mw1371.eqiad.wmnet mw1374.eqiad.wmnet mw1375.eqiad.wmnet
• 07:12 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw1352.eqiad.wmnet
• 06:51 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1304.eqiad.wmnet
• 06:51 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1304.eqiad.wmnet
• 06:51 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1303.eqiad.wmnet
• 06:51 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1303.eqiad.wmnet
• 06:51 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1301.eqiad.wmnet
• 06:51 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1301.eqiad.wmnet
• 06:51 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1296.eqiad.wmnet
• 06:51 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1296.eqiad.wmnet
• 06:51 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1294.eqiad.wmnet
• 06:50 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1294.eqiad.wmnet
• 06:50 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1290.eqiad.wmnet
• 06:50 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1290.eqiad.wmnet
• 06:50 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1289.eqiad.wmnet
• 06:50 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1289.eqiad.wmnet
• 06:50 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1286.eqiad.wmnet
• 06:50 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1286.eqiad.wmnet
• 06:50 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1284.eqiad.wmnet
• 06:50 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1284.eqiad.wmnet
• 06:50 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1282.eqiad.wmnet
• 06:50 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1282.eqiad.wmnet
• 06:50 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1281.eqiad.wmnet
• 06:50 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1281.eqiad.wmnet
• 06:50 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1279.eqiad.wmnet
• 06:50 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1279.eqiad.wmnet
• 06:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1278.eqiad.wmnet
• 06:49 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1278.eqiad.wmnet
• 06:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1274.eqiad.wmnet
• 06:49 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1274.eqiad.wmnet
• 06:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1273.eqiad.wmnet
• 06:49 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1273.eqiad.wmnet
• 06:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1271.eqiad.wmnet
• 06:49 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1271.eqiad.wmnet
• 06:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1270.eqiad.wmnet
• 06:49 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1270.eqiad.wmnet
• 06:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1265.eqiad.wmnet
• 06:49 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1265.eqiad.wmnet
• 06:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1264.eqiad.wmnet
• 06:49 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1264.eqiad.wmnet
• 06:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1262.eqiad.wmnet
• 06:48 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1262.eqiad.wmnet
• 06:48 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1260.eqiad.wmnet
• 06:48 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1260.eqiad.wmnet
• 06:48 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1259.eqiad.wmnet
• 06:48 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1259.eqiad.wmnet
• 06:48 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1258.eqiad.wmnet
• 06:48 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1258.eqiad.wmnet
• 06:48 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1257.eqiad.wmnet
• 06:48 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1257.eqiad.wmnet
• 06:48 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1254.eqiad.wmnet
• 06:48 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1254.eqiad.wmnet
• 06:48 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
• 06:48 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
• 06:48 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
• 06:48 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
• 06:47 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
• 06:47 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
• 06:47 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1249.eqiad.wmnet
• 06:47 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1249.eqiad.wmnet
• 06:44 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1247.eqiad.wmnet
• 06:44 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1247.eqiad.wmnet
• 06:44 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1245.eqiad.wmnet
• 06:44 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1245.eqiad.wmnet
• 06:43 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1247.eqiad.wmnet
• 06:43 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1247.eqiad.wmnet
• 06:43 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1246.eqiad.wmnet
• 06:43 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1246.eqiad.wmnet
• 06:43 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1245.eqiad.wmnet
• 06:43 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1245.eqiad.wmnet
• 06:43 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1244.eqiad.wmnet
• 06:43 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1244.eqiad.wmnet
• 06:43 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1243.eqiad.wmnet
• 06:43 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1243.eqiad.wmnet
• 06:43 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1242.eqiad.wmnet
• 06:43 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1242.eqiad.wmnet
• 06:43 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1241.eqiad.wmnet
• 06:42 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1241.eqiad.wmnet
• 06:42 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1240.eqiad.wmnet
• 06:42 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1240.eqiad.wmnet
• 03:26 vgutierrez: powercycle puppetserver1001 - T375839

2024-09-26

• 23:41 swfrench-wmf: silenced KubernetesAPILatency for 24h for resource=~"(blockaffinities|ipamblocks)" site="eqiad" (f33e2c42-d921-4686-a761-59d43ad14d84) - T369744
• 22:33 dduvall@deploy2002: Installation of scap version "4.107.0" completed for 212 hosts
• 22:29 dduvall@deploy2002: Installing scap version "4.107.0" for 212 hosts
• 21:19 milimetric@deploy2002: Finished deploy [airflow-dags/analytics@f6ea258]: Deploying to get fix for datahub ingestion (duration: 01m 08s)
• 21:18 milimetric@deploy2002: Started deploy [airflow-dags/analytics@f6ea258]: Deploying to get fix for datahub ingestion
• 21:07 dduvall@deploy2002: Installation of scap version "4.106.0" completed for 212 hosts
• 21:03 dduvall@deploy2002: Installing scap version "4.106.0" for 212 hosts
• 20:55 thcipriani@deploy2002: Finished scap sync-world: Backport for Change votewiki language to Ukrainian. (T302443) (duration: 12m 42s)
• 20:50 thcipriani@deploy2002: thcipriani, ahonc: Continuing with sync
• 20:44 thcipriani@deploy2002: thcipriani, ahonc: Backport for Change votewiki language to Ukrainian. (T302443) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:42 thcipriani@deploy2002: Started scap sync-world: Backport for Change votewiki language to Ukrainian. (T302443)
• 20:36 thcipriani@deploy2002: Finished scap sync-world: Backport for REST: Adjust REST Sandbox spec for new specs module (T375512) (duration: 09m 25s)
• 20:30 thcipriani@deploy2002: bpirkle, thcipriani: Continuing with sync
• 20:28 thcipriani@deploy2002: bpirkle, thcipriani: Backport for REST: Adjust REST Sandbox spec for new specs module (T375512) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:26 thcipriani@deploy2002: Started scap sync-world: Backport for REST: Adjust REST Sandbox spec for new specs module (T375512)
• 20:23 thcipriani@deploy2002: Finished scap sync-world: Backport for Bump coverage of the add-a-fact quicksurvey to 0.2 (duration: 13m 53s)
• 20:23 ebernhardson@deploy2002: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:22 ebernhardson@deploy2002: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:16 thcipriani@deploy2002: thcipriani, derenrich: Continuing with sync
• 20:13 thcipriani@deploy2002: thcipriani, derenrich: Backport for Bump coverage of the add-a-fact quicksurvey to 0.2 synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:11 ebernhardson@deploy2002: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:11 ebernhardson@deploy2002: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:09 thcipriani@deploy2002: Started scap sync-world: Backport for Bump coverage of the add-a-fact quicksurvey to 0.2
• 19:30 brennen@deploy2002: rebuilt and synchronized wikiversions files: group2 to 1.43.0-wmf.24 refs T373643
• 19:04 brennen@deploy2002: Sync cancelled.
• 19:04 brennen@deploy2002: brennen, jdlrobson: Backport for Limit a heading selector to Parsoid HTML (T375701) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 18:32 brennen@deploy2002: Started scap sync-world: Backport for Limit a heading selector to Parsoid HTML (T375701)
• 18:23 sukhe: repooling cp2037; downtimed removed for some time, looks good to repool: T375766
• 18:20 ebernhardson@deploy2002: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 18:20 ebernhardson@deploy2002: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 18:07 VRiley: rebooting msw-e1-eqiad for maintenance
• 17:40 ebernhardson@deploy2002: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:40 ebernhardson@deploy2002: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:36 ebernhardson@deploy2002: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:36 ebernhardson@deploy2002: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:14 bd808@deploy2002: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
• 17:14 bd808@deploy2002: helmfile [codfw] START helmfile.d/services/developer-portal: apply
• 17:14 bd808@deploy2002: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
• 17:13 bd808@deploy2002: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
• 17:13 bd808@deploy2002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
• 17:13 bd808@deploy2002: helmfile [staging] START helmfile.d/services/developer-portal: apply
• 17:12 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host snapshot1017.eqiad.wmnet
• 17:09 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: sync
• 17:09 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: sync
• 17:08 swfrench-wmf: cleared contents of a stale scap dsh group resource on deploy2002 - T370962
• 17:06 btullis@cumin1002: START - Cookbook sre.hosts.reboot-single for host snapshot1017.eqiad.wmnet
• 17:03 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2037.codfw.wmnet
• 17:03 sukhe@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp2037.codfw.wmnet
• 17:03 sukhe: removing downtime on cp2037 but still keeping it depooled: T375766
• 16:34 fnegri@cumin1002: END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=99)
• 16:32 fnegri@cumin1002: START - Cookbook sre.wikireplicas.update-views
• 16:31 jiawang@deploy2002: Finished deploy [airflow-dags/analytics_product@8f6a9ed]: (no justification provided) (duration: 01m 13s)
• 16:30 jiawang@deploy2002: Started deploy [airflow-dags/analytics_product@8f6a9ed]: (no justification provided)
• 16:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 16:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 16:27 swfrench-wmf: done with switchover day 3 deployment server switch - T370962
• 16:23 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp2037.codfw.wmnet with reason: T375766
• 16:23 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on cp2037.codfw.wmnet with reason: T375766
• 16:09 cwhite: `systemctl restart zuul` on contint1002 T233089
• 16:07 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 16:07 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
• 16:07 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 16:07 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 16:07 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
• 16:06 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
• 16:06 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
• 16:06 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
• 16:06 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
• 16:06 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
• 16:06 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 16:06 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 16:06 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 16:05 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 16:05 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 16:05 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
• 16:05 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 16:05 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 16:04 swfrench@deploy2002: Finished scap sync-world: No-op deployment to verify switchover - T370962 (duration: 29m 53s)
• 16:03 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 16:03 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 16:01 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 16:00 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 16:00 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 15:59 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 15:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:55 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 15:54 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 15:54 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 15:53 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 15:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:42 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 15:42 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 15:39 fnegri@cumin1002: END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=99)
• 15:38 fnegri@cumin1002: START - Cookbook sre.wikireplicas.update-views
• 15:34 swfrench@deploy2002: Started scap sync-world: No-op deployment to verify switchover - T370962
• 15:31 fnegri@cumin1002: END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=99)
• 15:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:22 fnegri@cumin1002: START - Cookbook sre.wikireplicas.update-views
• 15:22 jynus@cumin1002: dbctl commit (dc=all): 'repool db1206', diff saved to https://phabricator.wikimedia.org/P69426 and previous config saved to /var/cache/conftool/dbconfig/20240926-152204-jynus.json
• 15:17 jynus@cumin1002: dbctl commit (dc=all): 'depool db1206', diff saved to https://phabricator.wikimedia.org/P69425 and previous config saved to /var/cache/conftool/dbconfig/20240926-151716-jynus.json
• 15:04 swfrench-wmf: running authdns-update for deployment CNAME switch - T370962
• 15:01 swfrench-wmf: starting switchover day 3 deployment server switch - T370962
• 14:48 sukhe: depool cp2037.codfw.wmnet
• 14:41 fnegri@cumin1002: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0)
• 14:32 fnegri@cumin1002: START - Cookbook sre.wikireplicas.update-views
• 14:29 sukhe: sudo cumin 'O:postfix::mx_in' 'run-puppet-agent'
• 14:21 elukey@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host aux-k8s-worker1003.eqiad.wmnet
• 14:20 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-worker1003.eqiad.wmnet with OS bookworm
• 14:13 fnegri@cumin1002: END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=99)
• 14:11 fnegri@cumin1002: START - Cookbook sre.wikireplicas.update-views
• 14:06 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host cloudcephosd1004.eqiad.wmnet
• 14:03 fnegri@cumin1002: END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=99)
• 14:03 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-worker1003.eqiad.wmnet with reason: host reimage
• 14:02 kartik@deploy1003: Finished scap sync-world: Backport for Load styles for legacy message box markup (T375696) (duration: 24m 31s)
• 13:59 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudcephosd1004.eqiad.wmnet
• 13:57 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-worker1003.eqiad.wmnet with reason: host reimage
• 13:49 fnegri@cumin1002: START - Cookbook sre.wikireplicas.update-views
• 13:45 fnegri@cumin1002: END (FAIL) - Cookbook sre.wikireplicas.update-views (exit_code=99)
• 13:44 kartik@deploy1003: kartik, abi: Continuing with sync
• 13:42 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host aux-k8s-worker1003.eqiad.wmnet with OS bookworm
• 13:40 kartik@deploy1003: kartik, abi: Backport for Load styles for legacy message box markup (T375696) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:39 fnegri@cumin1002: START - Cookbook sre.wikireplicas.update-views
• 13:38 kartik@deploy1003: Started scap sync-world: Backport for Load styles for legacy message box markup (T375696)
• 13:37 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM aux-k8s-worker1003.eqiad.wmnet - elukey@cumin1002"
• 13:36 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM aux-k8s-worker1003.eqiad.wmnet - elukey@cumin1002"
• 13:36 elukey@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aux-k8s-worker1003.eqiad.wmnet on all recursors
• 13:36 elukey@cumin1002: START - Cookbook sre.dns.wipe-cache aux-k8s-worker1003.eqiad.wmnet on all recursors
• 13:36 elukey@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:36 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-worker1003.eqiad.wmnet - elukey@cumin1002"
• 13:36 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-worker1003.eqiad.wmnet - elukey@cumin1002"
• 13:36 kartik@deploy1003: Finished scap sync-world: Backport for Deploy Parsoid Read Views to shn/fr/el/vi/it wikivoyage (T374861) (duration: 25m 17s)
• 13:33 elukey@cumin1002: START - Cookbook sre.dns.netbox
• 13:32 elukey@cumin1002: START - Cookbook sre.ganeti.makevm for new host aux-k8s-worker1003.eqiad.wmnet
• 13:31 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host cloudcephosd1003.eqiad.wmnet
• 13:26 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host logging-hd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:23 elukey@cumin1002: START - Cookbook sre.hosts.provision for host logging-hd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:21 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host logging-hd2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:18 kartik@deploy1003: kartik, cscott: Continuing with sync
• 13:17 elukey@cumin1002: START - Cookbook sre.hosts.provision for host logging-hd2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:14 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudcephosd1003.eqiad.wmnet
• 13:13 kartik@deploy1003: kartik, cscott: Backport for Deploy Parsoid Read Views to shn/fr/el/vi/it wikivoyage (T374861) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:11 dcaro@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1041.eqiad.wmnet
• 13:10 kartik@deploy1003: Started scap sync-world: Backport for Deploy Parsoid Read Views to shn/fr/el/vi/it wikivoyage (T374861)
• 13:07 moritzm: installing QT security updates
• 13:05 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host cloudcephosd1002.eqiad.wmnet
• 13:04 dcaro@cumin1002: START - Cookbook sre.hosts.reboot-single for host cloudcephosd1041.eqiad.wmnet
• 13:04 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts vrts1001.eqiad.wmnet
• 13:03 aokoth@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:03 aokoth@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: vrts1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - aokoth@cumin1002"
• 12:58 aokoth@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: vrts1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - aokoth@cumin1002"
• 12:54 aokoth@cumin1002: START - Cookbook sre.dns.netbox
• 12:47 aokoth@cumin1002: START - Cookbook sre.hosts.decommission for hosts vrts1001.eqiad.wmnet
• 12:46 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on vrts1001.eqiad.wmnet with reason: Decom
• 12:46 aokoth@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on vrts1001.eqiad.wmnet with reason: Decom
• 12:37 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudcephosd1002.eqiad.wmnet
• 12:32 moritzm: installing glib2.0 bugfix updates from Bookworm point release
• 12:32 kart_: Updated cxserver to 2024-09-18-104433-production (T375017, T374815, T374644)
• 12:30 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
• 12:29 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply
• 12:28 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
• 12:27 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
• 12:27 jynus@cumin1002: dbctl commit (dc=all): 's8 weight tuning T375732', diff saved to https://phabricator.wikimedia.org/P69420 and previous config saved to /var/cache/conftool/dbconfig/20240926-122715-jynus.json
• 12:26 dcaro@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1040.eqiad.wmnet
• 12:22 jynus@cumin1002: dbctl commit (dc=all): 's8 weight tuning T375732', diff saved to https://phabricator.wikimedia.org/P69419 and previous config saved to /var/cache/conftool/dbconfig/20240926-122237-jynus.json
• 12:21 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
• 12:21 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply
• 12:18 dcaro@cumin1002: START - Cookbook sre.hosts.reboot-single for host cloudcephosd1040.eqiad.wmnet
• 12:11 jynus@cumin1002: dbctl commit (dc=all): 's4 weight tuning T375732', diff saved to https://phabricator.wikimedia.org/P69418 and previous config saved to /var/cache/conftool/dbconfig/20240926-121105-jynus.json
• 12:06 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
• 12:06 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/zotero: apply
• 12:02 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
• 12:01 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply
• 12:00 jynus@cumin1002: dbctl commit (dc=all): 's4 weight tuning T375732', diff saved to https://phabricator.wikimedia.org/P69417 and previous config saved to /var/cache/conftool/dbconfig/20240926-120013-jynus.json
• 12:00 akosiaris@deploy1003: helmfile [staging] DONE helmfile.d/services/zotero: apply
• 11:59 akosiaris@deploy1003: helmfile [staging] START helmfile.d/services/zotero: apply
• 11:52 jynus@cumin1002: dbctl commit (dc=all): 's4 weight tuning T375732', diff saved to https://phabricator.wikimedia.org/P69416 and previous config saved to /var/cache/conftool/dbconfig/20240926-115208-jynus.json
• 11:09 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/zotero: apply
• 11:09 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/zotero: apply
• 10:58 moritzm: prune now obsolete nginx packages from docker-registry hosts T329529
• 10:33 dcaro@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1041.eqiad.wmnet with OS bullseye
• 10:21 elukey: start dry run of docker distribution GC on registry1004 (info in https://phabricator.wikimedia.org/T375645#10176397, you can find a root tmux session named as the task on the host to stop)
• 10:17 dcaro@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1038.eqiad.wmnet
• 10:15 dcaro@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1041.eqiad.wmnet with reason: host reimage
• 10:11 dcaro@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1041.eqiad.wmnet with reason: host reimage
• 10:09 dcaro@cumin1002: START - Cookbook sre.hosts.reboot-single for host cloudcephosd1038.eqiad.wmnet
• 09:52 dcaro@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1041.eqiad.wmnet with OS bullseye
• 09:51 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry2003.codfw.wmnet
• 09:51 elukey@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:51 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: registry2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1002"
• 09:50 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: registry2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1002"
• 09:47 elukey@cumin1002: START - Cookbook sre.dns.netbox
• 09:43 elukey@cumin1002: START - Cookbook sre.hosts.decommission for hosts registry2003.codfw.wmnet
• 09:41 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry1003.eqiad.wmnet
• 09:41 elukey@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:41 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: registry1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1002"
• 09:40 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: registry1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1002"
• 09:40 dcaro@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1040.eqiad.wmnet with OS bullseye
• 09:37 elukey@cumin1002: START - Cookbook sre.dns.netbox
• 09:33 elukey@cumin1002: START - Cookbook sre.hosts.decommission for hosts registry1003.eqiad.wmnet
• 09:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:30 elukey@puppetserver1001: conftool action : set/pooled=inactive; selector: name=registry2003.codfw.wmnet,service=docker-registry,dc=eqiad
• 09:30 elukey@puppetserver1001: conftool action : set/pooled=inactive; selector: name=registry1003.eqiad.wmnet,service=docker-registry,dc=eqiad
• 09:27 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2004.codfw.wmnet with OS bookworm
• 09:21 moritzm: installing exim security updates
• 09:21 dcaro@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1040.eqiad.wmnet with reason: host reimage
• 09:19 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host cloudcephosd1030.eqiad.wmnet
• 09:17 dcaro@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1040.eqiad.wmnet with reason: host reimage
• 09:11 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage
• 09:11 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudcephosd1030.eqiad.wmnet
• 09:08 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage
• 08:59 dcaro@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1040.eqiad.wmnet with OS bullseye
• 08:58 dcaro@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1040.eqiad.wmnet with OS bullseye
• 08:52 dcaro@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1037.eqiad.wmnet
• 08:52 akosiaris@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 08:52 akosiaris@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
• 08:52 akosiaris@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 08:52 akosiaris@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 08:52 akosiaris@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
• 08:51 akosiaris@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
• 08:51 akosiaris@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
• 08:51 akosiaris@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
• 08:51 akosiaris@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
• 08:50 akosiaris@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
• 08:50 akosiaris@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 08:50 akosiaris@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 08:50 akosiaris@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 08:50 akosiaris@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 08:49 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host registry2004.codfw.wmnet with OS bookworm
• 08:48 akosiaris: deploy calico on all clusters to pick up the configuration for the lsw1-{e,f}{6,7} leaf switches. It is a noop in some clusters, but the config should be in-sync anyways
• 08:47 akosiaris@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 08:47 akosiaris@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
• 08:46 akosiaris@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 08:46 akosiaris@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 08:44 dcaro@cumin1002: START - Cookbook sre.hosts.reboot-single for host cloudcephosd1037.eqiad.wmnet
• 08:44 dcaro@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1036.eqiad.wmnet
• 08:38 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host cloudcephosd1031.eqiad.wmnet
• 08:35 dcaro@cumin1002: START - Cookbook sre.hosts.reboot-single for host cloudcephosd1036.eqiad.wmnet
• 08:27 dcaro@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1035.eqiad.wmnet
• 08:25 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudcephosd1031.eqiad.wmnet
• 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host cloudcephosd1032.eqiad.wmnet
• 08:20 dcaro@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1040.eqiad.wmnet with reason: host reimage
• 08:19 dcaro@cumin1002: START - Cookbook sre.hosts.reboot-single for host cloudcephosd1035.eqiad.wmnet
• 08:16 dcaro@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1040.eqiad.wmnet with reason: host reimage
• 08:10 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry1004.eqiad.wmnet with OS bookworm
• 08:08 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudcephosd1032.eqiad.wmnet
• 08:06 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host cloudcephosd1033.eqiad.wmnet
• 07:58 dcaro@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1040.eqiad.wmnet with OS bullseye
• 07:54 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudcephosd1033.eqiad.wmnet
• 07:50 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host cloudcephosd1034.eqiad.wmnet
• 07:47 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudcephosd1034.eqiad.wmnet
• 07:43 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
• 07:41 dcaro@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Run failed when reimaging cloudcephosd1039 and asked to run manually - dcaro@cumin1002 - T372814"
• 07:40 dcaro@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Run failed when reimaging cloudcephosd1039 and asked to run manually - dcaro@cumin1002 - T372814"
• 07:37 kartik@deploy1003: Finished scap sync-world: Backport for Revert^2 "Enable message group subscription feature for Test Wikipedia" (T372386) (duration: 13m 37s)
• 07:37 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on registry1004.eqiad.wmnet with reason: host reimage
• 07:36 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on registry1004.eqiad.wmnet with reason: host reimage
• 07:35 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
• 07:35 jmm@cumin2002: END (FAIL) - Cookbook sre.puppet.migrate-host (exit_code=99) for host cloudcephosd1034.eqiad.wmnet
• 07:34 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica to new version
• 07:32 kartik@deploy1003: kartik, abi: Continuing with sync
• 07:26 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica to new version
• 07:26 kartik@deploy1003: kartik, abi: Backport for Revert^2 "Enable message group subscription feature for Test Wikipedia" (T372386) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:24 kartik@deploy1003: Started scap sync-world: Backport for Revert^2 "Enable message group subscription feature for Test Wikipedia" (T372386)
• 07:20 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host registry1004.eqiad.wmnet with OS bookworm
• 07:19 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudcephosd1034.eqiad.wmnet
• 07:18 kartik@deploy1003: Finished scap sync-world: Backport for Translate: Add VirtualDomainsMapping (T372287) (duration: 13m 25s)
• 07:13 kartik@deploy1003: abi, kartik: Continuing with sync
• 07:07 kartik@deploy1003: abi, kartik: Backport for Translate: Add VirtualDomainsMapping (T372287) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:05 kartik@deploy1003: Started scap sync-world: Backport for Translate: Add VirtualDomainsMapping (T372287)
• 07:00 elukey: apt-get clean on grafana2001 to free some space in the root partition
• 00:41 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site eqiad [reason: repooling due to repeated port utilization alerts, T370962]
• 00:41 sukhe@cumin1002: START - Cookbook sre.dns.admin DNS admin: pool site eqiad [reason: repooling due to repeated port utilization alerts, T370962]

2024-09-25

• 21:26 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 21:25 dcaro@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1039.eqiad.wmnet with OS bullseye
• 21:25 dcaro@cumin1002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - dcaro@cumin1002"
• 21:25 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 21:25 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 21:25 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 21:24 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 21:23 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 21:19 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 21:18 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 21:18 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 21:17 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 21:16 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 21:16 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 21:12 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 21:11 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 21:11 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 21:10 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 21:09 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 21:08 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 18:34 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts vrts2001.codfw.wmnet
• 18:34 aokoth@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:34 aokoth@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: vrts2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - aokoth@cumin1002"
• 18:14 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.43.0-wmf.24 refs T373643
• 18:06 ejegg: fundraising civicrm upgraded from 8a5ab89b to cf27c789
• 18:05 brennen: 1.43.0-wmf.24 train status: no current blockers, logs clean-ish, rolling to group1
• 18:04 aokoth@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: vrts2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - aokoth@cumin1002"
• 18:02 moritzm: installing gtk+3.0 security updates
• 18:01 aokoth@cumin1002: START - Cookbook sre.dns.netbox
• 17:56 aokoth@cumin1002: START - Cookbook sre.hosts.decommission for hosts vrts2001.codfw.wmnet
• 17:55 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on vrts2001.codfw.wmnet with reason: Decom
• 17:55 aokoth@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on vrts2001.codfw.wmnet with reason: Decom
• 17:35 dcaro@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - dcaro@cumin1002"
• 17:17 dcaro@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1039.eqiad.wmnet with reason: host reimage
• 17:13 dcaro@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1039.eqiad.wmnet with reason: host reimage
• 16:55 dcaro@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1039.eqiad.wmnet with OS bullseye
• 16:47 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cr3-ulsfo with reason: router broken ☠️
• 16:47 ayounsi@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on cr3-ulsfo with reason: router broken ☠️
• 16:10 vgutierrez@cumin1002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site ulsfo [reason: cr3-ulsfo issues, T375345]
• 16:10 vgutierrez@cumin1002: START - Cookbook sre.dns.admin DNS admin: depool site ulsfo [reason: cr3-ulsfo issues, T375345]
• 15:57 swfrench-wmf: switchover day 2 manual steps are complete - T370962
• 15:56 swfrench@deploy1003: Finished scap sync-world: Backport for debug.json: order codfw (primary) DC backends first (T370962) (duration: 08m 05s)
• 15:51 swfrench@deploy1003: swfrench: Continuing with sync
• 15:50 swfrench@deploy1003: swfrench: Backport for debug.json: order codfw (primary) DC backends first (T370962) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 15:49 jayme: imported nerdctl 1.7.7-3 to bookworm-wikimedia - T362408
• 15:48 swfrench@deploy1003: Started scap sync-world: Backport for debug.json: order codfw (primary) DC backends first (T370962)
• 15:24 swfrench@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover - T370962 (duration: 58m 14s)
• 15:23 cwhite: reinstall phatality on logstash1023 and repool logstash1023 and logstash1032 T374880
• 15:21 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters (exit_code=0) for datacenter switchover from eqiad to codfw
• 15:21 Dreamy_Jazz: Running `foreachwikiindblist group2.dblist extensions/CheckUser/maintenance/populateCentralCheckUserIndexTables.php` on a tmux session for T375203
• 15:12 Dreamy_Jazz: Started MediaModeration script after datacenter switchover - https://wikitech.wikimedia.org/wiki/MediaModeration
• 15:12 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from eqiad to codfw
• 15:09 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.09-restore-ttl (exit_code=0) for datacenter switchover from eqiad to codfw
• 15:08 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from eqiad to codfw
• 15:07 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) for datacenter switchover from eqiad to codfw
• 15:05 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from eqiad to codfw
• 15:04 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner (exit_code=0) for datacenter switchover from eqiad to codfw
• 15:04 root@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: sync
• 15:03 root@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: sync
• 15:03 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from eqiad to codfw
• 15:01 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) for datacenter switchover from eqiad to codfw
• 15:01 swfrench@cumin1002: MediaWiki read-only period ends at: 2024-09-25 15:01:28.078892
• 14:58 swfrench@cumin1002: MediaWiki read-only period starts at: 2024-09-25 14:58:42.440378
• 14:58 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from eqiad to codfw
• 14:54 swfrench-wmf: reset failed units on mwmaint1002 to unblock 01-stop-maintenance - T370962
• 14:52 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) for datacenter switchover from eqiad to codfw
• 14:52 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to codfw
• 14:46 swfrench@cumin1002: END (FAIL) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=99) for datacenter switchover from eqiad to codfw
• 14:45 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to codfw
• 14:41 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) for datacenter switchover from eqiad to codfw
• 14:35 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from eqiad to codfw
• 14:34 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks (exit_code=0) for datacenter switchover from eqiad to codfw
• 14:34 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from eqiad to codfw
• 14:33 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) for datacenter switchover from eqiad to codfw
• 14:33 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.00-disable-puppet for datacenter switchover from eqiad to codfw
• 14:26 swfrench@deploy1003: Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover - T370962
• 14:24 jynus: run delete on cawiki (s7) db2181 (row format) T375507
• 14:23 kartik@deploy1003: Finished scap sync-world: Backport for CX3 Build 0.2.0+20240925 (T374387 T370746 T368422 T374567 T355780 T374559 T374886 T375410) (duration: 14m 06s)
• 14:18 kartik@deploy1003: kartik, sbisson: Continuing with sync
• 14:11 kartik@deploy1003: kartik, sbisson: Backport for CX3 Build 0.2.0+20240925 (T374387 T370746 T368422 T374567 T355780 T374559 T374886 T375410) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:09 kartik@deploy1003: Started scap sync-world: Backport for CX3 Build 0.2.0+20240925 (T374387 T370746 T368422 T374567 T355780 T374559 T374886 T375410)
• 13:50 swfrench-wmf: kartotherian repooled in eqiad due load issues - T370962
• 13:47 _joe_: repooling karthoterian in eqiad, a further roll restart in codfw
• 13:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 13:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 13:31 _joe_: rolling restart of kartotherian in codfw
• 13:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 13:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 13:28 _joe_: restart karthoterian on maps2005
• 13:27 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 13:26 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rdf-streaming-updater: apply
• 13:26 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/rdf-streaming-updater: apply
• 13:23 dcausse@deploy1003: helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply
• 13:23 dcausse@deploy1003: helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply
• 13:19 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 13:19 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 13:15 sgimeno@deploy1003: Finished scap sync-world: Backport for Add wgCommunityConfigurationCommonsApiURL (T369069) (duration: 08m 06s)
• 13:10 sgimeno@deploy1003: sgimeno: Continuing with sync
• 13:09 sgimeno@deploy1003: sgimeno: Backport for Add wgCommunityConfigurationCommonsApiURL (T369069) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:06 sgimeno@deploy1003: Started scap sync-world: Backport for Add wgCommunityConfigurationCommonsApiURL (T369069)
• 12:38 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 12:38 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 12:37 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 12:37 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 12:35 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host cloudcephosd1035.eqiad.wmnet
• 12:29 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
• 12:29 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
• 12:26 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
• 12:24 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudcephosd1035.eqiad.wmnet
• 12:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
• 12:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
• 12:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 12:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 11:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich-next: apply
• 11:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich-next: apply
• 11:55 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply
• 11:55 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply
• 11:51 jynus: running REPLACE into cawiki db2218 (s7) T375507
• 11:51 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 11:51 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 11:50 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 11:50 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 11:50 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 11:50 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
• 11:50 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 11:40 jynus: running DELETE + REPLACE on metawiki db2218 (s7) T375186
• 11:36 effie: updating mw-mcrouter
• 11:36 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
• 11:35 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
• 11:35 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
• 11:30 jynus: running DELETE + REPLACE on kowiki db2218 (s7) T375186
• 11:22 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich-next: apply
• 11:22 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich-next: apply
• 11:20 akosiaris@cumin1002: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool zotero in eqiad: maintenance
• 11:17 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich-next: apply
• 11:17 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich-next: apply
• 11:15 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) zotero.discovery.wmnet on all recursors
• 11:15 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache zotero.discovery.wmnet on all recursors
• 11:15 akosiaris@cumin1002: START - Cookbook sre.discovery.service-route depool zotero in eqiad: maintenance
• 11:15 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich-next: apply
• 11:15 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich-next: apply
• 11:13 akosiaris@cumin1002: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool zotero in codfw: maintenance
• 11:11 jmm@cumin2002: END (FAIL) - Cookbook sre.puppet.migrate-host (exit_code=99) for host cloudcephosd1036.eqiad.wmnet
• 11:10 jynus: running UPDATE into viwiki db2218 (s7) T375507
• 11:08 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) zotero.discovery.wmnet on all recursors
• 11:08 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache zotero.discovery.wmnet on all recursors
• 11:08 akosiaris@cumin1002: START - Cookbook sre.discovery.service-route pool zotero in codfw: maintenance
• 11:05 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
• 11:04 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/zotero: apply
• 11:04 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudcephosd1036.eqiad.wmnet
• 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host cloudcephosd1037.eqiad.wmnet
• 10:54 akosiaris@cumin1002: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool zotero in codfw: maintenance
• 10:53 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudcephosd1037.eqiad.wmnet
• 10:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) zotero.discovery.wmnet on all recursors
• 10:49 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache zotero.discovery.wmnet on all recursors
• 10:49 akosiaris@cumin1002: START - Cookbook sre.discovery.service-route depool zotero in codfw: maintenance
• 10:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
• 10:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
• 10:43 akosiaris@cumin1002: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool zotero in eqiad: maintenance
• 10:38 akosiaris@cumin1002: START - Cookbook sre.discovery.service-route pool zotero in eqiad: maintenance
• 10:25 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host cloudcephosd1025.eqiad.wmnet
• 10:24 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
• 10:24 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply
• 10:19 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/zotero: apply
• 10:19 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/zotero: apply
• 10:06 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudcephosd1025.eqiad.wmnet
• 09:39 moritzm: installing distro-info-data updates from bullseye/bookworm point updates
• 09:35 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host cloudcephosd1038.eqiad.wmnet
• 09:25 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudcephosd1038.eqiad.wmnet
• 09:22 slyngs: Upgrade idm2001 to Bitu version 0.0.9
• 09:19 dcaro@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 180 days, 0:00:00 on cloudcephosd1025.eqiad.wmnet with reason: Getting the disks shipped to dell T348643
• 09:19 dcaro@cumin1002: START - Cookbook sre.hosts.downtime for 180 days, 0:00:00 on cloudcephosd1025.eqiad.wmnet with reason: Getting the disks shipped to dell T348643
• 09:11 elukey: set max-catalog-entries (changes the default catalog pagination) to 25 for docker-registry - T348876
• 09:07 mvernon@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=apus,name=codfw
• 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging DNdubane out of all services on: 700 hosts
• 08:28 jmm@cumin2002: START - Cookbook sre.idm.logout Logging DNdubane out of all services on: 700 hosts
• 08:28 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging DNdubane out of all services on: 1540 hosts
• 08:27 jmm@cumin2002: START - Cookbook sre.idm.logout Logging DNdubane out of all services on: 1540 hosts
• 08:06 elukey: set max-catalog-entries (changes the default catalog pagination) to 50 for docker-registry - T348876
• 08:06 hashar@deploy1003: Finished deploy [integration/docroot@0482d53]: zuul: show change queues window value (duration: 00m 07s)
• 08:06 hashar@deploy1003: Started deploy [integration/docroot@0482d53]: zuul: show change queues window value
• 07:58 jynus: running REPLACE into dtpwiki db2123 (s5) T375507
• 07:55 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rdf-streaming-updater: apply
• 07:55 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/rdf-streaming-updater: apply
• 07:51 dcausse@deploy1003: helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply
• 07:51 dcausse@deploy1003: helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply
• 07:47 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 07:47 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 07:42 kartik@deploy1003: Finished scap sync-world: Backport for Revert "Enable message group subscription feature for Test Wikipedia" (duration: 06m 48s)
• 07:37 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling restart_daemons on A:ldap-replicas-eqiad
• 07:37 moritzm: restarting slapd on r/w LDAP servers to pick up GNUTLS security updates
• 07:37 kartik@deploy1003: kartik, trainbranchbot: Continuing with sync
• 07:37 kartik@deploy1003: kartik, trainbranchbot: Backport for Revert "Enable message group subscription feature for Test Wikipedia" synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:36 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling restart_daemons on A:ldap-replicas-eqiad
• 07:35 kartik@deploy1003: Started scap sync-world: Backport for Revert "Enable message group subscription feature for Test Wikipedia"
• 07:34 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling restart_daemons on A:ldap-replicas-codfw
• 07:32 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling restart_daemons on A:ldap-replicas-codfw
• 07:32 kartik@deploy1003: Sync cancelled.
• 07:19 jmm@cumin2002: END (PASS) - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors (exit_code=0) rolling restart_daemons on A:logstash-collector
• 07:06 kartik@deploy1003: abi, kartik: Backport for Enable message group subscription feature for Test Wikipedia (T372386) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:04 kartik@deploy1003: Started scap sync-world: Backport for Enable message group subscription feature for Test Wikipedia (T372386)
• 07:02 jmm@cumin2002: START - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors rolling restart_daemons on A:logstash-collector
• 06:51 moritzm: installing gnutls security updates on bullseye/bookworm
• 04:39 kart_: Updated MinT to 2024-09-19-120927-production
• 04:32 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
• 04:23 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
• 04:20 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
• 04:12 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
• 04:05 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
• 04:01 kartik@deploy1003: helmfile [staging] START helmfile.d/services/machinetranslation: apply

2024-09-24

• 22:42 zabe: zabe@mwmaint1002:~/T363538$ mwscript cleanupTitles.php --wiki=viwiki # T363538
• 22:42 zabe: zabe@mwmaint1002:~/T363538$ mwscript cleanupTitles.php --wiki=tumwiki # T363538
• 22:42 zabe: zabe@mwmaint1002:~/T363538$ mwscript cleanupTitles.php --wiki=testwiki # T363538
• 22:42 zabe: zabe@mwmaint1002:~/T363538$ mwscript cleanupTitles.php --wiki=pawiki # T363538
• 22:41 zabe: zabe@mwmaint1002:~/T363538$ mwscript cleanupTitles.php --wiki=orwiki # T363538
• 22:35 zabe: zabe@mwmaint1002:~/T363538$ mwscript cleanupTitles.php --wiki=metawiki # T363538
• 22:34 zabe: zabe@mwmaint1002:~/T363538$ mwscript cleanupTitles.php --wiki=gurwiki # T363538
• 22:34 zabe: zabe@mwmaint1002:~/T363538$ mwscript cleanupTitles.php --wiki=foundationwiki # T363538
• 22:34 zabe: zabe@mwmaint1002:~/T363538$ mwscript cleanupTitles.php --wiki=eowikinews # T363538
• 22:33 zabe: zabe@mwmaint1002:~/T363538$ mwscript cleanupTitles.php --wiki=cawiki # T363538
• 22:33 zabe: zabe@mwmaint1002:~/T363538$ mwscript cleanupTitles.php --wiki=elwiki # T363538
• 22:27 zabe@deploy1003: Finished scap sync-world: update interwiki cache (duration: 06m 52s)
• 22:20 zabe@deploy1003: Started scap sync-world: update interwiki cache
• 22:15 zabe: zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=shnwikinews --cluster=all 2>&1 | tee /tmp/shnwikinews.UpdateSearchIndexConfig.log # T375430
• 22:15 zabe@deploy1003: Finished scap sync-world: Creating shnwikinews (T375430) (duration: 06m 47s)
• 22:08 zabe@deploy1003: Started scap sync-world: Creating shnwikinews (T375430)
• 22:07 zabe: zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=gorwikiquote --cluster=all 2>&1 | tee /tmp/gorwikiquote.UpdateSearchIndexConfig.log # T375088
• 22:07 zabe@deploy1003: Finished scap sync-world: Creating gorwikiquote (T375088) (duration: 06m 56s)
• 22:00 zabe@deploy1003: Started scap sync-world: Creating gorwikiquote (T375088)
• 21:58 zabe: zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=moswiki --cluster=all 2>&1 | tee /tmp/moswiki.UpdateSearchIndexConfig.log # T374641
• 21:57 zabe@deploy1003: Finished scap sync-world: Creating moswiki (T374641) (duration: 06m 49s)
• 21:50 zabe@deploy1003: Started scap sync-world: Creating moswiki (T374641)
• 21:41 zabe: zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=kgewiki --cluster=all 2>&1 | tee /tmp/kgewiki.UpdateSearchIndexConfig.log # T374813
• 21:40 zabe@deploy1003: Finished scap sync-world: Creating kgewiki (T374813) (duration: 06m 35s)
• 21:34 zabe@deploy1003: Started scap sync-world: Creating kgewiki (T374813)
• 21:32 zabe: zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=madwiktionary --cluster=all 2>&1 | tee /tmp/madwiktionary.UpdateSearchIndexConfig.log # T374968
• 21:32 zabe@deploy1003: Finished scap sync-world: Creating madwiktionary (T374968) (duration: 06m 50s)
• 21:25 zabe@deploy1003: Started scap sync-world: Creating madwiktionary (T374968)
• 21:25 zabe: create Wiktionary Madurese # T374968
• 21:15 TheresNoTime: UTC late backport window done
• 21:14 samtar@deploy1003: Finished scap sync-world: Backport for Use `class` instead of `id` for scribunto errors (T375539) (duration: 09m 36s)
• 21:09 samtar@deploy1003: samtar, cscott: Continuing with sync
• 21:06 samtar@deploy1003: samtar, cscott: Backport for Use `class` instead of `id` for scribunto errors (T375539) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 21:04 samtar@deploy1003: Started scap sync-world: Backport for Use `class` instead of `id` for scribunto errors (T375539)
• 20:52 samtar@deploy1003: Finished scap sync-world: Backport for Remove unused wgStatsMethod, wgResourceLoaderClientPreferences (duration: 08m 13s)
• 20:47 zabe: zabe@mwmaint1002:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki 'Jasonb28' 'MichiganNJPat' # T375516
• 20:47 samtar@deploy1003: samtar, krinkle: Continuing with sync
• 20:45 samtar@deploy1003: samtar, krinkle: Backport for Remove unused wgStatsMethod, wgResourceLoaderClientPreferences synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:43 samtar@deploy1003: Started scap sync-world: Backport for Remove unused wgStatsMethod, wgResourceLoaderClientPreferences
• 20:42 samtar@deploy1003: Finished scap sync-world: Backport for Add a 0-coverage QuickSurvey to enwiki to advertise the Add A Fact Extension (duration: 07m 05s)
• 20:37 samtar@deploy1003: derenrich, samtar: Continuing with sync
• 20:37 samtar@deploy1003: derenrich, samtar: Backport for Add a 0-coverage QuickSurvey to enwiki to advertise the Add A Fact Extension synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:35 samtar@deploy1003: Started scap sync-world: Backport for Add a 0-coverage QuickSurvey to enwiki to advertise the Add A Fact Extension
• 20:32 samtar@deploy1003: Finished scap sync-world: Backport for Deploy donate link to all wikis (T373585) (duration: 11m 11s)
• 20:27 samtar@deploy1003: samtar, toyofuku: Continuing with sync
• 20:23 samtar@deploy1003: samtar, toyofuku: Backport for Deploy donate link to all wikis (T373585) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:21 samtar@deploy1003: Started scap sync-world: Backport for Deploy donate link to all wikis (T373585)
• 20:15 samtar@deploy1003: Finished scap sync-world: Backport for Set `wgMFCustomSiteModules` to false for mediawikiwiki (T375540) (duration: 19m 11s)
• 20:11 samtar@deploy1003: samtar, izno: Continuing with sync
• 19:58 samtar@deploy1003: samtar, izno: Backport for Set `wgMFCustomSiteModules` to false for mediawikiwiki (T375540) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 19:56 samtar@deploy1003: Started scap sync-world: Backport for Set `wgMFCustomSiteModules` to false for mediawikiwiki (T375540)
• 19:00 denisse@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1008.eqiad.wmnet
• 18:53 denisse@cumin2002: START - Cookbook sre.hosts.reboot-single for host prometheus1008.eqiad.wmnet
• 18:21 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.24 refs T373643
• 17:10 brennen: train presync finished successfully; going AFK for ~45 minutes but will return to roll group0 during train window (T373643, T375477)
• 17:09 brennen@deploy1003: Finished scap sync-world: testwikis to 1.43.0-wmf.24 refs T373643 (duration: 27m 45s)
• 17:03 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc2038.mgmt.codfw.wmnet with chassis set policy FORCE_RESTARTand with Dell SCP reboot policy FORCED
• 16:59 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host mc2038.mgmt.codfw.wmnet with chassis set policy FORCE_RESTARTand with Dell SCP reboot policy FORCED
• 16:58 sfaci@deploy1003: Finished deploy [airflow-dags/analytics@e1fb17b]: deploying new datahub ingestion (duration: 00m 52s)
• 16:57 sfaci@deploy1003: Started deploy [airflow-dags/analytics@e1fb17b]: deploying new datahub ingestion
• 16:54 sfaci: Deployed refinery using scap, then deployed onto hdfs
• 16:41 brennen@deploy1003: Started scap sync-world: testwikis to 1.43.0-wmf.24 refs T373643
• 16:39 swfrench-wmf: switched logstash.discovery.wmnet back to eqiad due to reports of stale dashboards - T370962
• 16:38 swfrench@cumin1002: conftool action : set/pooled=false; selector: dnsdisc=logstash,name=codfw
• 16:37 swfrench@cumin1002: conftool action : set/pooled=true; selector: dnsdisc=logstash,name=eqiad
• 16:30 sfaci@deploy1003: Finished deploy [analytics/refinery@cdcefda] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@cdcefda6] (duration: 03m 13s)
• 16:27 sfaci@deploy1003: Started deploy [analytics/refinery@cdcefda] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@cdcefda6]
• 16:27 sfaci@deploy1003: Finished deploy [analytics/refinery@cdcefda] (thin): Regular analytics weekly train THIN [analytics/refinery@cdcefda6] (duration: 04m 34s)
• 16:24 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc2038.mgmt.codfw.wmnet with chassis set policy FORCE_RESTARTand with Dell SCP reboot policy FORCED
• 16:22 sfaci@deploy1003: Started deploy [analytics/refinery@cdcefda] (thin): Regular analytics weekly train THIN [analytics/refinery@cdcefda6]
• 16:22 sfaci@deploy1003: Finished deploy [analytics/refinery@cdcefda]: Regular analytics weekly train [analytics/refinery@cdcefda6] (duration: 07m 46s)
• 16:18 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host mc2038.mgmt.codfw.wmnet with chassis set policy FORCE_RESTARTand with Dell SCP reboot policy FORCED
• 16:18 swfrench-wmf: repooled swift in eqiad to potentially mitigate issues with thumbor and swift - T370962
• 16:16 swfrench@cumin1002: conftool action : set/pooled=true; selector: dnsdisc=swift,name=eqiad
• 16:16 swfrench@cumin1002: conftool action : set/pooled=false; selector: dnsdisc=swift,name=eqiad
• 16:14 sfaci@deploy1003: Started deploy [analytics/refinery@cdcefda]: Regular analytics weekly train [analytics/refinery@cdcefda6]
• 16:12 sfaci: Deploying Refinery
• 16:07 mforns@deploy1003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
• 16:07 mforns@deploy1003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
• 16:07 swfrench-wmf: repooled swift-ro in eqiad to potentially mitigate issues with thumbor - T370962
• 16:06 swfrench@cumin1002: conftool action : set/pooled=true; selector: dnsdisc=swift-ro,name=eqiad
• 15:50 swfrench-wmf: switchover day 1 actions are complete - T370962
• 15:49 swfrench@cumin1002: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter Switchover - T370962
• 15:41 otto@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: sync
• 15:41 otto@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-analytics: sync
• 15:40 otto@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: sync
• 15:40 otto@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: sync
• 15:39 otto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: sync
• 15:39 ottomata: rolling restart of eventgate-analytics to pick up https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1073855
• 15:39 otto@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: sync
• 15:29 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1004.eqiad.wmnet with OS bookworm
• 15:23 swfrench@cumin1002: START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover - T370962
• 15:22 swfrench@cumin1002: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in all: None - None
• 15:22 swfrench@cumin1002: START - Cookbook sre.discovery.datacenter status all services in all: None - None
• 15:13 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1004.eqiad.wmnet with reason: host reimage
• 15:09 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1004.eqiad.wmnet with reason: host reimage
• 15:09 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: sync
• 15:07 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: sync
• 15:02 swfrench@cumin1002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site eqiad [reason: Datacenter Switchover, T370962]
• 15:01 swfrench@cumin1002: START - Cookbook sre.dns.admin DNS admin: depool site eqiad [reason: Datacenter Switchover, T370962]
• 15:01 swfrench-wmf: starting switchover day 1 - T370962
• 14:53 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1004.eqiad.wmnet with OS bookworm
• 14:50 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1003.eqiad.wmnet with OS bookworm
• 14:48 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1044.eqiad.wmnet with OS bookworm
• 14:47 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 14:37 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 14:33 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1003.eqiad.wmnet with reason: host reimage
• 14:31 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1003.eqiad.wmnet with reason: host reimage
• 14:24 dcaro@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1039.eqiad.wmnet with OS bullseye
• 14:23 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1044.eqiad.wmnet with reason: host reimage
• 14:21 swfrench-wmf: scaled up mw-web ahead of traffic+services switchover - T370962
• 14:21 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 14:21 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 14:20 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
• 14:20 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1044.eqiad.wmnet with reason: host reimage
• 14:20 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply
• 14:19 swfrench-wmf: scaled up mw-api-ext ahead of traffic+services switchover - T370962
• 14:19 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
• 14:18 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
• 14:18 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
• 14:17 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
• 14:15 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1003.eqiad.wmnet with OS bookworm
• 14:08 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1002.eqiad.wmnet with OS bookworm
• 14:04 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1044.eqiad.wmnet with OS bookworm
• 14:04 otto@deploy1003: Finished scap sync-world: Backport for EventStreamConfig: Disable regex steam hadoop ingestion (T361498) (duration: 10m 04s)
• 13:59 otto@deploy1003: otto, joal: Continuing with sync
• 13:55 otto@deploy1003: otto, joal: Backport for EventStreamConfig: Disable regex steam hadoop ingestion (T361498) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:53 otto@deploy1003: Started scap sync-world: Backport for EventStreamConfig: Disable regex steam hadoop ingestion (T361498)
• 13:51 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1002.eqiad.wmnet with reason: host reimage
• 13:48 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1002.eqiad.wmnet with reason: host reimage
• 13:46 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for Temporarily allow core password reset functionality (T151012) (duration: 10m 21s)
• 13:42 moritzm: installing krb5 security updates
• 13:41 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Continuing with sync
• 13:41 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Backport for Temporarily allow core password reset functionality (T151012) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:37 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site ulsfo [reason: cr3-ulsfo rebooted, repooling ulsfo, T375345]
• 13:37 sukhe@cumin1002: START - Cookbook sre.dns.admin DNS admin: pool site ulsfo [reason: cr3-ulsfo rebooted, repooling ulsfo, T375345]
• 13:35 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for Temporarily allow core password reset functionality (T151012)
• 13:34 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for SSO domain shouldn't have a mobile version (T375272) (duration: 12m 51s)
• 13:34 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1002.eqiad.wmnet with OS bookworm
• 13:27 jiji@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2127.codfw.wmnet
• 13:27 jiji@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2127.codfw.wmnet
• 13:27 jiji@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2127.codfw.wmnet
• 13:27 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Continuing with sync
• 13:27 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, matmarex: Backport for SSO domain shouldn't have a mobile version (T375272) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:25 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-worker1002.eqiad.wmnet with OS bookworm
• 13:25 dcaro@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1039.eqiad.wmnet with OS bullseye
• 13:23 kamila_: homer cr*codfw* commit T372878
• 13:21 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for SSO domain shouldn't have a mobile version (T375272)
• 13:21 effie: homer lsw1-a6-codfw* commit 'T372878
• 13:20 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for Apply videoscaler request limits and wall clock time limits to shellbox-video (T373517) (duration: 17m 12s)
• 13:18 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2127.codfw.wmnet with OS bullseye
• 13:11 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1002.eqiad.wmnet with OS bookworm
• 13:09 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, hnowlan: Continuing with sync
• 13:08 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, hnowlan: Backport for Apply videoscaler request limits and wall clock time limits to shellbox-video (T373517) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:02 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for Apply videoscaler request limits and wall clock time limits to shellbox-video (T373517)
• 12:58 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2127.codfw.wmnet with reason: host reimage
• 12:55 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2127.codfw.wmnet with reason: host reimage
• 12:48 akosiaris@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 12:48 akosiaris@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 12:44 akosiaris@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 12:40 akosiaris@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 12:36 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2127
• 12:36 jiji@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2127
• 12:36 jiji@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2127
• 12:36 jiji@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2127.codfw.wmnet 83.0.192.10.in-addr.arpa 3.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:36 jiji@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2127.codfw.wmnet 83.0.192.10.in-addr.arpa 3.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:36 jiji@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:36 jiji@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2127 - jiji@cumin1002"
• 12:36 jiji@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2127 - jiji@cumin1002"
• 12:33 jiji@cumin1002: START - Cookbook sre.dns.netbox
• 12:32 jiji@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2127
• 12:32 jiji@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2127.codfw.wmnet with OS bullseye
• 12:32 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
• 12:32 jiji@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2127.codfw.wmnet
• 12:32 elukey@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
• 12:31 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: sync
• 12:30 jiji@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2127.codfw.wmnet on all recursors
• 12:30 elukey@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: sync
• 12:30 jiji@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2127.codfw.wmnet on all recursors
• 12:30 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2427 to wikikube-worker2127
• 12:29 elukey@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: sync
• 12:29 jiji@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2127
• 12:29 elukey@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: sync
• 12:29 jiji@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2127
• 12:29 jiji@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:29 jiji@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2427 to wikikube-worker2127 - jiji@cumin1002"
• 12:28 jiji@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2427 to wikikube-worker2127 - jiji@cumin1002"
• 12:28 akosiaris@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 12:28 akosiaris@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 12:26 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on mc2038.codfw.wmnet with reason: CPU failure - T375495
• 12:25 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on mc2038.codfw.wmnet with reason: CPU failure - T375495
• 12:24 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1001.eqiad.wmnet with OS bullseye
• 12:21 jiji@cumin1002: START - Cookbook sre.dns.netbox
• 12:21 jiji@cumin1002: START - Cookbook sre.hosts.rename from mw2427 to wikikube-worker2127
• 12:17 jiji@cumin1002: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from mw2427 to wikikube-worker2127
• 12:17 jiji@cumin1002: START - Cookbook sre.hosts.rename from mw2427 to wikikube-worker2127
• 12:16 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2427.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL
• 12:16 akosiaris@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 12:16 akosiaris@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 12:13 akosiaris@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 12:13 akosiaris@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 12:12 jiji@cumin1002: START - Cookbook sre.hosts.provision for host mw2427.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL
• 12:12 jynus: running db-compare on s2, s3 T375186
• 12:05 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1001.eqiad.wmnet with reason: host reimage
• 12:05 jiji@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2427.codfw.wmnet
• 12:05 jiji@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2427.codfw.wmnet
• 12:01 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1001.eqiad.wmnet with reason: host reimage
• 11:59 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2427.codfw.wmnet with reason: reimage
• 11:59 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2427.codfw.wmnet with reason: reimage
• 11:48 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1001.eqiad.wmnet with OS bullseye
• 11:48 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-worker1001.eqiad.wmnet with OS bullseye
• 11:45 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 11:45 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 11:42 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1001.eqiad.wmnet with OS bullseye
• 11:39 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich-next: apply
• 11:39 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich-next: apply
• 11:38 jiji@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2126.codfw.wmnet
• 11:38 jiji@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2126.codfw.wmnet
• 11:38 jiji@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2126.codfw.wmnet
• 11:38 moritzm: installing systemd bugfix updates from Bookworm point release
• 11:31 effie: homer lsw1-a6-codfw* commit 'T372878'
• 11:31 effie: homer cr*codfw* commit 'T372878'
• 10:57 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2126.codfw.wmnet with OS bullseye
• 10:34 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2126.codfw.wmnet with reason: host reimage
• 10:32 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1001.eqiad.wmnet with OS bullseye
• 10:30 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2126.codfw.wmnet with reason: host reimage
• 10:25 godog: force deletion of older thanos blocks - T351927
• 10:17 stevemunene@cumin1002: START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1176.eqiad.wmnet
• 10:16 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=99) for hosts an-worker1176.eqiad.wmnet
• 10:14 stevemunene@cumin1002: START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1176.eqiad.wmnet
• 10:14 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=99) for hosts an-worker1176.eqiad.wmnet
• 10:13 stevemunene@cumin1002: START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker1176.eqiad.wmnet
• 10:13 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1001.eqiad.wmnet with reason: host reimage
• 10:11 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2126
• 10:11 jiji@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2126
• 10:11 jiji@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2126
• 10:11 jiji@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2126.codfw.wmnet 82.0.192.10.in-addr.arpa 2.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 10:11 jiji@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2126.codfw.wmnet 82.0.192.10.in-addr.arpa 2.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 10:11 jiji@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:11 jiji@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2126 - jiji@cumin1002"
• 10:11 jiji@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2126 - jiji@cumin1002"
• 10:10 jnuche@deploy1003: Started scap sync-world: testwikis to 1.43.0-wmf.24 refs T373643
• 10:09 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1001.eqiad.wmnet with reason: host reimage
• 10:06 jiji@cumin1002: START - Cookbook sre.dns.netbox
• 10:05 jiji@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2126
• 10:05 jiji@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2126.codfw.wmnet with OS bullseye
• 10:05 jiji@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2126.codfw.wmnet
• 10:03 jiji@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2126.codfw.wmnet on all recursors
• 10:03 jiji@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2126.codfw.wmnet on all recursors
• 10:02 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2426 to wikikube-worker2126
• 10:01 jiji@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2126
• 09:57 jiji@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2126
• 09:57 jiji@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:57 jiji@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2426 to wikikube-worker2126 - jiji@cumin1002"
• 09:56 jiji@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2426 to wikikube-worker2126 - jiji@cumin1002"
• 09:55 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1001.eqiad.wmnet with OS bullseye
• 09:52 jiji@cumin1002: START - Cookbook sre.dns.netbox
• 09:52 jiji@cumin1002: START - Cookbook sre.hosts.rename from mw2426 to wikikube-worker2126
• 09:51 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2426.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL
• 09:46 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-worker1001.eqiad.wmnet with OS bookworm
• 09:33 jiji@cumin1002: START - Cookbook sre.hosts.provision for host mw2426.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL
• 09:27 tappof: upgrade mtail on lists* and ncredir* https://phabricator.wikimedia.org/T375085
• 09:25 arnaudb@cumin1002: END (FAIL) - Cookbook sre.switchdc.databases.prepare (exit_code=93) for the switch from eqiad to codfw
• 09:25 arnaudb@cumin1002: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw
• 09:21 jnuche@deploy1003: scap failed: <UnboundLocalError> local variable 'e' referenced before assignment (scap version: 4.104.0-1) (duration: 38m 15s)
• 09:19 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bullseye
• 09:11 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1177.eqiad.wmnet with OS bullseye
• 09:08 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mw2427.codfw.wmnet with reason: reimage
• 09:07 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 3:00:00 on mw2427.codfw.wmnet with reason: reimage
• 09:07 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mw2426.codfw.wmnet with reason: reimage
• 09:07 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 3:00:00 on mw2426.codfw.wmnet with reason: reimage
• 09:05 jiji@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2427.codfw.wmnet
• 09:04 jiji@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2427.codfw.wmnet
• 09:04 jiji@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2426.codfw.wmnet
• 09:03 jiji@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2426.codfw.wmnet
• 09:03 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
• 08:59 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
• 08:55 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1177.eqiad.wmnet with reason: host reimage
• 08:51 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1177.eqiad.wmnet with reason: host reimage
• 08:46 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
• 08:43 jnuche@deploy1003: Started scap sync-world: testwikis to 1.43.0-wmf.24 refs T373643
• 08:37 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1177.eqiad.wmnet with OS bullseye
• 08:36 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on puppetmaster2001.codfw.wmnet with reason: WIP - working on puppet runs
• 08:36 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on puppetmaster2001.codfw.wmnet with reason: WIP - working on puppet runs
• 08:36 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cloudvirt1063.eqiad.wmnet with reason: cloudvirt1063 needs maintenance T375223
• 08:36 fnegri@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on cloudvirt1063.eqiad.wmnet with reason: cloudvirt1063 needs maintenance T375223
• 08:30 jnuche@deploy1003: Started scap sync-world: testwikis to 1.43.0-wmf.24 refs T373643
• 08:20 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
• 07:41 XioNoX: reboot cr3-ulsfo - T375345
• 07:28 dcausse: closing the backport window
• 07:27 dcausse@deploy1003: Finished scap sync-world: Backport for Add a private variant of the cirrus update stream (T374335) (duration: 24m 11s)
• 07:12 dcausse@deploy1003: dcausse, ebernhardson: Continuing with sync
• 07:07 dcausse@deploy1003: dcausse, ebernhardson: Backport for Add a private variant of the cirrus update stream (T374335) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:03 dcausse@deploy1003: Started scap sync-world: Backport for Add a private variant of the cirrus update stream (T374335)
• 07:02 tappof@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host centrallog1002.eqiad.wmnet
• 06:35 tappof@cumin2002: START - Cookbook sre.hosts.reboot-single for host centrallog1002.eqiad.wmnet
• 05:55 tappof: centrallog1002 upgrade to bookworm in progress https://phabricator.wikimedia.org/T353912
• 05:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on pc1013.eqiad.wmnet with reason: maintenance
• 05:03 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on pc1013.eqiad.wmnet with reason: maintenance
• 04:06 mwpresync@deploy1003: Pruned MediaWiki: 1.43.0-wmf.21 (duration: 06m 02s)
• 03:06 eevans@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: Server depooled. Has hardware issues
• 03:06 eevans@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: Server depooled. Has hardware issues
• 03:02 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.43.0-wmf.24 refs T373643
• 01:16 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 01:14 jclark@cumin1002: START - Cookbook sre.dns.netbox
• 01:08 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1052.eqiad.wmnet with OS bookworm
• 01:08 jclark@cumin1002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 01:08 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1044.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 01:03 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1044.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 01:00 pt1979@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1246.eqiad.wmnet with OS bookworm
• 00:58 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1043.eqiad.wmnet with OS bookworm
• 00:58 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 00:58 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 00:52 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1042.eqiad.wmnet with OS bookworm
• 00:52 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 00:46 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 00:43 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1043.eqiad.wmnet with reason: host reimage
• 00:40 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1043.eqiad.wmnet with reason: host reimage
• 00:39 pt1979@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage
• 00:35 pt1979@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage
• 00:32 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1042.eqiad.wmnet with reason: host reimage
• 00:29 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1042.eqiad.wmnet with reason: host reimage
• 00:23 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1051.eqiad.wmnet with OS bookworm
• 00:23 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 00:22 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 00:21 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1043.eqiad.wmnet with OS bookworm
• 00:20 pt1979@cumin1002: START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm
• 00:15 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1042.eqiad.wmnet with OS bookworm
• 00:11 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1044
• 00:11 pt1979@cumin1002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host db1246.eqiad.wmnet
• 00:10 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1044
• 00:10 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1043
• 00:08 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1043
• 00:08 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1041
• 00:08 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1041
• 00:08 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1051.eqiad.wmnet with reason: host reimage
• 00:05 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1042
• 00:04 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1051.eqiad.wmnet with reason: host reimage
• 00:03 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1042
• 00:03 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 00:01 jclark@cumin1002: START - Cookbook sre.dns.netbox

2024-09-23

• 23:43 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1050.eqiad.wmnet with OS bookworm
• 23:43 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 23:42 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 23:35 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 23:34 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1039.eqiad.wmnet with OS bookworm
• 23:34 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 23:34 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 23:34 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1049.eqiad.wmnet with OS bookworm
• 23:33 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 23:29 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 23:27 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1052.eqiad.wmnet with reason: host reimage
• 23:23 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1052.eqiad.wmnet with reason: host reimage
• 23:21 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1050.eqiad.wmnet with reason: host reimage
• 23:17 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1039.eqiad.wmnet with reason: host reimage
• 23:16 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1050.eqiad.wmnet with reason: host reimage
• 23:16 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2020.codfw.wmnet
• 23:14 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1047.eqiad.wmnet with OS bookworm
• 23:14 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 23:14 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1049.eqiad.wmnet with reason: host reimage
• 23:14 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1051.eqiad.wmnet with OS bookworm
• 23:13 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1039.eqiad.wmnet with reason: host reimage
• 23:11 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 23:10 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1049.eqiad.wmnet with reason: host reimage
• 23:09 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1052.eqiad.wmnet with OS bookworm
• 23:05 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1048.eqiad.wmnet with OS bookworm
• 23:05 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 23:04 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 23:02 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1050.eqiad.wmnet with OS bookworm
• 22:57 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1047.eqiad.wmnet with reason: host reimage
• 22:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1049.eqiad.wmnet with OS bookworm
• 22:53 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1047.eqiad.wmnet with reason: host reimage
• 22:49 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1048.eqiad.wmnet with reason: host reimage
• 22:46 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1048.eqiad.wmnet with reason: host reimage
• 22:40 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1047.eqiad.wmnet with OS bookworm
• 22:37 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1040.eqiad.wmnet with OS bookworm
• 22:37 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 22:37 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 22:32 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1041.eqiad.wmnet with OS bookworm
• 22:32 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 22:32 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1039.eqiad.wmnet with OS bookworm
• 22:32 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 22:31 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1048.eqiad.wmnet with OS bookworm
• 22:28 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1046.eqiad.wmnet with OS bookworm
• 22:28 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 22:27 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 22:24 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1045.eqiad.wmnet with OS bookworm
• 22:24 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 22:24 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 22:21 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1040.eqiad.wmnet with reason: host reimage
• 22:17 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1041.eqiad.wmnet with reason: host reimage
• 22:13 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1041.eqiad.wmnet with reason: host reimage
• 22:13 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1046.eqiad.wmnet with reason: host reimage
• 22:09 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1045.eqiad.wmnet with reason: host reimage
• 22:08 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1046.eqiad.wmnet with reason: host reimage
• 22:06 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1040.eqiad.wmnet with reason: host reimage
• 22:05 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1045.eqiad.wmnet with reason: host reimage
• 21:55 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1001.eqiad.wmnet with reason: host reimage
• 21:51 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1001.eqiad.wmnet with reason: host reimage
• 21:50 bking@cumin2002: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2020.codfw.wmnet
• 21:46 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1045.eqiad.wmnet with OS bookworm
• 21:39 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1046.eqiad.wmnet with OS bookworm
• 21:38 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 21:38 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove old switch stack - pt1979@cumin2002"
• 21:38 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove old switch stack - pt1979@cumin2002"
• 21:38 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1001.eqiad.wmnet with OS bookworm
• 21:34 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 21:30 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-worker1001.eqiad.wmnet with OS bookworm
• 21:25 bking@cumin2002: END (ERROR) - Cookbook sre.wdqs.categories-reload (exit_code=97) reloading categories to wdqs2020.codfw.wmnet
• 21:21 toyofuku@deploy1003: Finished scap sync-world: Backport for Do not apply table styling rules to Main page (T375245) (duration: 10m 44s)
• 21:17 bking@cumin2002: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2020.codfw.wmnet
• 21:15 toyofuku@deploy1003: jdlrobson, toyofuku: Continuing with sync
• 21:12 toyofuku@deploy1003: jdlrobson, toyofuku: Backport for Do not apply table styling rules to Main page (T375245) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 21:11 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1041.eqiad.wmnet with OS bookworm
• 21:10 toyofuku@deploy1003: Started scap sync-world: Backport for Do not apply table styling rules to Main page (T375245)
• 21:01 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1040.eqiad.wmnet with OS bookworm
• 20:56 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1001.eqiad.wmnet with OS bookworm
• 20:56 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1039.eqiad.wmnet with OS bookworm
• 20:56 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-worker1001.eqiad.wmnet with OS bookworm
• 20:40 toyofuku@deploy1003: Finished scap sync-world: Backport for Remove ProofreadPage dark mode namespaces exception, Promote dark mode for anons on tier 1 wikis (T374679) (duration: 27m 41s)
• 20:39 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1001.eqiad.wmnet with OS bookworm
• 20:25 toyofuku@deploy1003: jdlrobson, toyofuku, ebrahim: Continuing with sync
• 20:23 toyofuku@deploy1003: jdlrobson, toyofuku, ebrahim: Backport for Remove ProofreadPage dark mode namespaces exception, Promote dark mode for anons on tier 1 wikis (T374679) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:12 toyofuku@deploy1003: Started scap sync-world: Backport for Remove ProofreadPage dark mode namespaces exception, Promote dark mode for anons on tier 1 wikis (T374679)
• 18:47 jgleeson: SmashPig upgraded from 02ba8a7e to 697344d7
• 17:06 ebernhardson@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:05 ebernhardson@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:53 jgleeson: SmashPig upgraded from ac85ad1d to 02ba8a7e
• 16:49 ebernhardson@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:49 ebernhardson@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:43 ebernhardson@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:43 ebernhardson@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:43 pt1979@cumin1002: START - Cookbook sre.hosts.dhcp for host db1246.eqiad.wmnet
• 16:38 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
• 16:38 pt1979@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1246.eqiad.wmnet with OS bookworm
• 16:08 elukey@puppetserver1001: conftool action : set/pooled=no; selector: name=registry1003.eqiad.wmnet,service=docker-registry,dc=eqiad
• 16:08 elukey@puppetserver1001: conftool action : set/weight=10; selector: name=registry1005.eqiad.wmnet,service=docker-registry,dc=eqiad
• 16:08 elukey@puppetserver1001: conftool action : set/pooled=yes; selector: name=registry1005.eqiad.wmnet,service=docker-registry,dc=eqiad
• 16:05 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:05 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:04 elukey@puppetserver1001: conftool action : set/pooled=true,weight=10; selector: name=registry1005.eqiad.wmnet,service=docker-registry,dc=eqiad
• 16:03 dcausse@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:03 dcausse@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:59 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:59 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:46 pt1979@cumin1002: START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm
• 15:35 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart
• 15:20 stevemunene@deploy1003: Finished deploy [wdqs/wdqs@316bf7f]: allow 3 new endpoints T364233 T368085 T374195 (duration: 05m 51s)
• 15:15 stevemunene@deploy1003: Started deploy [wdqs/wdqs@316bf7f]: allow 3 new endpoints T364233 T368085 T374195
• 15:03 volans@cumin1002: dbctl commit (dc=all): 'emergency failover pc3 to pc1015', diff saved to https://phabricator.wikimedia.org/P69396 and previous config saved to /var/cache/conftool/dbconfig/20240923-150320-volans.json
• 14:51 moritzm: powercycle pc1013 (DIMM error in DIMM_A9)
• 14:50 elukey@puppetserver1001: conftool action : set/pooled=true,weight=10; selector: name=registry1005.eqiad.wmnet,service=docker-registry,dc=eqiad
• 14:49 elukey@puppetserver1001: conftool action : set/pooled=true; selector: name=registry1005.eqiad.wmnet
• 14:49 elukey@puppetserver1001: conftool action : set/weight=10; selector: name=registry1005.eqiad.wmnet
• 14:48 elukey@puppetserver1001: conftool action : set/pooled=true; selector: name=registry1005.eqiad.wmnet
• 14:38 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
• 14:31 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry1005.eqiad.wmnet with OS bookworm
• 14:30 sukhe: sudo cumin 'O:wikidough' 'run-puppet-agent'
• 14:30 jynus: restarting and moving replication source of pc1015 T375382
• 14:16 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-codfw
• 14:16 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cr2-codfw
• 14:16 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry1005.eqiad.wmnet with reason: host reimage
• 14:15 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr1-codfw
• 14:15 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cr1-codfw
• 14:14 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-esams
• 14:14 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cr2-esams
• 14:14 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-by27-esams
• 14:13 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device asw1-by27-esams
• 14:12 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on registry1005.eqiad.wmnet with reason: host reimage
• 14:04 xcollazo@deploy1003: Finished deploy [airflow-dags/analytics@3e2d3b8]: Deploy latest DAGs to analytics Airflow instance. T369868. (duration: 00m 48s)
• 14:03 xcollazo@deploy1003: Started deploy [airflow-dags/analytics@3e2d3b8]: Deploy latest DAGs to analytics Airflow instance. T369868.
• 14:02 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-bw27-esams
• 14:02 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device asw1-bw27-esams
• 14:00 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr1-esams
• 14:00 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cr1-esams
• 13:59 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host registry1005.eqiad.wmnet with OS bookworm
• 13:59 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-b13-drmrs
• 13:59 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device asw1-b13-drmrs
• 13:58 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-b12-drmrs
• 13:58 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device asw1-b12-drmrs
• 13:57 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-drmrs
• 13:57 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cr2-drmrs
• 13:56 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr1-drmrs
• 13:56 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cr1-drmrs
• 13:56 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-eqsin
• 13:55 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cr2-eqsin
• 13:52 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr3-eqsin
• 13:52 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cr3-eqsin
• 13:51 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr4-ulsfo
• 13:51 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cr4-ulsfo
• 13:50 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr3-ulsfo
• 13:50 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cr3-ulsfo
• 13:33 jiji@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker2125.codfw.wmnet
• 13:33 jiji@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2125.codfw.wmnet
• 13:33 jiji@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2125.codfw.wmnet
• 13:30 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cr3-ulsfo with reason: waiting for JTAC
• 13:30 ayounsi@cumin1002: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on cr3-ulsfo with reason: waiting for JTAC
• 13:21 effie: homer lsw1-a6-codfw* commit 'T372878'
• 13:21 effie: homer cr*codfw* commit 'T372878'
• 13:18 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2125.codfw.wmnet with OS bullseye
• 12:58 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2125.codfw.wmnet with reason: host reimage
• 12:54 mnz@deploy1003: Finished deploy [airflow-dags/research@3e2d3b8]: deploy reference risk DAG (duration: 00m 59s)
• 12:54 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2125.codfw.wmnet with reason: host reimage
• 12:54 mnz@deploy1003: Started deploy [airflow-dags/research@3e2d3b8]: deploy reference risk DAG
• 12:36 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2125
• 12:36 jiji@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2125
• 12:36 jiji@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2125
• 12:36 jiji@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2125.codfw.wmnet 81.0.192.10.in-addr.arpa 1.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:36 jiji@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2125.codfw.wmnet 81.0.192.10.in-addr.arpa 1.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:36 jiji@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:36 jiji@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2125 - jiji@cumin1002"
• 12:35 jiji@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2125 - jiji@cumin1002"
• 12:32 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 12:32 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 12:27 jiji@cumin1002: START - Cookbook sre.dns.netbox
• 12:27 jiji@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2125
• 12:27 jiji@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2125.codfw.wmnet with OS bullseye
• 12:27 jiji@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2125.codfw.wmnet
• 12:22 jiji@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2125.codfw.wmnet on all recursors
• 12:22 jiji@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2125.codfw.wmnet on all recursors
• 12:18 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2425 to wikikube-worker2125
• 12:18 jiji@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2125
• 12:18 jiji@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2125
• 12:18 jiji@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:18 jiji@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2425 to wikikube-worker2125 - jiji@cumin1002"
• 12:15 jiji@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2425 to wikikube-worker2125 - jiji@cumin1002"
• 12:12 jynus: restarting replication on pc1013 after crash T375382
• 12:12 jiji@cumin1002: START - Cookbook sre.dns.netbox
• 12:12 jiji@cumin1002: START - Cookbook sre.hosts.rename from mw2425 to wikikube-worker2125
• 12:10 jiji@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host mw2425.codfw.wmnet
• 12:10 jiji@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2425.codfw.wmnet
• 11:55 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2425.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL
• 11:52 jiji@cumin1002: START - Cookbook sre.hosts.provision for host mw2425.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL
• 11:51 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mw2425.codfw.wmnet with reason: reimage
• 11:51 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 3:00:00 on mw2425.codfw.wmnet with reason: reimage
• 11:45 moritzm: installing cups security updates
• 11:17 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 11:17 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 11:16 dcausse@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 11:16 dcausse@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
• 11:15 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 11:15 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 11:02 jmm@cumin2002: END (FAIL) - Cookbook sre.puppet.migrate-host (exit_code=99) for host registry1005.eqiad.wmnet
• 10:57 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host registry1005.eqiad.wmnet
• 10:56 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on registry1005.eqiad.wmnet with reason: WIP - working on puppet runs
• 10:56 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on registry1005.eqiad.wmnet with reason: WIP - working on puppet runs
• 10:51 elukey@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=97) for new host registry1005.eqiad.wmnet
• 10:51 elukey@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host registry1005.eqiad.wmnet with OS bookworm
• 10:51 jynus: starting db master table checks on s1 (db1163, db2203) T375186
• 10:43 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 10:43 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mw2425.codfw.wmnet with reason: reimage
• 10:43 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 3:00:00 on mw2425.codfw.wmnet with reason: reimage
• 10:43 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 10:38 Dreamy_Jazz: Restarting MediaModeration scanning script - https://wikitech.wikimedia.org/wiki/MediaModeration
• 10:34 jiji@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2425.codfw.wmnet
• 10:34 jiji@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2124.codfw.wmnet
• 10:34 jiji@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2124.codfw.wmnet
• 10:33 jiji@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2425.codfw.wmnet
• 10:31 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry1005.eqiad.wmnet with reason: host reimage
• 10:27 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on registry1005.eqiad.wmnet with reason: host reimage
• 10:25 effie: homer lsw1-a6-codfw* commit 'T372878'
• 10:18 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host registry1005.eqiad.wmnet with OS bookworm
• 10:16 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM registry1005.eqiad.wmnet - elukey@cumin1002"
• 10:16 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM registry1005.eqiad.wmnet - elukey@cumin1002"
• 10:15 elukey@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) registry1005.eqiad.wmnet on all recursors
• 10:15 elukey@cumin1002: START - Cookbook sre.dns.wipe-cache registry1005.eqiad.wmnet on all recursors
• 10:15 elukey@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:15 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM registry1005.eqiad.wmnet - elukey@cumin1002"
• 10:15 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM registry1005.eqiad.wmnet - elukey@cumin1002"
• 10:15 dcausse@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 10:14 dcausse@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
• 10:12 elukey@cumin1002: START - Cookbook sre.dns.netbox
• 10:12 elukey@cumin1002: START - Cookbook sre.ganeti.makevm for new host registry1005.eqiad.wmnet
• 10:11 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 10:11 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 10:08 moritzm: rolling out debmonitor-client updates T216832
• 09:59 jiji@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2124.codfw.wmnet
• 09:59 jiji@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2124.codfw.wmnet
• 09:58 jiji@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2124.codfw.wmnet
• 09:53 effie: homer cr*codfw* commit 'T372878'
• 09:51 jayme@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-jumbo-eqiad
• 09:48 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2124.codfw.wmnet with OS bullseye
• 09:46 jayme@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-eqiad
• 09:44 jayme@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-eqiad
• 09:29 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts poolcounter2004.codfw.wmnet
• 09:29 elukey@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:29 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: poolcounter2004.codfw.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1002"
• 09:28 jayme@cumin1002: conftool action : set/pooled=yes; selector: name=registry2004.codfw.wmnet
• 09:27 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2124.codfw.wmnet with reason: host reimage
• 09:26 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: poolcounter2004.codfw.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1002"
• 09:23 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2124.codfw.wmnet with reason: host reimage
• 09:22 jayme@cumin1002: conftool action : set/pooled=no; selector: name=registry200(3|4).codfw.wmnet
• 09:21 jayme@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-eqiad
• 09:21 jayme@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-codfw
• 09:20 elukey@cumin1002: START - Cookbook sre.dns.netbox
• 09:20 jayme@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-eqiad
• 09:18 jayme@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-codfw
• 09:16 elukey@cumin1002: START - Cookbook sre.hosts.decommission for hosts poolcounter2004.codfw.wmnet
• 09:12 jayme@cumin1002: conftool action : set/pooled=yes; selector: name=registry2005.codfw.wmnet
• 09:12 jayme@cumin1002: conftool action : set/weight=10; selector: name=registry2005.codfw.wmnet
• 09:07 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts poolcounter2003.codfw.wmnet
• 09:07 elukey@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:05 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2124
• 09:05 jiji@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2124
• 09:05 elukey@cumin1002: START - Cookbook sre.dns.netbox
• 09:05 jiji@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2124
• 09:05 jiji@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2124.codfw.wmnet 79.0.192.10.in-addr.arpa 9.7.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 09:05 jiji@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2124.codfw.wmnet 79.0.192.10.in-addr.arpa 9.7.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 09:05 jiji@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:05 jiji@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2124 - jiji@cumin1002"
• 09:04 jiji@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2124 - jiji@cumin1002"
• 09:01 jiji@cumin1002: START - Cookbook sre.dns.netbox
• 09:00 jiji@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 08:57 jayme: roll-restarting all kafka clusters for certificate changes - T374729
• 08:57 elukey@cumin1002: START - Cookbook sre.hosts.decommission for hosts poolcounter2003.codfw.wmnet
• 08:26 jiji@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2124.codfw.wmnet on all recursors
• 08:26 jiji@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2124.codfw.wmnet on all recursors
• 08:26 elukey@cumin1002: START - Cookbook sre.puppet.renew-cert for puppetmaster2001.codfw.wmnet: Renew puppet certificate - elukey@cumin1002
• 08:26 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for puppetmaster1001.eqiad.wmnet: Renew puppet certificate - elukey@cumin1002
• 08:25 hashar: Updated CI job operations-puppet-tests-bullseye to image rebuild for Puppet 7 # T330490
• 08:24 elukey@cumin1002: START - Cookbook sre.puppet.renew-cert for puppetmaster1001.eqiad.wmnet: Renew puppet certificate - elukey@cumin1002
• 08:21 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2424 to wikikube-worker2124
• 08:21 jiji@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2124
• 08:21 jiji@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2124
• 08:21 jiji@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:21 jiji@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2424 to wikikube-worker2124 - jiji@cumin1002"
• 08:20 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
• 08:18 moritzm: installing systemd bugfix updates from Bookworm point release
• 08:16 jiji@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2424 to wikikube-worker2124 - jiji@cumin1002"
• 08:16 elukey: elukey@puppetmaster1001:~$ sudo puppet cert destroy performance.discovery.wmnet
• 08:12 jiji@cumin1002: START - Cookbook sre.dns.netbox
• 08:12 jiji@cumin1002: START - Cookbook sre.hosts.rename from mw2424 to wikikube-worker2124
• 08:08 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
• 07:53 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts poolcounter1004.eqiad.wmnet
• 07:52 elukey@cumin1002: START - Cookbook sre.hosts.decommission for hosts poolcounter1004.eqiad.wmnet

2024-09-22

• 13:16 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cr3-ulsfo,cr3-ulsfo IPv6,cr3-ulsfo.mgmt with reason: cr3-ulsfo fpc restart and hw instability
• 13:16 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on cr3-ulsfo,cr3-ulsfo IPv6,cr3-ulsfo.mgmt with reason: cr3-ulsfo fpc restart and hw instability
• 13:15 vgutierrez: depooled ulsfo due to cr3-ulsfo issues
• 13:14 vgutierrez@cumin1002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site ulsfo [reason: cr3-ulsfo issues, no task ID specified]
• 13:14 vgutierrez@cumin1002: START - Cookbook sre.dns.admin DNS admin: depool site ulsfo [reason: cr3-ulsfo issues, no task ID specified]

2024-09-20

• 23:38 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bullseye
• 23:02 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1040.eqiad.wmnet with OS bookworm
• 23:01 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1040.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 23:01 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1040.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:59 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1039.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:59 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1039.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:31 dwisehaupt@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 21:31 dwisehaupt@cumin1002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: decommission frlog2001 and frpm2001 - dwisehaupt@cumin1002"
• 21:31 dwisehaupt@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: decommission frlog2001 and frpm2001 - dwisehaupt@cumin1002"
• 21:24 dwisehaupt@cumin1002: START - Cookbook sre.dns.netbox
• 19:28 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logging-hd2004.codfw.wmnet with OS bookworm
• 19:28 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logging-hd2005.codfw.wmnet with OS bookworm
• 19:28 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
• 19:28 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
• 18:42 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
• 18:36 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
• 18:22 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logging-hd2005.codfw.wmnet with reason: host reimage
• 18:18 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on logging-hd2005.codfw.wmnet with reason: host reimage
• 18:16 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logging-hd2004.codfw.wmnet with reason: host reimage
• 18:12 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on logging-hd2004.codfw.wmnet with reason: host reimage
• 17:46 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2005.codfw.wmnet with OS bookworm
• 17:41 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2004.codfw.wmnet with OS bookworm
• 17:37 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host logging-hd2005
• 17:37 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host logging-hd2005
• 17:37 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host logging-hd2004
• 17:37 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host logging-hd2004
• 17:34 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host logging-hd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:31 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host logging-hd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:29 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:29 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding logging-hd2005 to codfw - jhancock@cumin2002"
• 17:29 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding logging-hd2005 to codfw - jhancock@cumin2002"
• 17:25 jhancock@cumin2002: START - Cookbook sre.dns.netbox
• 17:23 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host logging-hd2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:20 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host logging-hd2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:18 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:18 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding logging-hd2004 to codfw - jhancock@cumin2002"
• 17:18 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding logging-hd2004 to codfw - jhancock@cumin2002"
• 17:13 jhancock@cumin2002: START - Cookbook sre.dns.netbox
• 16:03 xSavitar: T375261 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=metawiki 'Abderazack Ahmat Annour' 'Renamed user 5dcfa1a90396fed431e188c9cbe5af6a'
• 16:02 xSavitar: T375261 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki 'Abderazack Ahmat Annour' 'Renamed user 5dcfa1a90396fed431e188c9cbe5af6a'
• 16:02 xSavitar: T375261 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=loginwiki --logwiki=metawiki 'Abderazack Ahmat Annour' 'Renamed user 5dcfa1a90396fed431e188c9cbe5af6a'
• 16:02 xSavitar: T375261 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=frwiki --logwiki=metawiki 'Abderazack Ahmat Annour' 'Renamed user 5dcfa1a90396fed431e188c9cbe5af6a'
• 16:01 xSavitar: T375261 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=metawiki 'Abderazack Ahmat Annour' 'Renamed user 5dcfa1a90396fed431e188c9cbe5af6a'
• 16:01 xSavitar: T375261 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=metawiki 'Abderazack Ahmat Annour' 'Renamed user 5dcfa1a90396fed431e188c9cbe5af6a'
• 14:20 Dreamy_Jazz: Finished running `foreachwiki maintenance/fixAutoblockLogTitles.php` on a tmux session for T373929
• 14:08 Dreamy_Jazz: Running `foreachwiki maintenance/fixAutoblockLogTitles.php` on a tmux session for T373929
• 14:04 kcvelaga@deploy1003: Finished deploy [airflow-dags/analytics_product@6010568]: automoderator config dag T375062 (duration: 01m 47s)
• 14:03 kcvelaga@deploy1003: Started deploy [airflow-dags/analytics_product@6010568]: automoderator config dag T375062
• 13:20 moritzm: uploaded debmonitor-client 0.4.0-3 for buster/bullseye/bookworm to apt.wikimedia.org T216832
• 13:09 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on mw2425.codfw.wmnet with reason: reimage
• 13:09 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on mw2425.codfw.wmnet with reason: reimage
• 12:45 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2424.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL
• 12:38 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2425.codfw.wmnet with reason: reimage
• 12:37 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2425.codfw.wmnet with reason: reimage
• 12:36 jiji@cumin1002: START - Cookbook sre.hosts.provision for host mw2424.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL
• 12:24 jiji@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2424.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL
• 12:18 jiji@cumin1002: START - Cookbook sre.hosts.provision for host mw2424.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL
• 12:16 jiji@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2424.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL
• 12:11 jiji@cumin1002: START - Cookbook sre.hosts.provision for host mw2424.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL
• 12:07 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2425.codfw.wmnet with reason: reimage
• 12:07 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2425.codfw.wmnet with reason: reimage
• 11:58 jiji@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2424.codfw.wmnet
• 11:57 jiji@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2424.codfw.wmnet
• 11:46 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2092.codfw.wmnet with reason: Degraded RAID
• 11:46 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on wikikube-worker2092.codfw.wmnet with reason: Degraded RAID
• 11:18 moritzm: installing links2 updates from Bullseye point release
• 11:11 moritzm: installing nano updates from Bullseye point update
• 10:11 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f3-eqiad
• 10:11 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-f3-eqiad
• 10:08 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f2-eqiad
• 10:08 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-f2-eqiad
• 10:08 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e3-eqiad
• 10:07 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-e3-eqiad
• 10:07 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e2-eqiad
• 10:07 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-e2-eqiad
• 10:03 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e1-eqiad
• 10:03 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-e1-eqiad
• 09:42 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw1-f4-eqiad
• 09:40 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cloudsw1-f4-eqiad
• 09:38 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw1-e4-eqiad
• 09:35 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cloudsw1-e4-eqiad
• 09:34 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw1-b1-codfw
• 09:32 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cloudsw1-b1-codfw
• 08:54 btullis@cumin1002: END (PASS) - Cookbook sre.hadoop.reboot-workers (exit_code=0) for Hadoop analytics cluster
• 08:37 arnaudb@cumin1002: dbctl commit (dc=all): 'depool es1022 - T375257', diff saved to https://phabricator.wikimedia.org/P69377 and previous config saved to /var/cache/conftool/dbconfig/20240920-083722-arnaudb.json
• away: payments config changed from eee33e37 to e140fab5
• 07:50 tappof: T375085 testing mtail 3.0.9 using debian testing package on centrallog2002
• 07:33 moritzm: rebalance ganeti group D following the various switch maintenances T370630
• 07:28 jelto@cumin1002: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: test I946dd0 with dummy upgrade
• 07:27 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: test I946dd0 with dummy upgrade
• 07:18 vgutierrez: rolling upgrade of purged on magru, drmrs, esams and eqiad - T334078

2024-09-19

• 21:15 Dreamy_Jazz: Evening UTC backport window done
• 21:15 dreamyjazz@deploy1003: Finished scap sync-world: Backport for Re-order arguments to DataAccess::addTrackingCategory, Add a "duplicate-ids" lint category (T200517) (duration: 29m 20s)
• 21:02 dreamyjazz@deploy1003: dreamyjazz, cscott: Continuing with sync
• 21:00 dreamyjazz@deploy1003: dreamyjazz, cscott: Backport for Re-order arguments to DataAccess::addTrackingCategory, Add a "duplicate-ids" lint category (T200517) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for Re-order arguments to DataAccess::addTrackingCategory, Add a "duplicate-ids" lint category (T200517)
• 20:36 Dreamy_Jazz: Running `foreachwikiindblist group1.dblist extensions/CheckUser/maintenance/populateCentralCheckUserIndexTables.php` on a tmux session for T375203
• 20:34 dreamyjazz@deploy1003: Finished scap sync-world: Backport for Add ::caller to queries in populateCentralCheckUserIndexTables.php (T375221) (duration: 06m 50s)
• 20:29 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with sync
• 20:29 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for Add ::caller to queries in populateCentralCheckUserIndexTables.php (T375221) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:27 dreamyjazz@deploy1003: Started scap sync-world: Backport for Add ::caller to queries in populateCentralCheckUserIndexTables.php (T375221)
• 20:27 aqu@deploy1003: Finished deploy [airflow-dags/analytics@e0d8d78]: Fix canary events generation schedule [airflow-dags/analytics@e0d8d78a] (duration: 00m 42s)
• 20:26 aqu@deploy1003: Started deploy [airflow-dags/analytics@e0d8d78]: Fix canary events generation schedule [airflow-dags/analytics@e0d8d78a]
• 20:12 toyofuku@deploy1003: Finished scap sync-world: Backport for Deploy new donate link location to pilot wikis (take 2) (T373585) (duration: 08m 35s)
• 20:08 toyofuku@deploy1003: toyofuku: Continuing with sync
• 20:06 toyofuku@deploy1003: toyofuku: Backport for Deploy new donate link location to pilot wikis (take 2) (T373585) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:04 toyofuku@deploy1003: Started scap sync-world: Backport for Deploy new donate link location to pilot wikis (take 2) (T373585)
• 18:38 jnuche@deploy1003: Finished scap sync-world: Backport for DiscussionParser: Do not create User objects from subpages (T375212) (duration: 06m 38s)
• 18:33 jnuche@deploy1003: jnuche, kharlan: Continuing with sync
• 18:33 jnuche@deploy1003: jnuche, kharlan: Backport for DiscussionParser: Do not create User objects from subpages (T375212) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 18:31 jnuche@deploy1003: Started scap sync-world: Backport for DiscussionParser: Do not create User objects from subpages (T375212)
• 18:23 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:23 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove entries for sretest2002 - cmooney@cumin1002"
• 18:23 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove entries for sretest2002 - cmooney@cumin1002"
• 18:20 cmooney@cumin1002: START - Cookbook sre.dns.netbox
• 18:02 swfrench-wmf: scaling down mw-api-ext in eqiad after pre-switchover testing - T371273
• 18:02 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
• 18:02 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
• 18:02 swfrench-wmf: scaling down mw-web in eqiad after pre-switchover testing - T371273
• 18:01 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 18:01 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 18:01 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on vrts1001.eqiad.wmnet with reason: Migration
• 18:01 aokoth@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on vrts1001.eqiad.wmnet with reason: Migration
• 17:53 swfrench@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=codfw [reason: Reverting pre-switchover capacity validation - T371273]
• 17:51 brett@cumin2002: conftool action : set/pooled=yes; selector: name=cp5024.eqsin.wmnet
• 17:49 swfrench@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=codfw [reason: Reverting pre-switchover capacity validation - T371273]
• 17:47 dancy@deploy1003: Installation of scap version "4.104.0" completed for 211 hosts
• 17:46 swfrench@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=mw-api-int-ro,name=codfw [reason: Reverting pre-switchover capacity validation - T371273]
• 17:43 dancy@deploy1003: Installing scap version "4.104.0" for 211 hosts
• 17:38 brett@cumin2002: conftool action : set/pooled=no; selector: name=cp5024.eqsin.wmnet
• 17:27 Dreamy_Jazz: Finished running script for T375203 on `group0`
• 17:21 mforns@deploy1003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
• 17:21 mforns@deploy1003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
• 17:17 swfrench@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=mw-web-ro,name=codfw [reason: Pre-switchover capacity validation - T371273]
• 17:11 arnaudb@cumin1002: dbctl commit (dc=all): 'es2040 (re)pooling @ 100%: T373105', diff saved to https://phabricator.wikimedia.org/P69376 and previous config saved to /var/cache/conftool/dbconfig/20240919-171122-arnaudb.json
• 17:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 100%: T373105', diff saved to https://phabricator.wikimedia.org/P69375 and previous config saved to /var/cache/conftool/dbconfig/20240919-171117-arnaudb.json
• 17:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 100%: T373105', diff saved to https://phabricator.wikimedia.org/P69374 and previous config saved to /var/cache/conftool/dbconfig/20240919-171112-arnaudb.json
• 17:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2195 (re)pooling @ 100%: T373105', diff saved to https://phabricator.wikimedia.org/P69373 and previous config saved to /var/cache/conftool/dbconfig/20240919-171107-arnaudb.json
• 17:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 100%: T373105', diff saved to https://phabricator.wikimedia.org/P69372 and previous config saved to /var/cache/conftool/dbconfig/20240919-171101-arnaudb.json
• 17:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2181 (re)pooling @ 100%: T373105', diff saved to https://phabricator.wikimedia.org/P69371 and previous config saved to /var/cache/conftool/dbconfig/20240919-171057-arnaudb.json
• 17:10 arnaudb@cumin1002: dbctl commit (dc=all): 'db2174 (re)pooling @ 100%: T373105', diff saved to https://phabricator.wikimedia.org/P69370 and previous config saved to /var/cache/conftool/dbconfig/20240919-171052-arnaudb.json
• 17:10 arnaudb@cumin1002: dbctl commit (dc=all): 'db2173 (re)pooling @ 100%: T373105', diff saved to https://phabricator.wikimedia.org/P69369 and previous config saved to /var/cache/conftool/dbconfig/20240919-171047-arnaudb.json
• 17:10 arnaudb@cumin1002: dbctl commit (dc=all): 'db2152 (re)pooling @ 100%: T373105', diff saved to https://phabricator.wikimedia.org/P69368 and previous config saved to /var/cache/conftool/dbconfig/20240919-171042-arnaudb.json
• 17:10 arnaudb@cumin1002: dbctl commit (dc=all): 'db2131 (re)pooling @ 100%: T373105', diff saved to https://phabricator.wikimedia.org/P69367 and previous config saved to /var/cache/conftool/dbconfig/20240919-171037-arnaudb.json
• 17:08 swfrench@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=codfw [reason: Pre-switchover capacity validation - T371273]
• 17:02 swfrench@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=mw-api-int-ro,name=codfw [reason: Pre-switchover capacity validation - T371273]
• 16:56 arnaudb@cumin1002: dbctl commit (dc=all): 'es2040 (re)pooling @ 75%: T373105', diff saved to https://phabricator.wikimedia.org/P69366 and previous config saved to /var/cache/conftool/dbconfig/20240919-165617-arnaudb.json
• 16:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 75%: T373105', diff saved to https://phabricator.wikimedia.org/P69365 and previous config saved to /var/cache/conftool/dbconfig/20240919-165611-arnaudb.json
• 16:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 75%: T373105', diff saved to https://phabricator.wikimedia.org/P69364 and previous config saved to /var/cache/conftool/dbconfig/20240919-165606-arnaudb.json
• 16:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2195 (re)pooling @ 75%: T373105', diff saved to https://phabricator.wikimedia.org/P69363 and previous config saved to /var/cache/conftool/dbconfig/20240919-165602-arnaudb.json
• 16:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 75%: T373105', diff saved to https://phabricator.wikimedia.org/P69362 and previous config saved to /var/cache/conftool/dbconfig/20240919-165556-arnaudb.json
• 16:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db2181 (re)pooling @ 75%: T373105', diff saved to https://phabricator.wikimedia.org/P69361 and previous config saved to /var/cache/conftool/dbconfig/20240919-165551-arnaudb.json
• 16:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db2174 (re)pooling @ 75%: T373105', diff saved to https://phabricator.wikimedia.org/P69360 and previous config saved to /var/cache/conftool/dbconfig/20240919-165546-arnaudb.json
• 16:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db2173 (re)pooling @ 75%: T373105', diff saved to https://phabricator.wikimedia.org/P69359 and previous config saved to /var/cache/conftool/dbconfig/20240919-165542-arnaudb.json
• 16:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db2152 (re)pooling @ 75%: T373105', diff saved to https://phabricator.wikimedia.org/P69358 and previous config saved to /var/cache/conftool/dbconfig/20240919-165537-arnaudb.json
• 16:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db2131 (re)pooling @ 75%: T373105', diff saved to https://phabricator.wikimedia.org/P69357 and previous config saved to /var/cache/conftool/dbconfig/20240919-165531-arnaudb.json
• 16:52 sukhe: ulsfo was depooled between 15:55 and 16:12 for sre.dns.admin test, current state is pooled
• 16:50 swfrench-wmf: scaling up mw-web in eqiad for pre-switchover testing - T371273
• 16:50 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 16:50 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 16:48 swfrench-wmf: scaling up mw-api-ext in eqiad for pre-switchover testing - T371273
• 16:48 vgutierrez: updated to purged 0.24 in codfw, ulsfo and eqsin - T334078
• 16:47 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
• 16:47 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
• 16:44 vgutierrez: uploaded purged 0.24 to apt.wm.o (bullseye-wikimedia)
• 16:41 arnaudb@cumin1002: dbctl commit (dc=all): 'es2040 (re)pooling @ 50%: T373105', diff saved to https://phabricator.wikimedia.org/P69356 and previous config saved to /var/cache/conftool/dbconfig/20240919-164111-arnaudb.json
• 16:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 50%: T373105', diff saved to https://phabricator.wikimedia.org/P69355 and previous config saved to /var/cache/conftool/dbconfig/20240919-164106-arnaudb.json
• 16:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 50%: T373105', diff saved to https://phabricator.wikimedia.org/P69354 and previous config saved to /var/cache/conftool/dbconfig/20240919-164101-arnaudb.json
• 16:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db2195 (re)pooling @ 50%: T373105', diff saved to https://phabricator.wikimedia.org/P69353 and previous config saved to /var/cache/conftool/dbconfig/20240919-164056-arnaudb.json
• 16:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 50%: T373105', diff saved to https://phabricator.wikimedia.org/P69352 and previous config saved to /var/cache/conftool/dbconfig/20240919-164051-arnaudb.json
• 16:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db2181 (re)pooling @ 50%: T373105', diff saved to https://phabricator.wikimedia.org/P69351 and previous config saved to /var/cache/conftool/dbconfig/20240919-164046-arnaudb.json
• 16:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db2174 (re)pooling @ 50%: T373105', diff saved to https://phabricator.wikimedia.org/P69350 and previous config saved to /var/cache/conftool/dbconfig/20240919-164041-arnaudb.json
• 16:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db2173 (re)pooling @ 50%: T373105', diff saved to https://phabricator.wikimedia.org/P69349 and previous config saved to /var/cache/conftool/dbconfig/20240919-164036-arnaudb.json
• 16:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db2152 (re)pooling @ 50%: T373105', diff saved to https://phabricator.wikimedia.org/P69348 and previous config saved to /var/cache/conftool/dbconfig/20240919-164031-arnaudb.json
• 16:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db2131 (re)pooling @ 50%: T373105', diff saved to https://phabricator.wikimedia.org/P69347 and previous config saved to /var/cache/conftool/dbconfig/20240919-164026-arnaudb.json
• 16:38 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host parse2020.codfw.wmnet
• 16:38 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host parse2020.codfw.wmnet
• 16:38 topranks: disable LAG interface from asw-d-codfw to ssw1-dX-codfw
• 16:38 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host parse2019.codfw.wmnet
• 16:38 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host parse2019.codfw.wmnet
• 16:37 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host parse2018.codfw.wmnet
• 16:37 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host parse2018.codfw.wmnet
• 16:37 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2282.codfw.wmnet
• 16:37 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2282.codfw.wmnet
• 16:37 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2053.codfw.wmnet
• 16:37 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2053.codfw.wmnet
• 16:37 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2052.codfw.wmnet
• 16:36 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2052.codfw.wmnet
• 16:34 effie: restarting confd on all servers on: codfw, ulsof, eqsin - T373105
• 16:34 vgutierrez: testing purged 0.24 in cp2037 - T334078
• 16:27 Emperor: restart swift-object-replicator on thanos-be2002
• 16:27 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=(cp2041|cp2042).codfw.wmnet [reason: T373105 is done]
• 16:26 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2042.codfw.wmnet
• 16:26 sukhe@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp2042.codfw.wmnet
• 16:26 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2041.codfw.wmnet
• 16:26 sukhe@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp2041.codfw.wmnet
• 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'es2040 (re)pooling @ 25%: T373105', diff saved to https://phabricator.wikimedia.org/P69346 and previous config saved to /var/cache/conftool/dbconfig/20240919-162606-arnaudb.json
• 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 25%: T373105', diff saved to https://phabricator.wikimedia.org/P69345 and previous config saved to /var/cache/conftool/dbconfig/20240919-162601-arnaudb.json
• 16:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 25%: T373105', diff saved to https://phabricator.wikimedia.org/P69344 and previous config saved to /var/cache/conftool/dbconfig/20240919-162556-arnaudb.json
• 16:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2195 (re)pooling @ 25%: T373105', diff saved to https://phabricator.wikimedia.org/P69343 and previous config saved to /var/cache/conftool/dbconfig/20240919-162551-arnaudb.json
• 16:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 25%: T373105', diff saved to https://phabricator.wikimedia.org/P69342 and previous config saved to /var/cache/conftool/dbconfig/20240919-162546-arnaudb.json
• 16:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2181 (re)pooling @ 25%: T373105', diff saved to https://phabricator.wikimedia.org/P69341 and previous config saved to /var/cache/conftool/dbconfig/20240919-162541-arnaudb.json
• 16:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2174 (re)pooling @ 25%: T373105', diff saved to https://phabricator.wikimedia.org/P69340 and previous config saved to /var/cache/conftool/dbconfig/20240919-162536-arnaudb.json
• 16:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2173 (re)pooling @ 25%: T373105', diff saved to https://phabricator.wikimedia.org/P69339 and previous config saved to /var/cache/conftool/dbconfig/20240919-162531-arnaudb.json
• 16:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2152 (re)pooling @ 25%: T373105', diff saved to https://phabricator.wikimedia.org/P69338 and previous config saved to /var/cache/conftool/dbconfig/20240919-162526-arnaudb.json
• 16:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2131 (re)pooling @ 25%: T373105', diff saved to https://phabricator.wikimedia.org/P69337 and previous config saved to /var/cache/conftool/dbconfig/20240919-162521-arnaudb.json
• 16:17 topranks: migrating server uplinks in codfw rack D8 to new top-of-rack switch T373105
• 16:16 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:25:00 on 25 hosts with reason: Move server uplinks in codfw rack D8
• 16:16 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:25:00 on 25 hosts with reason: Move server uplinks in codfw rack D8
• 16:13 Dreamy_Jazz: Running `foreachwikiindblist group0.dblist extensions/CheckUser/maintenance/populateCentralCheckUserIndexTables.php` on a tmux session for T375203
• 16:12 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site ulsfo [reason: testing done, no task ID specified]
• 16:12 sukhe@cumin1002: START - Cookbook sre.dns.admin DNS admin: pool site ulsfo [reason: testing done, no task ID specified]
• 16:11 dreamyjazz@deploy1003: Finished scap sync-world: Backport for Call require_once on CheckUserQueryInterface in population script (T375203) (duration: 06m 47s)
• 16:11 topranks: migrating server uplinks in codfw rack D7 to new top-of-rack switch T373105
• 16:09 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 22 hosts with reason: Move server uplinks in codfw rack D7
• 16:08 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on 22 hosts with reason: Move server uplinks in codfw rack D7
• 16:07 dreamyjazz@deploy1003: dreamyjazz: Continuing with sync
• 16:06 dreamyjazz@deploy1003: dreamyjazz: Backport for Call require_once on CheckUserQueryInterface in population script (T375203) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 16:04 dreamyjazz@deploy1003: Started scap sync-world: Backport for Call require_once on CheckUserQueryInterface in population script (T375203)
• 16:04 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-eqiad
• 15:59 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cr2-eqiad
• 15:58 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw1-d5-eqiad
• 15:56 mforns@deploy1003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
• 15:56 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cloudsw1-d5-eqiad
• 15:55 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site ulsfo [reason: testing cookbook for actual change, no task ID specified]
• 15:55 sukhe@cumin1002: START - Cookbook sre.dns.admin DNS admin: depool site ulsfo [reason: testing cookbook for actual change, no task ID specified]
• 15:55 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr1-eqiad
• 15:51 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cr1-eqiad
• 15:49 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw1-c8-eqiad
• 15:47 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device cloudsw1-c8-eqiad
• 15:46 mforns@deploy1003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
• 15:45 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: sync
• 15:45 elukey@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: sync
• 15:45 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: sync
• 15:45 elukey@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: sync
• 15:44 elukey@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: sync
• 15:44 elukey@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: sync
• 15:41 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse2020.codfw.wmnet
• 15:40 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host parse2020.codfw.wmnet
• 15:40 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse2019.codfw.wmnet
• 15:39 mforns@deploy1003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
• 15:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 10 hosts with reason: network maintenance T373105
• 15:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on 10 hosts with reason: network maintenance T373105
• 15:38 arnaudb@cumin1002: dbctl commit (dc=all): 'depool db2131 db2152 db2173 db2174 db2181 db2182 db2195 db2219 db2220 es2040 - T373105', diff saved to https://phabricator.wikimedia.org/P69336 and previous config saved to /var/cache/conftool/dbconfig/20240919-153815-arnaudb.json
• 15:37 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host parse2019.codfw.wmnet
• 15:37 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse2018.codfw.wmnet
• 15:36 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host parse2018.codfw.wmnet
• 15:36 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2282.codfw.wmnet
• 15:33 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2282.codfw.wmnet
• 15:33 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2053.codfw.wmnet
• 15:32 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2053.codfw.wmnet
• 15:32 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2052.codfw.wmnet
• 15:31 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2052.codfw.wmnet
• 15:29 mforns@deploy1003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
• 15:26 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f1-eqiad
• 15:24 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-f1-eqiad
• 15:22 sukhe@cumin1002: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool site ulsfo [reason: testing cookbook for actual change, no task ID specified]
• 15:22 sukhe@cumin1002: START - Cookbook sre.dns.admin DNS admin: depool site ulsfo [reason: testing cookbook for actual change, no task ID specified]
• 15:09 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=(cp2041|cp2042).codfw.wmnet [reason: depool for T373105]
• 15:08 dreamyjazz@deploy1003: Finished scap sync-world: Backport for Add CheckUserQueryInterface to autoload classes (T375203), Add CheckUserQueryInterface to autoload classes (T375203) (duration: 07m 18s)
• 15:04 dreamyjazz@deploy1003: dreamyjazz: Continuing with sync
• 15:04 dreamyjazz@deploy1003: dreamyjazz: Backport for Add CheckUserQueryInterface to autoload classes (T375203), Add CheckUserQueryInterface to autoload classes (T375203) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 15:02 elukey@cumin1002: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-f1-eqiad
• 15:02 elukey@cumin1002: START - Cookbook sre.network.tls for network device lsw1-f1-eqiad
• 15:01 dreamyjazz@deploy1003: Started scap sync-world: Backport for Add CheckUserQueryInterface to autoload classes (T375203), Add CheckUserQueryInterface to autoload classes (T375203)
• 14:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 depool T373579', diff saved to https://phabricator.wikimedia.org/P69333 and previous config saved to /var/cache/conftool/dbconfig/20240919-145626-arnaudb.json
• 14:56 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:55 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2129.codfw.wmnet
• 14:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2129.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 14:53 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2129.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 14:50 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 14:46 moritzm: installing expat security updates on Bookworm
• 14:45 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2129.codfw.wmnet
• 14:40 tappof: manual replace of mtail binary on centrallog2002 (3.0.0-rc50 to 3.0.8) T375085
• 14:25 sukhe: sudo cumin -b11 "A:cp" 'run-puppet-agent --enable "merging CR 1054918"'
• 14:24 Lucas_WMDE: UTC afternoon backport+config window done
• 14:24 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for Revert^2 "Activate feature flag for moving wikibase item to Other Projects sidebar in pilot wikis." (T66315) (duration: 18m 31s)
• 14:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 depool T373579', diff saved to https://phabricator.wikimedia.org/P69332 and previous config saved to /var/cache/conftool/dbconfig/20240919-142046-arnaudb.json
• 14:19 lucaswerkmeister-wmde@deploy1003: seanleong-wmde, lucaswerkmeister-wmde: Continuing with sync
• 14:11 lucaswerkmeister-wmde@deploy1003: seanleong-wmde, lucaswerkmeister-wmde: Backport for Revert^2 "Activate feature flag for moving wikibase item to Other Projects sidebar in pilot wikis." (T66315) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:05 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for Revert^2 "Activate feature flag for moving wikibase item to Other Projects sidebar in pilot wikis." (T66315)
• 14:03 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for Bring back quality colors before dark mode fixes (T375114) (duration: 12m 39s)
• 13:59 sukhe: sudo cumin "A:cp" 'disable-puppet "merging CR 1054918"'
• 13:58 elukey@cumin1002: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-f1-eqiad
• 13:58 elukey@cumin1002: START - Cookbook sre.network.tls for network device lsw1-f1-eqiad
• 13:58 lucaswerkmeister-wmde@deploy1003: soda, lucaswerkmeister-wmde: Continuing with sync
• 13:57 elukey@cumin1002: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-f1-eqiad
• 13:57 elukey@cumin1002: START - Cookbook sre.network.tls for network device lsw1-f1-eqiad
• 13:57 lucaswerkmeister-wmde@deploy1003: soda, lucaswerkmeister-wmde: Backport for Bring back quality colors before dark mode fixes (T375114) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:50 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for Bring back quality colors before dark mode fixes (T375114)
• 13:42 XioNoX: update pfw codfw syslog target - T374658
• 13:28 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ipoid: apply
• 13:28 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/ipoid: apply
• 11:34 ayounsi@cumin2002: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-f1-eqiad
• 11:34 ayounsi@cumin2002: START - Cookbook sre.network.tls for network device lsw1-f1-eqiad
• 11:31 ayounsi@cumin1002: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-f1-eqiad
• 11:31 ayounsi@cumin1002: START - Cookbook sre.network.tls for network device lsw1-f1-eqiad
• 10:55 xSavitar: T375078 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=bhwiki --logwiki=metawiki 'SikAnderAhmedas' 'Renamed user ab8e0a47aa0e5d456f28ee3977f8c682'
• 10:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 10:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 10:30 elukey@deploy1003: Finished scap sync-world: Remove network policies for old poolcounter nodes. (duration: 08m 55s)
• 10:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ipoid: apply
• 10:26 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/ipoid: apply
• 10:26 XioNoX: enable gNMI on cloudsw
• 10:26 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ipoid: apply
• 10:26 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/ipoid: apply
• 10:23 moritzm: rebalance ganeti group C following the various switch maintenances T370630
• 10:21 elukey@deploy1003: Started scap sync-world: Remove network policies for old poolcounter nodes.
• 09:48 volans: uploaded python3-wmflib_1.2.7 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia,bookworm-wikimedia
• 09:44 vgutierrez: testing purged 0.24 in cp4038 - T334078
• 09:44 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ipoid: apply
• 09:44 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/ipoid: apply
• 09:42 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ipoid: apply
• 09:42 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/ipoid: apply
• 09:42 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/ipoid: apply
• 09:41 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/ipoid: apply
• 09:23 arnaudb@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db1246.eqiad.wmnet with OS bookworm
• 09:23 btullis@cumin1002: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
• 09:10 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm
• 09:10 arnaudb@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db1246.eqiad.wmnet with OS bookworm
• 08:51 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm
• 08:50 arnaudb@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db1246.eqiad.wmnet with OS bookworm
• 08:45 hashar: Restarting CI Jenkins with Java 17 # T359795
• 08:31 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm
• 08:31 _joe_: deployed conftool 3.2.4 T375059 T373449
• 08:30 ayounsi@cumin1002: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
• 08:29 ayounsi@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
• 08:16 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.43.0-wmf.23 refs T373642
• 08:04 ayounsi@cumin1002: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
• 08:04 ayounsi@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
• 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2018.codfw.wmnet
• 07:48 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2018.codfw.wmnet
• 07:43 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 16509
• 07:38 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 16509
• 07:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2229 (re)pooling @ 100%: post maintenance', diff saved to https://phabricator.wikimedia.org/P69318 and previous config saved to /var/cache/conftool/dbconfig/20240919-073543-arnaudb.json
• 07:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2229 (re)pooling @ 75%: post maintenance', diff saved to https://phabricator.wikimedia.org/P69317 and previous config saved to /var/cache/conftool/dbconfig/20240919-072037-arnaudb.json
• 07:19 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 16509
• 07:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2229 (re)pooling @ 50%: post maintenance', diff saved to https://phabricator.wikimedia.org/P69316 and previous config saved to /var/cache/conftool/dbconfig/20240919-070532-arnaudb.json
• 06:53 moritzm: adding Tiziano to pwstore
• 06:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2229 (re)pooling @ 25%: post maintenance', diff saved to https://phabricator.wikimedia.org/P69315 and previous config saved to /var/cache/conftool/dbconfig/20240919-065026-arnaudb.json
• 06:47 moritzm: cleanup some old Bacula restores (4G) on seaborgium
• 06:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2229 (re)pooling @ 15%: post maintenance', diff saved to https://phabricator.wikimedia.org/P69314 and previous config saved to /var/cache/conftool/dbconfig/20240919-063521-arnaudb.json
• 06:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2229 (re)pooling @ 10%: post maintenance', diff saved to https://phabricator.wikimedia.org/P69313 and previous config saved to /var/cache/conftool/dbconfig/20240919-062016-arnaudb.json
• 06:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2229 (re)pooling @ 5%: post maintenance', diff saved to https://phabricator.wikimedia.org/P69312 and previous config saved to /var/cache/conftool/dbconfig/20240919-060510-arnaudb.json
• 05:01 eileen: civicrm upgraded from ac29ff45 to 8af371aa
• 01:25 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 01:25 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns name for frack new switches - pt1979@cumin2002"
• 01:24 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns name for frack new switches - pt1979@cumin2002"
• 01:21 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 00:46 sukhe: sudo cumin 'puppetserver1003* or puppetserver2003*' 'systemctl start sync-puppet-volatile.service'
• 00:45 sukhe: sukhe@puppetserver1002:~$ sudo systemctl start sync-puppet-volatile.service
• 00:41 swfrench-wmf: force-reboot of puppetserver1001 via ipmitool (unresponsive for over 30m)

2024-09-18

• 22:43 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: sync
• 22:43 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: sync
• 22:19 jynus: inserting without binlog missing heartbeat reecod on x1 codfw hosts
• 22:11 ladsgroup@cumin1002: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw
• 21:55 mutante: seaborgium - apt-get clean (disk space before: 98% used, now: 76% used, was alerting)
• 20:59 ladsgroup@cumin1002: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw
• 20:45 toyofuku@deploy1003: Finished scap sync-world: Backport for Enable dark mode for all logged in users on all projects (T370099), Deploy Vector 2022 on several Wikimedia wikis (T374255), Limit quick surveys to wikis with messages defined (T374654) (duration: 12m 52s)
• 20:40 toyofuku@deploy1003: toyofuku, jdlrobson: Continuing with sync
• 20:35 toyofuku@deploy1003: toyofuku, jdlrobson: Backport for Enable dark mode for all logged in users on all projects (T370099), Deploy Vector 2022 on several Wikimedia wikis (T374255), Limit quick surveys to wikis with messages defined (T374654) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:32 toyofuku@deploy1003: Started scap sync-world: Backport for Enable dark mode for all logged in users on all projects (T370099), Deploy Vector 2022 on several Wikimedia wikis (T374255), Limit quick surveys to wikis with messages defined (T374654)
• 18:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 100%: T375050', diff saved to https://phabricator.wikimedia.org/P69310 and previous config saved to /var/cache/conftool/dbconfig/20240918-180800-arnaudb.json
• 17:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for Revert^2 "Create group for assigning checkuser-temporary-account right" (T369187) (duration: 08m 18s)
• 17:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 75%: T375050', diff saved to https://phabricator.wikimedia.org/P69309 and previous config saved to /var/cache/conftool/dbconfig/20240918-175255-arnaudb.json
• 17:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with sync
• 17:49 dreamyjazz@deploy1003: dreamyjazz: Backport for Revert^2 "Create group for assigning checkuser-temporary-account right" (T369187) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 17:47 dreamyjazz@deploy1003: Started scap sync-world: Backport for Revert^2 "Create group for assigning checkuser-temporary-account right" (T369187)
• 17:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 50%: T375050', diff saved to https://phabricator.wikimedia.org/P69308 and previous config saved to /var/cache/conftool/dbconfig/20240918-173749-arnaudb.json
• 17:29 sukhe: re-enable puppet on A:cp to finish rolling out T347114
• 17:29 sukhe: re-enable puppet on A:cp to finish rolling out T368755
• 17:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 25%: T375050', diff saved to https://phabricator.wikimedia.org/P69306 and previous config saved to /var/cache/conftool/dbconfig/20240918-172243-arnaudb.json
• 17:09 arnaudb@cumin1002: dbctl commit (dc=all): 'db2217 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69305 and previous config saved to /var/cache/conftool/dbconfig/20240918-170918-arnaudb.json
• 17:09 arnaudb@cumin1002: dbctl commit (dc=all): 'db2216 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69304 and previous config saved to /var/cache/conftool/dbconfig/20240918-170913-arnaudb.json
• 17:09 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69303 and previous config saved to /var/cache/conftool/dbconfig/20240918-170909-arnaudb.json
• 17:09 arnaudb@cumin1002: dbctl commit (dc=all): 'db2194 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69302 and previous config saved to /var/cache/conftool/dbconfig/20240918-170903-arnaudb.json
• 17:09 arnaudb@cumin1002: dbctl commit (dc=all): 'db2193 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69301 and previous config saved to /var/cache/conftool/dbconfig/20240918-170858-arnaudb.json
• 17:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69300 and previous config saved to /var/cache/conftool/dbconfig/20240918-170849-arnaudb.json
• 17:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2140 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69299 and previous config saved to /var/cache/conftool/dbconfig/20240918-170843-arnaudb.json
• 17:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2130 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69298 and previous config saved to /var/cache/conftool/dbconfig/20240918-170838-arnaudb.json
• 17:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69297 and previous config saved to /var/cache/conftool/dbconfig/20240918-170833-arnaudb.json
• 17:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 15%: T375050', diff saved to https://phabricator.wikimedia.org/P69296 and previous config saved to /var/cache/conftool/dbconfig/20240918-170738-arnaudb.json
• 17:02 mutante: copied vopsbot.db from alert1001 to alert1002; restarted vopsbot
• 16:57 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
• 16:56 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
• 16:56 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
• 16:56 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
• 16:54 arnaudb@cumin1002: dbctl commit (dc=all): 'db2217 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69294 and previous config saved to /var/cache/conftool/dbconfig/20240918-165412-arnaudb.json
• 16:54 arnaudb@cumin1002: dbctl commit (dc=all): 'db2216 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69293 and previous config saved to /var/cache/conftool/dbconfig/20240918-165407-arnaudb.json
• 16:54 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69292 and previous config saved to /var/cache/conftool/dbconfig/20240918-165403-arnaudb.json
• 16:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2194 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69291 and previous config saved to /var/cache/conftool/dbconfig/20240918-165357-arnaudb.json
• 16:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2193 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69290 and previous config saved to /var/cache/conftool/dbconfig/20240918-165352-arnaudb.json
• 16:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69289 and previous config saved to /var/cache/conftool/dbconfig/20240918-165344-arnaudb.json
• 16:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2140 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69288 and previous config saved to /var/cache/conftool/dbconfig/20240918-165337-arnaudb.json
• 16:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2130 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69287 and previous config saved to /var/cache/conftool/dbconfig/20240918-165332-arnaudb.json
• 16:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69286 and previous config saved to /var/cache/conftool/dbconfig/20240918-165327-arnaudb.json
• 16:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 10%: T375050', diff saved to https://phabricator.wikimedia.org/P69285 and previous config saved to /var/cache/conftool/dbconfig/20240918-165232-arnaudb.json
• 16:42 sukhe: sudo cumin "A:cp" 'disable-puppet "merging CR 1073453"': T347114
• 16:40 dreamyjazz@deploy1003: Finished scap sync-world: Backport for Autopromote users into checkuser-temporary-account-viewer (T369187 T327913) (duration: 14m 06s)
• 16:39 arnaudb@cumin1002: dbctl commit (dc=all): 'db2217 (re)pooling @ 50%: T373104', diff saved to https://phabricator.wikimedia.org/P69284 and previous config saved to /var/cache/conftool/dbconfig/20240918-163907-arnaudb.json
• 16:39 arnaudb@cumin1002: dbctl commit (dc=all): 'db2216 (re)pooling @ 50%: T373104', diff saved to https://phabricator.wikimedia.org/P69283 and previous config saved to /var/cache/conftool/dbconfig/20240918-163902-arnaudb.json
• 16:39 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 50%: T373104', diff saved to https://phabricator.wikimedia.org/P69282 and previous config saved to /var/cache/conftool/dbconfig/20240918-163857-arnaudb.json
• 16:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2194 (re)pooling @ 50%: T373104', diff saved to https://phabricator.wikimedia.org/P69281 and previous config saved to /var/cache/conftool/dbconfig/20240918-163852-arnaudb.json
• 16:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2193 (re)pooling @ 50%: T373104', diff saved to https://phabricator.wikimedia.org/P69280 and previous config saved to /var/cache/conftool/dbconfig/20240918-163847-arnaudb.json
• 16:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 50%: T373104', diff saved to https://phabricator.wikimedia.org/P69279 and previous config saved to /var/cache/conftool/dbconfig/20240918-163837-arnaudb.json
• 16:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2140 (re)pooling @ 50%: T373104', diff saved to https://phabricator.wikimedia.org/P69278 and previous config saved to /var/cache/conftool/dbconfig/20240918-163832-arnaudb.json
• 16:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2130 (re)pooling @ 50%: T373104', diff saved to https://phabricator.wikimedia.org/P69277 and previous config saved to /var/cache/conftool/dbconfig/20240918-163827-arnaudb.json
• 16:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 50%: T373104', diff saved to https://phabricator.wikimedia.org/P69276 and previous config saved to /var/cache/conftool/dbconfig/20240918-163822-arnaudb.json
• 16:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 5%: T375050', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20240918-163721-arnaudb.json
• 16:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 T375050', diff saved to https://phabricator.wikimedia.org/P69275 and previous config saved to /var/cache/conftool/dbconfig/20240918-163637-arnaudb.json
• 16:35 dreamyjazz@deploy1003: dreamyjazz: Continuing with sync
• 16:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2218 to s7 primary T375050', diff saved to https://phabricator.wikimedia.org/P69274 and previous config saved to /var/cache/conftool/dbconfig/20240918-163404-arnaudb.json
• 16:33 arnaudb: Starting s7 codfw failover from db2220 to db2218 - T375050
• 16:30 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host parse2017.codfw.wmnet
• 16:30 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host parse2017.codfw.wmnet
• 16:29 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host parse2016.codfw.wmnet
• 16:29 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host parse2016.codfw.wmnet
• 16:29 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2451.codfw.wmnet
• 16:29 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2451.codfw.wmnet
• 16:29 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2450.codfw.wmnet
• 16:29 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2450.codfw.wmnet
• 16:28 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2449.codfw.wmnet
• 16:28 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2449.codfw.wmnet
• 16:28 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2448.codfw.wmnet
• 16:28 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2448.codfw.wmnet
• 16:28 dreamyjazz@deploy1003: dreamyjazz: Backport for Autopromote users into checkuser-temporary-account-viewer (T369187 T327913) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 16:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Remove db2218 from API/vslow/dump T375050', diff saved to https://phabricator.wikimedia.org/P69273 and previous config saved to /var/cache/conftool/dbconfig/20240918-162822-arnaudb.json
• 16:28 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2447.codfw.wmnet
• 16:28 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2447.codfw.wmnet
• 16:28 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2446.codfw.wmnet
• 16:27 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2446.codfw.wmnet
• 16:27 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2445.codfw.wmnet
• 16:27 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2445.codfw.wmnet
• 16:27 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2444.codfw.wmnet
• 16:27 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2444.codfw.wmnet
• 16:27 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2051.codfw.wmnet
• 16:27 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2051.codfw.wmnet
• 16:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2218 with weight 0 T375050', diff saved to https://phabricator.wikimedia.org/P69272 and previous config saved to /var/cache/conftool/dbconfig/20240918-162703-arnaudb.json
• 16:26 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2050.codfw.wmnet
• 16:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 T375050
• 16:26 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2050.codfw.wmnet
• 16:26 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2049.codfw.wmnet
• 16:26 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2049.codfw.wmnet
• 16:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s7 T375050
• 16:26 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2048.codfw.wmnet
• 16:26 dreamyjazz@deploy1003: Started scap sync-world: Backport for Autopromote users into checkuser-temporary-account-viewer (T369187 T327913)
• 16:26 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2048.codfw.wmnet
• 16:25 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2024.codfw.wmnet
• 16:25 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2024.codfw.wmnet
• 16:25 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2014.codfw.wmnet
• 16:25 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2014.codfw.wmnet
• 16:24 arnaudb@cumin1002: dbctl commit (dc=all): 'db2218 (re)pooling @ 25%: T373104', diff saved to https://phabricator.wikimedia.org/P69271 and previous config saved to /var/cache/conftool/dbconfig/20240918-162406-arnaudb.json
• 16:24 arnaudb@cumin1002: dbctl commit (dc=all): 'db2217 (re)pooling @ 25%: T373104', diff saved to https://phabricator.wikimedia.org/P69270 and previous config saved to /var/cache/conftool/dbconfig/20240918-162401-arnaudb.json
• 16:24 arnaudb@cumin1002: dbctl commit (dc=all): 'db2216 (re)pooling @ 25%: T373104', diff saved to https://phabricator.wikimedia.org/P69269 and previous config saved to /var/cache/conftool/dbconfig/20240918-162357-arnaudb.json
• 16:23 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2013.codfw.wmnet
• 16:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 25%: T373104', diff saved to https://phabricator.wikimedia.org/P69268 and previous config saved to /var/cache/conftool/dbconfig/20240918-162351-arnaudb.json
• 16:23 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2013.codfw.wmnet
• 16:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2194 (re)pooling @ 25%: T373104', diff saved to https://phabricator.wikimedia.org/P69267 and previous config saved to /var/cache/conftool/dbconfig/20240918-162346-arnaudb.json
• 16:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2193 (re)pooling @ 25%: T373104', diff saved to https://phabricator.wikimedia.org/P69266 and previous config saved to /var/cache/conftool/dbconfig/20240918-162341-arnaudb.json
• 16:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 25%: T373104', diff saved to https://phabricator.wikimedia.org/P69265 and previous config saved to /var/cache/conftool/dbconfig/20240918-162331-arnaudb.json
• 16:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2140 (re)pooling @ 25%: T373104', diff saved to https://phabricator.wikimedia.org/P69264 and previous config saved to /var/cache/conftool/dbconfig/20240918-162326-arnaudb.json
• 16:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2130 (re)pooling @ 25%: T373104', diff saved to https://phabricator.wikimedia.org/P69263 and previous config saved to /var/cache/conftool/dbconfig/20240918-162321-arnaudb.json
• 16:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 25%: T373104', diff saved to https://phabricator.wikimedia.org/P69262 and previous config saved to /var/cache/conftool/dbconfig/20240918-162316-arnaudb.json
• 16:07 topranks: moving servers in codfw rack D6 from asw-d6-codfw to lsw1-d6-codfw T373104
• 16:06 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:25:00 on 24 hosts with reason: Move servers in codfw rack D6
• 16:06 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:25:00 on 24 hosts with reason: Move servers in codfw rack D6
• 16:01 hashar@deploy1003: Finished scap sync-world: Update termbox (mul support) - T373088 (duration: 06m 48s)
• 16:00 topranks: moving servers in codfw rack D5 from asw-d5-codfw to lsw1-d5-codfw T373104
• 15:58 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 23 hosts with reason: Move servers in codfw rack D5
• 15:58 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on 23 hosts with reason: Move servers in codfw rack D5
• 15:55 elukey: deploy python3-setuptools upgrades fleetwide
• 15:54 hashar@deploy1003: Started scap sync-world: Update termbox (mul support) - T373088
• 15:40 sukhe: finished updating TLS1.3 ciphers for cp hosts: T365327
• 15:40 bking@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rdf-streaming-updater: apply
• 15:40 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 12 hosts with reason: network maintenance T373104
• 15:40 bking@deploy1003: helmfile [eqiad] START helmfile.d/services/rdf-streaming-updater: apply
• 15:39 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on 12 hosts with reason: network maintenance T373104
• 15:39 arnaudb@cumin1002: dbctl commit (dc=all): 'depool db2129 db2130 db2140 db2172 db2187 db2193 db2194 db2215 db2216 db2217 db2218 - T373104', diff saved to https://phabricator.wikimedia.org/P69261 and previous config saved to /var/cache/conftool/dbconfig/20240918-153922-arnaudb.json
• 15:34 bking@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rdf-streaming-updater: apply
• 15:34 bking@deploy1003: helmfile [eqiad] START helmfile.d/services/rdf-streaming-updater: apply
• 15:32 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse2017.codfw.wmnet
• 15:31 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host parse2017.codfw.wmnet
• 15:31 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse2016.codfw.wmnet
• 15:30 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host parse2016.codfw.wmnet
• 15:30 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2451.codfw.wmnet
• 15:30 bking@deploy1003: helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply
• 15:30 bking@deploy1003: helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply
• 15:30 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2451.codfw.wmnet
• 15:30 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2450.codfw.wmnet
• 15:29 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2450.codfw.wmnet
• 15:29 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2449.codfw.wmnet
• 15:28 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2449.codfw.wmnet
• 15:28 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2448.codfw.wmnet
• 15:27 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2448.codfw.wmnet
• 15:27 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2447.codfw.wmnet
• 15:27 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2447.codfw.wmnet
• 15:27 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2446.codfw.wmnet
• 15:26 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2446.codfw.wmnet
• 15:26 bking@deploy1003: helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply
• 15:25 bking@deploy1003: helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply
• 15:22 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 15:22 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 15:21 denisse: Enable metamonitoring for the alert1002, and alert2002 hosts - T372418
• 15:20 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on vrts2001.codfw.wmnet with reason: Migration
• 15:20 aokoth@cumin1002: START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on vrts2001.codfw.wmnet with reason: Migration
• 15:20 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2445.codfw.wmnet
• 15:20 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for vrts2002.codfw.wmnet
• 15:20 aokoth@cumin1002: START - Cookbook sre.hosts.remove-downtime for vrts2002.codfw.wmnet
• 15:19 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2445.codfw.wmnet
• 15:19 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2444.codfw.wmnet
• 15:19 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 15:19 sukhe: rolling out TLS1.3 cipher suite priority order change CR 1073798 to all cp hosts
• 15:19 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2444.codfw.wmnet
• 15:19 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 15:19 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2051.codfw.wmnet
• 15:19 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on vrts2002.codfw.wmnet with reason: Migration
• 15:18 aokoth@cumin1002: START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on vrts2002.codfw.wmnet with reason: Migration
• 15:18 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2051.codfw.wmnet
• 15:18 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2050.codfw.wmnet
• 15:17 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2050.codfw.wmnet
• 15:17 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2049.codfw.wmnet
• 15:17 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2049.codfw.wmnet
• 15:16 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2048.codfw.wmnet
• 15:16 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2048.codfw.wmnet
• 15:16 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2024.codfw.wmnet
• 15:15 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2024.codfw.wmnet
• 15:15 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2014.codfw.wmnet
• 15:14 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2014.codfw.wmnet
• 15:14 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2013.codfw.wmnet
• 15:14 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2013.codfw.wmnet
• 15:08 denisse: Resolve alerts DNS queries to alert1002 - T372418
• 15:03 _joe_: uploading conftool 3.2.4 to apt T375059
• 15:02 sukhe: sudo cumin "A:cp" 'disable-puppet "merging CR 1073798"': T365327
• 15:01 denisse: Make alert1002 the active host - T372418
• 15:00 denisse: Disable meta-monitoring for the alert hosts - T372418
• 14:55 elukey: restart poolcounter on poolcounter100[4,5] (depooled nodes) to clear old/stale TCP conns for port 7531
• 14:54 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 14:54 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 14:54 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 14:54 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 14:53 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 55655
• 14:52 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 55655
• 14:50 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 14:49 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 14:47 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 14:46 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 14:45 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 14:45 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 14:42 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1052.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:40 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough and A:wikidough
• 14:36 elukey@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1052.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:26 sukhe@cumin1002: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough and A:wikidough
• 14:25 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 14:24 sukhe: run puppet agent on A:wikidough
• 14:23 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 14:19 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 14:19 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 14:07 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 14:07 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 13:53 elukey@deploy1003: Finished scap sync-world: Backport for Swap poolcounter1005 with poolcounter1007 (T332015) (duration: 07m 23s)
• 13:49 elukey@deploy1003: elukey: Continuing with sync
• 13:48 elukey@deploy1003: elukey: Backport for Swap poolcounter1005 with poolcounter1007 (T332015) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:46 elukey@deploy1003: Started scap sync-world: Backport for Swap poolcounter1005 with poolcounter1007 (T332015)
• 13:38 elukey@deploy1003: Finished scap sync-world: Backport for Swap poolcounter1004 with poolcounter1006 (T332015) (duration: 07m 15s)
• 13:34 elukey@deploy1003: elukey: Continuing with sync
• 13:33 elukey@deploy1003: elukey: Backport for Swap poolcounter1004 with poolcounter1006 (T332015) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:31 elukey@deploy1003: Started scap sync-world: Backport for Swap poolcounter1004 with poolcounter1006 (T332015)
• 13:25 Dreamy_Jazz: Afternoon UTC backport window done
• {{safesubst:SAL entry|1=13:20 dreamyjazz@deploy1003: Finished scap sync-world: Backport for GrowthExperiments: enable Community Updates module in testwiki (T374577), Check that throttling exceptions use valid public IP addresses (T374980), Hide temp account IP address viewing right from non-temp account wikis (T369187), [[gerrit:1073586|Lift IP cap on 2024-10-07/08 for edit-a-thon (T374964)]}}
• 13:18 elukey: restart puppetserver on puppetserver1002 - trashing - T373527
• 13:15 dreamyjazz@deploy1003: sgimeno, anzx, lucaswerkmeister-wmde, cscott, hnowlan, dreamyjazz: Continuing with sync
• {{safesubst:SAL entry|1=13:11 dreamyjazz@deploy1003: sgimeno, anzx, lucaswerkmeister-wmde, cscott, hnowlan, dreamyjazz: Backport for GrowthExperiments: enable Community Updates module in testwiki (T374577), Check that throttling exceptions use valid public IP addresses (T374980), Hide temp account IP address viewing right from non-temp account wikis (T369187), [[gerrit:1073586|Lift IP cap on}}
• 13:09 dreamyjazz@deploy1003: Started scap sync-world: Backport for GrowthExperiments: enable Community Updates module in testwiki (T374577), Check that throttling exceptions use valid public IP addresses (T374980), Hide temp account IP address viewing right from non-temp account wikis (T369187), Lift IP cap on 2024-10-07/08 for edit-a-thon (T374964)
• 12:46 vgutierrez: rolling upgrade to purged 0.23 in A:cp-ulsfo - T334078
• 12:44 vgutierrez: uploaded purged 0.23 to bullseye-wikimedia (apt.wm.o) - T334078
• 12:33 moritzm: uploaded cas 7.0.4.1+wmf12u3 T367487
• 12:28 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.43.0-wmf.23 refs T373642
• 12:21 tchin@deploy1003: Finished deploy [airflow-dags/analytics_test@e6cc31a]: Regular analytics weekly train (duration: 00m 20s)
• 12:21 tchin@deploy1003: Started deploy [airflow-dags/analytics_test@e6cc31a]: Regular analytics weekly train
• 12:18 tchin@deploy1003: Finished deploy [airflow-dags/analytics@e6cc31a]: Regular analytics weekly train (duration: 01m 18s)
• 12:18 tchin@deploy1003: Started deploy [airflow-dags/analytics@e6cc31a]: Regular analytics weekly train
• 12:12 dreamyjazz@deploy1003: Finished scap sync-world: Backport for Allow IP ranges in CentralAuth::getInstanceByName() (T375061), Allow IP ranges in CentralAuth::getInstanceByName() (T375061) (duration: 07m 00s)
• 12:10 tchin: Deployed refinery using scap, then deployed onto hdfs
• 12:08 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1177.eqiad.wmnet with OS bullseye
• 12:07 dreamyjazz@deploy1003: dreamyjazz: Continuing with sync
• 12:07 dreamyjazz@deploy1003: dreamyjazz: Backport for Allow IP ranges in CentralAuth::getInstanceByName() (T375061), Allow IP ranges in CentralAuth::getInstanceByName() (T375061) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 12:05 dreamyjazz@deploy1003: Started scap sync-world: Backport for Allow IP ranges in CentralAuth::getInstanceByName() (T375061), Allow IP ranges in CentralAuth::getInstanceByName() (T375061)
• 11:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for Hooks: Re-order checks to verify that request user is same as Special:Contributions user (T375061) (duration: 09m 03s)
• 11:50 dreamyjazz@deploy1003: dreamyjazz: Continuing with sync
• 11:48 dreamyjazz@deploy1003: dreamyjazz: Backport for Hooks: Re-order checks to verify that request user is same as Special:Contributions user (T375061) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 11:46 dreamyjazz@deploy1003: Started scap sync-world: Backport for Hooks: Re-order checks to verify that request user is same as Special:Contributions user (T375061)
• 11:43 XioNoX: update pfw3-codfw dhcp-relay target 0 T375011
• 11:43 tchin@deploy1003: Finished deploy [analytics/refinery@bc0be94] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bc0be94a] (duration: 03m 57s)
• 11:39 tchin@deploy1003: Started deploy [analytics/refinery@bc0be94] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bc0be94a]
• 11:39 tchin@deploy1003: Finished deploy [analytics/refinery@bc0be94] (thin): Regular analytics weekly train THIN [analytics/refinery@bc0be94a] (duration: 05m 50s)
• 11:33 tchin@deploy1003: Started deploy [analytics/refinery@bc0be94] (thin): Regular analytics weekly train THIN [analytics/refinery@bc0be94a]
• 11:32 tchin@deploy1003: Finished deploy [analytics/refinery@bc0be94]: Regular analytics weekly train [analytics/refinery@bc0be94a] (duration: 09m 06s)
• 11:23 tchin@deploy1003: Started deploy [analytics/refinery@bc0be94]: Regular analytics weekly train [analytics/refinery@bc0be94a]
• 11:23 tchin: Deploying refinery
• 11:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 11:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:54 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1177.eqiad.wmnet with OS bullseye
• 10:25 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye
• 10:20 elukey: restart poolcounterd on poolcounter2003 (not serving any traffic atm, tried to clear old/stale conns)
• 10:14 elukey@deploy1003: Finished scap sync-world: Backport for Swap poolcounter2004 with poolcounter2006 (T332015) (duration: 07m 08s)
• 10:09 elukey@deploy1003: elukey: Continuing with sync
• 10:09 elukey@deploy1003: elukey: Backport for Swap poolcounter2004 with poolcounter2006 (T332015) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 10:07 elukey@deploy1003: Started scap sync-world: Backport for Swap poolcounter2004 with poolcounter2006 (T332015)
• 09:26 tappof@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host centrallog2002.codfw.wmnet
• 09:11 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye
• 09:11 tappof@cumin2002: START - Cookbook sre.hosts.reboot-single for host centrallog2002.codfw.wmnet
• 09:01 moritzm: drain ganeti2026 T373104
• 08:41 tappof: centrallog2002 upgrade to bookworm in progress https://phabricator.wikimedia.org/T353912
• 08:32 elukey: install openjdk-17-jdk on puppetserver1002 to get some useful tools like jmap - T373527
• 08:30 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.23 refs T373642
• 08:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 08:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 08:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2017.codfw.wmnet
• 08:16 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2017.codfw.wmnet
• 08:15 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.43.0-wmf.23 refs T373642
• 07:45 volans@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 07:45 volans@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Fixed asset tag for db1179 - volans@cumin1002"
• 07:43 volans@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Fixed asset tag for db1179 - volans@cumin1002"
• 07:33 volans@cumin1002: START - Cookbook sre.dns.netbox
• 06:39 moritzm: installing curl security updates
• 06:05 arnaudb@cumin1002: dbctl commit (dc=all): 'T374807', diff saved to https://phabricator.wikimedia.org/P69250 and previous config saved to /var/cache/conftool/dbconfig/20240918-060549-arnaudb.json
• 06:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2220 to s7 primary T374807', diff saved to https://phabricator.wikimedia.org/P69249 and previous config saved to /var/cache/conftool/dbconfig/20240918-060332-arnaudb.json
• 06:02 arnaudb: Starting s7 codfw failover from db2218 to db2220 - T374807
• 05:49 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 T374807
• 05:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Remove db2220 from API/vslow/dump T374807', diff saved to https://phabricator.wikimedia.org/P69248 and previous config saved to /var/cache/conftool/dbconfig/20240918-054921-arnaudb.json
• 05:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2220 with weight 0 T374807', diff saved to https://phabricator.wikimedia.org/P69247 and previous config saved to /var/cache/conftool/dbconfig/20240918-054909-arnaudb.json
• 05:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s7 T374807
• 05:47 arnaudb@cumin1002: dbctl commit (dc=all): 'T374804', diff saved to https://phabricator.wikimedia.org/P69246 and previous config saved to /var/cache/conftool/dbconfig/20240918-054729-arnaudb.json
• 05:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2179 to s4 primary T374804', diff saved to https://phabricator.wikimedia.org/P69245 and previous config saved to /var/cache/conftool/dbconfig/20240918-054515-arnaudb.json
• 05:43 arnaudb: Starting s4 codfw failover from db2140 to db2179 - T374804
• 05:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Remove db2179 from API/vslow/dump T374804', diff saved to https://phabricator.wikimedia.org/P69244 and previous config saved to /var/cache/conftool/dbconfig/20240918-053807-arnaudb.json
• 05:37 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s4 T374804
• 05:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2179 with weight 0 T374804', diff saved to https://phabricator.wikimedia.org/P69243 and previous config saved to /var/cache/conftool/dbconfig/20240918-053633-arnaudb.json
• 05:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s4 T374804
• 05:33 arnaudb@cumin1002: dbctl commit (dc=all): 'T375047', diff saved to https://phabricator.wikimedia.org/P69242 and previous config saved to /var/cache/conftool/dbconfig/20240918-053357-arnaudb.json
• 05:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2214 to s6 primary T375047', diff saved to https://phabricator.wikimedia.org/P69241 and previous config saved to /var/cache/conftool/dbconfig/20240918-053115-arnaudb.json
• 05:30 arnaudb: Starting s6 codfw failover from db2129 to db2214 - T375047
• 05:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2214 with weight 0 T375047', diff saved to https://phabricator.wikimedia.org/P69240 and previous config saved to /var/cache/conftool/dbconfig/20240918-052446-arnaudb.json
• 05:24 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s6 T375047
• 05:24 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 25 hosts with reason: Primary switchover s6 T375047
• 00:11 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on gitlab2002.wikimedia.org with reason: version upgrade
• 00:11 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on gitlab2002.wikimedia.org with reason: version upgrade

2024-09-17

• 23:56 dzahn@cumin2002: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: security release 20240917
• 21:25 sbassett: Deployed mitigation for T374438
• 21:21 hashar: UTC late backport window is completed
• 21:18 dzahn@cumin2002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: security release 20240917
• 21:07 dzahn@cumin2002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: security release 20240917
• 20:56 hashar@deploy1003: Finished scap sync-world: Backport for logging: rm per channel 'error' logging (T228838) (duration: 08m 22s)
• 20:52 hashar@deploy1003: hashar: Continuing with sync
• 20:50 hashar@deploy1003: hashar: Backport for logging: rm per channel 'error' logging (T228838) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:48 dzahn@cumin2002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: security release 20240917
• 20:48 hashar@deploy1003: Started scap sync-world: Backport for logging: rm per channel 'error' logging (T228838)
• 20:46 hashar@deploy1003: Finished scap sync-world: Backport for Improve $wgFooterIcons override, simplify $wmgWikimediaIcon (duration: 08m 05s)
• 20:45 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
• 20:44 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
• 20:43 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
• 20:42 dzahn@cumin2002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: security release 20240917
• 20:42 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply
• 20:42 hashar@deploy1003: matmarex, hashar: Continuing with sync
• 20:42 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
• 20:41 bd808@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply
• 20:40 hashar@deploy1003: matmarex, hashar: Backport for Improve $wgFooterIcons override, simplify $wmgWikimediaIcon synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:38 hashar@deploy1003: Started scap sync-world: Backport for Improve $wgFooterIcons override, simplify $wmgWikimediaIcon
• 20:37 hashar@deploy1003: Finished scap sync-world: Backport for Revert "Enter deprecation trial for third-party cookie blocking" (T359957) (duration: 06m 37s)
• 20:32 hashar@deploy1003: tgr, hashar: Continuing with sync
• 20:32 hashar@deploy1003: tgr, hashar: Backport for Revert "Enter deprecation trial for third-party cookie blocking" (T359957) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:32 dzahn@cumin2002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: security release 20240917
• 20:30 hashar@deploy1003: Started scap sync-world: Backport for Revert "Enter deprecation trial for third-party cookie blocking" (T359957)
• 20:27 hashar@deploy1003: Finished scap sync-world: Backport for Assign the API portal to the Wikimedia group for CentralNotice (T270308) (duration: 12m 11s)
• 20:22 hashar@deploy1003: ejegg, hashar: Continuing with sync
• 20:17 hashar@deploy1003: ejegg, hashar: Backport for Assign the API portal to the Wikimedia group for CentralNotice (T270308) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:15 hashar@deploy1003: Started scap sync-world: Backport for Assign the API portal to the Wikimedia group for CentralNotice (T270308)
• 20:07 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 20:07 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 20:07 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 19:49 herron: pyrra upgraded to 0.7.7-1
• 18:27 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T370754, transfer categories jnl) xfer categories from wdqs2023.codfw.wmnet -> wdqs1024.eqiad.wmnet w/ force delete existing files, repooling both afterwards
• 18:16 ryankemper@cumin2002: START - Cookbook sre.wdqs.data-transfer (T370754, transfer categories jnl) xfer categories from wdqs2023.codfw.wmnet -> wdqs1024.eqiad.wmnet w/ force delete existing files, repooling both afterwards
• 18:16 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T370754, transfer categories jnl) xfer categories from wdqs2023.codfw.wmnet -> wdqs1023.eqiad.wmnet w/ force delete existing files, repooling both afterwards
• 18:05 ryankemper@cumin2002: START - Cookbook sre.wdqs.data-transfer (T370754, transfer categories jnl) xfer categories from wdqs2023.codfw.wmnet -> wdqs1023.eqiad.wmnet w/ force delete existing files, repooling both afterwards
• 18:04 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T370754, transfer categories jnl) xfer categories from wdqs2023.codfw.wmnet -> wdqs2024.codfw.wmnet w/ force delete existing files, repooling both afterwards
• 17:56 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on gerrit2003.wikimedia.org with reason: in setup
• 17:56 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on gerrit2003.wikimedia.org with reason: in setup
• 17:54 ryankemper@cumin2002: START - Cookbook sre.wdqs.data-transfer (T370754, transfer categories jnl) xfer categories from wdqs2023.codfw.wmnet -> wdqs2024.codfw.wmnet w/ force delete existing files, repooling both afterwards
• 17:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2209 (re)pooling @ 100%: T373103', diff saved to https://phabricator.wikimedia.org/P69237 and previous config saved to /var/cache/conftool/dbconfig/20240917-175321-arnaudb.json
• 17:53 arnaudb@cumin1002: dbctl commit (dc=all): 'es2023 (re)pooling @ 100%: T373103', diff saved to https://phabricator.wikimedia.org/P69236 and previous config saved to /var/cache/conftool/dbconfig/20240917-175311-arnaudb.json
• 17:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 100%: T373103', diff saved to https://phabricator.wikimedia.org/P69235 and previous config saved to /var/cache/conftool/dbconfig/20240917-175306-arnaudb.json
• 17:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 100%: T373103', diff saved to https://phabricator.wikimedia.org/P69234 and previous config saved to /var/cache/conftool/dbconfig/20240917-175302-arnaudb.json
• 17:44 dduvall@deploy1003: Finished deploy [releng/jenkins-deploy@d93f2c7] (releasing): Deploying https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/83 (duration: 01m 13s)
• 17:43 dduvall@deploy1003: Started deploy [releng/jenkins-deploy@d93f2c7] (releasing): Deploying https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/83
• 17:39 elukey@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host registry2005.codfw.wmnet
• 17:39 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2005.codfw.wmnet with OS bookworm
• 17:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2209 (re)pooling @ 75%: T373103', diff saved to https://phabricator.wikimedia.org/P69233 and previous config saved to /var/cache/conftool/dbconfig/20240917-173816-arnaudb.json
• 17:38 arnaudb@cumin1002: dbctl commit (dc=all): 'es2023 (re)pooling @ 75%: T373103', diff saved to https://phabricator.wikimedia.org/P69232 and previous config saved to /var/cache/conftool/dbconfig/20240917-173806-arnaudb.json
• 17:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 75%: T373103', diff saved to https://phabricator.wikimedia.org/P69231 and previous config saved to /var/cache/conftool/dbconfig/20240917-173801-arnaudb.json
• 17:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 75%: T373103', diff saved to https://phabricator.wikimedia.org/P69230 and previous config saved to /var/cache/conftool/dbconfig/20240917-173756-arnaudb.json
• 17:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2209 (re)pooling @ 50%: T373103', diff saved to https://phabricator.wikimedia.org/P69229 and previous config saved to /var/cache/conftool/dbconfig/20240917-172310-arnaudb.json
• 17:23 arnaudb@cumin1002: dbctl commit (dc=all): 'es2023 (re)pooling @ 50%: T373103', diff saved to https://phabricator.wikimedia.org/P69228 and previous config saved to /var/cache/conftool/dbconfig/20240917-172300-arnaudb.json
• 17:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 50%: T373103', diff saved to https://phabricator.wikimedia.org/P69227 and previous config saved to /var/cache/conftool/dbconfig/20240917-172255-arnaudb.json
• 17:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 50%: T373103', diff saved to https://phabricator.wikimedia.org/P69226 and previous config saved to /var/cache/conftool/dbconfig/20240917-172250-arnaudb.json
• 17:22 dduvall@deploy1003: Installation of scap version "4.103.0" completed for 211 hosts
• 17:18 dduvall@deploy1003: Installing scap version "4.103.0" for 211 hosts
• 17:10 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=(cp2039|cp2040).codfw.wmnet [reason: [maint done] depool for T373103]
• 17:08 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2376.codfw.wmnet
• 17:08 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2376.codfw.wmnet
• 17:08 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2375.codfw.wmnet
• 17:08 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2375.codfw.wmnet
• 17:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2209 (re)pooling @ 25%: T373103', diff saved to https://phabricator.wikimedia.org/P69225 and previous config saved to /var/cache/conftool/dbconfig/20240917-170805-arnaudb.json
• 17:08 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2374.codfw.wmnet
• 17:08 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2374.codfw.wmnet
• 17:07 arnaudb@cumin1002: dbctl commit (dc=all): 'es2023 (re)pooling @ 25%: T373103', diff saved to https://phabricator.wikimedia.org/P69224 and previous config saved to /var/cache/conftool/dbconfig/20240917-170755-arnaudb.json
• 17:07 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2373.codfw.wmnet
• 17:07 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2373.codfw.wmnet
• 17:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 25%: T373103', diff saved to https://phabricator.wikimedia.org/P69223 and previous config saved to /var/cache/conftool/dbconfig/20240917-170749-arnaudb.json
• 17:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 25%: T373103', diff saved to https://phabricator.wikimedia.org/P69222 and previous config saved to /var/cache/conftool/dbconfig/20240917-170745-arnaudb.json
• 17:07 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2372.codfw.wmnet
• 17:07 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2372.codfw.wmnet
• 17:07 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2371.codfw.wmnet
• 17:07 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2371.codfw.wmnet
• 17:07 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2370.codfw.wmnet
• 17:07 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2370.codfw.wmnet
• 17:07 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2369.codfw.wmnet
• 17:07 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2369.codfw.wmnet
• 17:06 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2368.codfw.wmnet
• 17:06 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2368.codfw.wmnet
• 17:06 swfrench@cumin2002: conftool action : set/pooled=yes; selector: dc=codfw,name=mw2279.codfw.wmnet
• 17:06 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2367.codfw.wmnet
• 17:06 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2367.codfw.wmnet
• 17:06 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2366.codfw.wmnet
• 17:06 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2366.codfw.wmnet
• 17:06 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=maps2008.codfw.wmnet
• 17:06 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2056.codfw.wmnet
• 17:06 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2056.codfw.wmnet
• 17:06 swfrench@cumin2002: conftool action : set/pooled=yes; selector: dc=codfw,name=mw2278.codfw.wmnet
• 17:05 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2047.codfw.wmnet
• 17:05 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2047.codfw.wmnet
• 17:05 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2046.codfw.wmnet
• 17:05 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2046.codfw.wmnet
• 17:05 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2022.codfw.wmnet
• 17:05 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2022.codfw.wmnet
• 16:58 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2376.codfw.wmnet
• 16:54 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2376.codfw.wmnet
• 16:54 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2375.codfw.wmnet
• 16:54 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2375.codfw.wmnet
• 16:54 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2374.codfw.wmnet
• 16:53 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2374.codfw.wmnet
• 16:53 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2373.codfw.wmnet
• 16:52 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2373.codfw.wmnet
• 16:52 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2372.codfw.wmnet
• 16:51 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2372.codfw.wmnet
• 16:51 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2371.codfw.wmnet
• 16:51 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2371.codfw.wmnet
• 16:51 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2370.codfw.wmnet
• 16:50 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2370.codfw.wmnet
• 16:50 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2369.codfw.wmnet
• 16:49 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 24 hosts with reason: reboot cloudsw1-c8-eqiad
• 16:48 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on 24 hosts with reason: reboot cloudsw1-c8-eqiad
• 16:47 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2369.codfw.wmnet
• 16:47 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2368.codfw.wmnet
• 16:46 swfrench@cumin2002: conftool action : set/pooled=no; selector: dc=codfw,name=mw2279.codfw.wmnet
• 16:46 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2368.codfw.wmnet
• 16:46 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2367.codfw.wmnet
• 16:45 swfrench@cumin2002: conftool action : set/pooled=no; selector: dc=codfw,name=mw2278.codfw.wmnet
• 16:45 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2367.codfw.wmnet
• 16:45 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2366.codfw.wmnet
• 16:45 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2366.codfw.wmnet
• 16:44 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2056.codfw.wmnet
• 16:44 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2056.codfw.wmnet
• 16:44 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2047.codfw.wmnet
• 16:43 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2047.codfw.wmnet
• 16:43 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2046.codfw.wmnet
• 16:42 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2046.codfw.wmnet
• 16:42 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2022.codfw.wmnet
• 16:42 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2022.codfw.wmnet
• 16:30 hashar@deploy1003: Finished scap sync-world: Backport for Lift IP cap on this dates 17/09,24/09 for edit-a-thon for eswiki, commons and wikidata (T373468) (duration: 07m 25s)
• 16:26 hashar@deploy1003: hashar, gergesshamon: Continuing with sync
• 16:26 hashar@deploy1003: hashar, gergesshamon: Backport for Lift IP cap on this dates 17/09,24/09 for edit-a-thon for eswiki, commons and wikidata (T373468) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 16:23 hashar@deploy1003: Started scap sync-world: Backport for Lift IP cap on this dates 17/09,24/09 for edit-a-thon for eswiki, commons and wikidata (T373468)
• 16:08 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db2209.codfw.wmnet with reason: move to new switch
• 16:08 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on db2209.codfw.wmnet with reason: move to new switch
• 16:06 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cloudsw1-c8-eqiad,cloudsw1-c8-eqiad IPv6,cloudsw1-c8-eqiad.mgmt,cr1-eqiad
• 16:06 cmooney@cumin1002: START - Cookbook sre.hosts.remove-downtime for cloudsw1-c8-eqiad,cloudsw1-c8-eqiad IPv6,cloudsw1-c8-eqiad.mgmt,cr1-eqiad
• 15:50 hnowlan@puppetmaster1001: conftool action : set/pooled=no; selector: name=maps2008.codfw.wmnet
• 15:50 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' .
• 15:47 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' .
• 15:44 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:45:00 on 6 hosts with reason: network maintenance T373103
• 15:44 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:45:00 on 6 hosts with reason: network maintenance T373103
• 15:44 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' .
• 15:43 arnaudb@cumin1002: dbctl commit (dc=all): 'depool db2213 db2214 es2023 pc2016 db2209 - T373103', diff saved to https://phabricator.wikimedia.org/P69221 and previous config saved to /var/cache/conftool/dbconfig/20240917-154355-arnaudb.json
• 15:40 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet
• 15:33 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet
• 15:33 dancy: dancy@deploy1003 Installation of scap version "4.102.1" completed for 211 hosts
• 15:28 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=(cp2039|cp2040).codfw.wmnet [reason: depool for T373103]
• 15:28 dancy@deploy1003: Finished deploy [releng/jenkins-deploy@d8093b9] (releasing): (no justification provided) (duration: 00m 43s)
• 15:28 dancy@deploy1003: Started deploy [releng/jenkins-deploy@d8093b9] (releasing): (no justification provided)
• 15:19 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on registry2005.codfw.wmnet with reason: WIP - working on puppet runs
• 15:19 moritzm: installing postgresql-13 security updates
• 15:18 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on registry2005.codfw.wmnet with reason: WIP - working on puppet runs
• 15:17 logmsgbot: dreamyjazz Deployed security patch for T372998
• 15:09 logmsgbot: dreamyjazz Deployed security patch for T372998
• 15:05 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on 24 hosts with reason: reboot cloudsw1-c8-eqiad
• 15:05 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 1:30:00 on 24 hosts with reason: reboot cloudsw1-c8-eqiad
• 15:04 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on cr1-eqiad with reason: reboot cloudsw1-c8-eqiad
• 15:04 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 1:30:00 on cr1-eqiad with reason: reboot cloudsw1-c8-eqiad
• 15:03 Dreamy_Jazz: Starting security deploy
• 14:52 dancy@deploy1003: Finished deploy [releng/phatality@84c7283]: T374880 (duration: 00m 06s)
• 14:52 dancy@deploy1003: Started deploy [releng/phatality@84c7283]: T374880
• 14:47 dancy@deploy1003: Finished deploy [releng/phatality@84c7283]: T374880 (duration: 00m 09s)
• 14:46 dancy@deploy1003: Started deploy [releng/phatality@84c7283]: T374880
• 14:45 dancy@deploy1003: Finished deploy [releng/phatality@84c7283]: (no justification provided) (duration: 00m 18s)
• 14:45 dancy@deploy1003: Started deploy [releng/phatality@84c7283]: (no justification provided)
• 14:29 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on cloudsw1-c8-eqiad,cloudsw1-c8-eqiad IPv6,cloudsw1-c8-eqiad.mgmt with reason: Reboot cloudsw1-c8-eqiad and upgrade JunOS
• 14:29 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 1:30:00 on cloudsw1-c8-eqiad,cloudsw1-c8-eqiad IPv6,cloudsw1-c8-eqiad.mgmt with reason: Reboot cloudsw1-c8-eqiad and upgrade JunOS
• 13:57 cmooney@cumin1002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2016.codfw.wmnet
• 13:54 cmooney@cumin1002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2016.codfw.wmnet
• 13:53 jayme@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-jumbo-eqiad
• 13:51 elukey: copy jwt-authorizer from bullseye-wikimedia to bookworm-wikimedia
• 13:48 hashar: UTC afternoon backport window completed (again!)
• 13:47 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 13:46 jayme@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-eqiad
• 13:46 hashar@deploy1003: Finished scap sync-world: Backport for Enable ContactPage extension on zhwiki (T359998) (duration: 08m 09s)
• 13:43 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 13:42 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 13:42 jayme@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-eqiad
• 13:41 hashar@deploy1003: hashar: Continuing with sync
• 13:40 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kafka-stretch1002 to an-worker1177
• 13:40 hashar@deploy1003: hashar: Backport for Enable ContactPage extension on zhwiki (T359998) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:39 stevemunene@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1177
• 13:39 stevemunene@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1177
• 13:39 stevemunene@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:38 stevemunene@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kafka-stretch1002 to an-worker1177 - stevemunene@cumin1002"
• 13:38 stevemunene@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kafka-stretch1002 to an-worker1177 - stevemunene@cumin1002"
• 13:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T367781)', diff saved to https://phabricator.wikimedia.org/P69219 and previous config saved to /var/cache/conftool/dbconfig/20240917-133805-arnaudb.json
• 13:37 hashar@deploy1003: Started scap sync-world: Backport for Enable ContactPage extension on zhwiki (T359998)
• 13:36 Lucas_WMDE: (for the record, refreshing the WDQS GUI cache five minutes ago seems to have worked well enough… that, or the cache just happened to expire around the same time ^^)
• 13:36 hashar@deploy1003: Finished scap sync-world: Backport for Configure ContactPage and IPBE contact form on zhwiki (T359998) (duration: 10m 00s)
• 13:35 stevemunene@cumin1002: START - Cookbook sre.dns.netbox
• 13:34 stevemunene@cumin1002: START - Cookbook sre.hosts.rename from kafka-stretch1002 to an-worker1177
• 13:31 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ for domain in query{,-{main,scholarly; do for path in / /index.html /i18n/en.json /{default,custom}-config.json; do printf 'https://%s.wikidata.org%s\n' "$domain" "$path"; done; done | mwscript purgeList enwiki # try to refresh WDQS GUI cache, don’t know if it’ll work}}
• 13:31 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kafka-stretch1001 to an-worker1176
• 13:31 hashar@deploy1003: hashar, hamishz: Continuing with sync
• 13:30 stevemunene@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1176
• 13:30 stevemunene@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1176
• 13:30 stevemunene@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:30 stevemunene@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kafka-stretch1001 to an-worker1176 - stevemunene@cumin1002"
• 13:28 hashar@deploy1003: hashar, hamishz: Backport for Configure ContactPage and IPBE contact form on zhwiki (T359998) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:26 stevemunene@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kafka-stretch1001 to an-worker1176 - stevemunene@cumin1002"
• 13:26 hashar@deploy1003: Started scap sync-world: Backport for Configure ContactPage and IPBE contact form on zhwiki (T359998)
• 13:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P69218 and previous config saved to /var/cache/conftool/dbconfig/20240917-132257-arnaudb.json
• 13:22 stevemunene@cumin1002: START - Cookbook sre.dns.netbox
• 13:22 stevemunene@cumin1002: START - Cookbook sre.hosts.rename from kafka-stretch1001 to an-worker1176
• 13:21 jayme@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-eqiad
• 13:20 jayme@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-test-eqiad
• 13:20 jayme@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-codfw
• 13:18 hashar: UTC afternoon backport window completed
• 13:17 jayme@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-eqiad
• 13:16 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage
• 13:14 jayme@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-codfw
• 13:13 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage
• 13:09 hashar@deploy1003: Finished scap sync-world: Backport for logging: Default to log any error (all wikis) (T228838) (duration: 07m 06s)
• 13:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P69217 and previous config saved to /var/cache/conftool/dbconfig/20240917-130750-arnaudb.json
• 13:04 hashar@deploy1003: hashar: Continuing with sync
• 13:04 hashar@deploy1003: hashar: Backport for logging: Default to log any error (all wikis) (T228838) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:02 hashar@deploy1003: Started scap sync-world: Backport for logging: Default to log any error (all wikis) (T228838)
• 12:57 jayme@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-codfw
• 12:57 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host registry2005.codfw.wmnet with OS bookworm
• 12:57 jayme@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-jumbo-eqiad
• 12:56 jayme@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-test-eqiad
• 12:54 jayme@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-codfw
• 12:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T367781)', diff saved to https://phabricator.wikimedia.org/P69216 and previous config saved to /var/cache/conftool/dbconfig/20240917-125242-arnaudb.json
• 12:51 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM registry2005.codfw.wmnet - elukey@cumin1002"
• 12:51 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM registry2005.codfw.wmnet - elukey@cumin1002"
• 12:51 elukey@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) registry2005.codfw.wmnet on all recursors
• 12:50 elukey@cumin1002: START - Cookbook sre.dns.wipe-cache registry2005.codfw.wmnet on all recursors
• 12:50 elukey@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:50 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM registry2005.codfw.wmnet - elukey@cumin1002"
• 12:50 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM registry2005.codfw.wmnet - elukey@cumin1002"
• 12:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2161 (T367781)', diff saved to https://phabricator.wikimedia.org/P69215 and previous config saved to /var/cache/conftool/dbconfig/20240917-125032-arnaudb.json
• 12:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2161.codfw.wmnet with reason: Maintenance
• 12:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2161.codfw.wmnet with reason: Maintenance
• 12:49 arnaudb@cumin1002: dbctl commit (dc=all): 'codfw/db2161 weight', diff saved to https://phabricator.wikimedia.org/P69214 and previous config saved to /var/cache/conftool/dbconfig/20240917-124927-arnaudb.json
• 12:47 elukey@cumin1002: START - Cookbook sre.dns.netbox
• 12:47 elukey@cumin1002: START - Cookbook sre.ganeti.makevm for new host registry2005.codfw.wmnet
• 12:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2165 to s8 primary T374946', diff saved to https://phabricator.wikimedia.org/P69213 and previous config saved to /var/cache/conftool/dbconfig/20240917-124638-arnaudb.json
• 12:46 arnaudb: Starting s8 codfw failover from db2161 to db2165 - T374946
• 12:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2165 with weight 0 T374946', diff saved to https://phabricator.wikimedia.org/P69212 and previous config saved to /var/cache/conftool/dbconfig/20240917-124022-arnaudb.json
• 12:40 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T374946
• 12:39 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s8 T374946
• 12:38 topranks: disable Equinix IXP BGP peers on cr2-eqiad before reconfiguring port as LAG T370696
• 12:30 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on cr1-eqiad with reason: enable ixp port cr1-eqiad
• 12:30 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:20:00 on cr1-eqiad with reason: enable ixp port cr1-eqiad
• 12:23 topranks: reconfigure cr1-eqiad xe-3/0/6 into LAG grou ae6 for Equinix IXP peering T370696
• 12:15 topranks: disable Equinix IXP peering on cr1-eqiad in advance of port move
• 12:15 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on cr1-eqiad with reason: enable ixp port cr1-eqiad
• 12:15 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:20:00 on cr1-eqiad with reason: enable ixp port cr1-eqiad
• 12:02 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' .
• 11:44 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
• 11:44 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] START helmfile.d/services/termbox: apply
• 11:44 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
• 11:44 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply
• 11:44 lucaswerkmeister-wmde@deploy1003: helmfile [staging] DONE helmfile.d/services/termbox: apply
• 11:44 lucaswerkmeister-wmde@deploy1003: helmfile [staging] START helmfile.d/services/termbox: apply
• 11:39 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
• 11:38 lucaswerkmeister-wmde@deploy1003: helmfile [codfw] START helmfile.d/services/termbox: apply
• 11:38 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
• 11:37 lucaswerkmeister-wmde@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply
• 11:36 lucaswerkmeister-wmde@deploy1003: helmfile [staging] DONE helmfile.d/services/termbox: apply
• 11:19 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1039.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 11:18 elukey@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1039.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 11:15 lucaswerkmeister-wmde@deploy1003: helmfile [staging] START helmfile.d/services/termbox: apply
• 11:07 volans: installed spicerack_8.13.1 on the cumin hosts
• 11:05 volans: uploaded spicerack_8.13.1 to apt.wikimedia.org bullseye-wikimedia
• 11:01 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host deploy1003.eqiad.wmnet
• 10:51 elukey@deploy1003: Finished scap sync-world: Backport for Swap poolcounter2003 with poolcounter2005 (T332015) (duration: 13m 19s)
• 10:45 elukey@deploy1003: elukey: Continuing with sync
• 10:45 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host deploy1003.eqiad.wmnet
• 10:45 elukey@deploy1003: elukey: Backport for Swap poolcounter2003 with poolcounter2005 (T332015) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 10:38 elukey@deploy1003: Started scap sync-world: Backport for Swap poolcounter2003 with poolcounter2005 (T332015)
• 10:37 elukey@deploy1003: Finished scap sync-world: Swap poolcounter2003 with poolcounter2005 (duration: 30m 00s)
• 10:07 elukey@deploy1003: Started scap sync-world: Swap poolcounter2003 with poolcounter2005
• 10:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2138.codfw.wmnet
• 10:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2138.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 10:05 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2138.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 10:01 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 09:58 jayme: re-enable puppet on all kafka brokers
• 09:57 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main2006.codfw.wmnet
• 09:57 jayme@cumin1002: START - Cookbook sre.hosts.remove-downtime for kafka-main2006.codfw.wmnet
• 09:57 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2138.codfw.wmnet
• 09:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2138 T374852', diff saved to https://phabricator.wikimedia.org/P69209 and previous config saved to /var/cache/conftool/dbconfig/20240917-095625-arnaudb.json
• 09:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2138 T374852', diff saved to https://phabricator.wikimedia.org/P69208 and previous config saved to /var/cache/conftool/dbconfig/20240917-095230-arnaudb.json
• 09:51 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on kafka-main2006.codfw.wmnet with reason: Rollout of 1073402
• 09:51 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 0:10:00 on kafka-main2006.codfw.wmnet with reason: Rollout of 1073402
• 09:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2137.codfw.wmnet
• 09:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2137.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 09:51 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2137.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 09:47 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 09:45 jayme: disabling puppet on all kafka brokers for rollout of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1073402
• 09:43 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2137.codfw.wmnet
• 09:42 arnaudb@cumin1002: dbctl commit (dc=all): 'db2137 T374851', diff saved to https://phabricator.wikimedia.org/P69207 and previous config saved to /var/cache/conftool/dbconfig/20240917-094241-arnaudb.json
• 09:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2137 T374851', diff saved to https://phabricator.wikimedia.org/P69206 and previous config saved to /var/cache/conftool/dbconfig/20240917-093850-arnaudb.json
• 09:32 elukey@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=helm-charts,name=eqiad
• 09:31 moritzm: installing python-jwcrypto security updates
• 09:28 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2127.codfw.wmnet
• 09:28 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:28 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2127.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 09:26 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host chartmuseum1001.eqiad.wmnet with OS bookworm
• 09:25 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2127.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 09:22 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 09:17 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2127.codfw.wmnet
• 09:17 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 T374849', diff saved to https://phabricator.wikimedia.org/P69205 and previous config saved to /var/cache/conftool/dbconfig/20240917-091706-arnaudb.json
• 09:10 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on chartmuseum1001.eqiad.wmnet with reason: host reimage
• 09:07 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on chartmuseum1001.eqiad.wmnet with reason: host reimage
• 09:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 T374849', diff saved to https://phabricator.wikimedia.org/P69204 and previous config saved to /var/cache/conftool/dbconfig/20240917-090733-arnaudb.json
• 08:56 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host chartmuseum1001.eqiad.wmnet with OS bookworm
• 08:55 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
• 08:55 elukey@cumin1002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1002"
• 08:55 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
• 08:55 elukey@cumin1002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1002"
• 08:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2125.codfw.wmnet
• 08:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2125.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 08:50 elukey@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=helm-charts,name=eqiad
• 08:48 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2125.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 08:45 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 08:40 arnaudb@cumin1002: dbctl commit (dc=all): 'T374848', diff saved to https://phabricator.wikimedia.org/P69203 and previous config saved to /var/cache/conftool/dbconfig/20240917-084036-arnaudb.json
• 08:40 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2125.codfw.wmnet
• 08:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2125 T374848', diff saved to https://phabricator.wikimedia.org/P69202 and previous config saved to /var/cache/conftool/dbconfig/20240917-083652-arnaudb.json
• 08:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2124.codfw.wmnet
• 08:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2124.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 08:31 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2124.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 08:26 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 08:18 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2124.codfw.wmnet
• 08:16 arnaudb@cumin1002: dbctl commit (dc=all): 'db2124 T374847', diff saved to https://phabricator.wikimedia.org/P69201 and previous config saved to /var/cache/conftool/dbconfig/20240917-081642-arnaudb.json
• 08:12 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.23 refs T373642
• 08:04 arnaudb@cumin1002: dbctl commit (dc=all): 'db2121 T374846', diff saved to https://phabricator.wikimedia.org/P69200 and previous config saved to /var/cache/conftool/dbconfig/20240917-080453-arnaudb.json
• 08:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2122.codfw.wmnet
• 08:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2122.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 08:02 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2122.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 07:58 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 07:54 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2122.codfw.wmnet
• 07:49 arnaudb@cumin1002: dbctl commit (dc=all): 'db2121 T374846', diff saved to https://phabricator.wikimedia.org/P69199 and previous config saved to /var/cache/conftool/dbconfig/20240917-074918-arnaudb.json
• 07:42 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2121.codfw.wmnet
• 07:42 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 07:42 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2121.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 07:38 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2121.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
• 07:35 hashar@deploy1003: Finished scap sync-world: Backport for logging: Default to log any error (on group1) (T228838) (duration: 15m 36s)
• 07:33 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
• 07:29 hashar@deploy1003: hashar: Continuing with sync
• 07:29 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2121.codfw.wmnet
• 07:28 hashar@deploy1003: hashar: Backport for logging: Default to log any error (on group1) (T228838) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:26 vgutierrez: testing purged 0.23 in cp4038 - T334078
• 07:20 hashar@deploy1003: Started scap sync-world: Backport for logging: Default to log any error (on group1) (T228838)
• 07:11 arnaudb@cumin1002: dbctl commit (dc=all): 'depool db2121 T374845', diff saved to https://phabricator.wikimedia.org/P69198 and previous config saved to /var/cache/conftool/dbconfig/20240917-071120-arnaudb.json
• 04:00 mwpresync@deploy1003: Pruned MediaWiki: 1.43.0-wmf.20 (duration: 00m 58s)
• 03:53 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.43.0-wmf.23 refs T373642 (duration: 50m 47s)
• 03:02 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.43.0-wmf.23 refs T373642

2024-09-16

• 22:49 Dreamy_Jazz: Restarting MediaModeration scanning script - https://wikitech.wikimedia.org/wiki/MediaModeration
• 20:53 jhathaway: reloading puppetserver to enable strict mode
• 20:47 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 20:47 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add frack new switches - pt1979@cumin2002"
• 20:47 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add frack new switches - pt1979@cumin2002"
• 20:42 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 20:28 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 20:28 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add frack new switches - pt1979@cumin2002"
• 20:28 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add frack new switches - pt1979@cumin2002"
• 20:25 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 20:25 toyofuku@deploy1003: Finished scap sync-world: Backport for Deploy Vector 2022 on small wikis (T374255), Disable quick surveys (T374743) (duration: 08m 53s)
• 20:24 pt1979@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 20:21 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 20:20 toyofuku@deploy1003: jdlrobson, toyofuku: Continuing with sync
• 20:18 toyofuku@deploy1003: jdlrobson, toyofuku: Backport for Deploy Vector 2022 on small wikis (T374255), Disable quick surveys (T374743) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:16 toyofuku@deploy1003: Started scap sync-world: Backport for Deploy Vector 2022 on small wikis (T374255), Disable quick surveys (T374743)
• 18:27 dancy@deploy1003: Finished deploy [releng/phatality@c2cb594]: Deploying https://gerrit.wikimedia.org/r/c/releng/phatality/+/1071836 (duration: 00m 06s)
• 18:27 dancy@deploy1003: Started deploy [releng/phatality@c2cb594]: Deploying https://gerrit.wikimedia.org/r/c/releng/phatality/+/1071836
• 18:27 dancy@deploy1003: Finished deploy [releng/phatality@c2cb594]: Deploying https://gerrit.wikimedia.org/r/c/releng/phatality/+/1071836 (duration: 00m 04s)
• 18:27 dancy@deploy1003: Started deploy [releng/phatality@c2cb594]: Deploying https://gerrit.wikimedia.org/r/c/releng/phatality/+/1071836
• 18:24 rzl: rzl@cumin1002:~$ sudo cumin O:logging::opensearch::collector 'chown -R opensearch-dashboards: /usr/share/opensearch-dashboards/plugins/phatality'
• 18:07 dancy@deploy1003: Finished deploy [releng/phatality@c2cb594]: Deploying https://gerrit.wikimedia.org/r/c/releng/phatality/+/1071836 (duration: 00m 04s)
• 18:06 dancy@deploy1003: Started deploy [releng/phatality@c2cb594]: Deploying https://gerrit.wikimedia.org/r/c/releng/phatality/+/1071836
• 18:06 dancy@deploy1003: Finished deploy [releng/phatality@c2cb594]: Deploying https://gerrit.wikimedia.org/r/c/releng/phatality/+/1071836 (duration: 00m 46s)
• 18:05 dancy@deploy1003: Started deploy [releng/phatality@c2cb594]: Deploying https://gerrit.wikimedia.org/r/c/releng/phatality/+/1071836
• 17:27 arnaudb@cumin1002: dbctl commit (dc=all): 'db2238 (re)pooling @ 100%: T374623', diff saved to https://phabricator.wikimedia.org/P69196 and previous config saved to /var/cache/conftool/dbconfig/20240916-172712-arnaudb.json
• 17:27 arnaudb@cumin1002: dbctl commit (dc=all): 'db2237 (re)pooling @ 100%: T374623', diff saved to https://phabricator.wikimedia.org/P69195 and previous config saved to /var/cache/conftool/dbconfig/20240916-172706-arnaudb.json
• 17:27 arnaudb@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 100%: T374623', diff saved to https://phabricator.wikimedia.org/P69194 and previous config saved to /var/cache/conftool/dbconfig/20240916-172701-arnaudb.json
• 17:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2225 (re)pooling @ 100%: T374623', diff saved to https://phabricator.wikimedia.org/P69193 and previous config saved to /var/cache/conftool/dbconfig/20240916-172656-arnaudb.json
• 17:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 100%: T374623', diff saved to https://phabricator.wikimedia.org/P69192 and previous config saved to /var/cache/conftool/dbconfig/20240916-172651-arnaudb.json
• 17:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2222 (re)pooling @ 100%: T374623', diff saved to https://phabricator.wikimedia.org/P69191 and previous config saved to /var/cache/conftool/dbconfig/20240916-172646-arnaudb.json
• 17:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2221 (re)pooling @ 100%: T374623', diff saved to https://phabricator.wikimedia.org/P69190 and previous config saved to /var/cache/conftool/dbconfig/20240916-172641-arnaudb.json
• 17:22 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@5ad6710]: (no justification provided) (duration: 00m 44s)
• 17:21 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@5ad6710]: (no justification provided)
• 17:16 dancy@deploy1003: Finished deploy [releng/phatality@b1a2a70]: testing (duration: 00m 05s)
• 17:16 dancy@deploy1003: Started deploy [releng/phatality@b1a2a70]: testing
• 17:14 dancy@deploy1003: Installation of scap version "4.101.3" completed for 2 hosts
• 17:12 dancy@deploy1003: Installing scap version "4.101.3" for 2 hosts
• 17:12 arnaudb@cumin1002: dbctl commit (dc=all): 'db2238 (re)pooling @ 75%: T374623', diff saved to https://phabricator.wikimedia.org/P69189 and previous config saved to /var/cache/conftool/dbconfig/20240916-171206-arnaudb.json
• 17:12 arnaudb@cumin1002: dbctl commit (dc=all): 'db2237 (re)pooling @ 75%: T374623', diff saved to https://phabricator.wikimedia.org/P69188 and previous config saved to /var/cache/conftool/dbconfig/20240916-171201-arnaudb.json
• 17:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 75%: T374623', diff saved to https://phabricator.wikimedia.org/P69187 and previous config saved to /var/cache/conftool/dbconfig/20240916-171155-arnaudb.json
• 17:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2225 (re)pooling @ 75%: T374623', diff saved to https://phabricator.wikimedia.org/P69186 and previous config saved to /var/cache/conftool/dbconfig/20240916-171150-arnaudb.json
• 17:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 75%: T374623', diff saved to https://phabricator.wikimedia.org/P69185 and previous config saved to /var/cache/conftool/dbconfig/20240916-171146-arnaudb.json
• 17:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2222 (re)pooling @ 75%: T374623', diff saved to https://phabricator.wikimedia.org/P69184 and previous config saved to /var/cache/conftool/dbconfig/20240916-171140-arnaudb.json
• 17:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2221 (re)pooling @ 75%: T374623', diff saved to https://phabricator.wikimedia.org/P69183 and previous config saved to /var/cache/conftool/dbconfig/20240916-171136-arnaudb.json
• 17:08 dancy@deploy1003: Installing scap version "4.101.3" for 1 hosts
• 17:03 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1002"
• 17:03 dancy@deploy1003: Installing scap version "4.101.3" for 211 hosts
• 16:58 dancy@deploy1003: Installation of scap version "4.102.0" completed for 211 hosts
• 16:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2238 (re)pooling @ 50%: T374623', diff saved to https://phabricator.wikimedia.org/P69182 and previous config saved to /var/cache/conftool/dbconfig/20240916-165700-arnaudb.json
• 16:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2237 (re)pooling @ 50%: T374623', diff saved to https://phabricator.wikimedia.org/P69181 and previous config saved to /var/cache/conftool/dbconfig/20240916-165655-arnaudb.json
• 16:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 50%: T374623', diff saved to https://phabricator.wikimedia.org/P69180 and previous config saved to /var/cache/conftool/dbconfig/20240916-165650-arnaudb.json
• 16:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2225 (re)pooling @ 50%: T374623', diff saved to https://phabricator.wikimedia.org/P69179 and previous config saved to /var/cache/conftool/dbconfig/20240916-165645-arnaudb.json
• 16:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 50%: T374623', diff saved to https://phabricator.wikimedia.org/P69178 and previous config saved to /var/cache/conftool/dbconfig/20240916-165640-arnaudb.json
• 16:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2222 (re)pooling @ 50%: T374623', diff saved to https://phabricator.wikimedia.org/P69177 and previous config saved to /var/cache/conftool/dbconfig/20240916-165635-arnaudb.json
• 16:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2221 (re)pooling @ 50%: T374623', diff saved to https://phabricator.wikimedia.org/P69176 and previous config saved to /var/cache/conftool/dbconfig/20240916-165630-arnaudb.json
• 16:54 dancy@deploy1003: Installing scap version "4.102.0" for 211 hosts
• 16:48 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
• 16:45 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
• 16:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2238 (re)pooling @ 25%: T374623', diff saved to https://phabricator.wikimedia.org/P69175 and previous config saved to /var/cache/conftool/dbconfig/20240916-164154-arnaudb.json
• 16:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2237 (re)pooling @ 25%: T374623', diff saved to https://phabricator.wikimedia.org/P69174 and previous config saved to /var/cache/conftool/dbconfig/20240916-164149-arnaudb.json
• 16:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 25%: T374623', diff saved to https://phabricator.wikimedia.org/P69173 and previous config saved to /var/cache/conftool/dbconfig/20240916-164144-arnaudb.json
• 16:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2225 (re)pooling @ 25%: T374623', diff saved to https://phabricator.wikimedia.org/P69172 and previous config saved to /var/cache/conftool/dbconfig/20240916-164139-arnaudb.json
• 16:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 25%: T374623', diff saved to https://phabricator.wikimedia.org/P69171 and previous config saved to /var/cache/conftool/dbconfig/20240916-164134-arnaudb.json
• 16:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2222 (re)pooling @ 25%: T374623', diff saved to https://phabricator.wikimedia.org/P69170 and previous config saved to /var/cache/conftool/dbconfig/20240916-164129-arnaudb.json
• 16:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2221 (re)pooling @ 25%: T374623', diff saved to https://phabricator.wikimedia.org/P69169 and previous config saved to /var/cache/conftool/dbconfig/20240916-164124-arnaudb.json
• 16:33 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2238 (re)pooling @ 16%: T374623', diff saved to https://phabricator.wikimedia.org/P69168 and previous config saved to /var/cache/conftool/dbconfig/20240916-162649-arnaudb.json
• 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2237 (re)pooling @ 16%: T374623', diff saved to https://phabricator.wikimedia.org/P69167 and previous config saved to /var/cache/conftool/dbconfig/20240916-162644-arnaudb.json
• 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 16%: T374623', diff saved to https://phabricator.wikimedia.org/P69166 and previous config saved to /var/cache/conftool/dbconfig/20240916-162638-arnaudb.json
• 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2225 (re)pooling @ 16%: T374623', diff saved to https://phabricator.wikimedia.org/P69165 and previous config saved to /var/cache/conftool/dbconfig/20240916-162633-arnaudb.json
• 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 16%: T374623', diff saved to https://phabricator.wikimedia.org/P69164 and previous config saved to /var/cache/conftool/dbconfig/20240916-162629-arnaudb.json
• 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2222 (re)pooling @ 16%: T374623', diff saved to https://phabricator.wikimedia.org/P69163 and previous config saved to /var/cache/conftool/dbconfig/20240916-162623-arnaudb.json
• 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2221 (re)pooling @ 16%: T374623', diff saved to https://phabricator.wikimedia.org/P69162 and previous config saved to /var/cache/conftool/dbconfig/20240916-162618-arnaudb.json
• 16:22 ebernhardson@deploy1003: Finished deploy [airflow-dags/search@5ad6710]: standardize created file permissions (duration: 00m 22s)
• 16:22 ebernhardson@deploy1003: Started deploy [airflow-dags/search@5ad6710]: standardize created file permissions
• 16:12 rzl: rzl@cumin1002:~$ sudo cumin logstash[2023,2025,2030-2032].codfw.wmnet,logstash[1025,1030,1032].eqiad.wmnet 'systemctl restart opensearch-dashboards' # only hosts where status is failed
• 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2238 (re)pooling @ 8%: T374623', diff saved to https://phabricator.wikimedia.org/P69161 and previous config saved to /var/cache/conftool/dbconfig/20240916-161143-arnaudb.json
• 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2237 (re)pooling @ 8%: T374623', diff saved to https://phabricator.wikimedia.org/P69160 and previous config saved to /var/cache/conftool/dbconfig/20240916-161138-arnaudb.json
• 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 8%: T374623', diff saved to https://phabricator.wikimedia.org/P69159 and previous config saved to /var/cache/conftool/dbconfig/20240916-161133-arnaudb.json
• 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2225 (re)pooling @ 8%: T374623', diff saved to https://phabricator.wikimedia.org/P69158 and previous config saved to /var/cache/conftool/dbconfig/20240916-161128-arnaudb.json
• 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 8%: T374623', diff saved to https://phabricator.wikimedia.org/P69157 and previous config saved to /var/cache/conftool/dbconfig/20240916-161123-arnaudb.json
• 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2222 (re)pooling @ 8%: T374623', diff saved to https://phabricator.wikimedia.org/P69156 and previous config saved to /var/cache/conftool/dbconfig/20240916-161117-arnaudb.json
• 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2221 (re)pooling @ 8%: T374623', diff saved to https://phabricator.wikimedia.org/P69155 and previous config saved to /var/cache/conftool/dbconfig/20240916-161113-arnaudb.json
• 16:09 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:07 jhathaway: testing strict mode on puppetservers
• 16:06 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:00 rzl: rzl@cumin1002:~$ sudo cumin 'O:logging::opensearch::collector and not logstash1032.eqiad.wmnet' '/usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin --allow-root remove phatality'
• 15:59 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1020.eqiad.wmnet,service=s5
• 15:59 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1020.eqiad.wmnet,service=s8
• 15:59 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns4003.wikimedia.org,service=recdns
• 15:59 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: name=dns4003.wikimedia.org,service=recdns
• 15:57 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2238 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69154 and previous config saved to /var/cache/conftool/dbconfig/20240916-155637-arnaudb.json
• 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2237 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69153 and previous config saved to /var/cache/conftool/dbconfig/20240916-155632-arnaudb.json
• 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69152 and previous config saved to /var/cache/conftool/dbconfig/20240916-155627-arnaudb.json
• 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2225 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69151 and previous config saved to /var/cache/conftool/dbconfig/20240916-155622-arnaudb.json
• 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69150 and previous config saved to /var/cache/conftool/dbconfig/20240916-155617-arnaudb.json
• 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2222 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69149 and previous config saved to /var/cache/conftool/dbconfig/20240916-155612-arnaudb.json
• 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2221 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69148 and previous config saved to /var/cache/conftool/dbconfig/20240916-155607-arnaudb.json
• 15:55 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddb1020.eqiad.wmnet
• 15:54 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 15:52 fnegri@cumin1002: START - Cookbook sre.hosts.reboot-single for host clouddb1020.eqiad.wmnet
• 15:49 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1020.eqiad.wmnet with reason: Upgrading mariadb on clouddb1020 T365424
• 15:49 herron: logstash1023:/usr/share/opensearch-dashboards/bin# /usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin remove phatality
• 15:48 fnegri@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on clouddb1020.eqiad.wmnet with reason: Upgrading mariadb on clouddb1020 T365424
• 15:48 rzl: rzl@logstash1032:~$ sudo systemctl restart opensearch-dashboards
• 15:48 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1020.eqiad.wmnet,service=s8
• 15:47 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1020.eqiad.wmnet,service=s5
• 15:46 dancy@deploy1003: Finished deploy [releng/phatality@b1a2a70]: Attempting to revert (duration: 00m 06s)
• 15:46 dancy@deploy1003: Started deploy [releng/phatality@b1a2a70]: Attempting to revert
• 15:44 rzl: rzl@logstash1032:~$ sudo systemctl restart opensearch-dashboards
• 15:42 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 15:42 dancy@deploy1003: Finished deploy [releng/phatality@c2cb594]: Deploying https://gerrit.wikimedia.org/r/c/releng/phatality/+/1071836 (duration: 00m 21s)
• 15:41 dancy@deploy1003: Started deploy [releng/phatality@c2cb594]: Deploying https://gerrit.wikimedia.org/r/c/releng/phatality/+/1071836
• 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2238 (re)pooling @ 2%: T374623', diff saved to https://phabricator.wikimedia.org/P69147 and previous config saved to /var/cache/conftool/dbconfig/20240916-154132-arnaudb.json
• 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2237 (re)pooling @ 2%: T374623', diff saved to https://phabricator.wikimedia.org/P69146 and previous config saved to /var/cache/conftool/dbconfig/20240916-154127-arnaudb.json
• 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 2%: T374623', diff saved to https://phabricator.wikimedia.org/P69145 and previous config saved to /var/cache/conftool/dbconfig/20240916-154121-arnaudb.json
• 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2225 (re)pooling @ 2%: T374623', diff saved to https://phabricator.wikimedia.org/P69144 and previous config saved to /var/cache/conftool/dbconfig/20240916-154116-arnaudb.json
• 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 2%: T374623', diff saved to https://phabricator.wikimedia.org/P69143 and previous config saved to /var/cache/conftool/dbconfig/20240916-154112-arnaudb.json
• 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2222 (re)pooling @ 2%: T374623', diff saved to https://phabricator.wikimedia.org/P69142 and previous config saved to /var/cache/conftool/dbconfig/20240916-154106-arnaudb.json
• 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2221 (re)pooling @ 2%: T374623', diff saved to https://phabricator.wikimedia.org/P69141 and previous config saved to /var/cache/conftool/dbconfig/20240916-154101-arnaudb.json
• 15:39 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 15:34 dancy@deploy1003: Finished deploy [releng/phatality@8ddb2fa]: (no justification provided) (duration: 00m 15s)
• 15:34 dancy@deploy1003: Started deploy [releng/phatality@8ddb2fa]: (no justification provided)
• 15:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2238 (re)pooling @ 1%: T374623', diff saved to https://phabricator.wikimedia.org/P69140 and previous config saved to /var/cache/conftool/dbconfig/20240916-152626-arnaudb.json
• 15:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2237 (re)pooling @ 1%: T374623', diff saved to https://phabricator.wikimedia.org/P69139 and previous config saved to /var/cache/conftool/dbconfig/20240916-152621-arnaudb.json
• 15:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 1%: T374623', diff saved to https://phabricator.wikimedia.org/P69138 and previous config saved to /var/cache/conftool/dbconfig/20240916-152616-arnaudb.json
• 15:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2225 (re)pooling @ 1%: T374623', diff saved to https://phabricator.wikimedia.org/P69137 and previous config saved to /var/cache/conftool/dbconfig/20240916-152611-arnaudb.json
• 15:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 1%: T374623', diff saved to https://phabricator.wikimedia.org/P69136 and previous config saved to /var/cache/conftool/dbconfig/20240916-152606-arnaudb.json
• 15:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2222 (re)pooling @ 1%: T374623', diff saved to https://phabricator.wikimedia.org/P69135 and previous config saved to /var/cache/conftool/dbconfig/20240916-152601-arnaudb.json
• 15:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2221 (re)pooling @ 1%: T374623', diff saved to https://phabricator.wikimedia.org/P69134 and previous config saved to /var/cache/conftool/dbconfig/20240916-152556-arnaudb.json
• 15:13 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters (exit_code=0) for datacenter switchover from codfw to eqiad
• 15:03 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from codfw to eqiad
• 15:02 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.09-restore-ttl (exit_code=0) for datacenter switchover from codfw to eqiad
• 15:01 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from codfw to eqiad
• 15:01 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) for datacenter switchover from codfw to eqiad
• 14:59 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from codfw to eqiad
• 14:58 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner (exit_code=0) for datacenter switchover from codfw to eqiad
• 14:58 root@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: sync
• 14:57 root@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: sync
• 14:57 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from codfw to eqiad
• 14:57 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) for datacenter switchover from codfw to eqiad
• 14:57 swfrench@cumin1002: [DRY-RUN] MediaWiki read-only period ends at: 2024-09-16 14:57:30.267664
• 14:57 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from codfw to eqiad
• 14:57 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) for datacenter switchover from codfw to eqiad
• 14:57 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite for datacenter switchover from codfw to eqiad
• 14:56 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) for datacenter switchover from codfw to eqiad
• 14:56 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from codfw to eqiad
• 14:55 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) for datacenter switchover from codfw to eqiad
• 14:54 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from codfw to eqiad
• 14:54 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) for datacenter switchover from codfw to eqiad
• 14:54 moritzm: installing gdk-pixbuf security updates
• 14:54 swfrench@cumin1002: [DRY-RUN] MediaWiki read-only period starts at: 2024-09-16 14:54:20.136310
• 14:54 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from codfw to eqiad
• 14:47 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) for datacenter switchover from codfw to eqiad
• 14:47 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from codfw to eqiad
• 14:47 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) for datacenter switchover from codfw to eqiad
• 14:43 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
• 14:42 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
• 14:41 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from codfw to eqiad
• 14:39 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks (exit_code=0) for datacenter switchover from codfw to eqiad
• 14:39 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from codfw to eqiad
• 14:37 swfrench@cumin1002: END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) for datacenter switchover from codfw to eqiad
• 14:37 swfrench@cumin1002: START - Cookbook sre.switchdc.mediawiki.00-disable-puppet for datacenter switchover from codfw to eqiad
• 14:31 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1019.eqiad.wmnet,service=s4
• 14:31 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1019.eqiad.wmnet,service=s6
• 14:27 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddb1019.eqiad.wmnet
• 14:24 fnegri@cumin1002: START - Cookbook sre.hosts.reboot-single for host clouddb1019.eqiad.wmnet
• 14:15 hashar: Afternoon backport window is complete
• 14:12 hashar@deploy1003: Finished scap sync-world: Backport for eswiki, commonswiki, wikidata: lift IP cap for edit-a-thon (T374621) (duration: 06m 41s)
• 14:10 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1019.eqiad.wmnet with reason: Upgrading mariadb on clouddb1019 T365424
• 14:09 fnegri@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on clouddb1019.eqiad.wmnet with reason: Upgrading mariadb on clouddb1019 T365424
• 14:07 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1019.eqiad.wmnet,service=s6
• 14:07 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1019.eqiad.wmnet,service=s4
• 14:07 hashar@deploy1003: hamishz, hashar: Continuing with sync
• 14:07 hashar@deploy1003: hamishz, hashar: Backport for eswiki, commonswiki, wikidata: lift IP cap for edit-a-thon (T374621) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:06 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1018.eqiad.wmnet,service=s2
• 14:06 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1018.eqiad.wmnet,service=s7
• 14:05 hashar@deploy1003: Started scap sync-world: Backport for eswiki, commonswiki, wikidata: lift IP cap for edit-a-thon (T374621)
• 14:04 hashar@deploy1003: Finished scap sync-world: Backport for logging: Default to log any error (on group0) (T228838) (duration: 07m 59s)
• 14:00 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddb1018.eqiad.wmnet
• 13:59 hashar@deploy1003: matmarex, hashar: Continuing with sync
• 13:58 hashar@deploy1003: matmarex, hashar: Backport for logging: Default to log any error (on group0) (T228838) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:57 fnegri@cumin1002: START - Cookbook sre.hosts.reboot-single for host clouddb1018.eqiad.wmnet
• 13:56 hashar@deploy1003: Started scap sync-world: Backport for logging: Default to log any error (on group0) (T228838)
• 13:50 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1018.eqiad.wmnet with reason: Upgrading mariadb on clouddb1018 T365424
• 13:50 fnegri@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on clouddb1018.eqiad.wmnet with reason: Upgrading mariadb on clouddb1018 T365424
• 13:49 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1018.eqiad.wmnet,service=s7
• 13:49 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1018.eqiad.wmnet,service=s2
• 13:49 hashar@deploy1003: Finished scap sync-world: Backport for Define MW_ENTRY_POINT in static.php (T374286) (duration: 10m 03s)
• 13:48 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet,service=s1
• 13:48 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet,service=s3
• 13:48 sukhe: sudo cumin -b11 "A:cp" 'run-puppet-agent --enable "merging CR 1072566"'
• 13:47 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet [reason: [done] testing NOOP CR but depooling to be extra sure]
• 13:43 hashar@deploy1003: hashar, matmarex: Continuing with sync
• 13:43 hashar@deploy1003: hashar, matmarex: Backport for Define MW_ENTRY_POINT in static.php (T374286) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:42 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet [reason: testing NOOP CR but depooling to be extra sure]
• 13:40 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddb1017.eqiad.wmnet
• 13:39 hashar@deploy1003: Started scap sync-world: Backport for Define MW_ENTRY_POINT in static.php (T374286)
• 13:37 hashar: mwmaint: mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki=sewikimedia --current --namespace 2 # T374089
• 13:37 fnegri@cumin1002: START - Cookbook sre.hosts.reboot-single for host clouddb1017.eqiad.wmnet
• 13:36 sukhe: sudo cumin "A:cp" 'disable-puppet "merging CR 1072566"'
• 13:36 hashar@deploy1003: Finished scap sync-world: Backport for [sewikimedia] Enable signatures in the User-namespace (T374089) (duration: 15m 35s)
• 13:31 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1017.eqiad.wmnet with reason: Upgrading mariadb on clouddb1017 T365424
• 13:31 fnegri@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on clouddb1017.eqiad.wmnet with reason: Upgrading mariadb on clouddb1017 T365424
• 13:31 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1016.eqiad.wmnet with reason: Upgrading mariadb on clouddb1017 T365424
• 13:30 fnegri@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on clouddb1016.eqiad.wmnet with reason: Upgrading mariadb on clouddb1017 T365424
• 13:30 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1017.eqiad.wmnet,service=s3
• 13:30 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1017.eqiad.wmnet,service=s1
• 13:25 hashar@deploy1003: gergesshamon, hashar: Continuing with sync
• 13:25 hashar@deploy1003: gergesshamon, hashar: Backport for [sewikimedia] Enable signatures in the User-namespace (T374089) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:21 hashar@deploy1003: Started scap sync-world: Backport for [sewikimedia] Enable signatures in the User-namespace (T374089)
• 13:20 hashar@deploy1003: Sync cancelled.
• 13:20 hashar@deploy1003: hashar, gergesshamon: Backport for [sewikimedia] Enable signatures in the User-namespace (T374089) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:14 hashar@deploy1003: Started scap sync-world: Backport for [sewikimedia] Enable signatures in the User-namespace (T374089)
• 13:13 hashar@deploy1003: Sync cancelled.
• 13:13 hashar@deploy1003: gergesshamon, hashar: Backport for [sewikimedia] Enable signatures in the User-namespace (T374089) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:01 hashar@deploy1003: Started scap sync-world: Backport for [sewikimedia] Enable signatures in the User-namespace (T374089)
• 12:51 moritzm: installing node-undici security updates
• 12:30 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
• 12:28 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
• 12:23 moritzm: installing glibc bugfix updates from bookworm 12.7 point release
• 12:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test1004.wikimedia.org
• 12:19 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host idp-test1004.wikimedia.org
• 12:18 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
• 11:00 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM chartmuseum1001.eqiad.wmnet
• 10:56 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1349.eqiad.wmnet
• 10:50 jmm@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM chartmuseum1001.eqiad.wmnet
• 10:50 hnowlan@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1349.eqiad.wmnet
• 10:33 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1016.eqiad.wmnet,service=s5
• 10:33 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1016.eqiad.wmnet,service=s8
• 10:29 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddb1016.eqiad.wmnet
• 10:25 fnegri@cumin1002: START - Cookbook sre.hosts.reboot-single for host clouddb1016.eqiad.wmnet
• 10:06 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1016.eqiad.wmnet with reason: Upgrading mariadb on clouddb1016 T365424
• 10:06 fnegri@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on clouddb1016.eqiad.wmnet with reason: Upgrading mariadb on clouddb1016 T365424
• 10:05 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1016.eqiad.wmnet,service=s8
• 10:05 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1016.eqiad.wmnet,service=s5
• 10:03 elukey@deploy1003: Finished scap sync-world: Update network policies to allow the new poolcounter vms. (duration: 04m 35s)
• 10:03 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1015.eqiad.wmnet,service=s4
• 10:03 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1015.eqiad.wmnet,service=s6
• 10:00 elukey@deploy1003: Started scap sync-world: Update network policies to allow the new poolcounter vms.
• 09:56 elukey@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=helm-charts,name=codfw
• 09:50 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddb1015.eqiad.wmnet
• 09:49 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host chartmuseum2001.codfw.wmnet with OS bookworm
• 09:47 fnegri@cumin1002: START - Cookbook sre.hosts.reboot-single for host clouddb1015.eqiad.wmnet
• 09:42 elukey: upload helm3 3.11.3-2 to bookworm-wikimedia
• 09:36 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1015.eqiad.wmnet,service=s6
• 09:36 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1015.eqiad.wmnet,service=s4
• 09:35 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1015.eqiad.wmnet with reason: Upgrading mariadb on clouddb1015 T365424
• 09:35 fnegri@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on clouddb1015.eqiad.wmnet with reason: Upgrading mariadb on clouddb1015 T365424
• 09:34 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet,service=s2
• 09:34 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet,service=s7
• 09:32 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1002"
• 09:29 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddb1014.eqiad.wmnet
• 09:26 fnegri@cumin1002: START - Cookbook sre.hosts.reboot-single for host clouddb1014.eqiad.wmnet
• 09:22 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1014.eqiad.wmnet with reason: Upgrading mariadb on clouddb1014 T365424
• 09:22 fnegri@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on clouddb1014.eqiad.wmnet with reason: Upgrading mariadb on clouddb1014 T365424
• 09:21 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1014.eqiad.wmnet,service=s7
• 09:21 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1014.eqiad.wmnet,service=s2
• 09:21 elukey: copy python3-docker-report from bullseye-wikimedia to bookworm-wikimedia
• 09:19 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet,service=s1
• 09:19 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet,service=s3
• 09:17 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
• 09:14 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
• 09:12 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddb1013.eqiad.wmnet
• 09:09 fnegri@cumin1002: START - Cookbook sre.hosts.reboot-single for host clouddb1013.eqiad.wmnet
• 09:04 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet,service=s3
• 09:04 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet,service=s1
• 09:03 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1013.eqiad.wmnet with reason: Upgrading mariadb on clouddb1013 T365424
• 09:03 fnegri@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on clouddb1013.eqiad.wmnet with reason: Upgrading mariadb on clouddb1013 T365424
• 09:03 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 08:55 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on chartmuseum2001.codfw.wmnet with reason: host reimage
• 08:53 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on chartmuseum2001.codfw.wmnet with reason: host reimage
• 08:38 moritzm: bump memory allocation of chartmuseum1001/2001 to 2G (Bookworm fails to install with just 1G) T331969
• 08:37 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host chartmuseum2001.codfw.wmnet with OS bookworm
• 08:33 elukey@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host chartmuseum2001.codfw.wmnet with OS bookworm
• 08:24 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host chartmuseum2001.codfw.wmnet with OS bookworm
• 08:19 elukey@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host chartmuseum2001.codfw.wmnet with OS bookworm
• 08:15 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host chartmuseum2001.codfw.wmnet with OS bookworm
• 08:14 elukey@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host chartmuseum2001.codfw.wmnet with OS bookworm
• 08:09 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host chartmuseum2001.codfw.wmnet with OS bookworm
• 08:08 elukey@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host chartmuseum2001.codfw.wmnet with OS bookworm
• 08:03 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 T374806', diff saved to https://phabricator.wikimedia.org/P69131 and previous config saved to /var/cache/conftool/dbconfig/20240916-080342-arnaudb.json
• 08:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2129 to s6 primary T374806', diff saved to https://phabricator.wikimedia.org/P69130 and previous config saved to /var/cache/conftool/dbconfig/20240916-080132-arnaudb.json
• 07:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 T374805', diff saved to https://phabricator.wikimedia.org/P69129 and previous config saved to /var/cache/conftool/dbconfig/20240916-075059-arnaudb.json
• 07:49 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s6 T374806
• 07:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Primary switchover s6 T374806
• 07:45 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host chartmuseum2001.codfw.wmnet with OS bookworm
• 07:43 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: sync
• 07:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2123 to s5 primary T374805', diff saved to https://phabricator.wikimedia.org/P69128 and previous config saved to /var/cache/conftool/dbconfig/20240916-074312-arnaudb.json
• 07:42 arnaudb: Starting s5 codfw failover from db2213 to db2123 - T374805
• 07:39 elukey@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: sync
• 07:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Remove db2123 from API/vslow/dump T374805', diff saved to https://phabricator.wikimedia.org/P69126 and previous config saved to /var/cache/conftool/dbconfig/20240916-073521-arnaudb.json
• 07:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T374805
• 07:33 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Primary switchover s5 T374805
• 07:33 elukey@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: sync
• 07:33 elukey@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: sync
• 07:23 elukey@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=helm-charts,name=codfw
• 06:24 moritzm: installing git security updates
• 06:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1125.eqiad.wmnet with reason: testing node
• 06:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1125.eqiad.wmnet with reason: testing node

2024-09-15

• 12:05 mvernon@cumin2002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw
• 12:03 mvernon@cumin2002: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw

2024-09-14

• 11:14 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P69122 and previous config saved to /var/cache/conftool/dbconfig/20240914-111434-ladsgroup.json
• 11:04 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P69121 and previous config saved to /var/cache/conftool/dbconfig/20240914-110416-ladsgroup.json
• 10:49 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P69120 and previous config saved to /var/cache/conftool/dbconfig/20240914-104909-ladsgroup.json
• 10:34 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T367856)', diff saved to https://phabricator.wikimedia.org/P69119 and previous config saved to /var/cache/conftool/dbconfig/20240914-103402-ladsgroup.json

2024-09-13

• 21:24 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on vrts2001.codfw.wmnet with reason: nftables migration
• 21:24 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 0:10:00 on vrts2001.codfw.wmnet with reason: nftables migration
• 21:14 dwisehaupt@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 21:14 dwisehaupt@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: decommissioning frban2001 - dwisehaupt@cumin1002"
• 21:14 dwisehaupt@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: decommissioning frban2001 - dwisehaupt@cumin1002"
• 21:11 dwisehaupt@cumin1002: START - Cookbook sre.dns.netbox
• 19:52 bking@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wdqs[2021-2024].codfw.wmnet
• 19:52 bking@cumin2002: START - Cookbook sre.hosts.remove-downtime for wdqs[2021-2024].codfw.wmnet
• 19:52 bking@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wdqs[1021-1024].eqiad.wmnet
• 19:52 bking@cumin2002: START - Cookbook sre.hosts.remove-downtime for wdqs[1021-1024].eqiad.wmnet
• 17:38 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.transfer-purged-positions (exit_code=0) rolling custom on P{cp[2028-2034,2038-2042].codfw.wmnet,cp[5017,5019-5020,5023,5027-5028,5030].eqsin.wmnet,cp[4038-4052].ulsfo.wmnet} and A:cp
• 16:59 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2123.codfw.wmnet
• 16:59 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2123.codfw.wmnet
• 16:59 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2123.codfw.wmnet
• 16:57 swfrench-wmf: running homer cr*codfw* commit 'T372878'
• 16:56 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on wdqs[1021-1024].eqiad.wmnet with reason: T373935
• 16:56 bking@cumin2002: START - Cookbook sre.hosts.downtime for 3:00:00 on wdqs[1021-1024].eqiad.wmnet with reason: T373935
• 16:55 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on wdqs[2021-2024].codfw.wmnet with reason: T373791
• 16:55 bking@cumin2002: START - Cookbook sre.hosts.downtime for 3:00:00 on wdqs[2021-2024].codfw.wmnet with reason: T373791
• 16:52 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2122.codfw.wmnet
• 16:52 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2122.codfw.wmnet
• 16:52 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2122.codfw.wmnet
• 16:50 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2123.codfw.wmnet with OS bullseye
• 16:46 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2121.codfw.wmnet
• 16:46 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2121.codfw.wmnet
• 16:46 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2121.codfw.wmnet
• 16:43 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2122.codfw.wmnet with OS bullseye
• 16:38 swfrench-wmf: running homer lsw1-b3-codfw* commit 'T372878'
• 16:35 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2121.codfw.wmnet with OS bullseye
• 16:31 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2123.codfw.wmnet with reason: host reimage
• 16:27 swfrench@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2123.codfw.wmnet with reason: host reimage
• 16:23 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2122.codfw.wmnet with reason: host reimage
• 16:18 swfrench@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2122.codfw.wmnet with reason: host reimage
• 16:18 vgutierrez@cumin1002: START - Cookbook sre.cdn.transfer-purged-positions rolling custom on P{cp[2028-2034,2038-2042].codfw.wmnet,cp[5017,5019-5020,5023,5027-5028,5030].eqsin.wmnet,cp[4038-4052].ulsfo.wmnet} and A:cp
• 16:16 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2121.codfw.wmnet with reason: host reimage
• 16:15 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.transfer-purged-positions (exit_code=0) rolling custom on P{cp2027*} and A:cp
• 16:13 vgutierrez@cumin1002: START - Cookbook sre.cdn.transfer-purged-positions rolling custom on P{cp2027*} and A:cp
• 16:12 swfrench@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2121.codfw.wmnet with reason: host reimage
• 16:10 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2123
• 16:10 swfrench@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2123
• 16:10 swfrench@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2123
• 16:10 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2123.codfw.wmnet 164.16.192.10.in-addr.arpa 4.6.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 16:09 swfrench@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2123.codfw.wmnet 164.16.192.10.in-addr.arpa 4.6.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 16:09 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:09 swfrench@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2123 - swfrench@cumin2002"
• 16:09 swfrench@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2123 - swfrench@cumin2002"
• 16:08 dduvall@deploy1003: Finished deploy [releng/jenkins-deploy@e8b4e0b] (releasing): (no justification provided) (duration: 00m 43s)
• 16:07 dduvall@deploy1003: Started deploy [releng/jenkins-deploy@e8b4e0b] (releasing): (no justification provided)
• 16:07 dduvall: performing friday deployment of jenkins-deploy (releases server) to fix broken job (see https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/81)
• 16:06 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:06 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:05 swfrench@cumin2002: START - Cookbook sre.dns.netbox
• 16:03 swfrench@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2123
• 16:03 swfrench@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2123.codfw.wmnet with OS bullseye
• 16:03 swfrench@cumin2002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2123.codfw.wmnet
• 16:02 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2122
• 16:02 swfrench@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2122
• 16:02 swfrench@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2122
• 16:01 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2122.codfw.wmnet 163.16.192.10.in-addr.arpa 3.6.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 16:01 swfrench@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2122.codfw.wmnet 163.16.192.10.in-addr.arpa 3.6.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 16:01 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:01 swfrench@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2122 - swfrench@cumin2002"
• 16:01 swfrench@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2122 - swfrench@cumin2002"
• 15:57 swfrench@cumin2002: START - Cookbook sre.dns.netbox
• 15:57 swfrench@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2122
• 15:57 dcausse@deploy1003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:57 swfrench@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2122.codfw.wmnet with OS bullseye
• 15:56 swfrench@cumin2002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2122.codfw.wmnet
• 15:56 dcausse@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:55 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2121
• 15:55 swfrench@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2121
• 15:55 swfrench@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2121
• 15:55 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2121.codfw.wmnet 162.16.192.10.in-addr.arpa 2.6.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 15:55 swfrench@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2121.codfw.wmnet 162.16.192.10.in-addr.arpa 2.6.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 15:55 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:55 swfrench@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2121 - swfrench@cumin2002"
• 15:55 swfrench@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2121 - swfrench@cumin2002"
• 15:51 swfrench@cumin2002: START - Cookbook sre.dns.netbox
• 15:50 swfrench@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2121
• 15:50 swfrench@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2121.codfw.wmnet with OS bullseye
• 15:50 swfrench@cumin2002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2121.codfw.wmnet
• 15:46 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2121.codfw.wmnet wikikube-worker2122.codfw.wmnet wikikube-worker2123.codfw.wmnet on all recursors
• 15:46 swfrench@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2121.codfw.wmnet wikikube-worker2122.codfw.wmnet wikikube-worker2123.codfw.wmnet on all recursors
• 15:39 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2315 to wikikube-worker2123
• 15:38 swfrench@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2123
• 15:38 swfrench@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2123
• 15:38 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:38 swfrench@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2315 to wikikube-worker2123 - swfrench@cumin2002"
• 15:38 swfrench@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2315 to wikikube-worker2123 - swfrench@cumin2002"
• 15:34 swfrench@cumin2002: START - Cookbook sre.dns.netbox
• 15:34 swfrench@cumin2002: START - Cookbook sre.hosts.rename from mw2315 to wikikube-worker2123
• 15:33 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2314 to wikikube-worker2122
• 15:32 swfrench@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2122
• 15:32 swfrench@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2122
• 15:32 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:32 swfrench@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2314 to wikikube-worker2122 - swfrench@cumin2002"
• 15:32 swfrench@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2314 to wikikube-worker2122 - swfrench@cumin2002"
• 15:28 swfrench@cumin2002: START - Cookbook sre.dns.netbox
• 15:28 swfrench@cumin2002: START - Cookbook sre.hosts.rename from mw2314 to wikikube-worker2122
• 15:27 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2313 to wikikube-worker2121
• 15:26 swfrench@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2121
• 15:26 swfrench@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2121
• 15:26 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:26 swfrench@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2313 to wikikube-worker2121 - swfrench@cumin2002"
• 15:26 swfrench@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2313 to wikikube-worker2121 - swfrench@cumin2002"
• 15:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2120.codfw.wmnet
• 15:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2120.codfw.wmnet
• 15:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2119.codfw.wmnet
• 15:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2119.codfw.wmnet
• 15:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2118.codfw.wmnet
• 15:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2118.codfw.wmnet
• 15:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2117.codfw.wmnet
• 15:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2117.codfw.wmnet
• 15:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2116.codfw.wmnet
• 15:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2116.codfw.wmnet
• 15:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2115.codfw.wmnet
• 15:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2115.codfw.wmnet
• 15:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2114.codfw.wmnet
• 15:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2114.codfw.wmnet
• 15:23 akosiaris@cumin1002: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) pool for host wikikube-worker2111.codfw.wmnet
• 15:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2111.codfw.wmnet
• 15:22 swfrench@cumin2002: START - Cookbook sre.dns.netbox
• 15:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2014.codfw.wmnet
• 15:22 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2014.codfw.wmnet
• 15:22 swfrench@cumin2002: START - Cookbook sre.hosts.rename from mw2313 to wikikube-worker2121
• 15:17 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker2120.codfw.wmnet
• 15:17 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2120.codfw.wmnet
• 15:17 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2120.codfw.wmnet
• 15:16 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2315.codfw.wmnet
• 15:16 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2315.codfw.wmnet
• 15:15 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2314.codfw.wmnet
• 15:15 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2314.codfw.wmnet
• 15:14 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2313.codfw.wmnet
• 15:14 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2313.codfw.wmnet
• 15:12 akosiaris: homer lsw1-a6-codfw* commit T372878
• 14:48 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
• 14:48 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
• 14:44 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2120.codfw.wmnet with OS bullseye
• 14:41 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2118.codfw.wmnet
• 14:41 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2118.codfw.wmnet
• 14:40 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2118.codfw.wmnet
• 14:39 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2117.codfw.wmnet
• 14:39 akosiaris@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2117.codfw.wmnet with OS bullseye
• 14:34 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.transfer-purged-positions (exit_code=0) rolling custom on P{cp2035*} and A:cp
• 14:33 akosiaris: homer lsw1-a6-codfw* commit 'T372878'
• 14:33 akosiaris: homer cr*codfw* commit 'T372878'
• 14:32 vgutierrez@cumin1002: START - Cookbook sre.cdn.transfer-purged-positions rolling custom on P{cp2035*} and A:cp
• 14:30 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2118.codfw.wmnet with OS bullseye
• 14:19 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1172.eqiad.wmnet with reason: Schema change (T367856)
• 14:19 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db1172.eqiad.wmnet with reason: Schema change (T367856)
• 14:19 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.transfer-purged-positions (exit_code=0) rolling custom on P{cp2036*} and A:cp
• 14:18 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2119.codfw.wmnet
• 14:18 akosiaris@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2119.codfw.wmnet with OS bullseye
• 14:17 vgutierrez@cumin1002: START - Cookbook sre.cdn.transfer-purged-positions rolling custom on P{cp2036*} and A:cp
• 14:11 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2118.codfw.wmnet with reason: host reimage
• 14:09 akosiaris@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2118.codfw.wmnet with reason: host reimage
• 14:09 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2120.codfw.wmnet with reason: host reimage
• 14:08 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2116.codfw.wmnet
• 14:08 akosiaris@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2116.codfw.wmnet with OS bullseye
• 14:05 akosiaris@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2120.codfw.wmnet with reason: host reimage
• 14:01 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2115.codfw.wmnet
• 14:01 akosiaris@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2115.codfw.wmnet with OS bullseye
• 14:01 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2114.codfw.wmnet
• 14:01 akosiaris@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2114.codfw.wmnet with OS bullseye
• 14:01 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2115.codfw.wmnet with OS bullseye
• 14:00 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2115.codfw.wmnet
• 14:00 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2115.codfw.wmnet
• 14:00 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2115.codfw.wmnet
• 13:59 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2114.codfw.wmnet with OS bullseye
• 13:58 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2114.codfw.wmnet
• 13:55 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2115.codfw.wmnet
• 13:55 akosiaris@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2115.codfw.wmnet with OS bullseye
• 13:53 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2114.codfw.wmnet
• 13:53 akosiaris@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2114.codfw.wmnet with OS bullseye
• 13:53 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2114.codfw.wmnet with OS bullseye
• 13:52 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2114.codfw.wmnet
• 13:52 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2118
• 13:52 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2118
• 13:42 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2114.codfw.wmnet
• 13:42 akosiaris@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2114.codfw.wmnet with OS bullseye
• 13:40 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2118
• 13:40 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2118.codfw.wmnet 173.0.192.10.in-addr.arpa 3.7.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:40 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2118.codfw.wmnet 173.0.192.10.in-addr.arpa 3.7.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:40 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:40 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2118 - akosiaris@cumin1002"
• 13:40 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2118 - akosiaris@cumin1002"
• 13:37 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 13:27 hashar@deploy1003: Finished scap sync-world: Backport for rdbms: only count replication sources toward "masterConns" in getServerConnection() (T374534) (duration: 10m 34s)
• 13:22 hashar@deploy1003: hashar: Continuing with sync
• 13:22 akosiaris@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2118
• 13:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2117
• 13:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2117
• 13:21 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2117
• 13:21 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2117.codfw.wmnet 172.0.192.10.in-addr.arpa 2.7.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:21 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2117.codfw.wmnet 172.0.192.10.in-addr.arpa 2.7.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:21 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:21 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2117 - akosiaris@cumin1002"
• 13:21 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2117 - akosiaris@cumin1002"
• 13:20 hashar@deploy1003: hashar: Backport for rdbms: only count replication sources toward "masterConns" in getServerConnection() (T374534) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:17 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 13:16 hashar@deploy1003: Started scap sync-world: Backport for rdbms: only count replication sources toward "masterConns" in getServerConnection() (T374534)
• 13:12 akosiaris@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2117
• 13:12 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2118.codfw.wmnet with OS bullseye
• 13:11 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2118.codfw.wmnet
• 13:11 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2120
• 13:11 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2120
• 13:10 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2118.codfw.wmnet
• 13:10 akosiaris@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2118.codfw.wmnet with OS bullseye
• 13:05 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2120
• 13:05 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2120.codfw.wmnet 175.0.192.10.in-addr.arpa 5.7.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:05 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2120.codfw.wmnet 175.0.192.10.in-addr.arpa 5.7.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:05 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:05 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2120 - akosiaris@cumin1002"
• 13:05 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2120 - akosiaris@cumin1002"
• 13:01 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 13:01 akosiaris@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2120
• 13:01 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2119
• 13:01 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2119
• 13:00 aqu@deploy1003: Finished deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow (duration: 00m 31s)
• 13:00 aqu@deploy1003: Started deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow
• 12:57 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2119
• 12:57 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2119.codfw.wmnet 174.0.192.10.in-addr.arpa 4.7.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:57 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2119.codfw.wmnet 174.0.192.10.in-addr.arpa 4.7.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:57 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:57 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2119 - akosiaris@cumin1002"
• 12:57 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2119 - akosiaris@cumin1002"
• 12:54 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2117.codfw.wmnet with OS bullseye
• 12:54 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2117.codfw.wmnet
• 12:54 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2117.codfw.wmnet on all recursors
• 12:53 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2117.codfw.wmnet on all recursors
• 12:53 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2117.codfw.wmnet
• 12:53 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2117.codfw.wmnet
• 12:52 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 12:52 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2120.codfw.wmnet with OS bullseye
• 12:52 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2302 to wikikube-worker2117
• 12:52 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2120.codfw.wmnet
• 12:52 akosiaris@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2119
• 12:52 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2116
• 12:51 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2116
• 12:51 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2117
• 12:49 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2117
• 12:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2302 to wikikube-worker2117 - akosiaris@cumin1002"
• 12:49 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2302 to wikikube-worker2117 - akosiaris@cumin1002"
• 12:49 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2116
• 12:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2116.codfw.wmnet 171.0.192.10.in-addr.arpa 1.7.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:49 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2116.codfw.wmnet 171.0.192.10.in-addr.arpa 1.7.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2116 - akosiaris@cumin1002"
• 12:45 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2119.codfw.wmnet with OS bullseye
• 12:45 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2116 - akosiaris@cumin1002"
• 12:45 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2119.codfw.wmnet
• 12:45 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 12:44 akosiaris@cumin1002: START - Cookbook sre.hosts.rename from mw2302 to wikikube-worker2117
• 12:41 godog: bounce thanos-query-frontend on titan eqiad
• 12:41 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2118.codfw.wmnet with OS bullseye
• 12:41 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2305 to wikikube-worker2120
• 12:40 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2120
• 12:40 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2118.codfw.wmnet
• 12:39 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 12:39 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2118.codfw.wmnet on all recursors
• 12:39 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2120
• 12:39 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2118.codfw.wmnet on all recursors
• 12:39 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:39 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2305 to wikikube-worker2120 - akosiaris@cumin1002"
• 12:39 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2305 to wikikube-worker2120 - akosiaris@cumin1002"
• 12:38 akosiaris@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2116
• 12:37 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2115
• 12:37 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2115
• 12:36 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2115
• 12:36 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2115.codfw.wmnet 124.0.192.10.in-addr.arpa 4.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:36 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2115.codfw.wmnet 124.0.192.10.in-addr.arpa 4.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:36 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:36 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2115 - akosiaris@cumin1002"
• 12:35 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 12:35 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2115 - akosiaris@cumin1002"
• 12:32 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2117.codfw.wmnet on all recursors
• 12:31 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2117.codfw.wmnet on all recursors
• 12:31 akosiaris@cumin1002: START - Cookbook sre.hosts.rename from mw2305 to wikikube-worker2120
• 12:30 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 12:30 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2304 to wikikube-worker2119
• 12:29 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2119
• 12:29 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2119
• 12:29 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:29 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2304 to wikikube-worker2119 - akosiaris@cumin1002"
• 12:29 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2304 to wikikube-worker2119 - akosiaris@cumin1002"
• 12:26 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2116.codfw.wmnet with OS bullseye
• 12:26 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2116.codfw.wmnet
• 12:25 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 12:25 akosiaris@cumin1002: START - Cookbook sre.hosts.rename from mw2304 to wikikube-worker2119
• 12:25 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2303 to wikikube-worker2118
• 12:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2118
• 12:24 akosiaris@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2115
• 12:24 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2118
• 12:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2303 to wikikube-worker2118 - akosiaris@cumin1002"
• 12:24 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2303 to wikikube-worker2118 - akosiaris@cumin1002"
• 12:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2114
• 12:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2114
• 12:22 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2114
• 12:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2114.codfw.wmnet 102.0.192.10.in-addr.arpa 2.0.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:22 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2114.codfw.wmnet 102.0.192.10.in-addr.arpa 2.0.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2114 - akosiaris@cumin1002"
• 12:20 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 12:18 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2114 - akosiaris@cumin1002"
• 12:15 akosiaris@cumin1002: START - Cookbook sre.hosts.rename from mw2303 to wikikube-worker2118
• 12:15 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 12:13 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2301 to wikikube-worker2116
• 12:13 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2116
• 12:12 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2116
• 12:12 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:12 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2301 to wikikube-worker2116 - akosiaris@cumin1002"
• 12:12 akosiaris@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2114
• 12:12 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2115.codfw.wmnet with OS bullseye
• 12:12 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2114.codfw.wmnet with OS bullseye
• 12:11 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2301 to wikikube-worker2116 - akosiaris@cumin1002"
• 12:11 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2115.codfw.wmnet
• 12:11 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2114.codfw.wmnet
• 12:07 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 12:07 akosiaris@cumin1002: START - Cookbook sre.hosts.rename from mw2301 to wikikube-worker2116
• 12:06 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2060 to wikikube-worker2115
• 12:05 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2115
• 12:05 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2115
• 12:05 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:05 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2060 to wikikube-worker2115 - akosiaris@cumin1002"
• 11:59 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2060 to wikikube-worker2115 - akosiaris@cumin1002"
• 11:54 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 11:54 akosiaris@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2060 to wikikube-worker2115
• 11:53 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2059 to wikikube-worker2114
• 11:53 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2114
• 11:52 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2114
• 11:52 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:52 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2059 to wikikube-worker2114 - akosiaris@cumin1002"
• 11:50 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2059 to wikikube-worker2114 - akosiaris@cumin1002"
• 11:47 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 11:47 akosiaris@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2059 to wikikube-worker2114
• 11:18 damilare: SmashPig upgraded from 08c79f4f to ac85ad1d
• 10:44 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on db1172.eqiad.wmnet with reason: Depooled recovering replag
• 10:44 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on db1172.eqiad.wmnet with reason: Depooled recovering replag
• 10:37 xSavitar: T374684 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=metawiki 'Monty.ch' 'MajorFault'
• 10:33 xSavitar: T374684 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=metawiki 'Mohamadanisahmad5' 'Vanished user a53a2dd4f79a7bde25cf2ea2b2a309cb'
• 10:29 xSavitar: T374684 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki 'Iosonopony' 'L.Sala'
• 10:29 xSavitar: T374684 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki 'IloveuFlyTek' 'Theology1937' --ignorestatus
• 10:25 xSavitar: T12345 Ran mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki 'IloveuFlyTek' 'Theology1937' --ignorestatus
• 10:18 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1172.eqiad.wmnet with reason: ongoing schema change
• 10:18 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db1172.eqiad.wmnet with reason: ongoing schema change
• 09:41 klausman@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-lab1001.eqiad.wmnet with OS bookworm
• 09:41 klausman@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - klausman@cumin1002"
• 09:41 klausman@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - klausman@cumin1002"
• 09:34 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-main2005.codfw.wmnet
• 09:34 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:34 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-main2005.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002"
• 09:34 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-main2005.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002"
• 09:28 klausman@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-lab1001.eqiad.wmnet with reason: host reimage
• 09:27 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 09:25 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2305.codfw.wmnet
• 09:24 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2305.codfw.wmnet
• 09:24 klausman@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-lab1001.eqiad.wmnet with reason: host reimage
• 09:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2304.codfw.wmnet
• 09:24 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2304.codfw.wmnet
• 09:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2303.codfw.wmnet
• 09:23 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2303.codfw.wmnet
• 09:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2302.codfw.wmnet
• 09:23 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply
• 09:23 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply
• 09:23 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 09:22 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2302.codfw.wmnet
• 09:22 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply
• 09:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2301.codfw.wmnet
• 09:22 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
• 09:22 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2301.codfw.wmnet
• 09:22 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2060.codfw.wmnet
• 09:21 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
• 09:21 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: apply
• 09:21 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2060.codfw.wmnet
• 09:21 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2059.codfw.wmnet
• 09:21 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: apply
• 09:21 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 09:20 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2059.codfw.wmnet
• 09:20 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
• 09:20 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
• 09:20 jayme@cumin1002: START - Cookbook sre.hosts.decommission for hosts kafka-main2005.codfw.wmnet
• 09:20 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply
• 09:20 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
• 09:19 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
• 09:15 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main2010.codfw.wmnet
• 09:15 jayme@cumin1002: START - Cookbook sre.hosts.remove-downtime for kafka-main2010.codfw.wmnet
• 09:14 jayme: restoring leadership for all partitions assigned to broker id 2005 on kafka-main-codfw - T363210
• 09:12 klausman@cumin1002: START - Cookbook sre.hosts.reimage for host ml-lab1001.eqiad.wmnet with OS bookworm
• 09:09 klausman@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-lab1001.eqiad.wmnet with OS bookworm
• 09:02 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 09:02 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 09:01 elukey@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host poolcounter1007.eqiad.wmnet
• 09:01 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host poolcounter1007.eqiad.wmnet with OS bookworm
• 08:48 elukey@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
• 08:47 klausman@cumin1002: START - Cookbook sre.hosts.reimage for host ml-lab1001.eqiad.wmnet with OS bookworm
• 08:46 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on poolcounter1007.eqiad.wmnet with reason: host reimage
• 08:45 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 08:43 klausman@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-lab1001.eqiad.wmnet with OS bookworm
• 08:42 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on poolcounter1007.eqiad.wmnet with reason: host reimage
• 08:39 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet
• 08:35 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet
• 08:32 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host poolcounter1007.eqiad.wmnet with OS bookworm
• 08:30 klausman@cumin1002: START - Cookbook sre.hosts.reimage for host ml-lab1001.eqiad.wmnet with OS bookworm
• 08:29 moritzm: rolling out djangorestbase update from Bookworm point release (replacing our previous bespoke build)
• 08:28 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM poolcounter1007.eqiad.wmnet - elukey@cumin1002"
• 08:28 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM poolcounter1007.eqiad.wmnet - elukey@cumin1002"
• 08:28 elukey@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) poolcounter1007.eqiad.wmnet on all recursors
• 08:28 elukey@cumin1002: START - Cookbook sre.dns.wipe-cache poolcounter1007.eqiad.wmnet on all recursors
• 08:28 elukey@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:28 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM poolcounter1007.eqiad.wmnet - elukey@cumin1002"
• 08:27 moritzm: remove djangorestframework 3.14.0-2+wmf12u1 from apt.wikimedia.org, the bug fixed in that custom build has been integrated into Debian Bookworm via a point update and is no longer needed
• 08:25 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM poolcounter1007.eqiad.wmnet - elukey@cumin1002"
• 08:19 elukey@cumin1002: START - Cookbook sre.dns.netbox
• 08:19 elukey@cumin1002: START - Cookbook sre.ganeti.makevm for new host poolcounter1007.eqiad.wmnet
• 08:18 elukey@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host poolcounter1006.eqiad.wmnet
• 08:18 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host poolcounter1006.eqiad.wmnet with OS bookworm
• 08:16 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: gerrit1004.wikimedia.org
• 08:16 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: gerrit1004.wikimedia.org
• 08:05 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on poolcounter1006.eqiad.wmnet with reason: host reimage
• 08:02 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on poolcounter1006.eqiad.wmnet with reason: host reimage
• 07:53 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host poolcounter1006.eqiad.wmnet with OS bookworm
• 07:52 moritzm: installing nano updates from Bookworm point release
• 07:51 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM poolcounter1006.eqiad.wmnet - elukey@cumin1002"
• 07:51 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM poolcounter1006.eqiad.wmnet - elukey@cumin1002"
• 07:50 elukey@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) poolcounter1006.eqiad.wmnet on all recursors
• 07:50 elukey@cumin1002: START - Cookbook sre.dns.wipe-cache poolcounter1006.eqiad.wmnet on all recursors
• 07:50 elukey@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 07:50 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM poolcounter1006.eqiad.wmnet - elukey@cumin1002"
• 07:50 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM poolcounter1006.eqiad.wmnet - elukey@cumin1002"
• 07:47 elukey@cumin1002: START - Cookbook sre.dns.netbox
• 07:47 elukey@cumin1002: START - Cookbook sre.ganeti.makevm for new host poolcounter1006.eqiad.wmnet
• 07:46 jayme@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-codfw
• 07:35 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 07:35 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
• 07:35 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 07:35 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 07:35 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
• 07:34 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
• 07:34 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
• 07:34 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
• 07:34 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
• 07:34 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
• 07:34 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 07:33 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 07:33 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 07:33 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 07:33 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 07:32 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
• 07:32 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 07:32 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 07:27 jayme@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-codfw
• 06:56 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on kafka-main[2005,2010].codfw.wmnet with reason: Hardware refresh
• 06:56 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on kafka-main[2005,2010].codfw.wmnet with reason: Hardware refresh
• 06:54 jayme: evacuating leadership for all partitions assigned to broker id 2005 on kafka-main-codfw - T363210

2024-09-12

• 23:00 dduvall@deploy1003: Finished deploy [releng/jenkins-deploy@35befba] (releasing): (no justification provided) (duration: 00m 38s)
• 22:59 dduvall@deploy1003: Started deploy [releng/jenkins-deploy@35befba] (releasing): (no justification provided)
• 22:34 dduvall@deploy1003: Finished deploy [releng/jenkins-deploy@6e810dc] (releasing): (no justification provided) (duration: 00m 34s)
• 22:33 dduvall@deploy1003: Started deploy [releng/jenkins-deploy@6e810dc] (releasing): (no justification provided)
• 22:30 dduvall@deploy1003: deploy aborted: (no justification provided) (duration: 01m 43s)
• 22:28 dduvall@deploy1003: Started deploy [releng/jenkins-deploy@6e810dc] (releasing): (no justification provided)
• 22:11 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
• 22:11 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
• 22:00 tzatziki: removing 1 file for legal compliance
• 21:56 tzatziki: removing 6 files for legal compliance
• 21:54 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
• 21:52 ladsgroup@deploy1003: Finished scap sync-world: Backport for Fix night mode excepted Wikidata namespaces (duration: 07m 09s)
• 21:52 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply
• 21:48 ladsgroup@deploy1003: ladsgroup, ebrahim: Continuing with sync
• 21:47 ladsgroup@deploy1003: ladsgroup, ebrahim: Backport for Fix night mode excepted Wikidata namespaces synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 21:45 ladsgroup@deploy1003: Started scap sync-world: Backport for Fix night mode excepted Wikidata namespaces
• 21:43 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
• 21:40 cjming: end of UTC late backport window
• 21:37 bd808@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply
• 21:37 cjming@deploy1003: Finished scap sync-world: Backport for Revert "Lift IP cap on this dates 10/09, 17/09, 24/09 for edit-a-thon for eswiki, commons and wikidata" (duration: 06m 37s)
• 21:33 cjming@deploy1003: cjming, gergesshamon: Continuing with sync
• 21:32 cjming@deploy1003: cjming, gergesshamon: Backport for Revert "Lift IP cap on this dates 10/09, 17/09, 24/09 for edit-a-thon for eswiki, commons and wikidata" synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 21:30 cjming@deploy1003: Started scap sync-world: Backport for Revert "Lift IP cap on this dates 10/09, 17/09, 24/09 for edit-a-thon for eswiki, commons and wikidata"
• 21:20 tzatziki: removing 1 file for legal compliance
• 21:16 cjming@deploy1003: Finished scap sync-world: Backport for Revert "Lift IP cap on this dates 10/09, 17/09, 24/09 for edit-a-thon for eswiki, commons and wikidata" (duration: 08m 57s)
• 21:13 tzatziki: removing 1 file for legal compliance
• 21:12 cjming@deploy1003: cjming, trainbranchbot: Continuing with sync
• 21:09 cjming@deploy1003: cjming, trainbranchbot: Backport for Revert "Lift IP cap on this dates 10/09, 17/09, 24/09 for edit-a-thon for eswiki, commons and wikidata" synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 21:07 cjming@deploy1003: Started scap sync-world: Backport for Revert "Lift IP cap on this dates 10/09, 17/09, 24/09 for edit-a-thon for eswiki, commons and wikidata"
• 21:05 cjming@deploy1003: Sync cancelled.
• 21:01 cjming@deploy1003: cjming, gergesshamon: Backport for Lift IP cap on this dates 10/09, 17/09, 24/09 for edit-a-thon for eswiki, commons and wikidata synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:59 cjming@deploy1003: Started scap sync-world: Backport for Lift IP cap on this dates 10/09, 17/09, 24/09 for edit-a-thon for eswiki, commons and wikidata
• 20:53 cjming@deploy1003: Finished scap sync-world: Backport for eswiki, commonswiki, wikidata: lift IP cap for edit-a-thon (T374484) (duration: 09m 21s)
• 20:49 cjming@deploy1003: cjming, superzerocool: Continuing with sync
• 20:46 cjming@deploy1003: cjming, superzerocool: Backport for eswiki, commonswiki, wikidata: lift IP cap for edit-a-thon (T374484) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:44 cjming@deploy1003: Started scap sync-world: Backport for eswiki, commonswiki, wikidata: lift IP cap for edit-a-thon (T374484)
• 20:43 cjming@deploy1003: Finished scap sync-world: Backport for Enable the dark mode in Portal namespace (T366380) (duration: 08m 57s)
• 20:38 cjming@deploy1003: ebrahim, cjming: Continuing with sync
• 20:36 cjming@deploy1003: ebrahim, cjming: Backport for Enable the dark mode in Portal namespace (T366380) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:34 cjming@deploy1003: Started scap sync-world: Backport for Enable the dark mode in Portal namespace (T366380)
• 20:33 cjming@deploy1003: Finished scap sync-world: Backport for Remove unused $wgAllowRequiringEmailForResets (T242406), Remove unused $wmgPoweredByMediaWikiIcon, Remove unused settings removed in T339959 (duration: 07m 19s)
• 20:28 cjming@deploy1003: matmarex, cjming: Continuing with sync
• 20:28 cjming@deploy1003: matmarex, cjming: Backport for Remove unused $wgAllowRequiringEmailForResets (T242406), Remove unused $wmgPoweredByMediaWikiIcon, Remove unused settings removed in T339959 synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:25 cjming@deploy1003: Started scap sync-world: Backport for Remove unused $wgAllowRequiringEmailForResets (T242406), Remove unused $wmgPoweredByMediaWikiIcon, Remove unused settings removed in T339959
• 20:19 cjming@deploy1003: Finished scap sync-world: Backport for Enable AutoModerator on ukwiki (T373823) (duration: 07m 01s)
• 20:14 cjming@deploy1003: kgraessle, cjming: Continuing with sync
• 20:14 cjming@deploy1003: kgraessle, cjming: Backport for Enable AutoModerator on ukwiki (T373823) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:12 cjming@deploy1003: Started scap sync-world: Backport for Enable AutoModerator on ukwiki (T373823)
• 20:11 cjming@deploy1003: Finished scap sync-world: Backport for u4cwiki: create case and case_talk namespaces (T374439) (duration: 07m 36s)
• 20:06 cjming@deploy1003: hamishz, cjming: Continuing with sync
• 20:06 cjming@deploy1003: hamishz, cjming: Backport for u4cwiki: create case and case_talk namespaces (T374439) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:03 cjming@deploy1003: Started scap sync-world: Backport for u4cwiki: create case and case_talk namespaces (T374439)
• 19:57 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on vrts2002.codfw.wmnet with reason: Migration
• 19:57 aokoth@cumin1002: START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on vrts2002.codfw.wmnet with reason: Migration
• 19:03 swfrench-wmf: rebuilt php8.1 production images to pick up php-uuid - T372602
• 18:34 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on vrts2001.codfw.wmnet with reason: Migration
• 18:33 aokoth@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on vrts2001.codfw.wmnet with reason: Migration
• 18:20 swfrench-wmf: ran systemctl reset-failed mediawiki_job_MachineVision_prioritize_uncategorized.service on mwmaint1002 to clear failed state for turned down job - T352884
• 18:18 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.43.0-wmf.22 refs T373641
• 17:28 eevans@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on aqs1014.eqiad.wmnet with reason: SSD device troubleshooting
• 17:27 eevans@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on aqs1014.eqiad.wmnet with reason: SSD device troubleshooting
• 17:16 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
• 17:15 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-debug: apply
• 17:14 jynus: restarting db1171:s8 mysql process T374610
• 17:11 isaranto@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
• 17:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2209 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69106 and previous config saved to /var/cache/conftool/dbconfig/20240912-170524-arnaudb.json
• 17:05 arnaudb@cumin1002: dbctl commit (dc=all): 'pc2014 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69105 and previous config saved to /var/cache/conftool/dbconfig/20240912-170524-arnaudb.json
• 17:05 arnaudb@cumin1002: dbctl commit (dc=all): 'es2039 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69104 and previous config saved to /var/cache/conftool/dbconfig/20240912-170514-arnaudb.json
• 17:05 arnaudb@cumin1002: dbctl commit (dc=all): 'es2034 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69103 and previous config saved to /var/cache/conftool/dbconfig/20240912-170509-arnaudb.json
• 17:05 arnaudb@cumin1002: dbctl commit (dc=all): 'es2033 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69102 and previous config saved to /var/cache/conftool/dbconfig/20240912-170504-arnaudb.json
• 17:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69101 and previous config saved to /var/cache/conftool/dbconfig/20240912-170459-arnaudb.json
• 17:04 arnaudb@cumin1002: dbctl commit (dc=all): 'db2211 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69100 and previous config saved to /var/cache/conftool/dbconfig/20240912-170453-arnaudb.json
• 17:04 arnaudb@cumin1002: dbctl commit (dc=all): 'db2171 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69099 and previous config saved to /var/cache/conftool/dbconfig/20240912-170449-arnaudb.json
• 17:04 arnaudb@cumin1002: dbctl commit (dc=all): 'db2170 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69098 and previous config saved to /var/cache/conftool/dbconfig/20240912-170444-arnaudb.json
• 17:04 arnaudb@cumin1002: dbctl commit (dc=all): 'db2151 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69097 and previous config saved to /var/cache/conftool/dbconfig/20240912-170439-arnaudb.json
• 17:04 arnaudb@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P69096 and previous config saved to /var/cache/conftool/dbconfig/20240912-170433-arnaudb.json
• 16:52 kcvelaga@deploy1003: Finished deploy [airflow-dags/analytics_product@d045bb2]: (no justification provided) (duration: 00m 30s)
• 16:51 kcvelaga@deploy1003: Started deploy [airflow-dags/analytics_product@d045bb2]: (no justification provided)
• 16:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2209 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69094 and previous config saved to /var/cache/conftool/dbconfig/20240912-165018-arnaudb.json
• 16:50 arnaudb@cumin1002: dbctl commit (dc=all): 'es2039 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69093 and previous config saved to /var/cache/conftool/dbconfig/20240912-165009-arnaudb.json
• 16:50 arnaudb@cumin1002: dbctl commit (dc=all): 'es2034 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69092 and previous config saved to /var/cache/conftool/dbconfig/20240912-165003-arnaudb.json
• 16:50 arnaudb@cumin1002: dbctl commit (dc=all): 'es2033 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69091 and previous config saved to /var/cache/conftool/dbconfig/20240912-164959-arnaudb.json
• 16:49 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69090 and previous config saved to /var/cache/conftool/dbconfig/20240912-164953-arnaudb.json
• 16:49 arnaudb@cumin1002: dbctl commit (dc=all): 'db2211 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69089 and previous config saved to /var/cache/conftool/dbconfig/20240912-164948-arnaudb.json
• 16:49 arnaudb@cumin1002: dbctl commit (dc=all): 'db2171 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69088 and previous config saved to /var/cache/conftool/dbconfig/20240912-164943-arnaudb.json
• 16:49 arnaudb@cumin1002: dbctl commit (dc=all): 'db2170 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69087 and previous config saved to /var/cache/conftool/dbconfig/20240912-164938-arnaudb.json
• 16:49 arnaudb@cumin1002: dbctl commit (dc=all): 'db2151 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69086 and previous config saved to /var/cache/conftool/dbconfig/20240912-164933-arnaudb.json
• 16:49 arnaudb@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P69085 and previous config saved to /var/cache/conftool/dbconfig/20240912-164927-arnaudb.json
• 16:36 topranks: disable ports for now unused ports on asw-d1-codfw and asw-d2-codfw T373102
• 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2209 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69084 and previous config saved to /var/cache/conftool/dbconfig/20240912-163513-arnaudb.json
• 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'es2039 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69083 and previous config saved to /var/cache/conftool/dbconfig/20240912-163503-arnaudb.json
• 16:34 arnaudb@cumin1002: dbctl commit (dc=all): 'es2034 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69082 and previous config saved to /var/cache/conftool/dbconfig/20240912-163458-arnaudb.json
• 16:34 arnaudb@cumin1002: dbctl commit (dc=all): 'es2033 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69081 and previous config saved to /var/cache/conftool/dbconfig/20240912-163453-arnaudb.json
• 16:34 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69080 and previous config saved to /var/cache/conftool/dbconfig/20240912-163448-arnaudb.json
• 16:34 arnaudb@cumin1002: dbctl commit (dc=all): 'db2211 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69079 and previous config saved to /var/cache/conftool/dbconfig/20240912-163443-arnaudb.json
• 16:34 arnaudb@cumin1002: dbctl commit (dc=all): 'db2171 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69078 and previous config saved to /var/cache/conftool/dbconfig/20240912-163438-arnaudb.json
• 16:34 arnaudb@cumin1002: dbctl commit (dc=all): 'db2170 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69077 and previous config saved to /var/cache/conftool/dbconfig/20240912-163433-arnaudb.json
• 16:34 arnaudb@cumin1002: dbctl commit (dc=all): 'db2151 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69076 and previous config saved to /var/cache/conftool/dbconfig/20240912-163427-arnaudb.json
• 16:34 arnaudb@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P69075 and previous config saved to /var/cache/conftool/dbconfig/20240912-163422-arnaudb.json
• 16:32 urandom: pooling ms-fe2012 moss-fe2002 & thanos-fe2003 — T373102
• 16:31 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2045.codfw.wmnet
• 16:31 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2045.codfw.wmnet
• 16:31 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2044.codfw.wmnet
• 16:31 claime: Repooling kubernetes2044.codfw.wmnet kubernetes2045.codfw.wmnet - T373102
• 16:31 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2044.codfw.wmnet
• 16:24 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
• 16:23 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
• 16:21 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet
• 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2209 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69073 and previous config saved to /var/cache/conftool/dbconfig/20240912-162007-arnaudb.json
• 16:19 arnaudb@cumin1002: dbctl commit (dc=all): 'es2039 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69072 and previous config saved to /var/cache/conftool/dbconfig/20240912-161957-arnaudb.json
• 16:19 arnaudb@cumin1002: dbctl commit (dc=all): 'es2034 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69071 and previous config saved to /var/cache/conftool/dbconfig/20240912-161952-arnaudb.json
• 16:19 arnaudb@cumin1002: dbctl commit (dc=all): 'es2033 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69070 and previous config saved to /var/cache/conftool/dbconfig/20240912-161947-arnaudb.json
• 16:19 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69069 and previous config saved to /var/cache/conftool/dbconfig/20240912-161942-arnaudb.json
• 16:19 arnaudb@cumin1002: dbctl commit (dc=all): 'db2211 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69068 and previous config saved to /var/cache/conftool/dbconfig/20240912-161937-arnaudb.json
• 16:19 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
• 16:19 arnaudb@cumin1002: dbctl commit (dc=all): 'db2171 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69067 and previous config saved to /var/cache/conftool/dbconfig/20240912-161932-arnaudb.json
• 16:19 arnaudb@cumin1002: dbctl commit (dc=all): 'db2170 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69066 and previous config saved to /var/cache/conftool/dbconfig/20240912-161927-arnaudb.json
• 16:19 arnaudb@cumin1002: dbctl commit (dc=all): 'db2151 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69065 and previous config saved to /var/cache/conftool/dbconfig/20240912-161922-arnaudb.json
• 16:19 arnaudb@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P69064 and previous config saved to /var/cache/conftool/dbconfig/20240912-161916-arnaudb.json
• 16:19 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
• 16:18 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns2006.wikimedia.org [reason: [end] T373102 codfw maintenance]
• 16:14 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
• 16:13 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
• 16:12 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
• 16:12 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
• 16:09 jynus: restart ms-backup200[12] after maintenance and upgrade
• 16:08 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
• 16:08 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
• 16:07 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dns2006.wikimedia.org
• 16:07 sukhe@cumin1002: START - Cookbook sre.hosts.remove-downtime for dns2006.wikimedia.org
• 16:01 topranks: move server uplinks in codfw rack D1 from asw-d1-codfw to lsw1-d1-codfw T373102
• 16:01 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker2110.codfw.wmnet
• 16:00 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2110.codfw.wmnet
• 16:00 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2110.codfw.wmnet
• 16:00 swfrench@cumin1002: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in all: None - None
• 16:00 swfrench@cumin1002: START - Cookbook sre.discovery.datacenter status all services in all: None - None
• 15:59 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2113.codfw.wmnet
• 15:59 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2113.codfw.wmnet
• 15:59 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2113.codfw.wmnet
• 15:58 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker2112.codfw.wmnet
• 15:58 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2112.codfw.wmnet
• 15:58 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2112.codfw.wmnet
• 15:56 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker2111.codfw.wmnet
• 15:56 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2111.codfw.wmnet
• 15:56 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2111.codfw.wmnet
• 15:51 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:40:00 on 21 hosts with reason: Move server uplinks codfw racks D2
• 15:50 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:40:00 on 21 hosts with reason: Move server uplinks codfw racks D2
• 15:50 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:40:00 on 21 hosts with reason: Move server uplinks codfw racks D1
• 15:50 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:40:00 on 21 hosts with reason: Move server uplinks codfw racks D1
• 15:49 cmooney@cumin1002: END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 0:20:00 on 21 hosts with reason: Move server uplinks codfw racks D1
• 15:49 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:20:00 on 21 hosts with reason: Move server uplinks codfw racks D1
• 15:48 swfrench@cumin1002: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in all: None - None
• 15:48 swfrench@cumin1002: START - Cookbook sre.discovery.datacenter status all services in all: None - None
• 15:48 swfrench@cumin1002: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in all: None - None
• 15:47 swfrench@cumin1002: START - Cookbook sre.discovery.datacenter status all services in all: None - None
• 15:42 urandom: depooling ms-fe2012 moss-fe2002 & thanos-fe2003 — T373102
• 15:40 arnaudb@cumin1002: dbctl commit (dc=all): 'depool es2034 which was perceived master for es3 - T370852', diff saved to https://phabricator.wikimedia.org/P69063 and previous config saved to /var/cache/conftool/dbconfig/20240912-154008-arnaudb.json
• 15:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:45:00 on 11 hosts with reason: network maintenance T373101
• 15:37 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:45:00 on 11 hosts with reason: network maintenance T373101
• 15:37 arnaudb@cumin1002: dbctl commit (dc=all): 'depool db2128 db2151 db2170 db2171 db2211 db2212 es2033 es2034 es2039 pc2014 db2209 - T370852', diff saved to https://phabricator.wikimedia.org/P69062 and previous config saved to /var/cache/conftool/dbconfig/20240912-153720-arnaudb.json
• 15:33 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2045.codfw.wmnet
• 15:30 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2045.codfw.wmnet
• 15:30 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2044.codfw.wmnet
• 15:29 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2044.codfw.wmnet
• 15:29 claime: Depooling kubernetes2044.codfw.wmnet kubernetes2045.codfw.wmnet - T373102
• 15:26 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=dns2006.wikimedia.org [reason: T373102 codfw maintenance]
• 15:20 zabe: zabe@mwmaint1002:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTable.php {fawikiquote,fawikisource,fawiktionary} --skip /home/zabe/text_table_cleanup/{fawikiquote,fawikisource,fawiktionary} --dump /home/zabe/text_table_dump/{fawikiquote,fawikisource,fawiktionary} --sleep 1 # T183490
• 15:11 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 15:11 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 15:00 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 15:00 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 14:59 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 14:59 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 14:52 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 14:52 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 14:47 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2113.codfw.wmnet with OS bullseye
• 14:38 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 14:38 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 14:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1209 (T367781)', diff saved to https://phabricator.wikimedia.org/P69060 and previous config saved to /var/cache/conftool/dbconfig/20240912-143813-arnaudb.json
• 14:30 cscott: cleanupTitles on enwiki complete (T363538)
• 14:28 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2113.codfw.wmnet with reason: host reimage
• 14:25 akosiaris@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2113.codfw.wmnet with reason: host reimage
• 14:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2112.codfw.wmnet with OS bullseye
• 14:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P69059 and previous config saved to /var/cache/conftool/dbconfig/20240912-142306-arnaudb.json
• 14:18 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2109.codfw.wmnet
• 14:18 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2109.codfw.wmnet
• 14:18 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2109.codfw.wmnet
• 14:17 sukhe: sudo cumin "A:dnsbox" "rm /etc/ntp.conf": cleaning up ntpd configuration file to avoid confusion with ntpsec.conf
• 14:10 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2110.codfw.wmnet with OS bullseye
• 14:09 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2113
• 14:09 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2113
• 14:09 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2113
• 14:09 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2113.codfw.wmnet 63.0.192.10.in-addr.arpa 3.6.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 14:09 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2113.codfw.wmnet 63.0.192.10.in-addr.arpa 3.6.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 14:09 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:09 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2113 - akosiaris@cumin1002"
• 14:09 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2113 - akosiaris@cumin1002"
• 14:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P69058 and previous config saved to /var/cache/conftool/dbconfig/20240912-140758-arnaudb.json
• 14:04 denisse: Make alert2002 the active host - T372418
• 14:04 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2112.codfw.wmnet with reason: host reimage
• 14:02 denisse: Disable meta-monitoring for the alert hosts - T372418
• 14:02 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2109.codfw.wmnet with OS bullseye
• 14:01 denisse: Enable the alert[12]002 hosts as alertmanagers - T372418
• 14:01 denisse: Enable the alert[12]002 hosts as alertmanagers
• 14:00 akosiaris@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2112.codfw.wmnet with reason: host reimage
• 13:56 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2108.codfw.wmnet with OS bullseye
• 13:56 cscott: mwscript cleanupTitles enwiki 2>&1 | tee ~/T363538-enwiki-cleanupTitles
• 13:56 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2107.codfw.wmnet
• 13:56 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2107.codfw.wmnet
• 13:56 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2107.codfw.wmnet
• 13:55 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2111.codfw.wmnet with reason: host reimage
• 13:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1209 (T367781)', diff saved to https://phabricator.wikimedia.org/P69057 and previous config saved to /var/cache/conftool/dbconfig/20240912-135251-arnaudb.json
• 13:52 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 13:52 akosiaris@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2113
• 13:52 akosiaris@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2111.codfw.wmnet with reason: host reimage
• 13:52 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2113.codfw.wmnet with OS bullseye
• 13:51 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2113.codfw.wmnet
• 13:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1209 (T367781)', diff saved to https://phabricator.wikimedia.org/P69056 and previous config saved to /var/cache/conftool/dbconfig/20240912-135142-arnaudb.json
• 13:51 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2113.codfw.wmnet on all recursors
• 13:51 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2113.codfw.wmnet on all recursors
• 13:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1209.eqiad.wmnet with reason: Maintenance
• 13:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1209.eqiad.wmnet with reason: Maintenance
• 13:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T367781)', diff saved to https://phabricator.wikimedia.org/P69055 and previous config saved to /var/cache/conftool/dbconfig/20240912-135131-arnaudb.json
• 13:50 akosiaris: homer cr*codfw* commit 'T372878'
• 13:50 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T370903)', diff saved to https://phabricator.wikimedia.org/P69054 and previous config saved to /var/cache/conftool/dbconfig/20240912-135003-ladsgroup.json
• 13:50 cscott: mwscript namespaceDupes enwiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-suffix=/T363538 --fix 2>&1 | tee ~/T363538-enwiki-namespaceDupes.take2
• 13:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2110.codfw.wmnet with reason: host reimage
• 13:49 cscott: namespaceDupes crashed on MOS:_OVERLINKING, re-running with --add-suffix
• 13:48 akosiaris: homer lsw1-a3-codfw* commit 'T372878'
• 13:47 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2107.codfw.wmnet with OS bullseye
• 13:47 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2113.codfw.wmnet
• 13:47 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2113.codfw.wmnet
• 13:46 akosiaris@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2113.codfw.wmnet
• 13:46 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2113.codfw.wmnet
• 13:45 akosiaris@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2110.codfw.wmnet with reason: host reimage
• 13:44 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2112
• 13:44 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2112
• 13:44 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2112
• 13:44 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2112.codfw.wmnet 62.0.192.10.in-addr.arpa 2.6.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:44 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2112.codfw.wmnet 62.0.192.10.in-addr.arpa 2.6.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:44 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:44 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2399 to wikikube-worker2113
• 13:43 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2113
• 13:43 cscott: mwscript namespaceDupes enwiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-prefix=T363538/ --fix | tee ~/T363538-enwiki-namespaceDupes
• 13:42 hashar: Afternoon backport deployments are completed . NamespaceDupe is being run on enwiki for T363538#10140642
• 13:42 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2109.codfw.wmnet with reason: host reimage
• 13:42 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2113
• 13:42 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:42 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2399 to wikikube-worker2113 - akosiaris@cumin1002"
• 13:42 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2399 to wikikube-worker2113 - akosiaris@cumin1002"
• 13:42 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 13:40 akosiaris@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2109.codfw.wmnet with reason: host reimage
• 13:38 hashar@deploy1003: Finished scap sync-world: Backport for Elevate pseudo-namespace MOS to a real namespace on enwiki (T363538) (duration: 06m 39s)
• 13:37 elukey@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host poolcounter2006.codfw.wmnet
• 13:37 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host poolcounter2006.codfw.wmnet with OS bookworm
• 13:36 akosiaris@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2112
• 13:36 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 13:36 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2111
• 13:36 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2111
• 13:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P69052 and previous config saved to /var/cache/conftool/dbconfig/20240912-133623-arnaudb.json
• 13:36 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet
• 13:36 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2111
• 13:36 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2111.codfw.wmnet 61.0.192.10.in-addr.arpa 1.6.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:36 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2111.codfw.wmnet 61.0.192.10.in-addr.arpa 1.6.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:36 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:36 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2111 - akosiaris@cumin1002"
• 13:35 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2111 - akosiaris@cumin1002"
• 13:35 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2108.codfw.wmnet with reason: host reimage
• 13:35 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet
• 13:34 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P69051 and previous config saved to /var/cache/conftool/dbconfig/20240912-133456-ladsgroup.json
• 13:34 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2112.codfw.wmnet with OS bullseye
• 13:34 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2112.codfw.wmnet
• 13:33 akosiaris@cumin1002: START - Cookbook sre.hosts.rename from mw2399 to wikikube-worker2113
• 13:33 hashar@deploy1003: cscott, hashar: Continuing with sync
• 13:33 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2398 to wikikube-worker2112
• 13:33 hashar@deploy1003: cscott, hashar: Backport for Elevate pseudo-namespace MOS to a real namespace on enwiki (T363538) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:32 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2112
• 13:32 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 13:32 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2112
• 13:32 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:32 akosiaris@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2108.codfw.wmnet with reason: host reimage
• 13:31 hashar@deploy1003: Started scap sync-world: Backport for Elevate pseudo-namespace MOS to a real namespace on enwiki (T363538)
• 13:30 akosiaris@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2111
• 13:30 cmooney@cumin1002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2025.codfw.wmnet
• 13:30 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2110
• 13:30 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2110
• 13:30 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 13:29 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2110
• 13:29 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2110.codfw.wmnet 60.0.192.10.in-addr.arpa 0.6.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:29 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2110.codfw.wmnet 60.0.192.10.in-addr.arpa 0.6.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:29 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:29 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2110 - akosiaris@cumin1002"
• 13:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T367781)', diff saved to https://phabricator.wikimedia.org/P69050 and previous config saved to /var/cache/conftool/dbconfig/20240912-132943-arnaudb.json
• 13:29 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2110 - akosiaris@cumin1002"
• 13:28 cmooney@cumin1002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2025.codfw.wmnet
• 13:28 hashar@deploy1003: Finished scap sync-world: Backport for logging: Fix WikimediaDebug "Verbose logging" option (T374583) (duration: 07m 06s)
• 13:28 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2107.codfw.wmnet with reason: host reimage
• 13:26 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2111.codfw.wmnet with OS bullseye
• 13:26 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2111.codfw.wmnet
• 13:26 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 13:25 akosiaris@cumin1002: START - Cookbook sre.hosts.rename from mw2398 to wikikube-worker2112
• 13:25 akosiaris@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2110
• 13:25 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2110.codfw.wmnet with OS bullseye
• 13:25 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2110.codfw.wmnet
• 13:25 akosiaris@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2107.codfw.wmnet with reason: host reimage
• 13:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2397 to wikikube-worker2111
• 13:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2109
• 13:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2109
• 13:24 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2109
• 13:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2109.codfw.wmnet 59.0.192.10.in-addr.arpa 9.5.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:24 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2109.codfw.wmnet 59.0.192.10.in-addr.arpa 9.5.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2109 - akosiaris@cumin1002"
• 13:24 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2109 - akosiaris@cumin1002"
• 13:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2111
• 13:24 hashar@deploy1003: matmarex, hashar: Continuing with sync
• 13:24 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2111
• 13:24 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:23 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2397 to wikikube-worker2111 - akosiaris@cumin1002"
• 13:23 hashar@deploy1003: matmarex, hashar: Backport for logging: Fix WikimediaDebug "Verbose logging" option (T374583) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:23 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2397 to wikikube-worker2111 - akosiaris@cumin1002"
• 13:22 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet
• 13:21 hashar@deploy1003: Started scap sync-world: Backport for logging: Fix WikimediaDebug "Verbose logging" option (T374583)
• 13:21 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on poolcounter2006.codfw.wmnet with reason: host reimage
• 13:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P69049 and previous config saved to /var/cache/conftool/dbconfig/20240912-132116-arnaudb.json
• 13:20 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 13:19 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P69048 and previous config saved to /var/cache/conftool/dbconfig/20240912-131948-ladsgroup.json
• 13:18 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on poolcounter2006.codfw.wmnet with reason: host reimage
• 13:15 akosiaris@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2109
• 13:15 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2108
• 13:15 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2108
• 13:15 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2108
• 13:15 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2108.codfw.wmnet 58.0.192.10.in-addr.arpa 8.5.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:15 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 13:15 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2108.codfw.wmnet 58.0.192.10.in-addr.arpa 8.5.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:14 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P69047 and previous config saved to /var/cache/conftool/dbconfig/20240912-131436-arnaudb.json
• 13:14 akosiaris@cumin1002: START - Cookbook sre.hosts.rename from mw2397 to wikikube-worker2111
• 13:12 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2396 to wikikube-worker2110
• 13:12 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 13:12 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2110
• 13:12 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2110
• 13:12 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:11 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2396 to wikikube-worker2110 - akosiaris@cumin1002"
• 13:11 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2396 to wikikube-worker2110 - akosiaris@cumin1002"
• 13:10 cmooney@cumin1002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2015.codfw.wmnet
• 13:09 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2109.codfw.wmnet with OS bullseye
• 13:09 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2109.codfw.wmnet
• 13:09 cmooney@cumin1002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2015.codfw.wmnet
• 13:08 akosiaris@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2108
• 13:08 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 13:08 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2107
• 13:08 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2107
• 13:08 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2107
• 13:08 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2107.codfw.wmnet 53.0.192.10.in-addr.arpa 3.5.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:08 akosiaris@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2107.codfw.wmnet 53.0.192.10.in-addr.arpa 3.5.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:08 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:08 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2107 - akosiaris@cumin1002"
• 13:08 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2107 - akosiaris@cumin1002"
• 13:06 akosiaris@cumin1002: START - Cookbook sre.hosts.rename from mw2396 to wikikube-worker2110
• 13:06 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2108.codfw.wmnet with OS bullseye
• 13:06 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2395 to wikikube-worker2109
• 13:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T367781)', diff saved to https://phabricator.wikimedia.org/P69046 and previous config saved to /var/cache/conftool/dbconfig/20240912-130608-arnaudb.json
• 13:06 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2108.codfw.wmnet
• 13:05 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2109
• 13:04 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 13:04 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T370903)', diff saved to https://phabricator.wikimedia.org/P69045 and previous config saved to /var/cache/conftool/dbconfig/20240912-130441-ladsgroup.json
• 13:04 akosiaris@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2107
• 13:04 akosiaris@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2107.codfw.wmnet with OS bullseye
• 13:04 akosiaris@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2107.codfw.wmnet
• 13:04 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2109
• 13:04 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:04 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2395 to wikikube-worker2109 - akosiaris@cumin1002"
• 13:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1192 (T367781)', diff saved to https://phabricator.wikimedia.org/P69044 and previous config saved to /var/cache/conftool/dbconfig/20240912-130400-arnaudb.json
• 13:03 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2395 to wikikube-worker2109 - akosiaris@cumin1002"
• 13:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1192.eqiad.wmnet with reason: Maintenance
• 13:03 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1192.eqiad.wmnet with reason: Maintenance
• 13:02 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host poolcounter2006.codfw.wmnet with OS bookworm
• 12:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P69043 and previous config saved to /var/cache/conftool/dbconfig/20240912-125928-arnaudb.json
• 12:59 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM poolcounter2006.codfw.wmnet - elukey@cumin1002"
• 12:59 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM poolcounter2006.codfw.wmnet - elukey@cumin1002"
• 12:58 elukey@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) poolcounter2006.codfw.wmnet on all recursors
• 12:58 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 12:58 elukey@cumin1002: START - Cookbook sre.dns.wipe-cache poolcounter2006.codfw.wmnet on all recursors
• 12:58 elukey@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:58 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM poolcounter2006.codfw.wmnet - elukey@cumin1002"
• 12:57 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM poolcounter2006.codfw.wmnet - elukey@cumin1002"
• 12:57 akosiaris@cumin1002: START - Cookbook sre.hosts.rename from mw2395 to wikikube-worker2109
• 12:55 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2394 to wikikube-worker2108
• 12:54 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2108
• 12:54 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2108
• 12:54 elukey@cumin1002: START - Cookbook sre.dns.netbox
• 12:54 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:54 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2394 to wikikube-worker2108 - akosiaris@cumin1002"
• 12:53 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2394 to wikikube-worker2108 - akosiaris@cumin1002"
• 12:53 elukey@cumin1002: START - Cookbook sre.ganeti.makevm for new host poolcounter2006.codfw.wmnet
• 12:50 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host deploy2002.codfw.wmnet
• 12:50 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 12:50 akosiaris@cumin1002: START - Cookbook sre.hosts.rename from mw2394 to wikikube-worker2108
• 12:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2390 to wikikube-worker2107
• 12:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2107
• 12:49 akosiaris@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2107
• 12:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:49 akosiaris@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2390 to wikikube-worker2107 - akosiaris@cumin1002"
• 12:48 akosiaris@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2390 to wikikube-worker2107 - akosiaris@cumin1002"
• 12:46 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2209 (T370903)', diff saved to https://phabricator.wikimedia.org/P69042 and previous config saved to /var/cache/conftool/dbconfig/20240912-124626-ladsgroup.json
• 12:46 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2139.codfw.wmnet with reason: Maintenance
• 12:46 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on db2139.codfw.wmnet with reason: Maintenance
• 12:46 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2209.codfw.wmnet with reason: Maintenance
• 12:45 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db2209.codfw.wmnet with reason: Maintenance
• 12:44 akosiaris@cumin1002: START - Cookbook sre.dns.netbox
• 12:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T367781)', diff saved to https://phabricator.wikimedia.org/P69041 and previous config saved to /var/cache/conftool/dbconfig/20240912-124421-arnaudb.json
• 12:44 akosiaris@cumin1002: START - Cookbook sre.hosts.rename from mw2390 to wikikube-worker2107
• 12:41 elukey: thumbor codfw on wikikube moved to poolcounter2005 - T332015
• 12:38 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: sync
• 12:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2123 (T367781)', diff saved to https://phabricator.wikimedia.org/P69040 and previous config saved to /var/cache/conftool/dbconfig/20240912-123631-arnaudb.json
• 12:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2123.codfw.wmnet with reason: Maintenance
• 12:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2123.codfw.wmnet with reason: Maintenance
• 12:35 elukey@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: sync
• 12:31 akosiaris: depool mw239[0456789] for re-numbering, renaming and reimaging.
• 12:29 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2399.codfw.wmnet
• 12:29 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host deploy2002.codfw.wmnet
• 12:28 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2399.codfw.wmnet
• 12:28 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2398.codfw.wmnet
• 12:28 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2398.codfw.wmnet
• 12:27 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2397.codfw.wmnet
• 12:27 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2397.codfw.wmnet
• 12:27 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2396.codfw.wmnet
• 12:26 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2396.codfw.wmnet
• 12:26 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2395.codfw.wmnet
• 12:26 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2395.codfw.wmnet
• 12:26 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2394.codfw.wmnet
• 12:25 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2394.codfw.wmnet
• 12:25 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2390.codfw.wmnet
• 12:24 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2390.codfw.wmnet
• 12:19 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1085.eqiad.wmnet
• 12:11 btullis@cumin1002: START - Cookbook sre.hosts.reboot-single for host an-worker1085.eqiad.wmnet
• 11:47 urbanecm@deploy1003: Finished scap sync-world: Backport for Babel: Set BabelUseCommunityConfiguration to false (T374611) (duration: 11m 28s)
• 11:47 jynus: restarting db1171:s7 mysql process T374610
• 11:38 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
• 11:35 urbanecm@deploy1003: Started scap sync-world: Backport for Babel: Set BabelUseCommunityConfiguration to false (T374611)
• 11:20 damilare: SmashPig upgraded from eb7807f8 to 08c79f4f
• 11:14 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
• 11:13 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
• 11:06 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2029.codfw.wmnet
• 11:06 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2029.codfw.wmnet
• 11:05 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2093.codfw.wmnet
• 11:05 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2093.codfw.wmnet
• 11:03 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
• 11:03 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
• 10:59 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
• 10:59 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
• 10:34 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T371742)', diff saved to https://phabricator.wikimedia.org/P69039 and previous config saved to /var/cache/conftool/dbconfig/20240912-103434-ladsgroup.json
• 10:33 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.0.e.f.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa. on all recursors
• 10:32 cmooney@cumin1002: START - Cookbook sre.dns.wipe-cache 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.0.e.f.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa. on all recursors
• 10:25 btullis: stopping envoyproxy on cephosd1001
• 10:25 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:25 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns entries for et-0-0-31-100.ssw1-f1-eqiad.eqiad.wmnet - cmooney@cumin1002"
• 10:25 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns entries for et-0-0-31-100.ssw1-f1-eqiad.eqiad.wmnet - cmooney@cumin1002"
• 10:19 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P69038 and previous config saved to /var/cache/conftool/dbconfig/20240912-101927-ladsgroup.json
• 10:08 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-wikifunctions: apply
• 10:08 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-wikifunctions: apply
• 10:08 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-wikifunctions: apply
• 10:08 btullis: restarted envoyproxy on cephosd1001
• 10:08 claime: Increasing mw-wikifunctions replicas to 6
• 09:59 btullis: stopping envoyproxy on cephosd1001
• 09:53 arnaudb@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1246.eqiad.wmnet with OS bookworm
• 09:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 75%: post db2229 bootstrap', diff saved to https://phabricator.wikimedia.org/P69034 and previous config saved to /var/cache/conftool/dbconfig/20240912-095335-arnaudb.json
• 09:49 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T371742)', diff saved to https://phabricator.wikimedia.org/P69033 and previous config saved to /var/cache/conftool/dbconfig/20240912-094912-ladsgroup.json
• 09:41 dcausse@deploy1003: helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply
• 09:41 dcausse@deploy1003: helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply
• 09:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 50%: post db2229 bootstrap', diff saved to https://phabricator.wikimedia.org/P69032 and previous config saved to /var/cache/conftool/dbconfig/20240912-093829-arnaudb.json
• 09:35 elukey@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
• 09:32 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 09:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 25%: post db2229 bootstrap', diff saved to https://phabricator.wikimedia.org/P69031 and previous config saved to /var/cache/conftool/dbconfig/20240912-092324-arnaudb.json
• 09:23 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-main2004.codfw.wmnet
• 09:23 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:23 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-main2004.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002"
• 09:22 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-main2004.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002"
• 09:19 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 09:16 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 09:14 jayme@cumin1002: START - Cookbook sre.hosts.decommission for hosts kafka-main2004.codfw.wmnet
• 09:12 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 09:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 15%: post db2229 bootstrap', diff saved to https://phabricator.wikimedia.org/P69030 and previous config saved to /var/cache/conftool/dbconfig/20240912-090818-arnaudb.json
• 09:07 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 09:06 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply
• 09:01 arnaudb@cumin1002: dbctl commit (dc=all): 'T374421', diff saved to https://phabricator.wikimedia.org/P69029 and previous config saved to /var/cache/conftool/dbconfig/20240912-090157-arnaudb.json
• 08:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2205 to s3 primary T374421', diff saved to https://phabricator.wikimedia.org/P69028 and previous config saved to /var/cache/conftool/dbconfig/20240912-085859-arnaudb.json
• 08:54 arnaudb: Starting s3 codfw failover from db2209 to db2205 - T374421
• 08:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 10%: post db2229 bootstrap', diff saved to https://phabricator.wikimedia.org/P69027 and previous config saved to /var/cache/conftool/dbconfig/20240912-085312-arnaudb.json
• 08:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2216 (T371742)', diff saved to https://phabricator.wikimedia.org/P69026 and previous config saved to /var/cache/conftool/dbconfig/20240912-085232-ladsgroup.json
• 08:52 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2216.codfw.wmnet with reason: Maintenance
• 08:52 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
• 08:52 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2216.codfw.wmnet with reason: Maintenance
• 08:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212 (T371742)', diff saved to https://phabricator.wikimedia.org/P69025 and previous config saved to /var/cache/conftool/dbconfig/20240912-085209-ladsgroup.json
• 08:50 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
• 08:49 jayme: restoring leadership for all partitions assigned to broker id 2004 on kafka-main-codfw - T363210
• 08:47 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main2009.codfw.wmnet
• 08:47 jayme@cumin1002: START - Cookbook sre.hosts.remove-downtime for kafka-main2009.codfw.wmnet
• 08:44 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
• 08:42 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica to new version
• 08:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 5%: post db2229 bootstrap', diff saved to https://phabricator.wikimedia.org/P69024 and previous config saved to /var/cache/conftool/dbconfig/20240912-083807-arnaudb.json
• 08:37 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P69023 and previous config saved to /var/cache/conftool/dbconfig/20240912-083701-ladsgroup.json
• 08:35 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica to new version
• 08:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 4%: post db2229 bootstrap', diff saved to https://phabricator.wikimedia.org/P69022 and previous config saved to /var/cache/conftool/dbconfig/20240912-082301-arnaudb.json
• 08:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P69021 and previous config saved to /var/cache/conftool/dbconfig/20240912-082154-ladsgroup.json
• 08:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 3%: post db2229 bootstrap', diff saved to https://phabricator.wikimedia.org/P69020 and previous config saved to /var/cache/conftool/dbconfig/20240912-080756-arnaudb.json
• 08:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212 (T371742)', diff saved to https://phabricator.wikimedia.org/P69019 and previous config saved to /var/cache/conftool/dbconfig/20240912-080647-ladsgroup.json
• 07:58 gmodena@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 07:58 gmodena@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply
• 07:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 2%: post db2229 bootstrap', diff saved to https://phabricator.wikimedia.org/P69018 and previous config saved to /var/cache/conftool/dbconfig/20240912-075250-arnaudb.json
• 07:46 gmodena@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 07:46 gmodena@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply
• 07:39 gmodena@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 07:38 gmodena@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply
• 07:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 1%: post db2229 bootstrap', diff saved to https://phabricator.wikimedia.org/P69017 and previous config saved to /var/cache/conftool/dbconfig/20240912-073744-arnaudb.json
• 07:34 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2129.codfw.wmnet onto db2229.codfw.wmnet
• 07:28 jayme@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-codfw
• 07:22 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 07:22 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
• 07:22 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 07:21 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 07:21 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
• 07:21 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
• 07:21 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
• 07:20 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
• 07:20 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
• 07:20 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
• 07:20 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 07:20 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 07:19 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 07:19 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 07:19 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 07:19 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
• 07:19 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 07:19 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 07:18 slyngshede@cumin1002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Sandeeps out of all services on: 2298 hosts
• 07:18 slyngshede@cumin1002: START - Cookbook sre.idm.logout Logging Sandeeps out of all services on: 2298 hosts
• 07:10 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2212 (T371742)', diff saved to https://phabricator.wikimedia.org/P69016 and previous config saved to /var/cache/conftool/dbconfig/20240912-071034-ladsgroup.json
• 07:10 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2212.codfw.wmnet with reason: Maintenance
• 07:10 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2212.codfw.wmnet with reason: Maintenance
• 07:09 jayme@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-codfw
• 06:58 arnaudb@cumin1002: START - Cookbook sre.mysql.clone of db2129.codfw.wmnet onto db2229.codfw.wmnet
• 06:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2129 in db2229 for T373579', diff saved to https://phabricator.wikimedia.org/P69015 and previous config saved to /var/cache/conftool/dbconfig/20240912-065641-arnaudb.json
• 06:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2229.codfw.wmnet with reason: provisionning db2229.codfw.wmnet - T373579
• 06:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2229.codfw.wmnet with reason: provisionning db2229.codfw.wmnet - T373579
• 06:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2129.codfw.wmnet with reason: provisionning db2229.codfw.wmnet - T373579
• 06:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2129.codfw.wmnet with reason: provisionning db2229.codfw.wmnet - T373579
• 06:34 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on kafka-main[2004,2009].codfw.wmnet with reason: Hardware refresh
• 06:34 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on kafka-main[2004,2009].codfw.wmnet with reason: Hardware refresh
• 06:33 jayme: evacuating leadership for all partitions assigned to broker id 2004 on kafka-main-codfw - T363210
• 06:19 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s3 T374421
• 06:19 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 25 hosts with reason: Primary switchover s3 T374421
• 06:16 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2202.codfw.wmnet with reason: Maintenance
• 06:16 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2202.codfw.wmnet with reason: Maintenance
• 06:16 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T371742)', diff saved to https://phabricator.wikimedia.org/P69014 and previous config saved to /var/cache/conftool/dbconfig/20240912-061639-ladsgroup.json
• 06:05 arnaudb@cumin1002: dbctl commit (dc=all): 'T374592', diff saved to https://phabricator.wikimedia.org/P69013 and previous config saved to /var/cache/conftool/dbconfig/20240912-060550-arnaudb.json
• 06:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote es2038 to es7 primary and set section read-write T374592', diff saved to https://phabricator.wikimedia.org/P69012 and previous config saved to /var/cache/conftool/dbconfig/20240912-060308-arnaudb.json
• 06:02 arnaudb: Starting es7 codfw failover from es2039 to es2038 - T374592
• 06:01 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P69011 and previous config saved to /var/cache/conftool/dbconfig/20240912-060131-ladsgroup.json
• 05:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Set es2038 with weight 0 T374592', diff saved to https://phabricator.wikimedia.org/P69010 and previous config saved to /var/cache/conftool/dbconfig/20240912-055903-arnaudb.json
• 05:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es7 T374592
• 05:58 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es7 T374592
• 05:46 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P69009 and previous config saved to /var/cache/conftool/dbconfig/20240912-054624-ladsgroup.json
• 05:44 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm
• 05:31 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T371742)', diff saved to https://phabricator.wikimedia.org/P69008 and previous config saved to /var/cache/conftool/dbconfig/20240912-053116-ladsgroup.json
• 04:37 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2188 (T371742)', diff saved to https://phabricator.wikimedia.org/P69007 and previous config saved to /var/cache/conftool/dbconfig/20240912-043701-ladsgroup.json
• 04:36 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2188.codfw.wmnet with reason: Maintenance
• 04:36 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2188.codfw.wmnet with reason: Maintenance
• 04:36 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T371742)', diff saved to https://phabricator.wikimedia.org/P69006 and previous config saved to /var/cache/conftool/dbconfig/20240912-043628-ladsgroup.json
• 04:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P69005 and previous config saved to /var/cache/conftool/dbconfig/20240912-042121-ladsgroup.json
• 04:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P69004 and previous config saved to /var/cache/conftool/dbconfig/20240912-040613-ladsgroup.json
• 03:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T371742)', diff saved to https://phabricator.wikimedia.org/P69003 and previous config saved to /var/cache/conftool/dbconfig/20240912-035105-ladsgroup.json
• 02:46 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2176 (T371742)', diff saved to https://phabricator.wikimedia.org/P69002 and previous config saved to /var/cache/conftool/dbconfig/20240912-024635-ladsgroup.json
• 02:46 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2176.codfw.wmnet with reason: Maintenance
• 02:46 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2176.codfw.wmnet with reason: Maintenance
• 02:46 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T371742)', diff saved to https://phabricator.wikimedia.org/P69001 and previous config saved to /var/cache/conftool/dbconfig/20240912-024612-ladsgroup.json
• 02:31 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P69000 and previous config saved to /var/cache/conftool/dbconfig/20240912-023105-ladsgroup.json
• 02:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P68999 and previous config saved to /var/cache/conftool/dbconfig/20240912-021557-ladsgroup.json
• 02:00 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T371742)', diff saved to https://phabricator.wikimedia.org/P68998 and previous config saved to /var/cache/conftool/dbconfig/20240912-020050-ladsgroup.json
• 00:58 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2174 (T371742)', diff saved to https://phabricator.wikimedia.org/P68996 and previous config saved to /var/cache/conftool/dbconfig/20240912-005830-ladsgroup.json
• 00:58 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2174.codfw.wmnet with reason: Maintenance
• 00:58 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2174.codfw.wmnet with reason: Maintenance
• 00:58 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T371742)', diff saved to https://phabricator.wikimedia.org/P68995 and previous config saved to /var/cache/conftool/dbconfig/20240912-005808-ladsgroup.json
• 00:43 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P68994 and previous config saved to /var/cache/conftool/dbconfig/20240912-004301-ladsgroup.json
• 00:27 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P68993 and previous config saved to /var/cache/conftool/dbconfig/20240912-002753-ladsgroup.json
• 00:12 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T371742)', diff saved to https://phabricator.wikimedia.org/P68992 and previous config saved to /var/cache/conftool/dbconfig/20240912-001246-ladsgroup.json
• 00:04 eileen: civicrm upgraded from 929101dc to ac29ff45

2024-09-11

• 23:19 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1041.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 23:18 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host ganeti1041.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 23:13 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1052.eqiad.wmnet with OS bookworm
• 23:13 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2173 (T371742)', diff saved to https://phabricator.wikimedia.org/P68991 and previous config saved to /var/cache/conftool/dbconfig/20240911-231311-ladsgroup.json
• 23:13 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
• 23:12 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
• 23:12 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2173.codfw.wmnet with reason: Maintenance
• 23:12 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2173.codfw.wmnet with reason: Maintenance
• 23:12 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T371742)', diff saved to https://phabricator.wikimedia.org/P68990 and previous config saved to /var/cache/conftool/dbconfig/20240911-231233-ladsgroup.json
• 22:57 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P68989 and previous config saved to /var/cache/conftool/dbconfig/20240911-225726-ladsgroup.json
• 22:46 jforrester@deploy1003: Finished scap sync-world: Backport for SpecialExpandTemplates: Replace use of deprecated OutputPage::addCategoryLinks() (T373830), SpecialExpandTemplates: Replace use of deprecated OutputPage::addCategoryLinks() (T373830) (duration: 07m 27s)
• 22:42 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P68988 and previous config saved to /var/cache/conftool/dbconfig/20240911-224218-ladsgroup.json
• 22:41 jforrester@deploy1003: jforrester: Continuing with sync
• 22:41 jforrester@deploy1003: jforrester: Backport for SpecialExpandTemplates: Replace use of deprecated OutputPage::addCategoryLinks() (T373830), SpecialExpandTemplates: Replace use of deprecated OutputPage::addCategoryLinks() (T373830) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 22:38 jforrester@deploy1003: Started scap sync-world: Backport for SpecialExpandTemplates: Replace use of deprecated OutputPage::addCategoryLinks() (T373830), SpecialExpandTemplates: Replace use of deprecated OutputPage::addCategoryLinks() (T373830)
• 22:28 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1041.eqiad.wmnet with OS bookworm
• 22:27 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:27 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:27 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T371742)', diff saved to https://phabricator.wikimedia.org/P68987 and previous config saved to /var/cache/conftool/dbconfig/20240911-222711-ladsgroup.json
• 22:26 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1041.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:26 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1041.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:21 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1052.eqiad.wmnet with OS bookworm
• 22:20 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1052.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1052.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:19 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:19 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:19 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1050.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:18 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1050.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:18 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1044.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:18 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1049.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:17 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1049.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:17 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1048.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:16 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1048.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:15 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1047.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:14 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1047.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:14 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1046.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:13 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1046.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:13 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1045.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:12 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1045.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:11 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1044.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:03 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1052
• 22:02 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1051
• 22:01 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1052
• 22:01 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1051
• 22:01 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1050
• 22:00 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 22:00 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 22:00 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 22:00 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 21:59 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1050
• 21:59 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1049
• 21:59 jforrester@deploy1003: Finished scap sync-world: Backport for ZObjectFactory::validatePersistentKeys: Disable use of JsonSchema, at least temporarily (T374241), ZObjectFactory::validatePersistentKeys: Disable use of JsonSchema, at least temporarily (T374241) (duration: 07m 51s)
• 21:58 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1049
• 21:58 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1048
• 21:57 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1048
• 21:57 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1047
• 21:56 inflatador: bking@deploy1003 test deploy of flink operator in staging cancelled with no changes T373195
• 21:56 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1047
• 21:56 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1046
• 21:54 jforrester@deploy1003: jforrester: Continuing with sync
• 21:54 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1046
• 21:54 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1045
• 21:53 jforrester@deploy1003: jforrester: Backport for ZObjectFactory::validatePersistentKeys: Disable use of JsonSchema, at least temporarily (T374241), ZObjectFactory::validatePersistentKeys: Disable use of JsonSchema, at least temporarily (T374241) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 21:53 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1045
• 21:53 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1043
• 21:52 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1043
• 21:51 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1044
• 21:51 jforrester@deploy1003: Started scap sync-world: Backport for ZObjectFactory::validatePersistentKeys: Disable use of JsonSchema, at least temporarily (T374241), ZObjectFactory::validatePersistentKeys: Disable use of JsonSchema, at least temporarily (T374241)
• 21:50 jclark@cumin1002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host ganeti1043
• 21:50 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1044
• 21:50 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1042
• 21:50 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1043
• 21:50 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1042
• 21:50 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1041
• 21:50 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1039
• 21:50 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1039
• 21:49 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1040
• 21:49 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1040
• 21:48 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1041
• 21:48 inflatador: bking@deploy1003 test deploying flink operator in staging T373195
• 21:45 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:44 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:43 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1039.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:43 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1039.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:41 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1040.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:41 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:41 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1042.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:40 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1041.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:40 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1041.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:40 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1040.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:39 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1039.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:37 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1039.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:34 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 21:33 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt ganeti10 - jclark@cumin1002"
• 21:33 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt ganeti10 - jclark@cumin1002"
• 21:30 jclark@cumin1002: START - Cookbook sre.dns.netbox
• 21:30 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 21:29 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 21:29 jclark@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 21:24 jclark@cumin1002: START - Cookbook sre.dns.netbox
• 21:22 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2170 (T371742)', diff saved to https://phabricator.wikimedia.org/P68986 and previous config saved to /var/cache/conftool/dbconfig/20240911-212208-ladsgroup.json
• 21:22 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2170.codfw.wmnet with reason: Maintenance
• 21:21 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2170.codfw.wmnet with reason: Maintenance
• 21:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T371742)', diff saved to https://phabricator.wikimedia.org/P68985 and previous config saved to /var/cache/conftool/dbconfig/20240911-212145-ladsgroup.json
• 21:14 cjming: end of UTC late backport window
• 21:11 cjming@deploy1003: Finished scap sync-world: Backport for Deploy Parsoid Read Views to bn/hi/ps/tr wikivoyage (T373229) (duration: 08m 19s)
• 21:07 cjming@deploy1003: cjming, cscott: Continuing with sync
• 21:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P68984 and previous config saved to /var/cache/conftool/dbconfig/20240911-210638-ladsgroup.json
• 21:05 cjming@deploy1003: cjming, cscott: Backport for Deploy Parsoid Read Views to bn/hi/ps/tr wikivoyage (T373229) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 21:03 cjming@deploy1003: Started scap sync-world: Backport for Deploy Parsoid Read Views to bn/hi/ps/tr wikivoyage (T373229)
• 21:01 cjming@deploy1003: Finished scap sync-world: Backport for Update wgSitename for tlywiki (T367009) (duration: 11m 51s)
• 20:56 cjming@deploy1003: cjming, nmw03: Continuing with sync
• 20:51 cjming@deploy1003: cjming, nmw03: Backport for Update wgSitename for tlywiki (T367009) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P68983 and previous config saved to /var/cache/conftool/dbconfig/20240911-205130-ladsgroup.json
• 20:49 cjming@deploy1003: Started scap sync-world: Backport for Update wgSitename for tlywiki (T367009)
• 20:47 cjming@deploy1003: Finished scap sync-world: Backport for Ensure that it is possible to override MFNamespacesWithLeadParagraphs (duration: 09m 54s)
• 20:43 cjming@deploy1003: jdlrobson, cjming: Continuing with sync
• 20:42 cjming@deploy1003: jdlrobson, cjming: Backport for Ensure that it is possible to override MFNamespacesWithLeadParagraphs synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:37 cjming@deploy1003: Started scap sync-world: Backport for Ensure that it is possible to override MFNamespacesWithLeadParagraphs
• 20:36 gmodena@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 20:36 cjming@deploy1003: Finished scap sync-world: Backport for Turn off feature flag to move donate link everywhere (T373585) (duration: 09m 42s)
• 20:36 gmodena@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply
• 20:36 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T371742)', diff saved to https://phabricator.wikimedia.org/P68982 and previous config saved to /var/cache/conftool/dbconfig/20240911-203623-ladsgroup.json
• 20:32 cjming@deploy1003: cjming, toyofuku: Continuing with sync
• 20:32 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox and not P{dns7001*} and A:dnsbox
• 20:31 ejegg: payments-wiki upgraded from 672c9fb6 to e191de03
• 20:30 cjming@deploy1003: cjming, toyofuku: Backport for Turn off feature flag to move donate link everywhere (T373585) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:27 cjming@deploy1003: Started scap sync-world: Backport for Turn off feature flag to move donate link everywhere (T373585)
• 20:23 cjming@deploy1003: Finished scap sync-world: Backport for Roll out appearance menu and font size change to sister projects (T371020) (duration: 13m 09s)
• 20:18 cjming@deploy1003: jdlrobson, cjming: Continuing with sync
• 20:13 cjming@deploy1003: jdlrobson, cjming: Backport for Roll out appearance menu and font size change to sister projects (T371020) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:10 cjming@deploy1003: Started scap sync-world: Backport for Roll out appearance menu and font size change to sister projects (T371020)
• 20:09 gmodena@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 20:09 gmodena@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply
• 19:56 gmodena@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 19:56 gmodena@deploy1003: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply
• 19:47 gmodena@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 19:47 gmodena@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply
• 19:46 gmodena@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 19:46 gmodena@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply
• 19:33 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2153 (T371742)', diff saved to https://phabricator.wikimedia.org/P68980 and previous config saved to /var/cache/conftool/dbconfig/20240911-193335-ladsgroup.json
• 19:33 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2153.codfw.wmnet with reason: Maintenance
• 19:33 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2153.codfw.wmnet with reason: Maintenance
• 19:33 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T371742)', diff saved to https://phabricator.wikimedia.org/P68979 and previous config saved to /var/cache/conftool/dbconfig/20240911-193312-ladsgroup.json
• 19:18 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P68978 and previous config saved to /var/cache/conftool/dbconfig/20240911-191805-ladsgroup.json
• 19:02 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P68977 and previous config saved to /var/cache/conftool/dbconfig/20240911-190257-ladsgroup.json
• 18:47 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T371742)', diff saved to https://phabricator.wikimedia.org/P68976 and previous config saved to /var/cache/conftool/dbconfig/20240911-184750-ladsgroup.json
• 18:42 sukhe: running agent on O:alerting_host
• 18:30 zabe: zabe@mwmaint1002:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTable.php test2wiki --skip text_table_cleanup/test2wiki text_table_dump/test2wiki --sleep 1 # T183490
• 18:25 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.43.0-wmf.22 refs T373641
• 18:25 aikochou@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• {{safesubst:SAL entry|1=18:02 zabe@deploy1003: Finished scap sync-world: Backport for migrateESRefToContentTable: Add option for not deleting text row (T183490), migrateESRefToContentTable: Add option to dump tt: -> es: reference (T183490), migrateESRefToContentTable: Add option for not deleting text row (T183490), [[gerrit:1072260|migrateESRefToContentTable: Add option to dump tt: -> es: re}}
• 17:58 sukhe@cumin1002: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox and not P{dns7001*} and A:dnsbox
• 17:52 sukhe: re-enable puppet on A:dnsbox and [run] agent
• 17:52 sukhe: re-enable puppet on A:dnsbox and enable agent
• 17:51 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 17:51 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 17:50 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 17:50 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 17:49 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 17:48 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 17:48 zabe@deploy1003: zabe: Continuing with sync
• 17:48 zabe@deploy1003: zabe: Backport for migrateESRefToContentTable: Add option for not deleting text row (T183490), migrateESRefToContentTable: Add option to dump tt: -> es: reference (T183490), migrateESRefToContentTable: Add option for not deleting text row (T183490), migrateESRefToContentTable: Add option to dump tt: -> es: reference (T183490)
• 17:48 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 17:48 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 17:47 sukhe: sukhe@dns7001:~$ sudo systemctl restart ntpsec.service
• 17:45 moritzm: installing postgresql-15 security updates
• 17:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2146 (T371742)', diff saved to https://phabricator.wikimedia.org/P68975 and previous config saved to /var/cache/conftool/dbconfig/20240911-174422-ladsgroup.json
• 17:44 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2146.codfw.wmnet with reason: Maintenance
• 17:44 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2146.codfw.wmnet with reason: Maintenance
• 17:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T371742)', diff saved to https://phabricator.wikimedia.org/P68974 and previous config saved to /var/cache/conftool/dbconfig/20240911-174400-ladsgroup.json
• 17:39 swfrench-wmf: imported php-uuid_1.2.0-12+wmf11u1 into component/php81 - T372507
• {{safesubst:SAL entry|1=17:30 zabe@deploy1003: Started scap sync-world: Backport for migrateESRefToContentTable: Add option for not deleting text row (T183490), migrateESRefToContentTable: Add option to dump tt: -> es: reference (T183490), migrateESRefToContentTable: Add option for not deleting text row (T183490), [[gerrit:1072260|migrateESRefToContentTable: Add option to dump tt: -> es: ref}}
• 17:30 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=wdqs-main,name=codfw
• 17:28 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P68973 and previous config saved to /var/cache/conftool/dbconfig/20240911-172852-ladsgroup.json
• 17:25 zabe@deploy1003: Started scap sync-world: Backport for migrateESRefToContentTable: Add option for not deleting text row (T183490), migrateESRefToContentTable: Add option to dump tt: -> es: reference (T183490)
• 17:17 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (None, T373791) xfer wikidata_main from wdqs2022.codfw.wmnet -> wdqs2021.codfw.wmnet w/ force delete existing files, repooling neither afterwards
• 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'es2038 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68972 and previous config saved to /var/cache/conftool/dbconfig/20240911-171739-arnaudb.json
• 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'es2022 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68971 and previous config saved to /var/cache/conftool/dbconfig/20240911-171734-arnaudb.json
• 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'db2210 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68970 and previous config saved to /var/cache/conftool/dbconfig/20240911-171729-arnaudb.json
• 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'db2180 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68969 and previous config saved to /var/cache/conftool/dbconfig/20240911-171724-arnaudb.json
• 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'db2179 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68968 and previous config saved to /var/cache/conftool/dbconfig/20240911-171719-arnaudb.json
• 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'db2168 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68967 and previous config saved to /var/cache/conftool/dbconfig/20240911-171714-arnaudb.json
• 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'db2167 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68966 and previous config saved to /var/cache/conftool/dbconfig/20240911-171709-arnaudb.json
• 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68965 and previous config saved to /var/cache/conftool/dbconfig/20240911-171704-arnaudb.json
• 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'db2116 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68964 and previous config saved to /var/cache/conftool/dbconfig/20240911-171700-arnaudb.json
• 17:16 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 100%: T373101', diff saved to https://phabricator.wikimedia.org/P68963 and previous config saved to /var/cache/conftool/dbconfig/20240911-171655-arnaudb.json
• 17:14 moritzm: installing gtk+2.0 security updates on bookworm
• 17:13 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P68962 and previous config saved to /var/cache/conftool/dbconfig/20240911-171346-ladsgroup.json
• 17:05 sukhe: sukhe@dns7001:~$ sudo systemctl restart ntpsec.service
• 17:02 arnaudb@cumin1002: dbctl commit (dc=all): 'es2038 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P68961 and previous config saved to /var/cache/conftool/dbconfig/20240911-170233-arnaudb.json
• 17:02 arnaudb@cumin1002: dbctl commit (dc=all): 'es2022 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P68960 and previous config saved to /var/cache/conftool/dbconfig/20240911-170228-arnaudb.json
• 17:02 arnaudb@cumin1002: dbctl commit (dc=all): 'db2210 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P68959 and previous config saved to /var/cache/conftool/dbconfig/20240911-170223-arnaudb.json
• 17:02 arnaudb@cumin1002: dbctl commit (dc=all): 'db2180 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P68958 and previous config saved to /var/cache/conftool/dbconfig/20240911-170218-arnaudb.json
• 17:02 arnaudb@cumin1002: dbctl commit (dc=all): 'db2179 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P68957 and previous config saved to /var/cache/conftool/dbconfig/20240911-170213-arnaudb.json
• 17:02 arnaudb@cumin1002: dbctl commit (dc=all): 'db2168 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P68956 and previous config saved to /var/cache/conftool/dbconfig/20240911-170208-arnaudb.json
• 17:02 arnaudb@cumin1002: dbctl commit (dc=all): 'db2167 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P68955 and previous config saved to /var/cache/conftool/dbconfig/20240911-170203-arnaudb.json
• 17:02 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P68954 and previous config saved to /var/cache/conftool/dbconfig/20240911-170158-arnaudb.json
• 17:01 arnaudb@cumin1002: dbctl commit (dc=all): 'db2116 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P68953 and previous config saved to /var/cache/conftool/dbconfig/20240911-170153-arnaudb.json
• 17:01 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 75%: T373101', diff saved to https://phabricator.wikimedia.org/P68952 and previous config saved to /var/cache/conftool/dbconfig/20240911-170149-arnaudb.json
• 16:59 sukhe: sudo cumin "A:dnsbox" 'disable-puppet "merging CR 1072209"'
• 16:58 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T371742)', diff saved to https://phabricator.wikimedia.org/P68951 and previous config saved to /var/cache/conftool/dbconfig/20240911-165838-ladsgroup.json
• 16:58 rzl@deploy1003: Finished scap sync-world: 1071714, 1071715 (T291192) (duration: 07m 37s)
• 16:57 rzl@deploy1003: rzl: Continuing with sync
• 16:57 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=(cp2037|cp2038).codfw.wmnet [reason: done T373101]
• 16:54 rzl@deploy1003: rzl: 1071714, 1071715 (T291192) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 16:53 rzl@deploy1003: Started scap sync-world: 1071714, 1071715 (T291192)
• 16:48 cdanis@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 16:47 cdanis@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 16:47 arnaudb@cumin1002: dbctl commit (dc=all): 'es2038 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P68950 and previous config saved to /var/cache/conftool/dbconfig/20240911-164728-arnaudb.json
• 16:47 arnaudb@cumin1002: dbctl commit (dc=all): 'es2022 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P68949 and previous config saved to /var/cache/conftool/dbconfig/20240911-164723-arnaudb.json
• 16:47 arnaudb@cumin1002: dbctl commit (dc=all): 'db2210 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P68948 and previous config saved to /var/cache/conftool/dbconfig/20240911-164718-arnaudb.json
• 16:47 cdanis@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 16:47 arnaudb@cumin1002: dbctl commit (dc=all): 'db2180 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P68947 and previous config saved to /var/cache/conftool/dbconfig/20240911-164713-arnaudb.json
• 16:47 arnaudb@cumin1002: dbctl commit (dc=all): 'db2179 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P68946 and previous config saved to /var/cache/conftool/dbconfig/20240911-164708-arnaudb.json
• 16:47 arnaudb@cumin1002: dbctl commit (dc=all): 'db2168 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P68945 and previous config saved to /var/cache/conftool/dbconfig/20240911-164703-arnaudb.json
• 16:46 arnaudb@cumin1002: dbctl commit (dc=all): 'db2167 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P68944 and previous config saved to /var/cache/conftool/dbconfig/20240911-164657-arnaudb.json
• 16:46 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P68943 and previous config saved to /var/cache/conftool/dbconfig/20240911-164653-arnaudb.json
• 16:46 arnaudb@cumin1002: dbctl commit (dc=all): 'db2116 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P68942 and previous config saved to /var/cache/conftool/dbconfig/20240911-164648-arnaudb.json
• 16:46 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 50%: T373101', diff saved to https://phabricator.wikimedia.org/P68941 and previous config saved to /var/cache/conftool/dbconfig/20240911-164644-arnaudb.json
• 16:46 cdanis@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 16:36 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2032.codfw.wmnet
• 16:36 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2032.codfw.wmnet
• 16:36 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2024.codfw.wmnet
• 16:36 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2024.codfw.wmnet
• 16:36 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2023.codfw.wmnet
• 16:36 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2023.codfw.wmnet
• 16:36 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2022.codfw.wmnet
• 16:36 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2022.codfw.wmnet
• 16:36 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2021.codfw.wmnet
• 16:36 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2021.codfw.wmnet
• 16:36 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2020.codfw.wmnet
• 16:36 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2020.codfw.wmnet
• 16:36 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2002.codfw.wmnet
• 16:36 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2002.codfw.wmnet
• 16:36 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host parse2015.codfw.wmnet
• 16:35 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host parse2015.codfw.wmnet
• 16:35 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host parse2014.codfw.wmnet
• 16:35 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host parse2014.codfw.wmnet
• 16:35 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2359.codfw.wmnet
• 16:35 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2359.codfw.wmnet
• 16:35 topranks: disable now unused ports on asw-c6-codfw after server move T373101
• 16:35 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2357.codfw.wmnet
• 16:35 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2357.codfw.wmnet
• 16:35 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2356.codfw.wmnet
• 16:35 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2356.codfw.wmnet
• 16:35 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2355.codfw.wmnet
• 16:35 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2355.codfw.wmnet
• 16:35 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2354.codfw.wmnet
• 16:35 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2354.codfw.wmnet
• 16:35 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2353.codfw.wmnet
• 16:35 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2353.codfw.wmnet
• 16:35 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2352.codfw.wmnet
• 16:35 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2352.codfw.wmnet
• 16:34 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2351.codfw.wmnet
• 16:34 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2351.codfw.wmnet
• 16:34 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2350.codfw.wmnet
• 16:34 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2350.codfw.wmnet
• 16:34 urandom: pooling thanos-fe2004.codfw.wmnet — T373101
• 16:34 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2043.codfw.wmnet
• 16:34 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2043.codfw.wmnet
• 16:34 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2042.codfw.wmnet
• 16:34 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2042.codfw.wmnet
• 16:34 claime: Repooling kubernetes2042.codfw.wmnet kubernetes2043.codfw.wmnet mw2350.codfw.wmnet mw2351.codfw.wmnet mw2352.codfw.wmnet mw2353.codfw.wmnet mw2354.codfw.wmnet mw2355.codfw.wmnet mw2356.codfw.wmnet mw2357.codfw.wmnet mw2359.codfw.wmnet parse2014.codfw.wmnet parse2015.codfw.wmnet wikikube-ctrl2002.codfw.wmnet wikikube-worker2020.codfw.wmnet wikikube-worker2021.codfw.wmnet
• 16:32 arnaudb@cumin1002: dbctl commit (dc=all): 'es2038 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P68940 and previous config saved to /var/cache/conftool/dbconfig/20240911-163222-arnaudb.json
• 16:32 arnaudb@cumin1002: dbctl commit (dc=all): 'es2022 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P68939 and previous config saved to /var/cache/conftool/dbconfig/20240911-163217-arnaudb.json
• 16:32 arnaudb@cumin1002: dbctl commit (dc=all): 'db2210 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P68938 and previous config saved to /var/cache/conftool/dbconfig/20240911-163212-arnaudb.json
• 16:32 arnaudb@cumin1002: dbctl commit (dc=all): 'db2180 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P68937 and previous config saved to /var/cache/conftool/dbconfig/20240911-163207-arnaudb.json
• 16:32 arnaudb@cumin1002: dbctl commit (dc=all): 'db2179 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P68936 and previous config saved to /var/cache/conftool/dbconfig/20240911-163202-arnaudb.json
• 16:31 arnaudb@cumin1002: dbctl commit (dc=all): 'db2168 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P68935 and previous config saved to /var/cache/conftool/dbconfig/20240911-163157-arnaudb.json
• 16:31 arnaudb@cumin1002: dbctl commit (dc=all): 'db2167 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P68934 and previous config saved to /var/cache/conftool/dbconfig/20240911-163152-arnaudb.json
• 16:31 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P68933 and previous config saved to /var/cache/conftool/dbconfig/20240911-163147-arnaudb.json
• 16:31 arnaudb@cumin1002: dbctl commit (dc=all): 'db2116 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P68932 and previous config saved to /var/cache/conftool/dbconfig/20240911-163142-arnaudb.json
• 16:31 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 25%: T373101', diff saved to https://phabricator.wikimedia.org/P68931 and previous config saved to /var/cache/conftool/dbconfig/20240911-163137-arnaudb.json
• 16:28 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2038.codfw.wmnet
• 16:28 sukhe@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp2038.codfw.wmnet
• 16:28 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2037.codfw.wmnet
• 16:27 sukhe@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp2037.codfw.wmnet
• 16:25 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (None, T373791) xfer wikidata_main from wdqs2022.codfw.wmnet -> wdqs2021.codfw.wmnet w/ force delete existing files, repooling neither afterwards
• 16:21 bking@deploy1003: Finished deploy [wdqs/wdqs@316bf7f]: 8 (duration: 00m 12s)
• 16:21 bking@deploy1003: Started deploy [wdqs/wdqs@316bf7f]: 8
• 16:20 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on 24 hosts with reason: Move server uplinks codfw racks C7
• 16:20 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:20:00 on 24 hosts with reason: Move server uplinks codfw racks C7
• 16:18 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db2135.codfw.wmnet with reason: network maintenance
• 16:18 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on db2135.codfw.wmnet with reason: network maintenance
• 16:08 topranks: begin server uplink moves from asw-c6-codfw to lsw1-c6-codfw T373101
• 16:07 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on 34 hosts with reason: Move server uplinks codfw racks C6
• 16:07 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:20:00 on 34 hosts with reason: Move server uplinks codfw racks C6
• 15:56 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2145 (T371742)', diff saved to https://phabricator.wikimedia.org/P68930 and previous config saved to /var/cache/conftool/dbconfig/20240911-155608-ladsgroup.json
• 15:56 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2145.codfw.wmnet with reason: Maintenance
• 15:55 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2145.codfw.wmnet with reason: Maintenance
• 15:55 mutante: moscovium - apt-get upgrade - installing new apache2 version and more package upgrades
• 15:50 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wdqs2021.codfw.wmnet with reason: T373791
• 15:50 bking@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wdqs2021.codfw.wmnet with reason: T373791
• 15:49 bking@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs2021.codfw.wmnet with OS bullseye
• 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'depool db2115 db2116 db2127 db2167 db2168 db2179 db2180 db2210 es2022 es2038 - T370852', diff saved to https://phabricator.wikimedia.org/P68929 and previous config saved to /var/cache/conftool/dbconfig/20240911-154114-arnaudb.json
• 15:37 urandom: depooling thanos-fe2004.codfw.wmnet — T373101
• 15:36 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on moscovium.eqiad.wmnet with reason: nftables migration
• 15:36 mutante: moscovium - rebooting for nftables migration
• 15:36 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 0:10:00 on moscovium.eqiad.wmnet with reason: nftables migration
• 15:35 mutante: phab2002 - rebooting for nftables migration
• 15:35 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on phab2002.codfw.wmnet with reason: nftables migration
• 15:35 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 0:10:00 on phab2002.codfw.wmnet with reason: nftables migration
• 15:31 topranks: push server and vlan configuration to lsw1-c6-codfw with Homer to prep physical moves T373101
• 15:28 dduvall@deploy1003: Finished deploy [releng/jenkins-deploy@4635fcb] (releasing): (no justification provided) (duration: 00m 35s)
• 15:27 dduvall@deploy1003: Started deploy [releng/jenkins-deploy@4635fcb] (releasing): (no justification provided)
• 15:26 dduvall@deploy1003: Finished deploy [releng/jenkins-deploy@71141b8] (releasing): (no justification provided) (duration: 00m 41s)
• 15:26 dduvall@deploy1003: Started deploy [releng/jenkins-deploy@71141b8] (releasing): (no justification provided)
• 15:21 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=(cp2037|cp2038).codfw.wmnet [reason: depooling for T373101]
• 15:17 arnaudb@cumin1002: dbctl commit (dc=all): 'db2138 (re)pooling @ 100%: post db2138 → db2238 repool', diff saved to https://phabricator.wikimedia.org/P68928 and previous config saved to /var/cache/conftool/dbconfig/20240911-151754-arnaudb.json
• 15:08 cmooney@cumin1002: END (ERROR) - Cookbook sre.ganeti.drain-node (exit_code=97) for draining ganeti node ganeti2014.codfw.wmnet
• 15:05 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2032.codfw.wmnet
• 15:05 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2032.codfw.wmnet
• 15:05 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2024.codfw.wmnet
• 15:04 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2024.codfw.wmnet
• 15:04 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2023.codfw.wmnet
• 15:03 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2023.codfw.wmnet
• 15:03 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2022.codfw.wmnet
• 15:03 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2022.codfw.wmnet
• 15:03 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2021.codfw.wmnet
• 15:02 arnaudb@cumin1002: dbctl commit (dc=all): 'db2138 (re)pooling @ 75%: post db2138 → db2238 repool', diff saved to https://phabricator.wikimedia.org/P68927 and previous config saved to /var/cache/conftool/dbconfig/20240911-150249-arnaudb.json
• 15:02 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2021.codfw.wmnet
• 15:02 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2020.codfw.wmnet
• 15:01 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2020.codfw.wmnet
• 15:01 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2002.codfw.wmnet
• 15:01 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2002.codfw.wmnet
• 15:01 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse2015.codfw.wmnet
• 15:00 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2141.codfw.wmnet with reason: Maintenance
• 15:00 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2141.codfw.wmnet with reason: Maintenance
• 15:00 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T371742)', diff saved to https://phabricator.wikimedia.org/P68926 and previous config saved to /var/cache/conftool/dbconfig/20240911-150011-ladsgroup.json
• 14:58 ladsgroup@cumin1002: dbctl commit (dc=all): 'Pool pc5 into production traffic (T374496)', diff saved to https://phabricator.wikimedia.org/P68925 and previous config saved to /var/cache/conftool/dbconfig/20240911-145844-ladsgroup.json
• 14:58 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host parse2015.codfw.wmnet
• 14:58 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse2014.codfw.wmnet
• 14:57 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host parse2014.codfw.wmnet
• 14:57 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2359.codfw.wmnet
• 14:57 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2359.codfw.wmnet
• 14:57 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2357.codfw.wmnet
• 14:56 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2357.codfw.wmnet
• 14:56 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2356.codfw.wmnet
• 14:55 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2356.codfw.wmnet
• 14:55 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2355.codfw.wmnet
• 14:55 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2355.codfw.wmnet
• 14:54 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2354.codfw.wmnet
• 14:54 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2354.codfw.wmnet
• 14:54 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2353.codfw.wmnet
• 14:53 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2353.codfw.wmnet
• 14:53 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2352.codfw.wmnet
• 14:53 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2352.codfw.wmnet
• 14:52 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2351.codfw.wmnet
• 14:52 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2351.codfw.wmnet
• 14:52 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2350.codfw.wmnet
• 14:51 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2350.codfw.wmnet
• 14:51 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2043.codfw.wmnet
• 14:51 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2043.codfw.wmnet
• 14:50 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2042.codfw.wmnet
• 14:50 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2042.codfw.wmnet
• 14:49 claime: Depooling kubernetes2042.codfw.wmnet kubernetes2043.codfw.wmnet mw2350.codfw.wmnet mw2351.codfw.wmnet mw2352.codfw.wmnet mw2353.codfw.wmnet mw2354.codfw.wmnet mw2355.codfw.wmnet mw2356.codfw.wmnet mw2357.codfw.wmnet mw2359.codfw.wmnet parse2014.codfw.wmnet parse2015.codfw.wmnet wikikube-ctrl2002.codfw.wmnet wikikube-worker2020.codfw.wmnet wikikube-worker2021.codfw.wmnet
• 14:48 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-ctrl2003.codfw.wmnet
• 14:48 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-ctrl2003.codfw.wmnet
• 14:48 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-ctrl2001.codfw.wmnet
• 14:48 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-ctrl2001.codfw.wmnet
• 14:47 arnaudb@cumin1002: dbctl commit (dc=all): 'db2138 (re)pooling @ 50%: post db2138 → db2238 repool', diff saved to https://phabricator.wikimedia.org/P68924 and previous config saved to /var/cache/conftool/dbconfig/20240911-144743-arnaudb.json
• 14:46 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 14:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 14:45 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P68923 and previous config saved to /var/cache/conftool/dbconfig/20240911-144504-ladsgroup.json
• 14:43 jayme: deployed changeprop-jobqueue changeprop cirrus-streaming-updater eventgate-main eventstreams mw-page-content-change-enrich rdf-streaming-updater for kafka connection string updates - T363210
• 14:42 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply
• 14:41 ladsgroup@cumin1002: dbctl commit (dc=all): 'Pool pc5 into production traffic (T374496)', diff saved to https://phabricator.wikimedia.org/P68922 and previous config saved to /var/cache/conftool/dbconfig/20240911-144147-ladsgroup.json
• 14:41 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply
• 14:41 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply
• 14:40 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply
• 14:40 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
• 14:39 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
• 14:39 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: apply
• 14:38 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-main2003.codfw.wmnet
• 14:38 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:38 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-main2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002"
• 14:38 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: apply
• 14:37 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 14:37 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-main2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002"
• 14:34 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
• 14:34 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
• 14:34 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply
• 14:34 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
• 14:33 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
• 14:32 cmooney@cumin1002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2014.codfw.wmnet
• 14:32 cmooney@cumin1002: END (ERROR) - Cookbook sre.ganeti.drain-node (exit_code=97) for draining ganeti node ganeti2014.codfw.wmnet
• 14:32 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 14:32 arnaudb@cumin1002: dbctl commit (dc=all): 'db2138 (re)pooling @ 25%: post db2138 → db2238 repool', diff saved to https://phabricator.wikimedia.org/P68921 and previous config saved to /var/cache/conftool/dbconfig/20240911-143237-arnaudb.json
• 14:31 jayme: last 7 helmfile deploys did not happen
• 14:30 jayme@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 14:30 jayme@deploy1003: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply
• 14:30 jayme@deploy1003: helmfile [staging] START helmfile.d/services/eventstreams: apply
• 14:30 jayme@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-main: apply
• 14:30 jayme@deploy1003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
• 14:30 jayme@deploy1003: helmfile [staging] START helmfile.d/services/changeprop: apply
• 14:30 jayme@deploy1003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
• 14:29 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P68920 and previous config saved to /var/cache/conftool/dbconfig/20240911-142956-ladsgroup.json
• 14:25 jayme@cumin1002: START - Cookbook sre.hosts.decommission for hosts kafka-main2003.codfw.wmnet
• 14:20 cmooney@cumin1002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2014.codfw.wmnet
• 14:19 cmooney@cumin1002: END (ERROR) - Cookbook sre.ganeti.drain-node (exit_code=97) for draining ganeti node ganeti2013.codfw.wmnet
• 14:17 arnaudb@cumin1002: dbctl commit (dc=all): 'db2138 (re)pooling @ 10%: post db2138 → db2238 repool', diff saved to https://phabricator.wikimedia.org/P68919 and previous config saved to /var/cache/conftool/dbconfig/20240911-141732-arnaudb.json
• 14:14 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T371742)', diff saved to https://phabricator.wikimedia.org/P68918 and previous config saved to /var/cache/conftool/dbconfig/20240911-141449-ladsgroup.json
• 14:14 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet
• 14:14 fabfur: reverted 1072172 and repooling cp4037 (T370668)
• 14:13 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main2008.codfw.wmnet
• 14:13 jayme@cumin1002: START - Cookbook sre.hosts.remove-downtime for kafka-main2008.codfw.wmnet
• 14:09 cmooney@cumin1002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2013.codfw.wmnet
• 14:05 aqu@deploy1003: Finished deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow (duration: 00m 11s)
• 14:05 aqu@deploy1003: Started deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow
• 13:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2138.codfw.wmnet onto db2238.codfw.wmnet
• 13:49 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on ganeti2012.codfw.wmnet with reason: Move ganeti2012 server uplink
• 13:49 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:20:00 on ganeti2012.codfw.wmnet with reason: Move ganeti2012 server uplink
• 13:47 Dreamy_Jazz: Afternoon UTC backport window done
• 13:44 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2021.codfw.wmnet with reason: host reimage
• 13:40 dreamyjazz@deploy1003: Finished scap sync-world: Backport for Remove ResourceLoaderUseObjectCacheForDeps (T343492), Generate special page name in English for central URLs (T374277), IPInfoLogFormatter: Avoid unnecessary User object creation (T374526), Add arbcom group to zhwiki (T374455), Remove redundant oathauth-enable flag (T374528), Allow ipblock-exempt-grantor to remove ipblock-exempt group flag on zhwiki (T374504), Raise RelatedArticlesCardLimit to 9 in zhwikinews (T374323), Enable Web team search suggestions survey (T373039)
• 13:40 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2021.codfw.wmnet with reason: host reimage
• 13:36 dreamyjazz@deploy1003: jdrewniak, hokwelum, dreamyjazz, hamishz: Continuing with sync
• 13:29 dreamyjazz@deploy1003: jdrewniak, hokwelum, dreamyjazz, hamishz: Backport for Remove ResourceLoaderUseObjectCacheForDeps (T343492), Generate special page name in English for central URLs (T374277), IPInfoLogFormatter: Avoid unnecessary User object creation (T374526), Add arbcom group to zhwiki (T374455), Remove redundant oathauth-enable flag (T374528), Allow ipblock-exempt-grantor to remove ipblock-exempt group flag on zhwiki (T374504), Raise RelatedArticlesCardLimit to 9 in zhwikinews (T374323), Enable Web team search suggestions survey (T373039)
• 13:29 brouberol@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-jumbo-eqiad
• 13:27 dreamyjazz@deploy1003: Started scap sync-world: Backport for Remove ResourceLoaderUseObjectCacheForDeps (T343492), Generate special page name in English for central URLs (T374277), IPInfoLogFormatter: Avoid unnecessary User object creation (T374526), Add arbcom group to zhwiki (T374455), Remove redundant oathauth-enable flag (T374528), Allow ipblock-exempt-grantor to remove ipblock-exempt group flag on zhwiki (T374504), Raise RelatedArticlesCardLimit to 9 in zhwikinews (T374323), Enable Web team search suggestions survey (T373039)
• 13:26 elukey@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host poolcounter2005.codfw.wmnet
• 13:26 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host poolcounter2005.codfw.wmnet with OS bookworm
• 13:26 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
• 13:26 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
• 13:26 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
• 13:25 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
• 13:24 klausman@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
• 13:23 bking@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2021.codfw.wmnet with OS bullseye
• 13:23 klausman@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
• 13:21 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
• 13:21 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
• 13:11 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on poolcounter2005.codfw.wmnet with reason: host reimage
• 13:07 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on poolcounter2005.codfw.wmnet with reason: host reimage
• 13:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2130 (T371742)', diff saved to https://phabricator.wikimedia.org/P68917 and previous config saved to /var/cache/conftool/dbconfig/20240911-130639-ladsgroup.json
• 13:06 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2130.codfw.wmnet with reason: Maintenance
• 13:06 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2130.codfw.wmnet with reason: Maintenance
• 13:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T371742)', diff saved to https://phabricator.wikimedia.org/P68916 and previous config saved to /var/cache/conftool/dbconfig/20240911-130618-ladsgroup.json
• 12:59 hashar@deploy1003: Finished scap sync-world: Backport for logging: Simplify extra debug logging configuration (duration: 06m 53s)
• 12:58 dcausse@deploy1003: helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply
• 12:58 dcausse@deploy1003: helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply
• 12:55 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: sync
• 12:55 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: sync
• 12:54 hashar@deploy1003: matmarex, hashar: Continuing with sync
• 12:54 hashar@deploy1003: matmarex, hashar: Backport for logging: Simplify extra debug logging configuration synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 12:54 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host poolcounter2005.codfw.wmnet with OS bookworm
• 12:53 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM poolcounter2005.codfw.wmnet - elukey@cumin1002"
• 12:53 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM poolcounter2005.codfw.wmnet - elukey@cumin1002"
• 12:52 elukey@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) poolcounter2005.codfw.wmnet on all recursors
• 12:52 elukey@cumin1002: START - Cookbook sre.dns.wipe-cache poolcounter2005.codfw.wmnet on all recursors
• 12:52 elukey@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:52 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM poolcounter2005.codfw.wmnet - elukey@cumin1002"
• 12:52 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM poolcounter2005.codfw.wmnet - elukey@cumin1002"
• 12:52 hashar@deploy1003: Started scap sync-world: Backport for logging: Simplify extra debug logging configuration
• 12:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P68915 and previous config saved to /var/cache/conftool/dbconfig/20240911-125110-ladsgroup.json
• 12:47 elukey@cumin1002: START - Cookbook sre.dns.netbox
• 12:47 elukey@cumin1002: START - Cookbook sre.ganeti.makevm for new host poolcounter2005.codfw.wmnet
• 12:42 hashar@deploy1003: Finished scap sync-world: Backport for logging: Replace 'blackhole' handler with no handlers at all (duration: 06m 43s)
• 12:37 hashar@deploy1003: matmarex, hashar: Continuing with sync
• 12:37 hashar@deploy1003: matmarex, hashar: Backport for logging: Replace 'blackhole' handler with no handlers at all synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 12:36 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P68914 and previous config saved to /var/cache/conftool/dbconfig/20240911-123603-ladsgroup.json
• 12:35 hashar@deploy1003: Started scap sync-world: Backport for logging: Replace 'blackhole' handler with no handlers at all
• 12:34 brouberol@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-jumbo-eqiad
• 12:33 arnaudb@cumin1002: START - Cookbook sre.mysql.clone of db2138.codfw.wmnet onto db2238.codfw.wmnet
• 12:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2138 in db2238 for T373579', diff saved to https://phabricator.wikimedia.org/P68913 and previous config saved to /var/cache/conftool/dbconfig/20240911-122910-arnaudb.json
• 12:28 moritzm: installing glibc bugfix updates from bookworm 12.7 point release
• 12:28 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: provisionning db2238.codfw.wmnet - T373579
• 12:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: provisionning db2238.codfw.wmnet - T373579
• 12:27 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: provisionning db2238.codfw.wmnet - T373579
• 12:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: provisionning db2238.codfw.wmnet - T373579
• 12:26 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 12:26 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
• 12:26 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 12:26 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 12:26 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
• 12:25 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
• 12:25 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
• 12:25 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
• 12:25 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
• 12:24 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
• 12:24 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 12:24 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 12:24 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 12:24 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 12:24 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 12:23 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
• 12:23 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 12:23 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 12:20 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T371742)', diff saved to https://phabricator.wikimedia.org/P68912 and previous config saved to /var/cache/conftool/dbconfig/20240911-122056-ladsgroup.json
• 12:19 hashar@deploy1003: Finished scap sync-world: Backport for logging: Fix local variables leaking into global scope (duration: 10m 38s)
• 12:18 topranks: re-activate Equinix IXP peers on cr1-eqiad T370696
• 12:15 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2227.codfw.wmnet onto db2205.codfw.wmnet
• 12:15 hashar@deploy1003: matmarex, hashar: Continuing with sync
• 12:12 jayme: restoring leadership for partitions assigned to broker id 2003 on kafka-main-codfw - T363210
• 12:11 hashar@deploy1003: matmarex, hashar: Backport for logging: Fix local variables leaking into global scope synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 12:09 hashar@deploy1003: Started scap sync-world: Backport for logging: Fix local variables leaking into global scope
• 12:08 topranks: test bundling xe-3/0/6 into ae6 on cr1-eqiad T370696
• 12:03 jayme@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-codfw
• 12:01 aqu@deploy1003: Finished deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow (duration: 00m 11s)
• 12:00 aqu@deploy1003: Started deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow
• 11:55 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr1-eqiad with reason: reconfigure equinix port into LAG
• 11:55 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on cr1-eqiad with reason: reconfigure equinix port into LAG
• 11:44 jayme@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-codfw
• 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: dragonfly::supernode
• 11:25 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: dragonfly::supernode
• 11:23 _joe_: uploaded conftool 3.2.3 to apt
• 11:22 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet
• 11:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2116 (T371742)', diff saved to https://phabricator.wikimedia.org/P68909 and previous config saved to /var/cache/conftool/dbconfig/20240911-111549-ladsgroup.json
• 11:15 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2116.codfw.wmnet with reason: Maintenance
• 11:15 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2116.codfw.wmnet with reason: Maintenance
• 11:14 arnaudb@cumin1002: START - Cookbook sre.mysql.clone of db2227.codfw.wmnet onto db2205.codfw.wmnet
• 11:01 mfossati@deploy1003: Finished deploy [airflow-dags/platform_eng@19cd97a]: (no justification provided) (duration: 00m 32s)
• 11:01 mfossati@deploy1003: Started deploy [airflow-dags/platform_eng@19cd97a]: (no justification provided)
• 10:59 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet
• 10:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2205.codfw.wmnet
• 10:51 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet
• 10:50 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2205.codfw.wmnet
• 10:50 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet
• 10:50 fabfur: repooling cp4037 to test haproxykafka (T374473)
• 10:40 elukey@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
• 10:38 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 10:27 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 10:26 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 10:22 elukey@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
• 10:19 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 09:42 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet
• 09:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:42 fabfur: depooling cp4037 to test haproxykafka (T374473)
• 09:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:33 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 09:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 09:33 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 09:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 09:30 jayme@deploy1003: helmfile [staging] DONE helmfile.d/services/benthos-cache-invalidator: apply
• 09:30 jayme@deploy1003: helmfile [staging] START helmfile.d/services/benthos-cache-invalidator: apply
• 09:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:21 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:12 brouberol@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 09:11 brouberol@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 09:11 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: sync
• 09:10 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: sync
• 08:53 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet with OS bookworm
• 08:39 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dragonfly-supernode1001.eqiad.wmnet with reason: host reimage
• 08:38 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 08:38 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 08:38 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T371742)', diff saved to https://phabricator.wikimedia.org/P68908 and previous config saved to /var/cache/conftool/dbconfig/20240911-083831-ladsgroup.json
• 08:36 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dragonfly-supernode1001.eqiad.wmnet with reason: host reimage
• 08:25 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host dragonfly-supernode1001.eqiad.wmnet with OS bookworm
• 08:23 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P68907 and previous config saved to /var/cache/conftool/dbconfig/20240911-082324-ladsgroup.json
• 08:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 100%: post fix', diff saved to https://phabricator.wikimedia.org/P68906 and previous config saved to /var/cache/conftool/dbconfig/20240911-082200-arnaudb.json
• 08:08 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P68905 and previous config saved to /var/cache/conftool/dbconfig/20240911-080817-ladsgroup.json
• 08:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 75%: post fix', diff saved to https://phabricator.wikimedia.org/P68904 and previous config saved to /var/cache/conftool/dbconfig/20240911-080654-arnaudb.json
• 08:03 arnaudb@cumin1002: dbctl commit (dc=all): 'db2137 (re)pooling @ 100%: post db2137 → db2237 repool', diff saved to https://phabricator.wikimedia.org/P68903 and previous config saved to /var/cache/conftool/dbconfig/20240911-080319-arnaudb.json
• 07:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T371742)', diff saved to https://phabricator.wikimedia.org/P68899 and previous config saved to /var/cache/conftool/dbconfig/20240911-075310-ladsgroup.json
• 07:53 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on kafka-main[2003,2008].codfw.wmnet with reason: Hardware refresh
• 07:52 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on kafka-main[2003,2008].codfw.wmnet with reason: Hardware refresh
• 07:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: post fix', diff saved to https://phabricator.wikimedia.org/P68898 and previous config saved to /var/cache/conftool/dbconfig/20240911-075149-arnaudb.json
• 07:49 jayme: evacuating leadership for all partitions assigned to broker id 2003 on kafka-main-codfw - T363210
• 07:48 arnaudb@cumin1002: dbctl commit (dc=all): 'db2137 (re)pooling @ 75%: post db2137 → db2237 repool', diff saved to https://phabricator.wikimedia.org/P68897 and previous config saved to /var/cache/conftool/dbconfig/20240911-074813-arnaudb.json
• 07:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: post fix', diff saved to https://phabricator.wikimedia.org/P68896 and previous config saved to /var/cache/conftool/dbconfig/20240911-073643-arnaudb.json
• 07:34 arnaudb@cumin1002: dbctl commit (dc=all): 'prod issue kmwiki.pagelinks', diff saved to https://phabricator.wikimedia.org/P68895 and previous config saved to /var/cache/conftool/dbconfig/20240911-073420-arnaudb.json
• 07:33 sgimeno@deploy1003: Finished scap sync-world: Backport for EventStreamConfig and stream registration for homepage modules analytics (T370907) (duration: 13m 56s)
• 07:33 arnaudb@cumin1002: dbctl commit (dc=all): 'db2137 (re)pooling @ 50%: post db2137 → db2237 repool', diff saved to https://phabricator.wikimedia.org/P68894 and previous config saved to /var/cache/conftool/dbconfig/20240911-073307-arnaudb.json
• 07:29 sgimeno@deploy1003: sgimeno: Continuing with sync
• 07:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2179 T374512', diff saved to https://phabricator.wikimedia.org/P68893 and previous config saved to /var/cache/conftool/dbconfig/20240911-072612-arnaudb.json
• 07:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2179 T374512', diff saved to https://phabricator.wikimedia.org/P68892 and previous config saved to /var/cache/conftool/dbconfig/20240911-072458-arnaudb.json
• 07:24 sgimeno@deploy1003: sgimeno: Backport for EventStreamConfig and stream registration for homepage modules analytics (T370907) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2140 to s4 primary T374512', diff saved to https://phabricator.wikimedia.org/P68891 and previous config saved to /var/cache/conftool/dbconfig/20240911-072210-arnaudb.json
• 07:21 arnaudb: Starting s4 codfw failover from db2179 to db2140 - T374512
• 07:19 sgimeno@deploy1003: Started scap sync-world: Backport for EventStreamConfig and stream registration for homepage modules analytics (T370907)
• 07:18 arnaudb@cumin1002: dbctl commit (dc=all): 'db2137 (re)pooling @ 25%: post db2137 → db2237 repool', diff saved to https://phabricator.wikimedia.org/P68890 and previous config saved to /var/cache/conftool/dbconfig/20240911-071802-arnaudb.json
• 07:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Remove db2140 from API/vslow/dump T374512', diff saved to https://phabricator.wikimedia.org/P68889 and previous config saved to /var/cache/conftool/dbconfig/20240911-071335-arnaudb.json
• 07:12 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 33 hosts with reason: Primary switchover s4 T374512
• 07:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2140 with weight 0 T374512', diff saved to https://phabricator.wikimedia.org/P68888 and previous config saved to /var/cache/conftool/dbconfig/20240911-071205-arnaudb.json
• 07:11 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 33 hosts with reason: Primary switchover s4 T374512
• 07:02 arnaudb@cumin1002: dbctl commit (dc=all): 'db2137 (re)pooling @ 10%: post db2137 → db2237 repool', diff saved to https://phabricator.wikimedia.org/P68887 and previous config saved to /var/cache/conftool/dbconfig/20240911-070254-arnaudb.json
• 06:34 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P68884 and previous config saved to /var/cache/conftool/dbconfig/20240911-063458-ladsgroup.json
• 06:19 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P68883 and previous config saved to /var/cache/conftool/dbconfig/20240911-061951-ladsgroup.json
• 06:04 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T371742)', diff saved to https://phabricator.wikimedia.org/P68882 and previous config saved to /var/cache/conftool/dbconfig/20240911-060444-ladsgroup.json
• 05:05 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1234 (T371742)', diff saved to https://phabricator.wikimedia.org/P68881 and previous config saved to /var/cache/conftool/dbconfig/20240911-050506-ladsgroup.json
• 05:04 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1234.eqiad.wmnet with reason: Maintenance
• 05:04 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1234.eqiad.wmnet with reason: Maintenance
• 05:04 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T371742)', diff saved to https://phabricator.wikimedia.org/P68880 and previous config saved to /var/cache/conftool/dbconfig/20240911-050444-ladsgroup.json
• 04:49 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P68879 and previous config saved to /var/cache/conftool/dbconfig/20240911-044936-ladsgroup.json
• 04:34 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P68878 and previous config saved to /var/cache/conftool/dbconfig/20240911-043429-ladsgroup.json
• 04:19 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T371742)', diff saved to https://phabricator.wikimedia.org/P68877 and previous config saved to /var/cache/conftool/dbconfig/20240911-041922-ladsgroup.json
• 03:16 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1232 (T371742)', diff saved to https://phabricator.wikimedia.org/P68876 and previous config saved to /var/cache/conftool/dbconfig/20240911-031643-ladsgroup.json
• 03:16 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1232.eqiad.wmnet with reason: Maintenance
• 03:16 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1232.eqiad.wmnet with reason: Maintenance
• 03:16 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T371742)', diff saved to https://phabricator.wikimedia.org/P68875 and previous config saved to /var/cache/conftool/dbconfig/20240911-031621-ladsgroup.json
• 03:01 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P68874 and previous config saved to /var/cache/conftool/dbconfig/20240911-030112-ladsgroup.json
• 02:46 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P68873 and previous config saved to /var/cache/conftool/dbconfig/20240911-024605-ladsgroup.json
• 02:31 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T371742)', diff saved to https://phabricator.wikimedia.org/P68872 and previous config saved to /var/cache/conftool/dbconfig/20240911-023058-ladsgroup.json
• 01:33 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1219 (T371742)', diff saved to https://phabricator.wikimedia.org/P68871 and previous config saved to /var/cache/conftool/dbconfig/20240911-013327-ladsgroup.json
• 01:33 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1219.eqiad.wmnet with reason: Maintenance
• 01:33 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1219.eqiad.wmnet with reason: Maintenance
• 01:33 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T371742)', diff saved to https://phabricator.wikimedia.org/P68870 and previous config saved to /var/cache/conftool/dbconfig/20240911-013305-ladsgroup.json
• 01:17 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P68869 and previous config saved to /var/cache/conftool/dbconfig/20240911-011758-ladsgroup.json
• 01:02 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P68868 and previous config saved to /var/cache/conftool/dbconfig/20240911-010250-ladsgroup.json
• 00:53 eileen: civicrm upgraded from e830e526 to 929101dc
• 00:47 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T371742)', diff saved to https://phabricator.wikimedia.org/P68867 and previous config saved to /var/cache/conftool/dbconfig/20240911-004743-ladsgroup.json

2024-09-10

• 23:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1218 (T371742)', diff saved to https://phabricator.wikimedia.org/P68866 and previous config saved to /var/cache/conftool/dbconfig/20240910-235102-ladsgroup.json
• 23:50 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1218.eqiad.wmnet with reason: Maintenance
• 23:50 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1218.eqiad.wmnet with reason: Maintenance
• 23:50 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T371742)', diff saved to https://phabricator.wikimedia.org/P68865 and previous config saved to /var/cache/conftool/dbconfig/20240910-235040-ladsgroup.json
• 23:49 cstone: civicrm upgraded from 5dd4edc1 to e830e526
• 23:35 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P68864 and previous config saved to /var/cache/conftool/dbconfig/20240910-233533-ladsgroup.json
• 23:20 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P68863 and previous config saved to /var/cache/conftool/dbconfig/20240910-232026-ladsgroup.json
• 23:05 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T371742)', diff saved to https://phabricator.wikimedia.org/P68862 and previous config saved to /var/cache/conftool/dbconfig/20240910-230518-ladsgroup.json
• 22:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1207 (T371742)', diff saved to https://phabricator.wikimedia.org/P68861 and previous config saved to /var/cache/conftool/dbconfig/20240910-220748-ladsgroup.json
• 22:07 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1207.eqiad.wmnet with reason: Maintenance
• 22:07 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1207.eqiad.wmnet with reason: Maintenance
• 22:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T371742)', diff saved to https://phabricator.wikimedia.org/P68860 and previous config saved to /var/cache/conftool/dbconfig/20240910-220726-ladsgroup.json
• 21:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P68859 and previous config saved to /var/cache/conftool/dbconfig/20240910-215219-ladsgroup.json
• 21:43 tzatziki: removing 1 file for legal compliance
• 21:37 tzatziki: removing 8 files for legal compliance
• 21:37 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P68858 and previous config saved to /var/cache/conftool/dbconfig/20240910-213712-ladsgroup.json
• 21:22 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T371742)', diff saved to https://phabricator.wikimedia.org/P68857 and previous config saved to /var/cache/conftool/dbconfig/20240910-212205-ladsgroup.json
• 21:14 kindrobot: finish UTC late backport window
• 21:14 kindrobot: purged AR wiki logos and taglines
• 21:02 kindrobot@deploy1003: Finished scap sync-world: Backport for [arwiki] Change the wordmark and the tagline (T374430), [arwiki] change Wikipedia logo (T374430) (duration: 34m 17s)
• 20:57 kindrobot@deploy1003: gergesshamon, kindrobot: Continuing with sync
• 20:31 tzatziki: removing 9 files for legal compliance
• 20:30 kindrobot@deploy1003: gergesshamon, kindrobot: Backport for [arwiki] Change the wordmark and the tagline (T374430), [arwiki] change Wikipedia logo (T374430) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:28 kindrobot@deploy1003: Started scap sync-world: Backport for [arwiki] Change the wordmark and the tagline (T374430), [arwiki] change Wikipedia logo (T374430)
• 20:23 kindrobot@deploy1003: Finished scap sync-world: Backport for Enable native MathML by default on group0 (T373703), Configure QuickSurvey for Web empty search state experiments (T373039) (duration: 09m 37s)
• 20:22 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1206 (T371742)', diff saved to https://phabricator.wikimedia.org/P68855 and previous config saved to /var/cache/conftool/dbconfig/20240910-202207-ladsgroup.json
• 20:22 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1206.eqiad.wmnet with reason: Maintenance
• 20:21 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1206.eqiad.wmnet with reason: Maintenance
• 20:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T371742)', diff saved to https://phabricator.wikimedia.org/P68854 and previous config saved to /var/cache/conftool/dbconfig/20240910-202145-ladsgroup.json
• 20:18 kindrobot@deploy1003: kindrobot, jdrewniak, physikerwelt: Continuing with sync
• 20:15 kindrobot@deploy1003: kindrobot, jdrewniak, physikerwelt: Backport for Enable native MathML by default on group0 (T373703), Configure QuickSurvey for Web empty search state experiments (T373039) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:13 kindrobot@deploy1003: Started scap sync-world: Backport for Enable native MathML by default on group0 (T373703), Configure QuickSurvey for Web empty search state experiments (T373039)
• 20:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P68853 and previous config saved to /var/cache/conftool/dbconfig/20240910-200637-ladsgroup.json
• 19:54 tzatziki: removing 6 files for legal compliance
• 19:51 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 19:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P68852 and previous config saved to /var/cache/conftool/dbconfig/20240910-195130-ladsgroup.json
• 19:51 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 19:50 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 19:50 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 19:47 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 19:47 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 19:36 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T371742)', diff saved to https://phabricator.wikimedia.org/P68851 and previous config saved to /var/cache/conftool/dbconfig/20240910-193622-ladsgroup.json
• 19:20 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 19:20 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 19:12 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 19:12 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 19:09 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 19:09 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 19:01 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 19:00 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 18:57 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 18:56 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 18:56 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 18:47 dduvall@deploy1003: Finished deploy [releng/jenkins-deploy@71141b8] (releasing): (no justification provided) (duration: 00m 35s)
• 18:47 dduvall@deploy1003: Started deploy [releng/jenkins-deploy@71141b8] (releasing): (no justification provided)
• 18:38 swfrench-wmf: ran authdns-update on dns1004 (18:25 UTC) for T372604
• 18:30 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1196 (T371742)', diff saved to https://phabricator.wikimedia.org/P68849 and previous config saved to /var/cache/conftool/dbconfig/20240910-183055-ladsgroup.json
• 18:30 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 18:30 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 18:30 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1196.eqiad.wmnet with reason: Maintenance
• 18:30 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1196.eqiad.wmnet with reason: Maintenance
• 18:30 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1195 (T371742)', diff saved to https://phabricator.wikimedia.org/P68848 and previous config saved to /var/cache/conftool/dbconfig/20240910-183016-ladsgroup.json
• 18:26 dduvall@deploy1003: Finished deploy [releng/jenkins-deploy@8be3b36] (releasing): (no justification provided) (duration: 05m 43s)
• 18:20 dduvall@deploy1003: Started deploy [releng/jenkins-deploy@8be3b36] (releasing): (no justification provided)
• 18:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P68847 and previous config saved to /var/cache/conftool/dbconfig/20240910-181508-ladsgroup.json
• 18:14 dduvall@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.22 refs T373641
• 18:04 swfrench-wmf: ran sre.dns.netbox after adding mwdebug-next LVS VIPs for T372604
• 18:02 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:01 swfrench@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add newly allocated LVS VIPs for mwdebug-next - swfrench@cumin2002"
• 18:01 swfrench@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add newly allocated LVS VIPs for mwdebug-next - swfrench@cumin2002"
• 18:00 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P68846 and previous config saved to /var/cache/conftool/dbconfig/20240910-180001-ladsgroup.json
• 17:57 swfrench@cumin2002: START - Cookbook sre.dns.netbox
• 17:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1195 (T371742)', diff saved to https://phabricator.wikimedia.org/P68845 and previous config saved to /var/cache/conftool/dbconfig/20240910-174454-ladsgroup.json
• 17:39 tzatziki: removing 4 files for legal compliance
• 17:27 tzatziki: removing 15 files for legal compliance
• 17:09 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2021.codfw.wmnet with OS bullseye
• 17:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db2192 (re)pooling @ 100%: T373097', diff saved to https://phabricator.wikimedia.org/P68844 and previous config saved to /var/cache/conftool/dbconfig/20240910-170753-arnaudb.json
• 17:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db2165 (re)pooling @ 100%: T373097', diff saved to https://phabricator.wikimedia.org/P68843 and previous config saved to /var/cache/conftool/dbconfig/20240910-170734-arnaudb.json
• 17:03 arnaudb@cumin1002: dbctl commit (dc=all): 'db2208 (re)pooling @ 100%: T373097', diff saved to https://phabricator.wikimedia.org/P68842 and previous config saved to /var/cache/conftool/dbconfig/20240910-170348-arnaudb.json
• 17:03 arnaudb@cumin1002: dbctl commit (dc=all): 'es2037 (re)pooling @ 100%: T373097', diff saved to https://phabricator.wikimedia.org/P68841 and previous config saved to /var/cache/conftool/dbconfig/20240910-170348-arnaudb.json
• 17:03 arnaudb@cumin1002: dbctl commit (dc=all): 'db2166 (re)pooling @ 100%: T373097', diff saved to https://phabricator.wikimedia.org/P68840 and previous config saved to /var/cache/conftool/dbconfig/20240910-170347-arnaudb.json
• 17:03 arnaudb@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 100%: T373097', diff saved to https://phabricator.wikimedia.org/P68839 and previous config saved to /var/cache/conftool/dbconfig/20240910-170347-arnaudb.json
• 16:59 cgoubert@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL
• 16:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2192 (re)pooling @ 75%: T373097', diff saved to https://phabricator.wikimedia.org/P68838 and previous config saved to /var/cache/conftool/dbconfig/20240910-165248-arnaudb.json
• 16:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2165 (re)pooling @ 75%: T373097', diff saved to https://phabricator.wikimedia.org/P68837 and previous config saved to /var/cache/conftool/dbconfig/20240910-165228-arnaudb.json
• 16:48 arnaudb@cumin1002: dbctl commit (dc=all): 'db2208 (re)pooling @ 75%: T373097', diff saved to https://phabricator.wikimedia.org/P68836 and previous config saved to /var/cache/conftool/dbconfig/20240910-164843-arnaudb.json
• 16:48 arnaudb@cumin1002: dbctl commit (dc=all): 'es2037 (re)pooling @ 75%: T373097', diff saved to https://phabricator.wikimedia.org/P68835 and previous config saved to /var/cache/conftool/dbconfig/20240910-164842-arnaudb.json
• 16:48 arnaudb@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 75%: T373097', diff saved to https://phabricator.wikimedia.org/P68834 and previous config saved to /var/cache/conftool/dbconfig/20240910-164842-arnaudb.json
• 16:48 arnaudb@cumin1002: dbctl commit (dc=all): 'db2166 (re)pooling @ 75%: T373097', diff saved to https://phabricator.wikimedia.org/P68833 and previous config saved to /var/cache/conftool/dbconfig/20240910-164842-arnaudb.json
• 16:43 sukhe: running authdns-update
• 16:42 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns2005.wikimedia.org [reason: end: T373097 codfw maintenance]
• 16:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1195 (T371742)', diff saved to https://phabricator.wikimedia.org/P68832 and previous config saved to /var/cache/conftool/dbconfig/20240910-163908-ladsgroup.json
• 16:39 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1195.eqiad.wmnet with reason: Maintenance
• 16:38 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1195.eqiad.wmnet with reason: Maintenance
• 16:38 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T371742)', diff saved to https://phabricator.wikimedia.org/P68831 and previous config saved to /var/cache/conftool/dbconfig/20240910-163846-ladsgroup.json
• 16:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2192 (re)pooling @ 50%: T373097', diff saved to https://phabricator.wikimedia.org/P68830 and previous config saved to /var/cache/conftool/dbconfig/20240910-163742-arnaudb.json
• 16:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2165 (re)pooling @ 50%: T373097', diff saved to https://phabricator.wikimedia.org/P68829 and previous config saved to /var/cache/conftool/dbconfig/20240910-163722-arnaudb.json
• 16:34 claime: Repooled kubernetes2040.codfw.wmnet kubernetes2041.codfw.wmnet kubernetes2058.codfw.wmnet mw2440.codfw.wmnet mw2442.codfw.wmnet mw2443.codfw.wmnet parse2011.codfw.wmnet parse2012.codfw.wmnet parse2013.codfw.wmnet wikikube-worker2039.codfw.wmnet - T373097
• 16:34 arnaudb@cumin1002: dbctl commit (dc=all): 'db2208 (re)pooling @ 50%: T373097', diff saved to https://phabricator.wikimedia.org/P68828 and previous config saved to /var/cache/conftool/dbconfig/20240910-163338-arnaudb.json
• 16:33 arnaudb@cumin1002: dbctl commit (dc=all): 'es2037 (re)pooling @ 50%: T373097', diff saved to https://phabricator.wikimedia.org/P68827 and previous config saved to /var/cache/conftool/dbconfig/20240910-163338-arnaudb.json
• 16:33 arnaudb@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 50%: T373097', diff saved to https://phabricator.wikimedia.org/P68826 and previous config saved to /var/cache/conftool/dbconfig/20240910-163337-arnaudb.json
• 16:33 arnaudb@cumin1002: dbctl commit (dc=all): 'db2166 (re)pooling @ 50%: T373097', diff saved to https://phabricator.wikimedia.org/P68825 and previous config saved to /var/cache/conftool/dbconfig/20240910-163337-arnaudb.json
• 16:33 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 100%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68824 and previous config saved to /var/cache/conftool/dbconfig/20240910-163300-arnaudb.json
• 16:30 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [CheckUser] Don't write to central indexes when no CentralAuth (T374462), Don't attempt to interact with central indexes for some wikis (T374462), Don't attempt to interact with central indexes for some wikis (T374462) (duration: 06m 49s)
• 16:26 cgoubert@cumin1002: START - Cookbook sre.hosts.provision for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTARTand with Dell SCP reboot policy GRACEFUL
• 16:25 dreamyjazz@deploy1003: dreamyjazz: Continuing with sync
• 16:25 dreamyjazz@deploy1003: dreamyjazz: Backport for [CheckUser] Don't write to central indexes when no CentralAuth (T374462), Don't attempt to interact with central indexes for some wikis (T374462), Don't attempt to interact with central indexes for some wikis (T374462) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 16:23 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P68823 and previous config saved to /var/cache/conftool/dbconfig/20240910-162339-ladsgroup.json
• 16:23 dreamyjazz@deploy1003: Started scap sync-world: Backport for [CheckUser] Don't write to central indexes when no CentralAuth (T374462), Don't attempt to interact with central indexes for some wikis (T374462), Don't attempt to interact with central indexes for some wikis (T374462)
• 16:21 cmooney@cumin1002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2012.codfw.wmnet
• 16:19 cmooney@cumin1002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2012.codfw.wmnet
• 16:18 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2039.codfw.wmnet
• 16:18 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet
• 16:18 arnaudb@cumin1002: dbctl commit (dc=all): 'db2208 (re)pooling @ 25%: T373097', diff saved to https://phabricator.wikimedia.org/P68822 and previous config saved to /var/cache/conftool/dbconfig/20240910-161832-arnaudb.json
• 16:18 arnaudb@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 25%: T373097', diff saved to https://phabricator.wikimedia.org/P68821 and previous config saved to /var/cache/conftool/dbconfig/20240910-161832-arnaudb.json
• 16:18 arnaudb@cumin1002: dbctl commit (dc=all): 'es2037 (re)pooling @ 25%: T373097', diff saved to https://phabricator.wikimedia.org/P68820 and previous config saved to /var/cache/conftool/dbconfig/20240910-161832-arnaudb.json
• 16:18 arnaudb@cumin1002: dbctl commit (dc=all): 'db2166 (re)pooling @ 25%: T373097', diff saved to https://phabricator.wikimedia.org/P68819 and previous config saved to /var/cache/conftool/dbconfig/20240910-161832-arnaudb.json
• 16:18 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host parse2013.codfw.wmnet
• 16:18 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host parse2013.codfw.wmnet
• 16:18 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host parse2012.codfw.wmnet
• 16:18 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host parse2012.codfw.wmnet
• 16:18 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=restbase2025.codfw.wmnet
• 16:18 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host parse2011.codfw.wmnet
• 16:18 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host parse2011.codfw.wmnet
• 16:18 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=restbase2033.codfw.wmnet
• 16:18 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2443.codfw.wmnet
• 16:18 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2443.codfw.wmnet
• 16:18 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:18 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2442.codfw.wmnet
• 16:18 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2442.codfw.wmnet
• 16:18 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:17 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 75%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68818 and previous config saved to /var/cache/conftool/dbconfig/20240910-161753-arnaudb.json
• 16:17 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2440.codfw.wmnet
• 16:17 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2440.codfw.wmnet
• 16:17 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2058.codfw.wmnet
• 16:17 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2058.codfw.wmnet
• 16:17 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2041.codfw.wmnet
• 16:17 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2041.codfw.wmnet
• 16:17 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2040.codfw.wmnet
• 16:17 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2040.codfw.wmnet
• 16:14 dcausse@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 16:13 dcausse@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
• 16:12 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2137.codfw.wmnet onto db2237.codfw.wmnet
• 16:09 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on 26 hosts with reason: Move server uplinks codfw racks C5
• 16:08 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:20:00 on 26 hosts with reason: Move server uplinks codfw racks C5
• 16:08 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P68817 and previous config saved to /var/cache/conftool/dbconfig/20240910-160831-ladsgroup.json
• 16:03 topranks: commence maintenance - move server uplinks from old to new switch codfw rack C4 T373097
• 16:02 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 50%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68816 and previous config saved to /var/cache/conftool/dbconfig/20240910-160247-arnaudb.json
• 16:02 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 23 hosts with reason: Move server uplinks codfw racks C4
• 16:02 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on 23 hosts with reason: Move server uplinks codfw racks C4
• 15:59 topranks: push server and vlan configuration to lsw1-c5-codfw with Homer to prep physical moves T373097
• 15:58 topranks: push server and vlan configuration to lsw1-c4-codfw with Homer to prep physical moves T373097
• 15:57 topranks: move server uplinks in Netbox from asw-c4-codfw to lsw1-c4-codfw to prep physical moves T373097
• 15:56 topranks: move server uplinks in Netbox from asw-c5-codfw to lsw1-c5-codfw to prep physical moves T373097
• 15:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T371742)', diff saved to https://phabricator.wikimedia.org/P68815 and previous config saved to /var/cache/conftool/dbconfig/20240910-155324-ladsgroup.json
• 15:48 bking@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs2021.codfw.wmnet with OS bullseye
• 15:47 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 25%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68814 and previous config saved to /var/cache/conftool/dbconfig/20240910-154740-arnaudb.json
• 15:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 6 hosts with reason: network maintenance T373097
• 15:46 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on 6 hosts with reason: network maintenance T373097
• 15:45 arnaudb@cumin1002: dbctl commit (dc=all): 'depool db2126 db2165 db2166 db2192 db2208 es2037 - T370852', diff saved to https://phabricator.wikimedia.org/P68813 and previous config saved to /var/cache/conftool/dbconfig/20240910-154540-arnaudb.json
• 15:39 mutante: enabling throttling on Gerrit hosts - T365259
• 15:39 jelto: enabling throttling on GitLab hosts - T366882
• 15:34 cmooney@cumin1002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2012.codfw.wmnet
• 15:34 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2092.codfw.wmnet with reason: Degraded RAID
• 15:34 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on wikikube-worker2092.codfw.wmnet with reason: Degraded RAID
• 15:33 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2092.codfw.wmnet
• 15:33 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2092.codfw.wmnet
• 15:32 cmooney@cumin1002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2012.codfw.wmnet
• 15:32 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 15%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68811 and previous config saved to /var/cache/conftool/dbconfig/20240910-153234-arnaudb.json
• 15:17 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 5%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68810 and previous config saved to /var/cache/conftool/dbconfig/20240910-151729-arnaudb.json
• 15:16 ebernhardson@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:16 ebernhardson@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:12 ebernhardson@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:12 ebernhardson@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:08 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=dns2005.wikimedia.org [reason: T373097 codfw maintenance]
• 15:06 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2205.codfw.wmnet with reason: T374425
• 15:06 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2205.codfw.wmnet with reason: T374425
• 15:05 brennen@deploy1003: Finished deploy [phabricator/deployment@84ada67]: deploy phab1004 for T374458 (duration: 00m 50s)
• 15:04 brennen@deploy1003: Started deploy [phabricator/deployment@84ada67]: deploy phab1004 for T374458
• 15:04 brennen@deploy1003: Finished deploy [phabricator/deployment@84ada67]: deploy phab2002 for T374458 (duration: 00m 36s)
• 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@84ada67]: deploy phab2002 for T374458
• 15:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator/Phorge update
• 15:02 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator/Phorge update
• 15:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator/Phorge update
• 15:02 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 4%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68809 and previous config saved to /var/cache/conftool/dbconfig/20240910-150222-arnaudb.json
• 15:02 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator/Phorge update
• 15:02 cmooney@cumin1002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2011.codfw.wmnet
• 15:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab.wmfusercontent.org with reason: Phabricator/Phorge update
• 15:01 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on phab.wmfusercontent.org with reason: Phabricator/Phorge update
• 14:57 cmooney@cumin1002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2011.codfw.wmnet
• 14:47 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1186 (T371742)', diff saved to https://phabricator.wikimedia.org/P68808 and previous config saved to /var/cache/conftool/dbconfig/20240910-144725-ladsgroup.json
• 14:47 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1186.eqiad.wmnet with reason: Maintenance
• 14:47 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 3%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68807 and previous config saved to /var/cache/conftool/dbconfig/20240910-144716-arnaudb.json
• 14:47 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1186.eqiad.wmnet with reason: Maintenance
• 14:47 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T371742)', diff saved to https://phabricator.wikimedia.org/P68806 and previous config saved to /var/cache/conftool/dbconfig/20240910-144703-ladsgroup.json
• 14:37 sukhe: sudo cumin -b11 "A:cp" 'run-puppet-agent --enable "merging CRs 1065286"'
• 14:35 hnowlan@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2033.codfw.wmnet
• 14:35 hnowlan@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2025.codfw.wmnet
• 14:32 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 2%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68805 and previous config saved to /var/cache/conftool/dbconfig/20240910-143211-arnaudb.json
• 14:31 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P68804 and previous config saved to /var/cache/conftool/dbconfig/20240910-143156-ladsgroup.json
• 14:29 Lucas_WMDE: UTC afternoon backport+config window done
• 14:28 sukhe: [end] rolling restart of {pdns-recursor,haproxy}.service on A:dnsbox
• 14:25 jayme: restoring leadership for partitions assigned to broker id 2002 on kafka-main-codfw - T363210
• 14:17 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 1%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68802 and previous config saved to /var/cache/conftool/dbconfig/20240910-141705-arnaudb.json
• 14:16 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P68801 and previous config saved to /var/cache/conftool/dbconfig/20240910-141649-ladsgroup.json
• 14:16 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2039.codfw.wmnet
• 14:16 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript cleanupTitles zhwiki | tee T363538-zhwiki-cleanupTitles
• 14:15 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes zhwiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-prefix=T363538/ --fix | tee T363538-zhwiki-namespaceDupes
• 14:15 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2039.codfw.wmnet
• 14:15 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse2013.codfw.wmnet
• 14:15 arnaudb@cumin1002: START - Cookbook sre.mysql.clone of db2137.codfw.wmnet onto db2237.codfw.wmnet
• 14:15 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host parse2013.codfw.wmnet
• 14:15 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse2012.codfw.wmnet
• 14:14 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript cleanupTitles thwiki | tee T363538-thwiki-cleanupTitles
• 14:14 sukhe: sudo cumin "A:cp" 'disable-puppet "merging CRs 1065286"'
• 14:14 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host parse2012.codfw.wmnet
• 14:14 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse2011.codfw.wmnet
• 14:14 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes thwiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-prefix=T363538/ --fix | tee T363538-thwiki-namespaceDupes
• 14:13 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript cleanupTitles slwiki | tee T363538-slwiki-cleanupTitles
• 14:13 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes slwiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-prefix=T363538/ --fix | tee T363538-slwiki-namespaceDupes
• 14:11 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript cleanupTitles simplewiki | tee T363538-simplewiki-cleanupTitles
• 14:11 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes simplewiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-prefix=T363538/ --fix | tee T363538-simplewiki-namespaceDupes
• 14:11 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host parse2011.codfw.wmnet
• 14:11 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2443.codfw.wmnet
• 14:11 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-main2002.codfw.wmnet
• 14:11 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:10 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-main2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002"
• 14:10 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-main2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002"
• 14:10 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2443.codfw.wmnet
• 14:10 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2442.codfw.wmnet
• 14:10 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript cleanupTitles mswiki | tee T363538-mswiki-cleanupTitles
• 14:10 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes mswiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-prefix=T363538/ --fix | tee T363538-mswiki-namespaceDupes
• 14:09 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2442.codfw.wmnet
• 14:09 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2440.codfw.wmnet
• 14:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2137 in db2237 for T373579', diff saved to https://phabricator.wikimedia.org/P68792 and previous config saved to /var/cache/conftool/dbconfig/20240910-140918-arnaudb.json
• 14:09 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2440.codfw.wmnet
• 14:09 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2058.codfw.wmnet
• 14:09 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2227.codfw.wmnet onto db2127.codfw.wmnet
• 14:08 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2058.codfw.wmnet
• 14:08 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2041.codfw.wmnet
• 14:08 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: provisionning db2237.codfw.wmnet - T373579
• 14:08 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: provisionning db2237.codfw.wmnet - T373579
• 14:08 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: provisionning db2237.codfw.wmnet - T373579
• 14:07 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2041.codfw.wmnet
• 14:07 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2040.codfw.wmnet
• 14:07 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: provisionning db2237.codfw.wmnet - T373579
• 14:07 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2040.codfw.wmnet
• 14:06 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 14:06 claime: Depooling kubernetes2040.codfw.wmnet kubernetes2041.codfw.wmnet kubernetes2058.codfw.wmnet mw2440.codfw.wmnet mw2442.codfw.wmnet mw2443.codfw.wmnet parse2011.codfw.wmnet parse2012.codfw.wmnet parse2013.codfw.wmnet wikikube-worker2039.codfw.wmnet - T373097
• 14:06 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript cleanupTitles jawiki | tee T363538-jawiki-cleanupTitles
• 14:06 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes jawiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-prefix=T363538/ --fix | tee T363538-jawiki-namespaceDupes
• 14:01 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T371742)', diff saved to https://phabricator.wikimedia.org/P68788 and previous config saved to /var/cache/conftool/dbconfig/20240910-140141-ladsgroup.json
• 14:01 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript cleanupTitles idwiki | tee T363538-idwiki-cleanupTitles
• 14:01 jayme@cumin1002: START - Cookbook sre.hosts.decommission for hosts kafka-main2002.codfw.wmnet
• 14:01 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes idwiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-prefix=T363538/ --fix | tee T363538-idwiki-namespaceDupes
• 13:59 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript cleanupTitles dagwiki | tee T363538-dagwiki-cleanupTitles
• 13:58 sukhe: sudo cumin -b11 "A:cp" 'run-puppet-agent --enable "merging CRs 1065283"'
• 13:57 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes dagwiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-prefix=T363538/ --fix | tee T363538-dagwiki-namespaceDupes
• 13:52 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript cleanupTitles bnwiki | tee T363538-bnwiki-cleanupTitles
• 13:51 jebe@deploy1003: Finished deploy [analytics/refinery@464c114] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@464c114d] (duration: 03m 43s)
• 13:51 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes bnwiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-suffix=/T363538 --fix | tee T363538-bnwiki-namespaceDupes
• 13:50 sukhe: sudo cumin "A:cp" 'disable-puppet "merging CRs 1065283"'
• 13:48 jebe@deploy1003: Started deploy [analytics/refinery@464c114] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@464c114d]
• 13:47 jebe@deploy1003: Finished deploy [analytics/refinery@464c114] (thin): Regular analytics weekly train THIN [analytics/refinery@464c114d] (duration: 04m 58s)
• 13:43 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript cleanupTitles aswiki | tee T363538-aswiki-cleanupTitles
• 13:42 jebe@deploy1003: Started deploy [analytics/refinery@464c114] (thin): Regular analytics weekly train THIN [analytics/refinery@464c114d]
• 13:42 jebe@deploy1003: Finished deploy [analytics/refinery@464c114]: Regular analytics weekly train [analytics/refinery@464c114d] (duration: 07m 22s)
• 13:42 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php aswiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --add-prefix=T363538/ --fix | tee T363538-aswiki-namespaceDupes-prefix
• 13:38 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes.php aswiki --source-pseudo-namespace MOS --dest-namespace 126 --move-talk --fix | tee T363538-aswiki-namespaceDupes # crashed, DBQueryError
• 13:35 jebe@deploy1003: Started deploy [analytics/refinery@464c114]: Regular analytics weekly train [analytics/refinery@464c114d]
• 13:35 logmsgbot: lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for Elevate pseudo-namespace MOS to a real namespace on most wikis which use it (T363538) (duration: 13m 45s)
• 13:30 logmsgbot: lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, cscott: Continuing with sync
• 13:28 logmsgbot: lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, cscott: Backport for Elevate pseudo-namespace MOS to a real namespace on most wikis which use it (T363538) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:23 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on pc2017.codfw.wmnet,pc1017.eqiad.wmnet with reason: T374355
• 13:23 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on pc2017.codfw.wmnet,pc1017.eqiad.wmnet with reason: T374355
• 13:21 logmsgbot: lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for Elevate pseudo-namespace MOS to a real namespace on most wikis which use it (T363538)
• 13:18 logmsgbot: lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for Support new heading layout (T373039 T374377) (duration: 12m 11s)
• 13:14 logmsgbot: lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, jdlrobson: Continuing with sync
• 13:10 logmsgbot: lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, jdlrobson: Backport for Support new heading layout (T373039 T374377) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:09 sukhe: rolling restart of {pdns-recursor,haproxy}.service on A:dnsbox
• 13:06 logmsgbot: lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for Support new heading layout (T373039 T374377)
• 12:58 arnaudb@cumin1002: START - Cookbook sre.mysql.clone of db2227.codfw.wmnet onto db2127.codfw.wmnet
• 12:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2205.codfw.wmnet with reason: maintenance
• 12:58 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2205.codfw.wmnet with reason: maintenance
• 12:57 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1184 (T371742)', diff saved to https://phabricator.wikimedia.org/P68777 and previous config saved to /var/cache/conftool/dbconfig/20240910-125705-ladsgroup.json
• 12:56 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1184.eqiad.wmnet with reason: Maintenance
• 12:56 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1184.eqiad.wmnet with reason: Maintenance
• 12:56 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T371742)', diff saved to https://phabricator.wikimedia.org/P68776 and previous config saved to /var/cache/conftool/dbconfig/20240910-125643-ladsgroup.json
• 12:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2227.codfw.wmnet with reason: provisionning db2127.codfw.wmnet - T373579
• 12:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2227.codfw.wmnet with reason: provisionning db2127.codfw.wmnet - T373579
• 12:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2127.codfw.wmnet with reason: provisionning db2127.codfw.wmnet - T373579
• 12:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2127.codfw.wmnet with reason: provisionning db2127.codfw.wmnet - T373579
• 12:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2127.codfw.wmnet
• 12:45 Amir1: dropping bv2013_edits table everywhere
• 12:43 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2127.codfw.wmnet
• 12:41 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P68774 and previous config saved to /var/cache/conftool/dbconfig/20240910-124136-ladsgroup.json
• 12:26 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P68773 and previous config saved to /var/cache/conftool/dbconfig/20240910-122629-ladsgroup.json
• 12:24 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
• 12:23 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main2007.codfw.wmnet
• 12:23 jayme@cumin1002: START - Cookbook sre.hosts.remove-downtime for kafka-main2007.codfw.wmnet
• 12:19 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
• 12:18 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
• 12:13 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
• 12:11 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T371742)', diff saved to https://phabricator.wikimedia.org/P68772 and previous config saved to /var/cache/conftool/dbconfig/20240910-121122-ladsgroup.json
• 12:03 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T370903)', diff saved to https://phabricator.wikimedia.org/P68771 and previous config saved to /var/cache/conftool/dbconfig/20240910-120357-ladsgroup.json
• 11:58 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
• 11:58 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
• 11:48 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P68770 and previous config saved to /var/cache/conftool/dbconfig/20240910-114850-ladsgroup.json
• 11:33 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P68769 and previous config saved to /var/cache/conftool/dbconfig/20240910-113342-ladsgroup.json
• 11:18 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T370903)', diff saved to https://phabricator.wikimedia.org/P68768 and previous config saved to /var/cache/conftool/dbconfig/20240910-111835-ladsgroup.json
• 11:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1169 (T371742)', diff saved to https://phabricator.wikimedia.org/P68767 and previous config saved to /var/cache/conftool/dbconfig/20240910-110614-ladsgroup.json
• 11:06 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1169.eqiad.wmnet with reason: Maintenance
• 11:05 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1169.eqiad.wmnet with reason: Maintenance
• 11:04 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2123 (T370903)', diff saved to https://phabricator.wikimedia.org/P68766 and previous config saved to /var/cache/conftool/dbconfig/20240910-110409-ladsgroup.json
• 11:04 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2123.codfw.wmnet with reason: Maintenance
• 11:04 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db2123.codfw.wmnet with reason: Maintenance
• 10:51 vgutierrez: switching purged in codfw and ulsfo to use main-eqiad kafka cluster - T373189
• 10:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:49 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:44 moritzm: installing bind9 security updates (client-side tools/libs only)
• 10:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:14 hnowlan@deploy1003: Finished scap sync-world: Backport for Enable Copyupload-allowed-domain on testwiki, disable on test2 (T356241) (duration: 09m 39s)
• 10:09 hnowlan@deploy1003: hnowlan: Continuing with sync
• 10:08 hnowlan@deploy1003: hnowlan: Backport for Enable Copyupload-allowed-domain on testwiki, disable on test2 (T356241) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 10:05 hnowlan@deploy1003: Started scap sync-world: Backport for Enable Copyupload-allowed-domain on testwiki, disable on test2 (T356241)
• 09:35 cgoubert@cumin1002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2431.codfw.wmnet
• 09:35 cgoubert@cumin1002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2431.codfw.wmnet
• 09:11 arnaudb@cumin1002: dbctl commit (dc=all): 'T374421 → replag not catching up on exec, ^C to debug', diff saved to https://phabricator.wikimedia.org/P68763 and previous config saved to /var/cache/conftool/dbconfig/20240910-091114-arnaudb.json
• 09:10 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 09:10 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 09:03 jayme@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-codfw
• 09:02 hashar: Restarting CI Jenkins
• 09:02 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 09:02 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
• 09:02 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 09:01 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 09:01 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
• 09:01 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
• 09:01 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
• 09:00 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
• 09:00 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
• 09:00 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
• 09:00 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 08:59 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 08:59 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 08:59 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 08:59 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 08:59 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
• 08:59 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 08:58 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 08:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2205 with weight 0 T374421', diff saved to https://phabricator.wikimedia.org/P68761 and previous config saved to /var/cache/conftool/dbconfig/20240910-085854-arnaudb.json
• 08:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s3 T374421
• 08:58 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 25 hosts with reason: Primary switchover s3 T374421
• 08:51 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: sync on production
• 08:47 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply on production
• 08:46 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: sync on production
• 08:46 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply on production
• 08:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: sync on staging
• 08:44 jayme@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-codfw
• 08:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply on staging
• 08:39 moritzm: installing Java security updates on puppetservers
• 08:23 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on kafka-main[2002,2007].codfw.wmnet with reason: Hardware refresh
• 08:23 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on kafka-main[2002,2007].codfw.wmnet with reason: Hardware refresh
• 08:13 dcausse@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 08:12 dcausse@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 08:09 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: https://phabricator.wikimedia.org/T374215 → server depooled has hardware issues
• 08:09 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: https://phabricator.wikimedia.org/T374215 → server depooled has hardware issues
• 08:08 dcausse@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 08:07 dcausse@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
• 07:55 jayme: evacuating leadership for all partitions assigned to broker id 2002 on kafka-main-codfw - T363210
• 07:20 dcausse@deploy1003: Finished scap sync-world: Backport for search: use the stem field when searching mul labels (T371401) (duration: 17m 22s)
• 07:15 dcausse@deploy1003: dcausse: Continuing with sync
• 07:10 dcausse@deploy1003: dcausse: Backport for search: use the stem field when searching mul labels (T371401) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:03 dcausse@deploy1003: Started scap sync-world: Backport for search: use the stem field when searching mul labels (T371401)
• 06:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc2017.codfw.wmnet with OS bookworm
• 06:37 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on pc2017.codfw.wmnet with reason: host reimage
• 06:34 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1017.eqiad.wmnet with reason: host reimage
• 06:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on pc1017.eqiad.wmnet with reason: host reimage
• 06:18 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host pc2017.codfw.wmnet with OS bookworm
• 06:16 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host pc1017.eqiad.wmnet with OS bookworm
• 06:11 kart_: Updated cxserver to 2024-08-28-053620-production
• 06:11 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
• 06:10 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
• 05:47 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
• 05:46 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply
• 05:37 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
• 05:36 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply
• 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.43.0-wmf.19 (duration: 00m 58s)
• 03:47 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.43.0-wmf.22 refs T373641 (duration: 45m 06s)
• 03:02 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.43.0-wmf.22 refs T373641

2024-09-09

• 23:00 toyofuku@deploy1003: Finished scap sync-world: Backport for Enable appearance menu for all logged in users on all projects (T371020) (duration: 12m 40s)
• 22:56 toyofuku@deploy1003: toyofuku, jdlrobson: Continuing with sync
• 22:50 toyofuku@deploy1003: toyofuku, jdlrobson: Backport for Enable appearance menu for all logged in users on all projects (T371020) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 22:48 toyofuku@deploy1003: Started scap sync-world: Backport for Enable appearance menu for all logged in users on all projects (T371020)
• 21:37 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 21:37 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 21:35 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 21:35 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 21:35 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 21:35 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 21:29 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 21:29 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 21:27 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 21:26 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 21:24 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 21:24 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 21:23 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 21:19 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 21:19 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 21:19 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 21:19 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 21:18 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 21:17 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 21:15 bking@deploy1003: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
• 21:15 bking@deploy1003: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
• 21:02 kindrobot: finished UTC late backport window. BACKPORTED: 1071281, 1071202, NOT backported (ran out of time): 1071037, 1070347, 1070354, 1070975
• 21:01 kindrobot@deploy1003: Finished scap sync-world: Backport for Release donate link to pilot wikis (French Wikipedia and Wikifunctions) (T373585), Fix typo in browser vendor prefix (T374180) (duration: 12m 39s)
• 20:57 kindrobot@deploy1003: jforrester, toyofuku, kindrobot: Continuing with sync
• 20:51 kindrobot@deploy1003: jforrester, toyofuku, kindrobot: Backport for Release donate link to pilot wikis (French Wikipedia and Wikifunctions) (T373585), Fix typo in browser vendor prefix (T374180) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:49 kindrobot@deploy1003: Started scap sync-world: Backport for Release donate link to pilot wikis (French Wikipedia and Wikifunctions) (T373585), Fix typo in browser vendor prefix (T374180)
• 19:33 jforrester@deploy1003: Finished scap sync-world: Backport for ZObjectStructureValidator::validate: use set_time_limit() to limit in the case of run-away JsonSchema (T374241) (duration: 08m 19s)
• 19:28 jforrester@deploy1003: jforrester: Continuing with sync
• 19:27 jforrester@deploy1003: jforrester: Backport for ZObjectStructureValidator::validate: use set_time_limit() to limit in the case of run-away JsonSchema (T374241) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 19:25 jforrester@deploy1003: Started scap sync-world: Backport for ZObjectStructureValidator::validate: use set_time_limit() to limit in the case of run-away JsonSchema (T374241)
• 18:16 ejegg: payments-wiki upgraded from 4f3500f7 to 672c9fb6
• 17:44 jforrester@deploy1003: Finished scap sync-world: Backport for ZObjectStore::findZTesterResult: Trim our own error so we don't break logstash (T374241) (duration: 12m 05s)
• 17:39 jforrester@deploy1003: jforrester: Continuing with sync
• 17:33 jforrester@deploy1003: jforrester: Backport for ZObjectStore::findZTesterResult: Trim our own error so we don't break logstash (T374241) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 17:31 jforrester@deploy1003: Started scap sync-world: Backport for ZObjectStore::findZTesterResult: Trim our own error so we don't break logstash (T374241)
• 17:15 ebernhardson@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:15 ebernhardson@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:08 ebernhardson@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:07 ebernhardson@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:05 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:05 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add nes frack firewalls - pt1979@cumin2002"
• 17:05 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add nes frack firewalls - pt1979@cumin2002"
• 17:02 ebernhardson@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:02 ebernhardson@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:01 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 16:54 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Manuel Merz (WMDE) out of all services on: 1552 hosts
• 16:53 jmm@cumin2002: START - Cookbook sre.idm.logout Logging Manuel Merz (WMDE) out of all services on: 1552 hosts
• 16:52 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Manuel Merz (WMDE) out of all services on: 677 hosts
• 16:52 jmm@cumin2002: START - Cookbook sre.idm.logout Logging Manuel Merz (WMDE) out of all services on: 677 hosts
• 16:46 ejegg: payments-wiki upgraded from e47e61cb to 4f3500f7
• 16:26 hnowlan@deploy1003: Finished scap sync-world: Backport for Enable async uploads on test2wiki (T356241) (duration: 11m 11s)
• 16:20 hnowlan@deploy1003: hnowlan: Continuing with sync
• 16:20 cgoubert@cumin1002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 2 hosts: mw[2428-2429].codfw.wmnet
• 16:20 cgoubert@cumin1002: START - Cookbook sre.debmonitor.remove-hosts for 2 hosts: mw[2428-2429].codfw.wmnet
• 16:19 hnowlan@deploy1003: hnowlan: Backport for Enable async uploads on test2wiki (T356241) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 16:17 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2106.codfw.wmnet
• 16:17 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2106.codfw.wmnet
• 16:17 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2106.codfw.wmnet
• 16:17 elukey@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
• 16:15 hnowlan@deploy1003: Started scap sync-world: Backport for Enable async uploads on test2wiki (T356241)
• 16:13 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 16:13 elukey@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
• 16:10 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2106.codfw.wmnet with OS bullseye
• 16:09 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2105.codfw.wmnet
• 16:09 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2105.codfw.wmnet
• 16:09 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2105.codfw.wmnet
• 16:09 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2095.codfw.wmnet
• 16:09 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2095.codfw.wmnet
• 16:05 hnowlan: homer lsw1-b5-codfw* commit
• 16:05 aqu@deploy1003: Finished deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow (duration: 00m 10s)
• 16:04 aqu@deploy1003: Started deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow
• 16:04 claime: homer lsw1-b6-codfw* commit 'T372878'
• 16:03 claime: homer cr*codfw* commit 'T372878'
• 16:01 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2105.codfw.wmnet with OS bullseye
• 15:53 cmooney@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1002.eqiad.wmnet with reason: Release v0.7.0 plugin update for cephosd bgp - cmooney@cumin1002
• 15:48 hnowlan@cumin1002: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) pool for host wikikube-worker2095.codfw.wmnet
• 15:48 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2095.codfw.wmnet
• 15:47 cmooney@cumin1002: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1002.eqiad.wmnet with reason: Release v0.7.0 plugin update for cephosd bgp - cmooney@cumin1002
• 15:46 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2095.codfw.wmnet with OS bullseye
• 15:44 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2106.codfw.wmnet with reason: host reimage
• 15:41 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2105.codfw.wmnet with reason: host reimage
• 15:41 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2106.codfw.wmnet with reason: host reimage
• 15:41 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 15:40 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw2-d-eqiad
• 15:40 cmooney@cumin1002: START - Cookbook sre.hosts.remove-downtime for asw2-d-eqiad
• 15:37 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2105.codfw.wmnet with reason: host reimage
• 15:32 sukhe: restart bird on durum1001
• 15:27 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on asw2-d-eqiad with reason: repalce vcp link from d2 port 51 to d4 port 52
• 15:27 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on asw2-d-eqiad with reason: repalce vcp link from d2 port 51 to d4 port 52
• 15:26 hnowlan@cumin1002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker2095.codfw.wmnet with reason: host reimage
• 15:26 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2095.codfw.wmnet with reason: host reimage
• 15:09 moritzm: installing imagemagick security updates
• 15:08 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2095.codfw.wmnet with OS bullseye
• 14:55 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2106
• 14:55 cgoubert@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2106
• 14:54 cgoubert@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2106
• 14:54 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2106.codfw.wmnet 57.16.192.10.in-addr.arpa 7.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 14:54 cgoubert@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2106.codfw.wmnet 57.16.192.10.in-addr.arpa 7.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 14:54 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:54 cgoubert@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2106 - cgoubert@cumin1002"
• 14:54 cgoubert@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2106 - cgoubert@cumin1002"
• 14:53 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
• 14:53 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
• 14:53 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
• 14:53 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
• 14:51 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 14:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140 (T371742)', diff saved to https://phabricator.wikimedia.org/P68759 and previous config saved to /var/cache/conftool/dbconfig/20240909-145145-ladsgroup.json
• 14:51 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 14:51 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 14:50 cgoubert@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2106
• 14:50 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2106.codfw.wmnet with OS bullseye
• 14:49 cgoubert@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2106.codfw.wmnet
• 14:49 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2105
• 14:49 cgoubert@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2105
• 14:49 cgoubert@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2105
• 14:49 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2105.codfw.wmnet 56.16.192.10.in-addr.arpa 6.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 14:49 cgoubert@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2105.codfw.wmnet 56.16.192.10.in-addr.arpa 6.5.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 14:49 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:49 cgoubert@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2105 - cgoubert@cumin1002"
• 14:49 cgoubert@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2105 - cgoubert@cumin1002"
• 14:49 sukhe: sudo cumin -b1 -s300 'A:dnsbox and A:edges' 'systemctl restart ntpsec.service'
• 14:45 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 14:45 cgoubert@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2105
• 14:45 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2105.codfw.wmnet with OS bullseye
• 14:44 cgoubert@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2105.codfw.wmnet
• 14:44 sukhe@cumin1002: END (ERROR) - Cookbook sre.dns.roll-restart-ntp (exit_code=97) rolling restart_daemons on A:dnsbox and not P{dns1004* or dns1005*} and A:dnsbox
• 14:43 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2105.codfw.wmnet wikikube-worker2106.codfw.wmnet on all recursors
• 14:43 cgoubert@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2105.codfw.wmnet wikikube-worker2106.codfw.wmnet on all recursors
• 14:40 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2429 to wikikube-worker2106
• 14:39 cgoubert@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2106
• 14:39 cgoubert@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2106
• 14:39 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:39 cgoubert@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2429 to wikikube-worker2106 - cgoubert@cumin1002"
• 14:38 cgoubert@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2429 to wikikube-worker2106 - cgoubert@cumin1002"
• 14:36 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140', diff saved to https://phabricator.wikimedia.org/P68758 and previous config saved to /var/cache/conftool/dbconfig/20240909-143638-ladsgroup.json
• 14:34 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 14:34 cgoubert@cumin1002: START - Cookbook sre.hosts.rename from mw2429 to wikikube-worker2106
• 14:31 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2428 to wikikube-worker2105
• 14:30 cgoubert@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2105
• 14:30 cgoubert@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2105
• 14:30 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:30 cgoubert@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2428 to wikikube-worker2105 - cgoubert@cumin1002"
• 14:28 cgoubert@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2428 to wikikube-worker2105 - cgoubert@cumin1002"
• 14:25 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 14:24 cgoubert@cumin1002: START - Cookbook sre.hosts.rename from mw2428 to wikikube-worker2105
• 14:22 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2429.codfw.wmnet
• 14:21 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2429.codfw.wmnet
• 14:21 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2428.codfw.wmnet
• 14:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140', diff saved to https://phabricator.wikimedia.org/P68757 and previous config saved to /var/cache/conftool/dbconfig/20240909-142131-ladsgroup.json
• 14:20 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2428.codfw.wmnet
• 14:17 jforrester@deploy1003: Finished scap sync-world: Backport for Revert "Activate feature flag for moving wikibase item to Other Projects sidebar in pilot wikis." (T66315), Enable Copyupload-allowed-domains on test2wiki (T356241) (duration: 12m 16s)
• 14:13 jforrester@deploy1003: seanleong-wmde, jforrester, hnowlan: Continuing with sync
• 14:08 jforrester@deploy1003: seanleong-wmde, jforrester, hnowlan: Backport for Revert "Activate feature flag for moving wikibase item to Other Projects sidebar in pilot wikis." (T66315), Enable Copyupload-allowed-domains on test2wiki (T356241) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:07 sukhe@cumin1002: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox and not P{dns1004* or dns1005*} and A:dnsbox
• 14:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140 (T371742)', diff saved to https://phabricator.wikimedia.org/P68756 and previous config saved to /var/cache/conftool/dbconfig/20240909-140623-ladsgroup.json
• 14:05 jforrester@deploy1003: Started scap sync-world: Backport for Revert "Activate feature flag for moving wikibase item to Other Projects sidebar in pilot wikis." (T66315), Enable Copyupload-allowed-domains on test2wiki (T356241)
• 14:03 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:01 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2104.codfw.wmnet
• 14:01 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2104.codfw.wmnet
• 14:01 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2104.codfw.wmnet
• 14:00 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough and A:wikidough
• {{safesubst:SAL entry|1=13:58 jforrester@deploy1003: Finished scap sync-world: Backport for Define wgCheckUserCentralIndexRangesToExclude to exclude WMCS (T373021), tests: Disable all Beta Cluster CI testing, all failing (T374242), Don't pass empty type/returnType to zobject lookup when undefined (T374199), [[gerrit:1071265|Use default width/height on gallery to avoid parser instance (T37414}}
• 13:55 kamila_: homer lsw1-b6-codfw* commit 'T372878'
• 13:55 kamila_: homer cr*codfw* commit 'T372878'
• 13:54 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2104.codfw.wmnet with OS bullseye
• 13:52 jforrester@deploy1003: dreamyjazz, jforrester: Continuing with sync
• {{safesubst:SAL entry|1=13:50 jforrester@deploy1003: dreamyjazz, jforrester: Backport for Define wgCheckUserCentralIndexRangesToExclude to exclude WMCS (T373021), tests: Disable all Beta Cluster CI testing, all failing (T374242), Don't pass empty type/returnType to zobject lookup when undefined (T374199), [[gerrit:1071265|Use default width/height on gallery to avoid parser instance (T374146)}}
• 13:49 aqu@deploy1003: Finished deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow (duration: 00m 11s)
• 13:49 aqu@deploy1003: Started deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow
• 13:47 mszabo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ipoid: sync
• 13:46 sukhe@cumin1002: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough and A:wikidough
• 13:46 mszabo@deploy1003: helmfile [eqiad] START helmfile.d/services/ipoid: sync
• {{safesubst:SAL entry|1=13:46 jforrester@deploy1003: Started scap sync-world: Backport for Define wgCheckUserCentralIndexRangesToExclude to exclude WMCS (T373021), tests: Disable all Beta Cluster CI testing, all failing (T374242), Don't pass empty type/returnType to zobject lookup when undefined (T374199), [[gerrit:1071265|Use default width/height on gallery to avoid parser instance (T374146}}
• 13:45 James_F: kill 4135240 # scap thread with no attached screen
• 13:41 mszabo@deploy1003: helmfile [codfw] DONE helmfile.d/services/ipoid: apply
• 13:40 mszabo@deploy1003: helmfile [codfw] START helmfile.d/services/ipoid: apply
• {{safesubst:SAL entry|1=13:40 jforrester@deploy1003: dreamyjazz, jforrester: Backport for Define wgCheckUserCentralIndexRangesToExclude to exclude WMCS (T373021), tests: Disable all Beta Cluster CI testing, all failing (T374242), Don't pass empty type/returnType to zobject lookup when undefined (T374199), [[gerrit:1071265|Use default width/height on gallery to avoid parser instance (T374146)}}
• 13:39 mszabo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ipoid: apply
• 13:39 mszabo@deploy1003: helmfile [eqiad] START helmfile.d/services/ipoid: apply
• 13:38 mszabo@deploy1003: helmfile [staging] DONE helmfile.d/services/ipoid: apply
• 13:37 mszabo@deploy1003: helmfile [staging] START helmfile.d/services/ipoid: apply
• {{safesubst:SAL entry|1=13:37 jforrester@deploy1003: Started scap sync-world: Backport for Define wgCheckUserCentralIndexRangesToExclude to exclude WMCS (T373021), tests: Disable all Beta Cluster CI testing, all failing (T374242), Don't pass empty type/returnType to zobject lookup when undefined (T374199), [[gerrit:1071265|Use default width/height on gallery to avoid parser instance (T374146}}
• 13:34 sukhe: sudo cumin "A:dnsbox" 'disable-puppet "merging CR 1071616"'
• 13:32 milimetric@deploy1003: Finished deploy [airflow-dags/platform_eng@574f0de]: (no justification provided) (duration: 00m 26s)
• 13:32 milimetric@deploy1003: Started deploy [airflow-dags/platform_eng@574f0de]: (no justification provided)
• 13:13 dreamyjazz@deploy1003: dreamyjazz: Continuing with sync
• 13:13 dreamyjazz@deploy1003: dreamyjazz: Backport for Define wgCheckUserCentralIndexRangesToExclude to exclude WMCS (T373021) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:09 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2104.codfw.wmnet with reason: host reimage
• 13:08 dreamyjazz@deploy1003: Started scap sync-world: Backport for Define wgCheckUserCentralIndexRangesToExclude to exclude WMCS (T373021)
• 13:06 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2104.codfw.wmnet with reason: host reimage
• 12:59 dreamyjazz@deploy1003: Started scap sync-world: Backport for Define wgCheckUserCentralIndexRangesToExclude to exclude WMCS (T373021)
• 12:48 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2104
• 12:48 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2104
• 12:48 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2104
• 12:48 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2104.codfw.wmnet 61.16.192.10.in-addr.arpa 1.6.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:48 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2104.codfw.wmnet 61.16.192.10.in-addr.arpa 1.6.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:48 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:48 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2104 - kamila@cumin1002"
• 12:48 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2104 - kamila@cumin1002"
• 12:43 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 12:43 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2104
• 12:43 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2104.codfw.wmnet with OS bullseye
• 12:43 kamila@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2104.codfw.wmnet
• 12:41 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2104.codfw.wmnet on all recursors
• 12:41 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2104.codfw.wmnet on all recursors
• 12:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 12:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 12:38 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2431 to wikikube-worker2104
• 12:37 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2104
• 12:37 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2104
• 12:37 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:37 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2431 to wikikube-worker2104 - kamila@cumin1002"
• 12:36 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2431 to wikikube-worker2104 - kamila@cumin1002"
• 12:33 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
• 12:33 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 12:33 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw2431 to wikikube-worker2104
• 12:29 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2431.codfw.wmnet
• 12:28 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2431.codfw.wmnet
• 12:23 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T370903)', diff saved to https://phabricator.wikimedia.org/P68755 and previous config saved to /var/cache/conftool/dbconfig/20240909-122321-ladsgroup.json
• 12:18 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 12:18 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 12:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 12:11 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host dragonfly-supernode2001.codfw.wmnet
• 12:08 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P68754 and previous config saved to /var/cache/conftool/dbconfig/20240909-120814-ladsgroup.json
• 12:07 Dreamy_Jazz: Restarting MediaModeration scanning script - https://wikitech.wikimedia.org/wiki/MediaModeration
• 11:54 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 11:53 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 11:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P68753 and previous config saved to /var/cache/conftool/dbconfig/20240909-115306-ladsgroup.json
• 11:38 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2140 (T371742)', diff saved to https://phabricator.wikimedia.org/P68752 and previous config saved to /var/cache/conftool/dbconfig/20240909-113849-ladsgroup.json
• 11:38 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2140.codfw.wmnet with reason: Maintenance
• 11:38 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2140.codfw.wmnet with reason: Maintenance
• 11:38 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T370903)', diff saved to https://phabricator.wikimedia.org/P68751 and previous config saved to /var/cache/conftool/dbconfig/20240909-113759-ladsgroup.json
• 11:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 100%: post db2227 clone repool', diff saved to https://phabricator.wikimedia.org/P68750 and previous config saved to /var/cache/conftool/dbconfig/20240909-113613-arnaudb.json
• 11:31 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2220 (T370903)', diff saved to https://phabricator.wikimedia.org/P68749 and previous config saved to /var/cache/conftool/dbconfig/20240909-113110-ladsgroup.json
• 11:31 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2220.codfw.wmnet with reason: Maintenance
• 11:30 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db2220.codfw.wmnet with reason: Maintenance
• 11:25 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
• 11:25 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
• 11:21 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
• 11:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 75%: post db2227 clone repool', diff saved to https://phabricator.wikimedia.org/P68748 and previous config saved to /var/cache/conftool/dbconfig/20240909-112107-arnaudb.json
• 11:09 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2379.codfw.wmnet
• 11:09 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2379.codfw.wmnet
• 11:09 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2435.codfw.wmnet
• 11:09 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2435.codfw.wmnet
• 11:09 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2434.codfw.wmnet
• 11:09 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2434.codfw.wmnet
• 11:09 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2430.codfw.wmnet
• 11:09 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2430.codfw.wmnet
• 11:08 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2423.codfw.wmnet
• 11:08 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2423.codfw.wmnet
• 11:08 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2422.codfw.wmnet
• 11:08 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2422.codfw.wmnet
• 11:08 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2420.codfw.wmnet
• 11:08 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2420.codfw.wmnet
• 11:07 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2407.codfw.wmnet
• 11:07 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2407.codfw.wmnet
• 11:07 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2406.codfw.wmnet
• 11:07 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2406.codfw.wmnet
• 11:07 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2402.codfw.wmnet
• 11:07 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2402.codfw.wmnet
• 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2389.codfw.wmnet
• 11:06 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2389.codfw.wmnet
• 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2388.codfw.wmnet
• 11:06 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2388.codfw.wmnet
• 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2387.codfw.wmnet
• 11:06 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2387.codfw.wmnet
• 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2386.codfw.wmnet
• 11:06 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2386.codfw.wmnet
• 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2385.codfw.wmnet
• 11:06 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2385.codfw.wmnet
• 11:06 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2384.codfw.wmnet
• 11:06 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2384.codfw.wmnet
• 11:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 50%: post db2227 clone repool', diff saved to https://phabricator.wikimedia.org/P68747 and previous config saved to /var/cache/conftool/dbconfig/20240909-110601-arnaudb.json
• 11:05 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2383.codfw.wmnet
• 11:05 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2383.codfw.wmnet
• 11:05 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2382.codfw.wmnet
• 11:05 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2382.codfw.wmnet
• 11:05 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2381.codfw.wmnet
• 11:05 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2381.codfw.wmnet
• 11:05 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2380.codfw.wmnet
• 11:05 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2380.codfw.wmnet
• 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2378.codfw.wmnet
• 11:03 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2378.codfw.wmnet
• 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2377.codfw.wmnet
• 11:03 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2377.codfw.wmnet
• 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2319.codfw.wmnet
• 11:02 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2319.codfw.wmnet
• 11:02 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2318.codfw.wmnet
• 11:02 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2318.codfw.wmnet
• 11:02 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2317.codfw.wmnet
• 11:02 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2317.codfw.wmnet
• 11:02 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2316.codfw.wmnet
• 11:02 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2316.codfw.wmnet
• 11:02 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2312.codfw.wmnet
• 11:02 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2312.codfw.wmnet
• 11:01 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2296.codfw.wmnet
• 11:01 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2296.codfw.wmnet
• 11:01 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2295.codfw.wmnet
• 11:01 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2295.codfw.wmnet
• 11:01 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2293.codfw.wmnet
• 11:01 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2293.codfw.wmnet
• 10:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 25%: post db2227 clone repool', diff saved to https://phabricator.wikimedia.org/P68746 and previous config saved to /var/cache/conftool/dbconfig/20240909-105056-arnaudb.json
• 10:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2127.codfw.wmnet onto db2227.codfw.wmnet
• 10:34 jelto@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 10:33 jelto@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 10:33 jelto@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 10:32 jelto@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
• 10:31 jelto@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 10:30 jelto@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 10:19 jelto@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 10:18 jelto@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 09:44 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' .
• 09:44 isaranto@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' .
• 09:42 isaranto@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' .
• 09:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 09:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 09:27 arnaudb@cumin1002: START - Cookbook sre.mysql.clone of db2127.codfw.wmnet onto db2227.codfw.wmnet
• 09:25 moritzm: removing libssl1.1 from prometheus hosts which were dist-upgraded from bullseye to bookworm
• 09:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2127 in db2227 for T373579', diff saved to https://phabricator.wikimedia.org/P68745 and previous config saved to /var/cache/conftool/dbconfig/20240909-092404-arnaudb.json
• 09:22 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2227.codfw.wmnet with reason: provisionning db2227.codfw.wmnet - T373579
• 09:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2227.codfw.wmnet with reason: provisionning db2227.codfw.wmnet - T373579
• 09:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2127.codfw.wmnet with reason: provisionning db2227.codfw.wmnet - T373579
• 09:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2127.codfw.wmnet with reason: provisionning db2227.codfw.wmnet - T373579
• 09:18 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: gerrit1004.wikimedia.org
• 09:18 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: gerrit1004.wikimedia.org
• 09:07 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript namespaceDupes.php --wiki=mnwiki --add-prefix=BROKEN --fix # T366271
• 08:57 moritzm: restarting postfix on mx-in/mx-out to pick up openssl updates
• 08:54 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet with OS bookworm
• 08:51 arnaudb@cumin1002: dbctl commit (dc=all): 'API/vslow/dump T374086', diff saved to https://phabricator.wikimedia.org/P68744 and previous config saved to /var/cache/conftool/dbconfig/20240909-085122-arnaudb.json
• 08:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2213 to s5 primary T374086', diff saved to https://phabricator.wikimedia.org/P68743 and previous config saved to /var/cache/conftool/dbconfig/20240909-084810-arnaudb.json
• 08:47 arnaudb: Starting s5 codfw failover from db2123 to db2213 - T374086
• 08:41 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists2001.wikimedia.org
• 08:40 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dragonfly-supernode2001.codfw.wmnet with reason: host reimage
• 08:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Remove db2213 from API/vslow/dump T374086', diff saved to https://phabricator.wikimedia.org/P68742 and previous config saved to /var/cache/conftool/dbconfig/20240909-083910-arnaudb.json
• 08:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T374086
• 08:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Primary switchover s5 T374086
• 08:37 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kubernetes2054.codfw.wmnet
• 08:37 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kubernetes2054.codfw.wmnet
• 08:36 elukey@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on dragonfly-supernode2001.codfw.wmnet with reason: host reimage
• 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kubernetes2055.codfw.wmnet
• 08:36 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kubernetes2055.codfw.wmnet
• 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kubernetes2057.codfw.wmnet
• 08:35 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kubernetes2057.codfw.wmnet
• 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kubernetes2055.codfw.wmnet
• 08:35 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kubernetes2055.codfw.wmnet
• 08:35 jelto@cumin1002: START - Cookbook sre.hosts.reboot-single for host lists2001.wikimedia.org
• 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kubernetes2035.codfw.wmnet
• 08:35 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kubernetes2035.codfw.wmnet
• 08:35 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kubernetes2033.codfw.wmnet
• 08:35 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kubernetes2033.codfw.wmnet
• 08:34 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kubernetes2029.codfw.wmnet
• 08:34 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kubernetes2029.codfw.wmnet
• 08:34 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kubernetes2028.codfw.wmnet
• 08:34 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kubernetes2028.codfw.wmnet
• 08:34 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kubernetes2027.codfw.wmnet
• 08:34 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kubernetes2027.codfw.wmnet
• 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kubernetes2025.codfw.wmnet
• 08:33 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kubernetes2025.codfw.wmnet
• 08:31 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kubernetes2018.codfw.wmnet
• 08:31 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kubernetes2018.codfw.wmnet
• 08:31 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kubernetes2010.codfw.wmnet
• 08:31 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kubernetes2010.codfw.wmnet
• 08:31 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kubernetes2008.codfw.wmnet
• 08:31 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kubernetes2008.codfw.wmnet
• 08:27 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kubernetes2034.codfw.wmnet
• 08:27 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kubernetes2034.codfw.wmnet
• 08:26 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: kubernetes2031.codfw.wmnet
• 08:26 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: kubernetes2031.codfw.wmnet
• 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2332.codfw.wmnet
• 08:25 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2332.codfw.wmnet
• 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2322.codfw.wmnet
• 08:25 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2322.codfw.wmnet
• 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2321.codfw.wmnet
• 08:25 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2321.codfw.wmnet
• 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: mw2320.codfw.wmnet
• 08:25 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: mw2320.codfw.wmnet
• 08:23 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
• 08:23 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
• 08:22 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubestage2002.codfw.wmnet with OS bookworm
• 08:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Reconfig db2140 T373330', diff saved to https://phabricator.wikimedia.org/P68741 and previous config saved to /var/cache/conftool/dbconfig/20240909-082053-arnaudb.json
• 08:20 elukey@cumin2002: START - Cookbook sre.hosts.reimage for host dragonfly-supernode2001.codfw.wmnet with OS bookworm
• 08:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2179 to s4 primary T373330', diff saved to https://phabricator.wikimedia.org/P68740 and previous config saved to /var/cache/conftool/dbconfig/20240909-081750-arnaudb.json
• 08:17 arnaudb: Starting s4 codfw failover from db2140 to db2179 - T373330
• 08:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 08:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Remove db2179 from API/vslow/dump T373330', diff saved to https://phabricator.wikimedia.org/P68739 and previous config saved to /var/cache/conftool/dbconfig/20240909-080956-arnaudb.json
• 08:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 08:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2179 with weight 0 T373330', diff saved to https://phabricator.wikimedia.org/P68738 and previous config saved to /var/cache/conftool/dbconfig/20240909-080935-arnaudb.json
• 08:09 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s4 T373330
• 08:08 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s4 T373330
• 08:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 T373175', diff saved to https://phabricator.wikimedia.org/P68737 and previous config saved to /var/cache/conftool/dbconfig/20240909-080558-arnaudb.json
• 08:04 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestage2002.codfw.wmnet with reason: host reimage
• 08:04 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 T373175', diff saved to https://phabricator.wikimedia.org/P68736 and previous config saved to /var/cache/conftool/dbconfig/20240909-080422-arnaudb.json
• 08:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2218 to s7 primary T373175', diff saved to https://phabricator.wikimedia.org/P68735 and previous config saved to /var/cache/conftool/dbconfig/20240909-080108-arnaudb.json
• 08:00 arnaudb: Starting s7 codfw failover from db2220 to db2218 - T373175
• 08:00 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on kubestage2002.codfw.wmnet with reason: host reimage
• 07:58 moritzm: installing openssl security updates
• 07:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Remove db2218 from API/vslow/dump T373175', diff saved to https://phabricator.wikimedia.org/P68734 and previous config saved to /var/cache/conftool/dbconfig/20240909-075258-arnaudb.json
• 07:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2218 with weight 0 T373175', diff saved to https://phabricator.wikimedia.org/P68733 and previous config saved to /var/cache/conftool/dbconfig/20240909-075145-arnaudb.json
• 07:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 29 hosts with reason: Primary switchover s7 T373175
• 07:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 29 hosts with reason: Primary switchover s7 T373175
• 07:39 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host kubestage2002.codfw.wmnet with OS bookworm
• 07:37 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host kubestage2002.codfw.wmnet
• 07:37 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubestage2002.codfw.wmnet with OS bookworm
• 07:37 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host kubestage2002.codfw.wmnet with OS bookworm
• 07:36 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
• 07:33 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
• 07:33 jayme@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host kubestage2002.codfw.wmnet
• 07:33 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
• 07:33 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
• 07:17 moritzm: installing Linux 5.10.223 on bullseye hosts
• 07:06 moritzm: roll out debmonitor-client 0.4.0-2+deb11u1 on bullseye hosts
• 06:56 moritzm: installing aom security updates

2024-09-07

• 14:44 ebernhardson@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 14:44 ebernhardson@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 14:38 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 14:38 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 14:38 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 14:38 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 14:38 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 14:38 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
• 14:34 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 14:34 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 14:34 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 14:34 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 14:34 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 14:34 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
• 10:28 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on db1246.eqiad.wmnet with reason: https://phabricator.wikimedia.org/T374215 → server depooled has hardware issues
• 10:28 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on db1246.eqiad.wmnet with reason: https://phabricator.wikimedia.org/T374215 → server depooled has hardware issues

2024-09-06

• 22:35 ebernhardson@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 22:35 ebernhardson@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 19:07 dduvall@deploy1003: Finished deploy [releng/jenkins-deploy@6ca00a7] (releasing): (no justification provided) (duration: 00m 43s)
• 19:06 dduvall@deploy1003: Started deploy [releng/jenkins-deploy@6ca00a7] (releasing): (no justification provided)
• 19:01 ebernhardson@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 19:00 ebernhardson@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 18:57 ebernhardson@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 18:57 ebernhardson@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
• 18:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 7:00:00 on db2200.codfw.wmnet with reason: Maintenance
• 18:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 7:00:00 on db2200.codfw.wmnet with reason: Maintenance
• 18:12 brett: Import ncmonitor 1.2.1-1 into bookworm-wikimedia apt archive
• 17:55 brett: Import corto 0.3.1-1 into bookworm-wikimedia apt archive
• 16:46 kamila_: ran homer on cr*codfw* for T372878
• 16:30 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2103.codfw.wmnet
• 16:30 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2103.codfw.wmnet
• 16:30 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2103.codfw.wmnet
• 16:24 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2103.codfw.wmnet with OS bullseye
• 16:06 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling reboot on A:durum and A:durum
• 16:04 hnowlan@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2095.codfw.wmnet with OS bullseye
• 16:02 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2103.codfw.wmnet with reason: host reimage
• 15:59 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2103.codfw.wmnet with reason: host reimage
• 15:52 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 15:49 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 15:44 dduvall@deploy1003: Finished deploy [releng/jenkins-deploy@ad2c434] (releasing): (no justification provided) (duration: 00m 41s)
• 15:43 dduvall@deploy1003: Started deploy [releng/jenkins-deploy@ad2c434] (releasing): (no justification provided)
• 15:43 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1017.eqiad.wmnet
• 15:43 cmooney@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs1017.eqiad.wmnet
• 15:42 topranks: enabling PyBal on lvs1017 to make primary again after repairing faulty fiber link T374247
• 15:42 elukey: install spicerack 8.13.0 on cumin1002
• 15:40 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2103
• 15:40 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2103
• 15:39 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2103
• 15:39 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2103.codfw.wmnet 60.16.192.10.in-addr.arpa 0.6.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 15:39 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2103.codfw.wmnet 60.16.192.10.in-addr.arpa 0.6.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 15:39 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:39 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2103 - kamila@cumin1002"
• 15:39 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2103 - kamila@cumin1002"
• 15:34 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 15:34 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 15:32 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2103
• 15:32 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2103.codfw.wmnet with OS bullseye
• 15:31 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 15:29 kamila@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2103.codfw.wmnet
• 15:28 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2103.codfw.wmnet on all recursors
• 15:28 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2103.codfw.wmnet on all recursors
• 15:27 mutante: rolling restarts on durum machines
• 15:27 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=93) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 15:27 dzahn@cumin2002: START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum and A:durum
• 15:24 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2430 to wikikube-worker2103
• 15:23 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2103
• 15:23 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 15:22 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2103
• 15:22 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:22 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2430 to wikikube-worker2103 - kamila@cumin1002"
• 15:19 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2430 to wikikube-worker2103 - kamila@cumin1002"
• 15:16 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 15:15 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw2430 to wikikube-worker2103
• 15:14 topranks: disabling PyBal on lvs1017 to shift traffic to lvs1020 and allow work to fix faulty fibre link T374247
• 15:13 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1017.eqiad.wmnet with reason: Move traffic off lvs1017 to lvs1020 to troubleshooot faulty link
• 15:13 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs1017.eqiad.wmnet with reason: Move traffic off lvs1017 to lvs1020 to troubleshooot faulty link
• 15:07 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2430.codfw.wmnet
• 15:07 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2430.codfw.wmnet
• 15:02 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2098.codfw.wmnet
• 15:02 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2098.codfw.wmnet
• 15:02 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2098.codfw.wmnet
• 14:52 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2098.codfw.wmnet with OS bullseye
• 14:51 btullis@cumin1002: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database bdrwiki (T371759)
• 14:44 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2095.codfw.wmnet with OS bullseye
• 14:42 hnowlan@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2095.codfw.wmnet
• 14:42 hnowlan@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2095.codfw.wmnet with OS bullseye
• 14:42 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2095.codfw.wmnet with OS bullseye
• 14:42 hnowlan@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2095.codfw.wmnet
• 14:41 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubestage2001.codfw.wmnet with OS bookworm
• 14:28 akosiaris: repool kubernetes1059 T365993
• 14:28 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes1059.eqiad.wmnet
• 14:27 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes1059.eqiad.wmnet
• 14:25 btullis@cumin1002: START - Cookbook sre.wikireplicas.add-wiki for database bdrwiki (T371759)
• 14:23 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestage2001.codfw.wmnet with reason: host reimage
• 14:20 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on kubestage2001.codfw.wmnet with reason: host reimage
• 14:17 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2102.codfw.wmnet
• 14:17 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2102.codfw.wmnet
• 14:17 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2102.codfw.wmnet
• 14:13 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2102.codfw.wmnet with OS bullseye
• 14:10 akosiaris: restart pybal on lvs1019
• 14:07 akosiaris: silence alerts based on alertname=PHPFPMTooBusy,deployment=mw-wikifunctions,site=codfw T374241
• 13:59 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host kubestage2001.codfw.wmnet with OS bookworm
• 13:58 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
• 13:58 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
• 13:56 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host kubestage2001.codfw.wmnet
• 13:56 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
• 13:56 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
• 13:52 btullis@cumin1002: END (FAIL) - Cookbook sre.wikireplicas.add-wiki (exit_code=99) for database bdrwiki (T371759)
• 13:52 jayme: homer lsw1-a6-codfw* commit 'T372878'
• 13:52 btullis@cumin1002: START - Cookbook sre.wikireplicas.add-wiki for database bdrwiki (T371759)
• 13:48 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2102.codfw.wmnet with reason: host reimage
• 13:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet
• 13:44 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2102.codfw.wmnet with reason: host reimage
• 13:36 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubestage2001.codfw.wmnet with OS bullseye
• 13:32 cgoubert@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker2101.codfw.wmnet
• 13:32 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2101.codfw.wmnet
• 13:32 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2101.codfw.wmnet
• 13:29 cgoubert@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker2100.codfw.wmnet
• 13:29 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2100.codfw.wmnet
• 13:29 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2100.codfw.wmnet
• 13:29 hnowlan@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2095.codfw.wmnet
• 13:28 hnowlan@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2095.codfw.wmnet with OS bullseye
• 13:28 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2099.codfw.wmnet
• 13:28 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2099.codfw.wmnet
• 13:28 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2099.codfw.wmnet
• 13:27 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2101.codfw.wmnet with OS bullseye
• 13:26 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2102
• 13:26 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2102
• 13:25 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2100.codfw.wmnet with OS bullseye
• 13:24 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet
• 13:23 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2102
• 13:23 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2102.codfw.wmnet 226.16.192.10.in-addr.arpa 6.2.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:23 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2102.codfw.wmnet 226.16.192.10.in-addr.arpa 6.2.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:22 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:22 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2102 - jayme@cumin1002"
• 13:22 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2102 - jayme@cumin1002"
• 13:21 claime: homer lsw1-b6-codfw* commit 'T372878'
• 13:18 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 13:17 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
• 13:16 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
• 13:13 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestage2001.codfw.wmnet with reason: host reimage
• 13:10 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on kubestage2001.codfw.wmnet with reason: host reimage
• 13:10 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2098.codfw.wmnet with reason: host reimage
• 13:07 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2098.codfw.wmnet with reason: host reimage
• 13:06 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2099.codfw.wmnet with OS bullseye
• 13:02 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2102
• 13:02 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2102.codfw.wmnet with OS bullseye
• 13:02 jayme@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2102.codfw.wmnet
• 13:01 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2034 to wikikube-worker2102
• 13:00 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2102
• 12:58 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2102
• 12:58 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:58 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2034 to wikikube-worker2102 - jayme@cumin1002"
• 12:58 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2034 to wikikube-worker2102 - jayme@cumin1002"
• 12:57 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2096.codfw.wmnet
• 12:57 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2096.codfw.wmnet
• 12:57 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2096.codfw.wmnet
• 12:55 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 12:54 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2034 to wikikube-worker2102
• 12:54 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2096.codfw.wmnet with OS bullseye
• 12:53 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kubestage2001
• 12:53 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kubestage2001
• 12:52 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2101.codfw.wmnet with reason: host reimage
• 12:52 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host kubestage2001
• 12:52 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kubestage2001.codfw.wmnet 195.0.192.10.in-addr.arpa 5.9.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:52 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache kubestage2001.codfw.wmnet 195.0.192.10.in-addr.arpa 5.9.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:52 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:52 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kubestage2001 - jayme@cumin1002"
• 12:52 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kubestage2001 - jayme@cumin1002"
• 12:50 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2101.codfw.wmnet with reason: host reimage
• 12:49 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 12:49 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host kubestage2001
• 12:48 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host kubestage2001.codfw.wmnet with OS bullseye
• 12:48 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
• 12:48 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2100.codfw.wmnet with reason: host reimage
• 12:48 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2097.codfw.wmnet
• 12:48 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2097.codfw.wmnet
• 12:48 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2097.codfw.wmnet
• 12:47 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
• 12:47 jayme@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host kubestage2001.codfw.wmnet
• 12:45 hnowlan: homer lsw1-b3-codfw* commit
• 12:45 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw1486.eqiad.wmnet
• 12:45 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw1486.eqiad.wmnet
• 12:44 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2100.codfw.wmnet with reason: host reimage
• 12:43 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2097.codfw.wmnet with OS bullseye
• 12:43 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2099.codfw.wmnet with reason: host reimage
• 12:39 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2099.codfw.wmnet with reason: host reimage
• 12:38 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2066.codfw.wmnet
• 12:38 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2066.codfw.wmnet
• 12:37 claime: homer cr*codfw* commit 'T372878'
• 12:33 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2101
• 12:33 cgoubert@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2101
• 12:33 cgoubert@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2101
• 12:33 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2101.codfw.wmnet 203.16.192.10.in-addr.arpa 3.0.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:33 cgoubert@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2101.codfw.wmnet 203.16.192.10.in-addr.arpa 3.0.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:33 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:33 cgoubert@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2101 - cgoubert@cumin1002"
• 12:33 cgoubert@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2101 - cgoubert@cumin1002"
• 12:32 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2096.codfw.wmnet with reason: host reimage
• 12:30 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 12:29 cgoubert@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2101
• 12:29 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2101.codfw.wmnet with OS bullseye
• 12:29 cgoubert@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2101.codfw.wmnet
• 12:28 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2100
• 12:28 cgoubert@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2100
• 12:28 cgoubert@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2100
• 12:28 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2100.codfw.wmnet 202.16.192.10.in-addr.arpa 2.0.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:28 cgoubert@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2100.codfw.wmnet 202.16.192.10.in-addr.arpa 2.0.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:28 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:28 cgoubert@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2100 - cgoubert@cumin1002"
• 12:28 cgoubert@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2100 - cgoubert@cumin1002"
• 12:27 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2096.codfw.wmnet with reason: host reimage
• 12:24 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 12:24 cgoubert@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2100
• 12:24 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2100.codfw.wmnet with OS bullseye
• 12:23 cgoubert@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2100.codfw.wmnet
• 12:22 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2099
• 12:22 cgoubert@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2099
• 12:22 cgoubert@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2099
• 12:22 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2099.codfw.wmnet 201.16.192.10.in-addr.arpa 1.0.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:22 cgoubert@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2099.codfw.wmnet 201.16.192.10.in-addr.arpa 1.0.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:21 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:21 cgoubert@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2099 - cgoubert@cumin1002"
• 12:21 cgoubert@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2099 - cgoubert@cumin1002"
• 12:19 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2097.codfw.wmnet with reason: host reimage
• 12:18 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 12:18 cgoubert@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2099
• 12:18 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2099.codfw.wmnet with OS bullseye
• 12:18 cgoubert@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2099.codfw.wmnet
• 12:17 cgoubert@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2029.codfw.wmnet
• 12:17 cgoubert@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2029.codfw.wmnet
• 12:16 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2097.codfw.wmnet with reason: host reimage
• 12:16 cgoubert@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2029.codfw.wmnet
• 12:16 cgoubert@cumin1002: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host wikikube-worker2029.codfw.wmnet
• 12:16 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2098
• 12:16 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2098
• 12:16 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2029.codfw.wmnet
• 12:16 cgoubert@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2029.codfw.wmnet
• 12:16 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2098
• 12:16 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2098.codfw.wmnet 176.16.192.10.in-addr.arpa 6.7.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:16 hnowlan@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2098.codfw.wmnet 176.16.192.10.in-addr.arpa 6.7.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:16 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:16 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2098 - hnowlan@cumin1002"
• 12:15 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2098 - hnowlan@cumin1002"
• 12:15 cgoubert@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2099.codfw.wmnet
• 12:15 cgoubert@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2099.codfw.wmnet
• 12:15 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit1003.wikimedia.org
• 12:14 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2099.codfw.wmnet wikikube-worker2100.codfw.wmnet wikikube-worker2101.codfw.wmnet on all recursors
• 12:14 cgoubert@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2099.codfw.wmnet wikikube-worker2100.codfw.wmnet wikikube-worker2101.codfw.wmnet on all recursors
• 12:13 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2334 to wikikube-worker2101
• 12:12 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 12:12 cgoubert@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2101
• 12:12 cgoubert@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2101
• 12:12 hnowlan@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2098
• 12:12 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:10 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2096
• 12:10 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2096
• 12:10 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 12:09 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2096
• 12:09 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2096.codfw.wmnet 173.16.192.10.in-addr.arpa 3.7.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:09 hnowlan@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2096.codfw.wmnet 173.16.192.10.in-addr.arpa 3.7.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:09 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:09 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2096 - hnowlan@cumin1002"
• 12:09 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2096 - hnowlan@cumin1002"
• 12:08 jelto@cumin1002: START - Cookbook sre.hosts.reboot-single for host gerrit1003.wikimedia.org
• 12:08 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2095
• 12:08 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2095
• 12:08 moritzm: upgrade ganeti-test2003 to bookworm for some bullseye->bookworm VM migration tests
• 12:08 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2095
• 12:08 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2095.codfw.wmnet 222.16.192.10.in-addr.arpa 2.2.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:08 hnowlan@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2095.codfw.wmnet 222.16.192.10.in-addr.arpa 2.2.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:08 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:08 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2095 - hnowlan@cumin1002"
• 12:06 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2095 - hnowlan@cumin1002"
• 12:05 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 12:04 cgoubert@cumin1002: START - Cookbook sre.hosts.rename from mw2334 to wikikube-worker2101
• 12:04 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on gerrit1003.wikimedia.org with reason: Gerrit reboot
• 12:03 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 0:15:00 on gerrit1003.wikimedia.org with reason: Gerrit reboot
• 12:03 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:15:00 on gerrit.wikimedia.org with reason: Gerrit reboot
• 12:03 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 0:15:00 on gerrit.wikimedia.org with reason: Gerrit reboot
• 12:03 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2333 to wikikube-worker2100
• 12:02 cgoubert@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2100
• 12:02 cgoubert@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2100
• 12:02 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:02 cgoubert@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2333 to wikikube-worker2100 - cgoubert@cumin1002"
• 12:02 cgoubert@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2333 to wikikube-worker2100 - cgoubert@cumin1002"
• 12:01 hnowlan@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2096
• 12:00 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 12:00 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2097
• 12:00 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2097
• 11:55 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 11:54 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2097
• 11:54 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2097.codfw.wmnet 175.16.192.10.in-addr.arpa 5.7.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:54 hnowlan@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2097.codfw.wmnet 175.16.192.10.in-addr.arpa 5.7.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:54 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:54 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2097 - hnowlan@cumin1002"
• 11:54 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2097 - hnowlan@cumin1002"
• 11:52 cgoubert@cumin1002: START - Cookbook sre.hosts.rename from mw2333 to wikikube-worker2100
• 11:51 hnowlan@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2095
• 11:51 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2098.codfw.wmnet with OS bullseye
• 11:51 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2095.codfw.wmnet with OS bullseye
• 11:50 hnowlan@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2098.codfw.wmnet
• 11:50 hnowlan@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2095.codfw.wmnet
• 11:50 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2332 to wikikube-worker2099
• 11:50 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 11:50 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2096.codfw.wmnet with OS bullseye
• 11:50 hnowlan@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2097
• 11:50 hnowlan@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2096.codfw.wmnet
• 11:50 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2097.codfw.wmnet with OS bullseye
• 11:50 hnowlan@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2097.codfw.wmnet
• 11:49 cgoubert@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2099
• 11:49 cgoubert@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2099
• 11:49 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:49 cgoubert@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2332 to wikikube-worker2099 - cgoubert@cumin1002"
• 11:49 cgoubert@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2332 to wikikube-worker2099 - cgoubert@cumin1002"
• 11:48 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2095.codfw.wmnet wikikube-worker2096.codfw.wmnet wikikube-worker2097.codfw.wmnet wikikube-worker2098.codfw.wmnet on all recursors
• 11:48 hnowlan@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2095.codfw.wmnet wikikube-worker2096.codfw.wmnet wikikube-worker2097.codfw.wmnet wikikube-worker2098.codfw.wmnet on all recursors
• 11:47 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2322 to wikikube-worker2098
• 11:46 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2098
• 11:46 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2098
• 11:46 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:46 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2322 to wikikube-worker2098 - hnowlan@cumin1002"
• 11:45 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 11:43 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2322 to wikikube-worker2098 - hnowlan@cumin1002"
• 11:41 cgoubert@cumin1002: START - Cookbook sre.hosts.rename from mw2332 to wikikube-worker2099
• 11:40 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2321 to wikikube-worker2097
• 11:40 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 11:40 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from mw2322 to wikikube-worker2098
• 11:40 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2097
• 11:39 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2097
• 11:39 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:39 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2321 to wikikube-worker2097 - hnowlan@cumin1002"
• 11:39 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2321 to wikikube-worker2097 - hnowlan@cumin1002"
• 11:35 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 11:35 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2334.codfw.wmnet
• 11:35 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2031 to wikikube-worker2095
• 11:35 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from mw2321 to wikikube-worker2097
• 11:35 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2334.codfw.wmnet
• 11:34 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2333.codfw.wmnet
• 11:34 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2095
• 11:34 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2333.codfw.wmnet
• 11:34 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2320 to wikikube-worker2096
• 11:34 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2332.codfw.wmnet
• 11:33 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2095
• 11:33 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:33 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2031 to wikikube-worker2095 - hnowlan@cumin1002"
• 11:33 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2332.codfw.wmnet
• 11:33 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2096
• 11:32 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2096
• 11:32 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:30 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 11:27 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2031 to wikikube-worker2095 - hnowlan@cumin1002"
• 11:26 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2093.codfw.wmnet
• 11:24 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 11:24 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from mw2320 to wikikube-worker2096
• 11:24 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2031 to wikikube-worker2095
• 11:23 jayme: homer lsw1-b6-codfw* commit 'T372878'
• 11:15 moritzm: installing Linux 5.10.223 on bullseye hosts
• 11:09 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2093.codfw.wmnet with OS bullseye
• 11:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2002.wikimedia.org
• 11:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti-test2003.codfw.wmnet
• 11:02 moritzm: rolling out debmonitor-client 0.4.0-2+deb11u1 on bullseye-wikimedia on bullseye hosts T372472
• 11:00 eoghan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on lists1004.wikimedia.org with reason: T373980
• 11:00 jelto@cumin1002: START - Cookbook sre.hosts.reboot-single for host gitlab2002.wikimedia.org
• 11:00 eoghan@cumin1002: START - Cookbook sre.hosts.downtime for 0:20:00 on lists1004.wikimedia.org with reason: T373980
• 10:43 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2003.codfw.wmnet
• 10:40 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 10:39 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 10:39 moritzm: uploaded debmonitor-client 0.4.0-2+deb11u1 on bullseye-wikimedia (didn't rebuild the other suites since the fix is specific to Bullseye) T372472
• 10:38 elukey: factory reset of sretest2001
• 10:33 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on db1246.eqiad.wmnet with reason: Server failed, rebooted in emergency/single user mode
• 10:33 akosiaris@cumin1002: START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on db1246.eqiad.wmnet with reason: Server failed, rebooted in emergency/single user mode
• 10:33 akosiaris@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1246.eqiad.wmnet with reason: Server failed, rebooted in emergency/single user mode
• 10:32 akosiaris@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1246.eqiad.wmnet with reason: Server failed, rebooted in emergency/single user mode
• 10:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:25 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool db1246 (It's sad)', diff saved to https://phabricator.wikimedia.org/P68731 and previous config saved to /var/cache/conftool/dbconfig/20240906-102551-ladsgroup.json
• 10:23 elukey: install spicerack 8.13.0 on cumin2002
• 10:17 elukey: uploaded spicerack_8.13.0 to apt.wikimedia.org bullseye-wikimedia
• 10:09 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2093.codfw.wmnet with reason: host reimage
• 10:05 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2093.codfw.wmnet with reason: host reimage
• 09:57 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2094.codfw.wmnet
• 09:57 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2094.codfw.wmnet
• 09:50 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2094.codfw.wmnet
• 09:48 jayme: homer cr*codfw* commit 'T372878'
• 09:47 jayme: homer lsw1-b6-codfw* commit 'T372878'
• 09:46 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=maps2007.codfw.wmnet
• 09:45 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2094.codfw.wmnet with OS bullseye
• 09:35 klausman@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2005.codfw.wmnet
• 09:28 klausman@cumin2002: START - Cookbook sre.hosts.reboot-single for host ml-serve2005.codfw.wmnet
• 09:26 klausman@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2005.codfw.wmnet
• 09:20 klausman@cumin2002: START - Cookbook sre.hosts.reboot-single for host ml-serve2005.codfw.wmnet
• 09:15 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2094.codfw.wmnet with reason: host reimage
• 09:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:12 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2094.codfw.wmnet with reason: host reimage
• 09:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 08:57 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: sync
• 08:55 elukey@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: sync
• 08:54 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2094
• 08:54 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2094
• 08:52 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2094
• 08:52 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2094.codfw.wmnet 224.16.192.10.in-addr.arpa 4.2.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 08:52 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2094.codfw.wmnet 224.16.192.10.in-addr.arpa 4.2.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 08:52 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:52 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2094 - jayme@cumin1002"
• 08:51 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2094 - jayme@cumin1002"
• 08:49 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: sync
• 08:48 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 08:48 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2094
• 08:48 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2094.codfw.wmnet with OS bullseye
• 08:48 elukey@deploy1003: helmfile [codfw] START helmfile.d/services/proton: sync
• 08:48 jayme@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2094.codfw.wmnet
• 08:47 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2093
• 08:47 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2093
• 08:46 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2093
• 08:46 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2093.codfw.wmnet 135.16.192.10.in-addr.arpa 5.3.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 08:46 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2093.codfw.wmnet 135.16.192.10.in-addr.arpa 5.3.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 08:46 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:46 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2093 - jayme@cumin1002"
• 08:46 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2093 - jayme@cumin1002"
• 08:43 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 08:43 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2093
• 08:43 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2093.codfw.wmnet with OS bullseye
• 08:42 jayme@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2093.codfw.wmnet
• 08:41 elukey@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: sync
• 08:40 elukey@deploy1003: helmfile [staging] START helmfile.d/services/proton: sync
• 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts acmechief2001.codfw.wmnet
• 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: acmechief2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
• 08:36 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: acmechief2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
• 08:36 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2087.codfw.wmnet
• 08:36 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2087.codfw.wmnet
• 08:32 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2033 to wikikube-worker2094
• 08:32 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2094
• 08:32 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2020 to wikikube-worker2093
• 08:31 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2094
• 08:31 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:31 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2033 to wikikube-worker2094 - jayme@cumin1002"
• 08:31 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2033 to wikikube-worker2094 - jayme@cumin1002"
• 08:31 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2093
• 08:31 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2093
• 08:31 jayme@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 08:28 jmm@cumin2002: START - Cookbook sre.dns.netbox
• 08:24 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts acmechief2001.codfw.wmnet
• 08:24 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2087.codfw.wmnet with OS bullseye
• 08:23 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 08:18 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2033 to wikikube-worker2094
• 08:18 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 08:18 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2020 to wikikube-worker2093
• 08:01 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts acmechief1001.eqiad.wmnet
• 08:01 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:01 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: acmechief1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
• 08:00 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host kubernetes2033.codfw.wmnet
• 08:00 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2033.codfw.wmnet
• 08:00 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2020.codfw.wmnet
• 08:00 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: acmechief1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
• 07:59 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2020.codfw.wmnet
• 07:55 jmm@cumin2002: START - Cookbook sre.dns.netbox
• 07:51 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts acmechief1001.eqiad.wmnet
• 07:49 aqu@deploy1003: Finished deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow (duration: 00m 10s)
• 07:49 aqu@deploy1003: Started deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow
• 07:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 07:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 07:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 07:21 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2087.codfw.wmnet with reason: host reimage
• 07:18 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2087.codfw.wmnet with reason: host reimage
• 07:00 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2087.codfw.wmnet with OS bullseye
• 06:57 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2087.codfw.wmnet
• 06:57 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2087.codfw.wmnet with OS bullseye
• 06:57 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2087.codfw.wmnet with OS bullseye
• 06:57 jayme@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2087.codfw.wmnet
• 05:37 vgutierrez: repool cp2041
• 04:57 vgutierrez: repool cp2038
• 04:24 vgutierrez: restarting purged in cp2038 && cp2041 - T334078
• 04:16 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp(2038|2041).codfw.wmnet
• 04:15 vgutierrez: depooling cp2041 && cp2038 due to high purged lag
• 02:37 ejegg: restarted donations queue consumer
• 02:31 ejegg: fundraising civicrm upgraded from 67ee99ce to 5dd4edc1
• 02:29 ejegg: disabled donations queue consumer for civi deploy

2024-09-05

• 23:43 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1019.eqiad.wmnet
• 23:43 cmooney@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs1019.eqiad.wmnet
• 23:36 topranks: re-enable PyBal on lvs1019 after fixing faulty link with replacement optic T374155
• 22:54 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: Move traffic off lvs1019 to lvs1029 to troubleshooot faulty link
• 22:54 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs1019.eqiad.wmnet with reason: Move traffic off lvs1019 to lvs1029 to troubleshooot faulty link
• 22:53 topranks: disable PyBal on lvs1019 to swing traffic to lvs1020 and allow for intrusive work to correct link errors T374155
• 22:50 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on gerrit2002.wikimedia.org with reason: T373980
• 22:49 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 0:10:00 on gerrit2002.wikimedia.org with reason: T373980
• 22:49 mutante: gerrit-replica.wikimedia.org (gerrit2002) - rebooting T373980
• 22:03 dancy@deploy1003: Installation of scap version "4.101.3" completed for 211 hosts
• 21:59 dancy@deploy1003: Installing scap version "4.101.3" for 211 hosts
• 21:57 inflatador: bking@grafana1002 apply grizzly SLO dashboard updates slo-Search added slo-apigw updated P68729 T328330
• 21:57 inflatador: bking@grafana1002 apply grizzly SLO dashboard updates slo-Search added slo-apigw updated P68729
• 21:54 dancy@deploy1003: Installing scap version "4.101.3" for 211 hosts
• 21:54 dancy@deploy1003: install-world aborted: (duration: 02m 00s)
• 21:52 dancy@deploy1003: Installing scap version "4.101.3" for 211 hosts
• 21:33 mutante: gitlab1004 systemct list-units --state=failed listed wmf_auto_restart_ssh-gitlab.service but at the same time it's 'Service ssh-gitlab not present or not running'.(?). Did a systemctl reset-failed to clear monitoring and it doesn't seem to come back. T374106
• 21:32 dancy@deploy1003: Sync cancelled.
• 21:31 dancy@deploy1003: dancy: Backport for Revert "Set parser for image gallery in CampaignPageFormatter" synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 21:29 dancy@deploy1003: Started scap sync-world: Backport for Revert "Set parser for image gallery in CampaignPageFormatter"
• 21:15 topranks: add interface qos scheduler config to remaining switches T339850
• 21:11 dduvall@deploy1003: Finished deploy [releng/jenkins-deploy@b47c79e] (releasing): (no justification provided) (duration: 00m 39s)
• 21:10 dduvall@deploy1003: Started deploy [releng/jenkins-deploy@b47c79e] (releasing): (no justification provided)
• 21:09 dancy@deploy1003: Sync cancelled.
• 21:02 dancy@deploy1003: dancy: Backport for Set parser for image gallery in CampaignPageFormatter (T374146) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 21:00 dancy@deploy1003: Started scap sync-world: Backport for Set parser for image gallery in CampaignPageFormatter (T374146)
• 20:46 dancy@deploy1003: install-world aborted: (duration: 02m 28s)
• 20:44 dancy@deploy1003: Installing scap version "4.101.2" for 211 hosts
• 20:41 topranks: add interface qos scheduler config to eqiad switches T339850
• 20:40 dancy@deploy1003: Finished scap sync-world: Backport for Remove redundandant setting of $wgDefaultUserOptions['math'] (T373703) (duration: 10m 36s)
• 20:35 dancy@deploy1003: dancy, physikerwelt: Continuing with sync
• 20:31 dancy@deploy1003: dancy, physikerwelt: Backport for Remove redundandant setting of $wgDefaultUserOptions['math'] (T373703) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:29 dancy@deploy1003: Started scap sync-world: Backport for Remove redundandant setting of $wgDefaultUserOptions['math'] (T373703)
• 19:51 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm
• 19:36 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1009.eqiad.wmnet with reason: host reimage
• 19:33 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1009.eqiad.wmnet with reason: host reimage
• 19:21 topranks: add interface qos scheduler config to codfw switches T339850
• 19:20 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm
• 18:37 dancy@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.43.0-wmf.21 refs T373640
• 18:06 topranks: add interface qos scheduler config to remaining CRs T339850
• 17:59 ebernhardson@deploy1003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:59 ebernhardson@deploy1003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2169 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68725 and previous config saved to /var/cache/conftool/dbconfig/20240905-175316-arnaudb.json
• 17:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2191 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68724 and previous config saved to /var/cache/conftool/dbconfig/20240905-175316-arnaudb.json
• 17:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68723 and previous config saved to /var/cache/conftool/dbconfig/20240905-175315-arnaudb.json
• 17:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2169 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68722 and previous config saved to /var/cache/conftool/dbconfig/20240905-173811-arnaudb.json
• 17:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2191 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68721 and previous config saved to /var/cache/conftool/dbconfig/20240905-173810-arnaudb.json
• 17:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68720 and previous config saved to /var/cache/conftool/dbconfig/20240905-173810-arnaudb.json
• 17:33 ebernhardson@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:32 ebernhardson@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:28 ebernhardson@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 17:28 ebernhardson@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 17:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2169 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68719 and previous config saved to /var/cache/conftool/dbconfig/20240905-172306-arnaudb.json
• 17:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68718 and previous config saved to /var/cache/conftool/dbconfig/20240905-172305-arnaudb.json
• 17:10 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2019.codfw.wmnet
• 17:10 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2019.codfw.wmnet
• 17:10 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2419.codfw.wmnet
• 17:10 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2419.codfw.wmnet
• 17:10 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2418.codfw.wmnet
• 17:10 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2418.codfw.wmnet
• 17:09 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2417.codfw.wmnet
• 17:09 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2417.codfw.wmnet
• 17:09 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2416.codfw.wmnet
• 17:09 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2416.codfw.wmnet
• 17:09 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2415.codfw.wmnet
• 17:09 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2415.codfw.wmnet
• 17:09 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2414.codfw.wmnet
• 17:09 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2414.codfw.wmnet
• 17:09 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2413.codfw.wmnet
• 17:09 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2413.codfw.wmnet
• 17:09 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2412.codfw.wmnet
• 17:09 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2412.codfw.wmnet
• 17:09 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2338.codfw.wmnet
• 17:09 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2338.codfw.wmnet
• 17:09 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2337.codfw.wmnet
• 17:09 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2337.codfw.wmnet
• 17:08 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2336.codfw.wmnet
• 17:08 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2336.codfw.wmnet
• 17:08 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2335.codfw.wmnet
• 17:08 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2335.codfw.wmnet
• 17:08 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2039.codfw.wmnet
• 17:08 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2039.codfw.wmnet
• 17:08 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2038.codfw.wmnet
• 17:08 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2038.codfw.wmnet
• 17:08 claime: Repooling kubernetes nodes after T373096 - kubernetes2017 kubernetes2021 kubernetes2038 kubernetes2039 mw2335 mw2336 mw2337 mw2338 mw2412 mw2413 mw2414 mw2415 mw2416 mw2417 mw2418 mw2419 wikikube-worker2019
• 17:08 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2021.codfw.wmnet
• 17:08 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2021.codfw.wmnet
• 17:08 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2017.codfw.wmnet
• 17:08 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2017.codfw.wmnet
• 17:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2169 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68717 and previous config saved to /var/cache/conftool/dbconfig/20240905-170801-arnaudb.json
• 17:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2191 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68716 and previous config saved to /var/cache/conftool/dbconfig/20240905-170800-arnaudb.json
• 17:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68715 and previous config saved to /var/cache/conftool/dbconfig/20240905-170800-arnaudb.json
• 17:06 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp2036.ulsfo.wmnet
• 17:06 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp2035.ulsfo.wmnet
• 17:05 Emperor: pool moss-fe2001 ms-fe2011 T373096
• 17:04 Emperor: moss-be2003 exit maintenance mode T373096
• 17:03 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2036.codfw.wmnet
• 17:03 fabfur@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp2036.codfw.wmnet
• 17:03 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2035.codfw.wmnet
• 17:03 fabfur@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp2035.codfw.wmnet
• 16:59 topranks: move server uplinks codfw rack c3 from asw-c3-codfw to lsw1-c3-codfw T373096
• 16:58 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2091.codfw.wmnet
• 16:58 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2091.codfw.wmnet
• 16:58 cmooney@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 34 hosts with reason: Move server uplinks codfw racks C3
• 16:57 cmooney@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on 34 hosts with reason: Move server uplinks codfw racks C3
• 16:55 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2091.codfw.wmnet
• 16:55 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2091.codfw.wmnet
• 16:55 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2091.codfw.wmnet
• 16:51 cmooney@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:30:00 on 35 hosts with reason: Move server uplinks codfw racks C3
• 16:51 cmooney@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on 35 hosts with reason: Move server uplinks codfw racks C3
• 16:51 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2091.codfw.wmnet with OS bullseye
• 16:49 dancy@deploy1003: Installing scap version "4.101.1" for 211 hosts
• 16:49 kamila@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2092.codfw.wmnet
• 16:49 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2092.codfw.wmnet with OS bullseye
• 16:42 topranks: move server uplinks codfw rack c2 from asw-c2-codfw to lsw1-c2-codfw T373096
• 16:42 jhathaway: disabling puppet on cp nodes to test puppet change
• 16:39 cmooney@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 23 hosts with reason: Move server uplinks codfw racks C2
• 16:38 cmooney@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on 23 hosts with reason: Move server uplinks codfw racks C2
• 16:38 dancy@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.43.0-wmf.21 refs T373640
• 16:30 Emperor: depool moss-fe2001 ms-fe2011 T373096
• 16:30 Emperor: moss-be2003 to maintenance mode T373096
• 16:27 hnowlan@puppetmaster1001: conftool action : set/pooled=no; selector: name=maps2007.codfw.wmnet
• 16:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 6 hosts with reason: network maintenance T370852
• 16:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on 6 hosts with reason: network maintenance T370852
• 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'depool db2141 db2144 db2150 db2169 db2186 db2191 - T370852', diff saved to https://phabricator.wikimedia.org/P68714 and previous config saved to /var/cache/conftool/dbconfig/20240905-162057-arnaudb.json
• 16:10 arnaudb@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 100%: post clone repool', diff saved to https://phabricator.wikimedia.org/P68713 and previous config saved to /var/cache/conftool/dbconfig/20240905-161051-arnaudb.json
• 16:02 claime: homer cr*-codfw* commit 'T372878'
• 15:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 75%: post clone repool', diff saved to https://phabricator.wikimedia.org/P68712 and previous config saved to /var/cache/conftool/dbconfig/20240905-155545-arnaudb.json
• 15:54 ebernhardson@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
• 15:54 ebernhardson@deploy1003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
• 15:48 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on backup2006.codfw.wmnet with reason: Move backup2006 uplink to lsw1-c2-codfw
• 15:48 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on backup2006.codfw.wmnet with reason: Move backup2006 uplink to lsw1-c2-codfw
• 15:43 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2322.codfw.wmnet
• 15:42 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2322.codfw.wmnet
• 15:42 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2321.codfw.wmnet
• 15:42 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2321.codfw.wmnet
• 15:42 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2320.codfw.wmnet
• 15:41 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2092.codfw.wmnet with reason: host reimage
• 15:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 50%: post clone repool', diff saved to https://phabricator.wikimedia.org/P68711 and previous config saved to /var/cache/conftool/dbconfig/20240905-154040-arnaudb.json
• 15:39 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2092.codfw.wmnet with reason: host reimage
• 15:39 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2320.codfw.wmnet
• 15:38 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2031.codfw.wmnet
• 15:38 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2031.codfw.wmnet
• 15:35 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2091.codfw.wmnet with reason: host reimage
• 15:31 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2091.codfw.wmnet with reason: host reimage
• 15:30 topranks: prep lsw1-c3-codfw for server migration from asw-c3-codfw T373096
• 15:28 Emperor: restart swift-proxy on ms-fe1014 T360913
• 15:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 25%: post clone repool', diff saved to https://phabricator.wikimedia.org/P68710 and previous config saved to /var/cache/conftool/dbconfig/20240905-152534-arnaudb.json
• 15:22 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2092
• 15:22 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2092
• 15:21 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2092
• 15:21 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2092.codfw.wmnet 77.0.192.10.in-addr.arpa 7.7.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 15:21 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2092.codfw.wmnet 77.0.192.10.in-addr.arpa 7.7.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 15:21 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:21 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2092 - kamila@cumin1002"
• 15:21 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2092 - kamila@cumin1002"
• 15:21 topranks: prep lsw1-c2-codfw for server migration from asw-c2-codfw T373096
• 15:18 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 15:18 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2125.codfw.wmnet onto db2225.codfw.wmnet
• 15:17 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2092
• 15:17 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2092.codfw.wmnet with OS bullseye
• 15:17 kamila@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2092.codfw.wmnet
• 15:13 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2091
• 15:12 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2091
• 15:09 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp2036.ulsfo.wmnet
• 15:08 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp2035.ulsfo.wmnet
• 15:08 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on cp2036.codfw.wmnet with reason: T373096
• 15:08 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2091
• 15:08 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2091.codfw.wmnet 9.0.192.10.in-addr.arpa 9.0.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 15:08 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2091.codfw.wmnet 9.0.192.10.in-addr.arpa 9.0.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 15:08 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:08 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2091 - kamila@cumin1002"
• 15:08 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 3:00:00 on cp2036.codfw.wmnet with reason: T373096
• 15:07 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2091 - kamila@cumin1002"
• 15:07 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on cp2035.codfw.wmnet with reason: T373096
• 15:07 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 3:00:00 on cp2035.codfw.wmnet with reason: T373096
• 15:07 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2019.codfw.wmnet
• 15:06 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2019.codfw.wmnet
• 15:06 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2419.codfw.wmnet
• 15:05 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2419.codfw.wmnet
• 15:05 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2418.codfw.wmnet
• 15:05 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2418.codfw.wmnet
• 15:05 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2417.codfw.wmnet
• 15:04 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2417.codfw.wmnet
• 15:04 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2416.codfw.wmnet
• 15:04 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 15:04 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2091
• 15:03 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2416.codfw.wmnet
• 15:03 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2091.codfw.wmnet with OS bullseye
• 15:03 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2415.codfw.wmnet
• 15:03 kamila@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2091.codfw.wmnet
• 15:03 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2415.codfw.wmnet
• 15:03 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2414.codfw.wmnet
• 15:02 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2414.codfw.wmnet
• 15:02 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2413.codfw.wmnet
• 15:01 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2413.codfw.wmnet
• 15:01 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2412.codfw.wmnet
• 15:01 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2421 to wikikube-worker2092
• 15:01 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2412.codfw.wmnet
• 15:01 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2338.codfw.wmnet
• 15:00 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2092
• 15:00 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2092
• 15:00 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:00 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2421 to wikikube-worker2092 - kamila@cumin1002"
• 15:00 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2338.codfw.wmnet
• 15:00 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2421 to wikikube-worker2092 - kamila@cumin1002"
• 15:00 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2337.codfw.wmnet
• 15:00 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2420 to wikikube-worker2091
• 15:00 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2337.codfw.wmnet
• 14:59 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2336.codfw.wmnet
• 14:59 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2091
• 14:59 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2336.codfw.wmnet
• 14:59 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2335.codfw.wmnet
• 14:58 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2091
• 14:58 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:58 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2420 to wikikube-worker2091 - kamila@cumin1002"
• 14:58 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2335.codfw.wmnet
• 14:58 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2039.codfw.wmnet
• 14:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db2124 (re)pooling @ 100%: post db2224 clone repool', diff saved to https://phabricator.wikimedia.org/P68709 and previous config saved to /var/cache/conftool/dbconfig/20240905-145755-arnaudb.json
• 14:57 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2039.codfw.wmnet
• 14:57 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2038.codfw.wmnet
• 14:57 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2038.codfw.wmnet
• 14:57 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2021.codfw.wmnet
• 14:56 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2021.codfw.wmnet
• 14:56 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2017.codfw.wmnet
• 14:56 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 14:55 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2017.codfw.wmnet
• 14:55 claime: depooling kubernetes nodes for T373096 - kubernetes2017 kubernetes2021 kubernetes2038 kubernetes2039 mw2335 mw2336 mw2337 mw2338 mw2412 mw2413 mw2414 mw2415 mw2416 mw2417 mw2418 mw2419 wikikube-worker2019
• 14:55 hashar: UTC afternoon backport window completed
• 14:54 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2420 to wikikube-worker2091 - kamila@cumin1002"
• 14:52 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw2421 to wikikube-worker2092
• 14:51 volans: deploying python3-wmflib 1.2.6 fleet-wide
• 14:51 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 14:50 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw2420 to wikikube-worker2091
• 14:50 hashar@deploy1003: Finished scap sync-world: Backport for Revert "Fix missing wikibase link in Minerva sidebar" (T66315) (duration: 09m 27s)
• 14:46 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2421.codfw.wmnet
• 14:46 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2420.codfw.wmnet
• 14:46 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2421.codfw.wmnet
• 14:46 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2420.codfw.wmnet
• 14:45 hashar@deploy1003: hashar: Continuing with sync
• 14:43 hashar@deploy1003: hashar: Backport for Revert "Fix missing wikibase link in Minerva sidebar" (T66315) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:42 arnaudb@cumin1002: dbctl commit (dc=all): 'db2124 (re)pooling @ 75%: post db2224 clone repool', diff saved to https://phabricator.wikimedia.org/P68707 and previous config saved to /var/cache/conftool/dbconfig/20240905-144249-arnaudb.json
• 14:41 hashar@deploy1003: Started scap sync-world: Backport for Revert "Fix missing wikibase link in Minerva sidebar" (T66315)
• 14:32 hashar: UTC afternoon backport window not completed! https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/1070953 pending
• 14:30 hashar: UTC afternoon backport window completed
• 14:27 arnaudb@cumin1002: dbctl commit (dc=all): 'db2124 (re)pooling @ 50%: post db2224 clone repool', diff saved to https://phabricator.wikimedia.org/P68706 and previous config saved to /var/cache/conftool/dbconfig/20240905-142743-arnaudb.json
• 14:14 hashar@deploy1003: Finished scap sync-world: Backport for Replace confusing uses of $wgDebugLogFile with $wmgExtraLogFile (duration: 10m 37s)
• 14:14 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:13 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:12 arnaudb@cumin1002: dbctl commit (dc=all): 'db2124 (re)pooling @ 25%: post db2224 clone repool', diff saved to https://phabricator.wikimedia.org/P68705 and previous config saved to /var/cache/conftool/dbconfig/20240905-141238-arnaudb.json
• 14:11 topranks: add interface qos schedulers on cr1-codfw T339850
• 14:10 hashar@deploy1003: matmarex, hashar: Continuing with sync
• 14:06 hashar@deploy1003: matmarex, hashar: Backport for Replace confusing uses of $wgDebugLogFile with $wmgExtraLogFile synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:06 volans: uploaded python3-wmflib_1.2.6 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia,bookworm-wikimedia
• 14:04 hashar@deploy1003: Started scap sync-world: Backport for Replace confusing uses of $wgDebugLogFile with $wmgExtraLogFile
• 13:59 hashar@deploy1003: Finished scap sync-world: Backport for Fix missing wikibase link in Minerva sidebar (T66315) (duration: 19m 21s)
• 13:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2124 (re)pooling @ 15%: post db2224 clone repool', diff saved to https://phabricator.wikimedia.org/P68704 and previous config saved to /var/cache/conftool/dbconfig/20240905-135731-arnaudb.json
• 13:55 filippo@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply
• 13:54 filippo@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply
• 13:54 hashar@deploy1003: hashar, joelyrookewmde: Continuing with sync
• 13:52 arnaudb@cumin1002: START - Cookbook sre.mysql.clone of db2125.codfw.wmnet onto db2225.codfw.wmnet
• 13:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2125 in db2225 for T373579', diff saved to https://phabricator.wikimedia.org/P68703 and previous config saved to /var/cache/conftool/dbconfig/20240905-134929-arnaudb.json
• 13:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: provisionning db2225.codfw.wmnet - T373579
• 13:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: provisionning db2225.codfw.wmnet - T373579
• 13:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2125.codfw.wmnet with reason: provisionning db2225.codfw.wmnet - T373579
• 13:47 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2125.codfw.wmnet with reason: provisionning db2225.codfw.wmnet - T373579
• 13:42 arnaudb@cumin1002: dbctl commit (dc=all): 'db2124 (re)pooling @ 5%: post db2224 clone repool', diff saved to https://phabricator.wikimedia.org/P68702 and previous config saved to /var/cache/conftool/dbconfig/20240905-134225-arnaudb.json
• 13:41 hashar@deploy1003: hashar, joelyrookewmde: Backport for Fix missing wikibase link in Minerva sidebar (T66315) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:39 hashar@deploy1003: Started scap sync-world: Backport for Fix missing wikibase link in Minerva sidebar (T66315)
• 13:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2168 (re)pooling @ 100%: post clone repool', diff saved to https://phabricator.wikimedia.org/P68701 and previous config saved to /var/cache/conftool/dbconfig/20240905-133649-arnaudb.json
• 13:34 jmm@cumin2002: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling restart_daemons on A:ncredir
• 13:31 hashar@deploy1003: Finished scap sync-world: Backport for search: use mul fallback for fine-tuned search profiles (T371401) (duration: 06m 28s)
• 13:29 aqu@deploy1003: Finished deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow (duration: 00m 14s)
• 13:29 aqu@deploy1003: Started deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow
• 13:26 hashar@deploy1003: hashar, dcausse: Continuing with sync
• 13:26 hashar@deploy1003: hashar, dcausse: Backport for search: use mul fallback for fine-tuned search profiles (T371401) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:25 jmm@cumin2002: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling restart_daemons on A:ncredir
• 13:24 hashar@deploy1003: Started scap sync-world: Backport for search: use mul fallback for fine-tuned search profiles (T371401)
• 13:22 hashar@deploy1003: Finished scap sync-world: Backport for Allow copyuploads on test2wiki (T356241) (duration: 11m 45s)
• 13:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2168 (re)pooling @ 75%: post clone repool', diff saved to https://phabricator.wikimedia.org/P68700 and previous config saved to /var/cache/conftool/dbconfig/20240905-132144-arnaudb.json
• 13:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2124.codfw.wmnet onto db2224.codfw.wmnet
• 13:18 hashar@deploy1003: hnowlan, hashar: Continuing with sync
• 13:16 jmm@cumin2002: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling restart_daemons on A:ncredir-eqsin
• 13:15 jmm@cumin2002: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling restart_daemons on A:ncredir-eqsin
• 13:14 hashar@deploy1003: hnowlan, hashar: Backport for Allow copyuploads on test2wiki (T356241) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:11 hashar@deploy1003: Started scap sync-world: Backport for Allow copyuploads on test2wiki (T356241)
• 13:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2168 (re)pooling @ 50%: post clone repool', diff saved to https://phabricator.wikimedia.org/P68699 and previous config saved to /var/cache/conftool/dbconfig/20240905-130638-arnaudb.json
• 12:58 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2029.codfw.wmnet
• 12:58 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2029.codfw.wmnet
• 12:58 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2029.codfw.wmnet
• 12:57 cmooney@cumin1002: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
• 12:56 cmooney@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
• 12:56 cmooney@cumin1002: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
• 12:56 cmooney@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
• 12:54 claime: homer lsw1-b6-codfw* commit 'T372878'
• 12:54 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2029.codfw.wmnet with OS bullseye
• 12:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2168 (re)pooling @ 25%: post clone repool', diff saved to https://phabricator.wikimedia.org/P68698 and previous config saved to /var/cache/conftool/dbconfig/20240905-125133-arnaudb.json
• 12:38 arnaudb@cumin1002: START - Cookbook sre.mysql.clone of db2124.codfw.wmnet onto db2224.codfw.wmnet
• 12:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2168 (re)pooling @ 15%: post clone repool', diff saved to https://phabricator.wikimedia.org/P68697 and previous config saved to /var/cache/conftool/dbconfig/20240905-123627-arnaudb.json
• 12:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2124 in db2224 for T373579', diff saved to https://phabricator.wikimedia.org/P68696 and previous config saved to /var/cache/conftool/dbconfig/20240905-123619-arnaudb.json
• 12:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2224.codfw.wmnet with reason: provisionning db2224.codfw.wmnet - T373579
• 12:35 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2224.codfw.wmnet with reason: provisionning db2224.codfw.wmnet - T373579
• 12:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2124.codfw.wmnet with reason: provisionning db2224.codfw.wmnet - T373579
• 12:34 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2124.codfw.wmnet with reason: provisionning db2224.codfw.wmnet - T373579
• 12:31 eoghan@cumin1002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host lists1004.wikimedia.org
• 12:26 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2029.codfw.wmnet with reason: host reimage
• 12:24 ladsgroup@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 100%: Maint over', diff saved to https://phabricator.wikimedia.org/P68695 and previous config saved to /var/cache/conftool/dbconfig/20240905-122408-ladsgroup.json
• 12:23 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2029.codfw.wmnet with reason: host reimage
• 12:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2168 (re)pooling @ 5%: post clone repool', diff saved to https://phabricator.wikimedia.org/P68694 and previous config saved to /var/cache/conftool/dbconfig/20240905-122108-arnaudb.json
• 12:20 eoghan@cumin1002: START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org
• 12:12 claime: homer cr*codfw* commit 'T372878'
• 12:09 ladsgroup@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 75%: Maint over', diff saved to https://phabricator.wikimedia.org/P68693 and previous config saved to /var/cache/conftool/dbconfig/20240905-120903-ladsgroup.json
• 12:06 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2029
• 12:06 cgoubert@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2029
• 12:06 cgoubert@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2029
• 12:06 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2029.codfw.wmnet 199.16.192.10.in-addr.arpa 9.9.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:06 cgoubert@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2029.codfw.wmnet 199.16.192.10.in-addr.arpa 9.9.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:06 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:06 cgoubert@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2029 - cgoubert@cumin1002"
• 12:05 cgoubert@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2029 - cgoubert@cumin1002"
• 12:04 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2029.codfw.wmnet on all recursors
• 12:04 cmooney@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2029.codfw.wmnet on all recursors
• 11:58 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:55 cmooney@cumin1002: START - Cookbook sre.dns.netbox
• 11:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 25%: Maint over', diff saved to https://phabricator.wikimedia.org/P68692 and previous config saved to /var/cache/conftool/dbconfig/20240905-115357-ladsgroup.json
• 11:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2003.wikimedia.org
• 11:50 samtar@deploy1003: Finished scap sync-world: Backport for IS: Enable CommunityRequests on Meta (T372527) (duration: 07m 09s)
• 11:46 jelto@cumin1002: START - Cookbook sre.hosts.reboot-single for host gerrit2003.wikimedia.org
• 11:45 samtar@deploy1003: samtar: Continuing with sync
• 11:45 samtar@deploy1003: samtar: Backport for IS: Enable CommunityRequests on Meta (T372527) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 11:43 samtar@deploy1003: Started scap sync-world: Backport for IS: Enable CommunityRequests on Meta (T372527)
• 11:42 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: apply
• 11:41 jayme@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: apply
• 11:41 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-logging-external: apply
• 11:41 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 11:41 samtar@deploy1003: Finished scap sync-world: Backport for CS: Load CommunityRequests (T372527) (duration: 08m 32s)
• 11:41 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 11:40 jayme@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-logging-external: apply
• 11:40 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply
• 11:39 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 11:39 jayme@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply
• 11:39 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
• 11:38 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
• 11:38 ladsgroup@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 10%: Maint over', diff saved to https://phabricator.wikimedia.org/P68691 and previous config saved to /var/cache/conftool/dbconfig/20240905-113852-ladsgroup.json
• 11:38 jiji@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply
• 11:38 jayme@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
• 11:38 jiji@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply
• 11:37 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 11:36 samtar@deploy1003: samtar: Continuing with sync
• 11:36 samtar@deploy1003: samtar: Backport for CS: Load CommunityRequests (T372527) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 11:36 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 11:36 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 11:35 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance
• 11:35 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance
• 11:34 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2129.codfw.wmnet with reason: Maintenance
• 11:34 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db2129.codfw.wmnet with reason: Maintenance
• 11:33 samtar@deploy1003: Started scap sync-world: Backport for CS: Load CommunityRequests (T372527)
• 11:32 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 11:32 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host etherpad2002.codfw.wmnet
• 11:31 ladsgroup@cumin2002: dbctl commit (dc=all): 'Depool db2129 T374087', diff saved to https://phabricator.wikimedia.org/P68690 and previous config saved to /var/cache/conftool/dbconfig/20240905-113153-ladsgroup.json
• 11:31 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 11:30 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply
• 11:29 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply
• 11:29 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply
• 11:29 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply
• 11:28 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
• 11:28 ladsgroup@cumin2002: dbctl commit (dc=all): 'Promote db2214 to s6 primary T374087', diff saved to https://phabricator.wikimedia.org/P68689 and previous config saved to /var/cache/conftool/dbconfig/20240905-112846-ladsgroup.json
• 11:28 jelto@cumin1002: START - Cookbook sre.hosts.reboot-single for host etherpad2002.codfw.wmnet
• 11:28 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
• 11:27 Amir1: Starting s6 codfw failover from db2129 to db2214 - T374087
• 11:27 cgoubert@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2029
• 11:26 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2029.codfw.wmnet with OS bullseye
• 11:26 jayme@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-logging-external: apply
• 11:25 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2029.codfw.wmnet
• 11:25 jayme@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-logging-external: apply
• 11:25 jayme@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply
• 11:25 jayme@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply
• 11:25 jayme@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
• 11:25 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2029.codfw.wmnet
• 11:25 cgoubert@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2029.codfw.wmnet
• 11:24 jayme@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
• 11:23 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: apply
• 11:23 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: apply
• 11:22 jayme@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-main: apply
• 11:22 jayme@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-main: apply
• 11:22 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2090.codfw.wmnet
• 11:22 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2090.codfw.wmnet
• 11:22 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2090.codfw.wmnet
• 11:21 jayme: homer cr*codfw* commit 'T372878'
• 11:21 ladsgroup@cumin2002: dbctl commit (dc=all): 'Set db2214 with weight 0 T374087', diff saved to https://phabricator.wikimedia.org/P68688 and previous config saved to /var/cache/conftool/dbconfig/20240905-112121-ladsgroup.json
• 11:20 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host etherpad2002.codfw.wmnet
• 11:20 ladsgroup@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s6 T374087
• 11:20 ladsgroup@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Primary switchover s6 T374087
• 11:19 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2090.codfw.wmnet with OS bullseye
• 11:19 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
• 11:17 jelto@cumin1002: START - Cookbook sre.hosts.reboot-single for host etherpad2002.codfw.wmnet
• 11:09 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling restart_daemons on A:schema-eqiad
• 11:08 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling restart_daemons on A:schema-eqiad
• 11:05 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
• 11:00 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2084.codfw.wmnet
• 11:00 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2084.codfw.wmnet
• 11:00 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2084.codfw.wmnet
• 10:58 hnowlan: homer lsw1-b3-codfw* commit
• 10:57 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2084.codfw.wmnet with OS bullseye
• 10:51 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
• 10:51 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
• 10:42 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade/restart of Apache Traffic Server on A:cp-text_esams for 9.2.5-1wm2
• 10:38 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2084.codfw.wmnet with reason: host reimage
• 10:36 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2084.codfw.wmnet with reason: host reimage
• 10:30 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2168.codfw.wmnet onto db2222.codfw.wmnet
• 10:29 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:19 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2084
• 10:19 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2084
• 10:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:14 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2084
• 10:14 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2084.codfw.wmnet 170.16.192.10.in-addr.arpa 0.7.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 10:14 hnowlan@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2084.codfw.wmnet 170.16.192.10.in-addr.arpa 0.7.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 10:14 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:14 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2084 - hnowlan@cumin1002"
• 10:14 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2084 - hnowlan@cumin1002"
• 10:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on puppetmaster1003.eqiad.wmnet with reason: hardware fix
• 10:11 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on puppetmaster1003.eqiad.wmnet with reason: hardware fix
• 10:04 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 10:03 hnowlan@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2084
• 10:03 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2084.codfw.wmnet with OS bullseye
• 10:03 hnowlan@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2084.codfw.wmnet
• 09:59 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:59 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2089.codfw.wmnet
• 09:59 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2089.codfw.wmnet
• 09:59 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2089.codfw.wmnet
• 09:54 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2089.codfw.wmnet with OS bookworm
• 09:53 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:53 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:43 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org
• 09:42 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade/restart of Apache Traffic Server on A:cp-text_esams for 9.2.5-1wm2
• 09:38 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2090.codfw.wmnet with reason: host reimage
• 09:37 jelto@cumin1002: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org
• 09:37 oblivian@deploy1003: Finished scap sync-world: Backport for BounceHandler: add IPs for the new mx servers (T338761) (duration: 11m 16s)
• 09:36 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1003.wikimedia.org
• 09:36 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2090.codfw.wmnet with reason: host reimage
• 09:35 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2089.codfw.wmnet with reason: host reimage
• 09:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aphlict2001.codfw.wmnet
• 09:32 oblivian@deploy1003: oblivian: Continuing with sync
• 09:32 oblivian@deploy1003: oblivian: Backport for BounceHandler: add IPs for the new mx servers (T338761) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 09:32 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2089.codfw.wmnet with reason: host reimage
• 09:30 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aphlict2001.codfw.wmnet
• 09:30 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host etherpad1004.eqiad.wmnet
• 09:30 jelto@cumin1002: START - Cookbook sre.hosts.reboot-single for host gitlab1003.wikimedia.org
• 09:28 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aphlict1002.eqiad.wmnet
• 09:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host etherpad1004.eqiad.wmnet
• 09:26 oblivian@deploy1003: Started scap sync-world: Backport for BounceHandler: add IPs for the new mx servers (T338761)
• 09:25 arnaudb@cumin1002: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2222.codfw.wmnet
• 09:24 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host aphlict1002.eqiad.wmnet
• 09:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2168 in db2222 for T373579', diff saved to https://phabricator.wikimedia.org/P68686 and previous config saved to /var/cache/conftool/dbconfig/20240905-092339-arnaudb.json
• 09:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: provisionning db2222.codfw.wmnet - T373579
• 09:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: provisionning db2222.codfw.wmnet - T373579
• 09:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: provisionning db2222.codfw.wmnet - T373579
• 09:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: provisionning db2222.codfw.wmnet - T373579
• 09:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doc2002.codfw.wmnet
• 09:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host miscweb1003.eqiad.wmnet
• 09:19 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2088.codfw.wmnet
• 09:19 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2088.codfw.wmnet
• 09:19 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2088.codfw.wmnet
• 09:18 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade/restart of Apache Traffic Server on A:cp-upload_esams for 9.2.5-1wm2
• 09:18 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2090
• 09:18 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2090
• 09:18 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2090
• 09:18 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2090
• 09:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host doc2002.codfw.wmnet
• 09:16 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host miscweb1003.eqiad.wmnet
• 09:16 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2090
• 09:16 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2090.codfw.wmnet 123.16.192.10.in-addr.arpa 3.2.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 09:16 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:16 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2090.codfw.wmnet 123.16.192.10.in-addr.arpa 3.2.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 09:16 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:16 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2090 - jayme@cumin1002"
• 09:16 jayme: homer lsw1-b8-codfw* commit 'T372878'
• 09:16 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2090 - jayme@cumin1002"
• 09:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2168.codfw.wmnet onto db2221.codfw.wmnet
• 09:12 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2088.codfw.wmnet with OS bookworm
• 09:12 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 09:11 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2090
• 09:10 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2089
• 09:10 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2089
• 09:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host planet1003.eqiad.wmnet
• 09:07 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2089
• 09:07 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2089.codfw.wmnet 122.16.192.10.in-addr.arpa 2.2.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 09:07 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2089.codfw.wmnet 122.16.192.10.in-addr.arpa 2.2.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 09:07 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:07 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2089 - jayme@cumin1002"
• 09:07 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2089 - jayme@cumin1002"
• 09:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doc1003.eqiad.wmnet
• 09:05 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2090.codfw.wmnet with OS bullseye
• 09:04 jayme@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2090.codfw.wmnet
• 09:04 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 09:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host planet1003.eqiad.wmnet
• 09:04 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2089
• 09:04 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2089.codfw.wmnet with OS bookworm
• 09:03 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host doc1003.eqiad.wmnet
• 09:03 jayme@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2089.codfw.wmnet
• 09:03 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2089.codfw.wmnet
• 09:02 jayme@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2089.codfw.wmnet
• 09:02 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2089.codfw.wmnet wikikube-worker2090.codfw.wmnet on all recursors
• 09:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host miscweb2003.codfw.wmnet
• 09:02 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2089.codfw.wmnet wikikube-worker2090.codfw.wmnet on all recursors
• 09:02 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2089.codfw.wmnet
• 09:02 jayme@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2089.codfw.wmnet
• 09:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host planet2003.codfw.wmnet
• 09:01 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2435 to wikikube-worker2090
• 09:00 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2090
• 09:00 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2090
• 09:00 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:00 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2435 to wikikube-worker2090 - jayme@cumin1002"
• 09:00 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2435 to wikikube-worker2090 - jayme@cumin1002"
• 09:00 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2434 to wikikube-worker2089
• 08:59 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2089
• 08:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host miscweb2003.codfw.wmnet
• 08:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host planet2003.codfw.wmnet
• 08:57 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2089
• 08:57 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:57 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2434 to wikikube-worker2089 - jayme@cumin1002"
• 08:56 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 08:56 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2434 to wikikube-worker2089 - jayme@cumin1002"
• 08:55 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2088.codfw.wmnet with reason: host reimage
• 08:54 vgutierrez: acmechief1002 is now the acme-chief active host
• 08:52 jayme@cumin1002: START - Cookbook sre.hosts.rename from mw2435 to wikikube-worker2090
• 08:51 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 08:51 jayme@cumin1002: START - Cookbook sre.hosts.rename from mw2434 to wikikube-worker2089
• 08:51 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2088.codfw.wmnet with reason: host reimage
• 08:50 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host mw2435.codfw.wmnet
• 08:49 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2435.codfw.wmnet
• 08:49 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2434.codfw.wmnet
• 08:49 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2434.codfw.wmnet
• 08:34 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2088
• 08:34 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2088
• 08:34 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2088
• 08:34 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2088.codfw.wmnet 231.16.192.10.in-addr.arpa 1.3.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 08:34 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2088.codfw.wmnet 231.16.192.10.in-addr.arpa 1.3.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 08:34 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:34 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2088 - jayme@cumin1002"
• 08:34 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2088 - jayme@cumin1002"
• 08:30 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 08:28 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2088
• 08:28 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2088.codfw.wmnet with OS bookworm
• 08:28 jayme@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2088.codfw.wmnet
• 08:21 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 08:20 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade/restart of Apache Traffic Server on A:cp-upload_esams for 9.2.5-1wm2
• 08:18 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 08:09 arnaudb@cumin1002: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2221.codfw.wmnet
• 08:08 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 08:08 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 08:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host etherpad2002.codfw.wmnet
• 08:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2168 in db2221 for T373579', diff saved to https://phabricator.wikimedia.org/P68685 and previous config saved to /var/cache/conftool/dbconfig/20240905-080540-arnaudb.json
• 08:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: provisionning db2221.codfw.wmnet - T373579
• 08:03 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: provisionning db2221.codfw.wmnet - T373579
• 08:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: provisionning db2221.codfw.wmnet - T373579
• 08:03 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: provisionning db2221.codfw.wmnet - T373579
• 08:02 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host etherpad2002.codfw.wmnet
• 07:55 elukey: re-enabling puppet on all nodes after puppetdb restarts
• 07:39 elukey: restart puppetdb on puppetdb[12]003 to pick up the new jvm
• 07:16 kartik@deploy1003: Finished scap sync-world: Backport for aswiki: Set MT threshold for CX to 80% (T369417) (duration: 09m 37s)
• 07:15 elukey: disable puppet fleetwide to restart puppetdb jvms (without impacting ongoing puppet runs, getting noise etc..)
• 07:11 kartik@deploy1003: kartik: Continuing with sync
• 07:09 kartik@deploy1003: kartik: Backport for aswiki: Set MT threshold for CX to 80% (T369417) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:06 kartik@deploy1003: Started scap sync-world: Backport for aswiki: Set MT threshold for CX to 80% (T369417)
• 07:02 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling restart_daemons on A:schema-eqiad
• 07:02 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling restart_daemons on A:schema-eqiad
• 06:58 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling restart_daemons on A:schema-codfw
• 06:57 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling restart_daemons on A:schema-codfw

2024-09-04

• 21:03 dancy@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.21 refs T373640
• 20:54 mutante: gerrit servers: upgraded git package version
• 20:52 dancy@deploy1003: Finished scap sync-world: Backport for NetworkSession: Only enable for private wikis (T373826) (duration: 06m 34s)
• 20:48 dancy@deploy1003: ebernhardson, dancy: Continuing with sync
• 20:48 dancy@deploy1003: ebernhardson, dancy: Backport for NetworkSession: Only enable for private wikis (T373826) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:46 dancy@deploy1003: Started scap sync-world: Backport for NetworkSession: Only enable for private wikis (T373826)
• 20:44 dancy@deploy1003: Finished scap sync-world: Backport for Turn on Parsoid Read Views for eswikivoyage (T374029) (duration: 09m 46s)
• 20:40 dancy@deploy1003: cscott, dancy: Continuing with sync
• 20:36 dancy@deploy1003: cscott, dancy: Backport for Turn on Parsoid Read Views for eswikivoyage (T374029) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:34 dancy@deploy1003: Started scap sync-world: Backport for Turn on Parsoid Read Views for eswikivoyage (T374029)
• 20:30 dancy@deploy1003: Finished scap sync-world: Backport for ParserOutput: Turn off noisy log - we have the info we need for now (T374046) (duration: 13m 33s)
• 20:26 dancy@deploy1003: dancy, cscott: Continuing with sync
• 20:21 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
• 20:19 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
• 20:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
• 20:19 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
• 20:19 dancy@deploy1003: dancy, cscott: Backport for ParserOutput: Turn off noisy log - we have the info we need for now (T374046) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:17 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
• 20:17 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
• 20:17 dancy@deploy1003: Started scap sync-world: Backport for ParserOutput: Turn off noisy log - we have the info we need for now (T374046)
• 20:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
• 20:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
• 19:56 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host phab1005.eqiad.wmnet with OS bookworm
• 19:56 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 19:54 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 19:49 dancy@deploy1003: Sync cancelled.
• 19:38 dancy@deploy1003: jdlrobson, dancy: Backport for Fixes: Echo icon not visible after click (T373936) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 19:37 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab1005.eqiad.wmnet with reason: host reimage
• 19:36 dancy@deploy1003: Started scap sync-world: Backport for Fixes: Echo icon not visible after click (T373936)
• 19:34 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on phab1005.eqiad.wmnet with reason: host reimage
• 19:31 eileen: civicrm upgraded from 27b1f673 to 67ee99ce
• 19:15 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host phab1005.eqiad.wmnet with OS bookworm
• 19:07 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host phab1005.eqiad.wmnet with OS bookworm
• 19:07 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-lab1001.eqiad.wmnet with OS bookworm
• 17:59 dancy@deploy1003: Finished scap sync-world: testwikis to 1.43.0-wmf.21 refs T373640 (duration: 06m 48s)
• 17:52 dancy@deploy1003: Started scap sync-world: testwikis to 1.43.0-wmf.21 refs T373640
• 17:16 dancy@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.21 refs T373640
• 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'es2036 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68684 and previous config saved to /var/cache/conftool/dbconfig/20240904-171447-arnaudb.json
• 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'es2032 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68683 and previous config saved to /var/cache/conftool/dbconfig/20240904-171431-arnaudb.json
• 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'es2031 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68682 and previous config saved to /var/cache/conftool/dbconfig/20240904-171415-arnaudb.json
• 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68681 and previous config saved to /var/cache/conftool/dbconfig/20240904-171402-arnaudb.json
• 17:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db2206 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68680 and previous config saved to /var/cache/conftool/dbconfig/20240904-171351-arnaudb.json
• 17:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db2190 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68679 and previous config saved to /var/cache/conftool/dbconfig/20240904-171332-arnaudb.json
• 17:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68678 and previous config saved to /var/cache/conftool/dbconfig/20240904-171317-arnaudb.json
• 17:12 arnaudb@cumin1002: dbctl commit (dc=all): 'db2138 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68677 and previous config saved to /var/cache/conftool/dbconfig/20240904-171254-arnaudb.json
• 17:12 arnaudb@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 100%: T370852', diff saved to https://phabricator.wikimedia.org/P68676 and previous config saved to /var/cache/conftool/dbconfig/20240904-171232-arnaudb.json
• 17:09 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2083.codfw.wmnet
• 17:09 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2083.codfw.wmnet
• 17:09 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2083.codfw.wmnet
• 17:00 hnowlan: homer lsw1-b3-codfw* commit 'T372878'
• 16:59 arnaudb@cumin1002: dbctl commit (dc=all): 'es2036 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68675 and previous config saved to /var/cache/conftool/dbconfig/20240904-165941-arnaudb.json
• 16:59 arnaudb@cumin1002: dbctl commit (dc=all): 'es2032 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68674 and previous config saved to /var/cache/conftool/dbconfig/20240904-165926-arnaudb.json
• 16:59 arnaudb@cumin1002: dbctl commit (dc=all): 'es2031 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68673 and previous config saved to /var/cache/conftool/dbconfig/20240904-165909-arnaudb.json
• 16:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68672 and previous config saved to /var/cache/conftool/dbconfig/20240904-165857-arnaudb.json
• 16:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db2206 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68671 and previous config saved to /var/cache/conftool/dbconfig/20240904-165846-arnaudb.json
• 16:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db2190 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68670 and previous config saved to /var/cache/conftool/dbconfig/20240904-165827-arnaudb.json
• 16:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68669 and previous config saved to /var/cache/conftool/dbconfig/20240904-165811-arnaudb.json
• 16:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2138 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68668 and previous config saved to /var/cache/conftool/dbconfig/20240904-165749-arnaudb.json
• 16:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 75%: T370852', diff saved to https://phabricator.wikimedia.org/P68667 and previous config saved to /var/cache/conftool/dbconfig/20240904-165727-arnaudb.json
• 16:49 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2083.codfw.wmnet with OS bullseye
• 16:48 dancy@deploy1003: Finished scap sync-world: Backport for Do not consume 'centralauthtoken' on api.php OPTIONS requests (T373925) (duration: 10m 16s)
• 16:47 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
• 16:44 arnaudb@cumin1002: dbctl commit (dc=all): 'es2036 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68666 and previous config saved to /var/cache/conftool/dbconfig/20240904-164435-arnaudb.json
• 16:44 arnaudb@cumin1002: dbctl commit (dc=all): 'es2032 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68665 and previous config saved to /var/cache/conftool/dbconfig/20240904-164421-arnaudb.json
• 16:44 arnaudb@cumin1002: dbctl commit (dc=all): 'es2031 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68664 and previous config saved to /var/cache/conftool/dbconfig/20240904-164404-arnaudb.json
• 16:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68663 and previous config saved to /var/cache/conftool/dbconfig/20240904-164351-arnaudb.json
• 16:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2206 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68662 and previous config saved to /var/cache/conftool/dbconfig/20240904-164340-arnaudb.json
• 16:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2190 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68661 and previous config saved to /var/cache/conftool/dbconfig/20240904-164321-arnaudb.json
• 16:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68660 and previous config saved to /var/cache/conftool/dbconfig/20240904-164305-arnaudb.json
• 16:42 arnaudb@cumin1002: dbctl commit (dc=all): 'db2138 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68659 and previous config saved to /var/cache/conftool/dbconfig/20240904-164243-arnaudb.json
• 16:42 arnaudb@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 50%: T370852', diff saved to https://phabricator.wikimedia.org/P68658 and previous config saved to /var/cache/conftool/dbconfig/20240904-164221-arnaudb.json
• 16:41 dancy@deploy1003: matmarex, dancy: Continuing with sync
• 16:40 dancy@deploy1003: matmarex, dancy: Backport for Do not consume 'centralauthtoken' on api.php OPTIONS requests (T373925) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 16:37 dancy@deploy1003: Started scap sync-world: Backport for Do not consume 'centralauthtoken' on api.php OPTIONS requests (T373925)
• 16:29 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2083.codfw.wmnet with reason: host reimage
• 16:29 claime: T373095 repool kubernetes2011, kubernetes2012, kubernetes2036, kubernetes2037, wikikube-worker2037, wikikube-worker2038, mw2436, mw2437
• 16:29 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2038.codfw.wmnet
• 16:29 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet
• 16:29 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2037.codfw.wmnet
• 16:29 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2037.codfw.wmnet
• 16:28 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2437.codfw.wmnet
• 16:28 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2437.codfw.wmnet
• 16:28 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host mw2436.codfw.wmnet
• 16:28 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host mw2436.codfw.wmnet
• 16:27 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2037.codfw.wmnet
• 16:27 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2037.codfw.wmnet
• 16:27 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2036.codfw.wmnet
• 16:27 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2036.codfw.wmnet
• 16:27 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2012.codfw.wmnet
• 16:27 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2012.codfw.wmnet
• 16:27 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubernetes2011.codfw.wmnet
• 16:27 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host kubernetes2011.codfw.wmnet
• 16:26 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2083.codfw.wmnet with reason: host reimage
• 16:18 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2036.codfw.wmnet
• 16:18 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2036.codfw.wmnet
• 16:18 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2036.codfw.wmnet
• 16:18 arnaudb@cumin1002: dbctl commit (dc=all): 'es2036 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68657 and previous config saved to /var/cache/conftool/dbconfig/20240904-161806-arnaudb.json
• 16:18 arnaudb@cumin1002: dbctl commit (dc=all): 'es2031 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68656 and previous config saved to /var/cache/conftool/dbconfig/20240904-161806-arnaudb.json
• 16:18 arnaudb@cumin1002: dbctl commit (dc=all): 'db2206 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68655 and previous config saved to /var/cache/conftool/dbconfig/20240904-161806-arnaudb.json
• 16:18 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68653 and previous config saved to /var/cache/conftool/dbconfig/20240904-161806-arnaudb.json
• 16:18 arnaudb@cumin1002: dbctl commit (dc=all): 'db2190 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68654 and previous config saved to /var/cache/conftool/dbconfig/20240904-161806-arnaudb.json
• 16:18 arnaudb@cumin1002: dbctl commit (dc=all): 'db2138 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68652 and previous config saved to /var/cache/conftool/dbconfig/20240904-161806-arnaudb.json
• 16:18 arnaudb@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 25%: T370852', diff saved to https://phabricator.wikimedia.org/P68651 and previous config saved to /var/cache/conftool/dbconfig/20240904-161806-arnaudb.json
• 16:16 claime: homer lsw1-b8-codfw* commit 'T372878'
• 16:16 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2036.codfw.wmnet with OS bullseye
• 16:13 swfrench-wmf: running homer 'cr*codfw*' commit 'T374018'
• 16:09 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2083
• 16:09 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2083
• 16:09 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2083
• 16:09 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2083.codfw.wmnet 167.16.192.10.in-addr.arpa 7.6.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 16:09 hnowlan@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2083.codfw.wmnet 167.16.192.10.in-addr.arpa 7.6.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 16:09 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:09 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2083 - hnowlan@cumin1002"
• 16:07 effie: restarting mcrouter on codfw
• 16:06 topranks: migrating servers in codfw rack C1 from asw-c-codfw to lsw1-c1-codfw T373095
• 16:06 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
• 16:04 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw[2260,2267].codfw.wmnet
• 16:04 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:03 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2083 - hnowlan@cumin1002"
• 16:02 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-lab1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 16:01 swfrench@cumin2002: START - Cookbook sre.dns.netbox
• 16:00 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 27 hosts with reason: Move server uplinks codfw racks C1
• 15:59 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on 27 hosts with reason: Move server uplinks codfw racks C1
• 15:58 elukey@cumin2002: START - Cookbook sre.hosts.provision for host ml-lab1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 15:56 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 15:56 hnowlan@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2083
• 15:56 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2083.codfw.wmnet with OS bullseye
• 15:56 hnowlan@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2083.codfw.wmnet
• 15:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 10 hosts with reason: network maintenance T370852
• 15:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on 10 hosts with reason: network maintenance T370852
• 15:55 arnaudb@cumin1002: dbctl commit (dc=all): 'depool db2125 db2138 db2149 db2190 db2206 db2207 es2031 es2032 es2036 - T370852', diff saved to https://phabricator.wikimedia.org/P68650 and previous config saved to /var/cache/conftool/dbconfig/20240904-155459-arnaudb.json
• 15:53 ladsgroup@deploy1003: Finished scap sync-world: Backport for Fix bug causing review form to disappear on unreviewed pages (T373582) (duration: 10m 31s)
• 15:53 swfrench@cumin2002: START - Cookbook sre.hosts.decommission for hosts mw[2260,2267].codfw.wmnet
• 15:48 ladsgroup@deploy1003: ladsgroup: Continuing with sync
• 15:47 ladsgroup@deploy1003: ladsgroup: Backport for Fix bug causing review form to disappear on unreviewed pages (T373582) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 15:46 bking@cumin2002: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99)
• 15:43 aqu@deploy1003: Finished deploy [airflow-dags/analytics@3b0d8ba]: Regular analytics weekly train [airflow-dags@3b0d8ba1] (duration: 00m 48s)
• 15:43 topranks: configure lsw1-c1-codfw interfaces for servers in advance of move T373095
• 15:43 ladsgroup@deploy1003: Started scap sync-world: Backport for Fix bug causing review form to disappear on unreviewed pages (T373582)
• 15:42 aqu@deploy1003: Started deploy [airflow-dags/analytics@3b0d8ba]: Regular analytics weekly train [airflow-dags@3b0d8ba1]
• 15:25 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2267.codfw.wmnet
• 15:24 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2267.codfw.wmnet
• 15:21 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2081.codfw.wmnet
• 15:20 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2081.codfw.wmnet
• 15:15 vgutierrez@cumin1002: conftool action : set/pooled=yes; selector: name=cp7015.magru.wmnet
• 15:14 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-lab1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 15:13 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7015.magru.wmnet
• 15:13 vgutierrez@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp7015.magru.wmnet
• 15:12 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host phab1005.eqiad.wmnet with OS bookworm
• 15:12 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host phab1005.eqiad.wmnet with OS bookworm
• 15:11 elukey@cumin2002: START - Cookbook sre.hosts.provision for host ml-lab1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 15:09 hashar@deploy1003: Finished scap sync-world: Backport for ParserOutput::collectMetadata: Log if given value is non-numeric and also non-string, for easier debugging, and don't fatal (T373920) (duration: 08m 37s)
• 15:06 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-lab1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 15:06 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2036.codfw.wmnet with reason: host reimage
• 15:05 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ml-lab1001.eqiad.wmnet with OS bookworm
• 15:04 hashar@deploy1003: hashar: Continuing with sync
• 15:04 hashar@deploy1003: hashar: Backport for ParserOutput::collectMetadata: Log if given value is non-numeric and also non-string, for easier debugging, and don't fatal (T373920) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 15:02 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2036.codfw.wmnet with reason: host reimage
• 15:02 elukey@cumin2002: START - Cookbook sre.hosts.provision for host ml-lab1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 15:00 hashar@deploy1003: Started scap sync-world: Backport for ParserOutput::collectMetadata: Log if given value is non-numeric and also non-string, for easier debugging, and don't fatal (T373920)
• 14:50 claime: homer cr*codfw* commit 'T372878'
• 14:44 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2036
• 14:44 cgoubert@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2036
• 14:44 cgoubert@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2036
• 14:44 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2036.codfw.wmnet 121.16.192.10.in-addr.arpa 1.2.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 14:44 cgoubert@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2036.codfw.wmnet 121.16.192.10.in-addr.arpa 1.2.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 14:44 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:44 cgoubert@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2036 - cgoubert@cumin1002"
• 14:44 cgoubert@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2036 - cgoubert@cumin1002"
• 14:40 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:40 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 14:40 bking@cumin2002: START - Cookbook sre.wdqs.restart
• 14:40 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:40 cgoubert@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2036
• 14:40 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
• 14:39 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2036.codfw.wmnet with OS bullseye
• 14:39 arnaudb@cumin1002: dbctl commit (dc=all): 'swap masters for es1 - T373095', diff saved to https://phabricator.wikimedia.org/P68648 and previous config saved to /var/cache/conftool/dbconfig/20240904-143928-arnaudb.json
• 14:38 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2036.codfw.wmnet
• 14:38 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:38 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2036.codfw.wmnet
• 14:38 cgoubert@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2036.codfw.wmnet
• 14:37 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:37 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:36 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-lab1001.eqiad.wmnet with OS bookworm
• 14:35 jclark@cumin1002: START - Cookbook sre.dns.netbox
• 14:33 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2082.codfw.wmnet
• 14:33 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2082.codfw.wmnet
• 14:30 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2088.codfw.wmnet on all recursors
• 14:30 cgoubert@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2088.codfw.wmnet on all recursors
• 14:29 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2054 to wikikube-worker2088
• 14:29 cgoubert@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2088
• 14:29 cgoubert@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2088
• 14:29 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:29 cgoubert@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2054 to wikikube-worker2088 - cgoubert@cumin1002"
• 14:28 cgoubert@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2054 to wikikube-worker2088 - cgoubert@cumin1002"
• 14:25 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 14:25 cgoubert@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2054 to wikikube-worker2088
• 14:22 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:22 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:17 hnowlan@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2082
• 14:17 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:16 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:16 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2087
• 14:16 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2087.codfw.wmnet with OS bookworm
• 14:15 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:15 hnowlan: homer lsw1-b3-codfw* commit
• 14:13 TheresNoTime: gerrit:1070561 reverted, fwiw
• 14:11 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:09 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:08 TheresNoTime: scap failed, gerrit:1070561 merged but undeployed currently
• 14:00 samtar@deploy1003: Started scap sync-world: Backport for Allow copyuploads on test2wiki (T356241)
• 13:57 TheresNoTime: scap failed: Exception K8s Deployment, rolled back
• 13:56 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2082.codfw.wmnet with OS bullseye
• 13:52 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 13:52 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 13:50 bking@cumin2002: START - Cookbook sre.wdqs.restart
• 13:48 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 13:47 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 13:44 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 13:44 samtar@deploy1003: Started scap sync-world: Backport for Allow copyuploads on test2wiki (T356241)
• 13:42 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2086.codfw.wmnet
• 13:42 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2086.codfw.wmnet
• 13:41 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2085.codfw.wmnet
• 13:41 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2085.codfw.wmnet
• 13:40 samtar@deploy1003: Finished scap sync-world: Backport for Enable CampaignEvents Invitation Lists on igwiki and swwiki (T372582) (duration: 08m 28s)
• 13:40 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 13:40 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ml-lab1001.eqiad.wmnet with OS bookworm
• 13:38 jayme: homer 'lsw1-b6-codfw*' commit 'T372878'
• 13:36 samtar@deploy1003: samtar, daimona: Continuing with sync
• 13:34 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2082.codfw.wmnet with reason: host reimage
• 13:34 samtar@deploy1003: samtar, daimona: Backport for Enable CampaignEvents Invitation Lists on igwiki and swwiki (T372582) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:34 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2086
• 13:34 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2086.codfw.wmnet with OS bookworm
• 13:33 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host phab1005.eqiad.wmnet with OS bookworm
• 13:32 samtar@deploy1003: Started scap sync-world: Backport for Enable CampaignEvents Invitation Lists on igwiki and swwiki (T372582)
• 13:31 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2082.codfw.wmnet with reason: host reimage
• 13:28 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2085
• 13:28 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2085.codfw.wmnet with OS bookworm
• 13:15 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2086.codfw.wmnet with reason: host reimage
• 13:14 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-worker2079.codfw.wmnet
• 13:14 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker2079.codfw.wmnet
• 13:13 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2082
• 13:13 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2082
• 13:12 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2086.codfw.wmnet with reason: host reimage
• 13:11 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2082
• 13:11 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2082.codfw.wmnet 166.16.192.10.in-addr.arpa 6.6.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:11 hnowlan@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2082.codfw.wmnet 166.16.192.10.in-addr.arpa 6.6.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:11 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:11 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2082 - hnowlan@cumin1002"
• 13:11 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2082 - hnowlan@cumin1002"
• 13:09 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2085.codfw.wmnet with reason: host reimage
• 13:06 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 13:06 hnowlan@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2082
• 13:06 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2085.codfw.wmnet with reason: host reimage
• 13:06 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2082.codfw.wmnet with OS bullseye
• 13:05 hnowlan@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2082
• 13:01 hashar@deploy1003: Finished deploy [releng/jenkins-deploy@26ba1ab] (releasing): do not set keepUndefinedParameters - T133737 (duration: 00m 38s)
• 13:00 hashar@deploy1003: Started deploy [releng/jenkins-deploy@26ba1ab] (releasing): do not set keepUndefinedParameters - T133737
• 12:57 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade/restart of Apache Traffic Server on A:cp-text_eqiad for 9.2.5-1wm2
• 12:56 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2087
• 12:56 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2087
• 12:55 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2087
• 12:55 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2087.codfw.wmnet 225.16.192.10.in-addr.arpa 5.2.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:55 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2087.codfw.wmnet 225.16.192.10.in-addr.arpa 5.2.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:55 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:55 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2087 - jayme@cumin1002"
• 12:55 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2087 - jayme@cumin1002"
• 12:52 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 12:52 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2086
• 12:52 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2086
• 12:52 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2086
• 12:52 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2086.codfw.wmnet 212.16.192.10.in-addr.arpa 2.1.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:52 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2086.codfw.wmnet 212.16.192.10.in-addr.arpa 2.1.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:52 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:52 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2086 - jayme@cumin1002"
• 12:52 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2086 - jayme@cumin1002"
• 12:50 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2087
• 12:50 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2087.codfw.wmnet with OS bookworm
• 12:50 jayme@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2087
• 12:49 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2087
• 12:49 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host wikikube-worker2087.codfw.wmnet with OS bookworm
• 12:49 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2087.codfw.wmnet with OS bookworm
• 12:49 jayme@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2087
• 12:48 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 12:46 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2086
• 12:45 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2085
• 12:45 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2085
• 12:45 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2085
• 12:45 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2085.codfw.wmnet 138.16.192.10.in-addr.arpa 8.3.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:45 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2085.codfw.wmnet 138.16.192.10.in-addr.arpa 8.3.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:45 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:45 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2085 - jayme@cumin1002"
• 12:45 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2085 - jayme@cumin1002"
• 12:42 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2086.codfw.wmnet with OS bookworm
• 12:42 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:41 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 12:41 jayme@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2086
• 12:41 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:41 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2085
• 12:41 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2085.codfw.wmnet with OS bookworm
• 12:40 jayme@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2085
• 12:36 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2035 to wikikube-worker2087
• 12:35 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2087
• 12:35 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:35 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2087
• 12:35 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:35 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2035 to wikikube-worker2087 - jayme@cumin1002"
• 12:34 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2035 to wikikube-worker2087 - jayme@cumin1002"
• 12:33 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:32 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2010 to wikikube-worker2086
• 12:31 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2086
• 12:31 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 12:31 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2086
• 12:31 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:31 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2010 to wikikube-worker2086 - jayme@cumin1002"
• 12:31 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2010 to wikikube-worker2086 - jayme@cumin1002"
• 12:29 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2035 to wikikube-worker2087
• 12:28 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2009 to wikikube-worker2085
• 12:27 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 12:27 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2085
• 12:27 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2010 to wikikube-worker2086
• 12:27 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2085
• 12:27 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:27 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2009 to wikikube-worker2085 - jayme@cumin1002"
• 12:27 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2191.codfw.wmnet with reason: provisionning db2221.codfw.wmnet - T373579
• 12:27 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2009 to wikikube-worker2085 - jayme@cumin1002"
• 12:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2191.codfw.wmnet with reason: provisionning db2221.codfw.wmnet - T373579
• 12:27 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: provisionning db2221.codfw.wmnet - T373579
• 12:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: provisionning db2221.codfw.wmnet - T373579
• 12:20 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 12:20 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2009 to wikikube-worker2085
• 12:08 hnowlan@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2081
• 12:08 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2081.codfw.wmnet with OS bullseye
• 12:05 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2054.codfw.wmnet
• 12:04 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2054.codfw.wmnet
• 12:04 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2035.codfw.wmnet
• 12:04 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2035.codfw.wmnet
• 12:04 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2010.codfw.wmnet
• 12:03 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2010.codfw.wmnet
• 12:03 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2009.codfw.wmnet
• 12:02 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2009.codfw.wmnet
• 12:00 hashar@deploy1003: Finished deploy [releng/jenkins-deploy@10f309b] (releasing): do not set keepUndefinedParameters - T133737 (duration: 00m 14s)
• 12:00 hashar@deploy1003: Started deploy [releng/jenkins-deploy@10f309b] (releasing): do not set keepUndefinedParameters - T133737
• 11:59 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade/restart of Apache Traffic Server on A:cp-text_eqiad for 9.2.5-1wm2
• 11:50 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade/restart of Apache Traffic Server on A:cp-upload_eqiad for 9.2.5-1wm2
• 11:46 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2081.codfw.wmnet with reason: host reimage
• 11:43 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2081.codfw.wmnet with reason: host reimage
• 11:26 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2081
• 11:26 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2081
• 11:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218 (T367781)', diff saved to https://phabricator.wikimedia.org/P68643 and previous config saved to /var/cache/conftool/dbconfig/20240904-112507-arnaudb.json
• 11:24 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2081
• 11:24 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2081.codfw.wmnet 165.16.192.10.in-addr.arpa 5.6.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:24 hnowlan@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2081.codfw.wmnet 165.16.192.10.in-addr.arpa 5.6.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:24 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:24 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2081 - hnowlan@cumin1002"
• 11:24 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2081 - hnowlan@cumin1002"
• 11:20 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 11:20 hnowlan@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2081
• 11:19 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2081.codfw.wmnet with OS bullseye
• 11:19 hnowlan@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2081
• 11:12 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2079.codfw.wmnet
• 11:12 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2079.codfw.wmnet
• 11:12 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2079.codfw.wmnet
• 11:11 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 11:11 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 11:11 claime: homer lsw1-b6-codfw* commit 'T372878'
• 11:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P68642 and previous config saved to /var/cache/conftool/dbconfig/20240904-110959-arnaudb.json
• 11:09 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2317 to wikikube-worker2082
• 11:09 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2319 to wikikube-worker2084
• 11:09 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2318 to wikikube-worker2083
• 11:09 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2082
• 11:09 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2082
• 11:08 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2084
• 11:08 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2083
• 11:08 hnowlan@cumin1002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wikikube-worker2082
• 11:07 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2084
• 11:07 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:07 topranks: migrating VMs off ganeti2009 in advance of network maintence codfw rack C1 - T373095
• 11:06 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2083
• 11:06 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:06 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2318 to wikikube-worker2083 - hnowlan@cumin1002"
• 11:06 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2318 to wikikube-worker2083 - hnowlan@cumin1002"
• 11:05 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2082
• 11:05 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:05 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2317 to wikikube-worker2082 - hnowlan@cumin1002"
• 11:05 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 11:04 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2079.codfw.wmnet with OS bullseye
• 11:04 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 11:04 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 11:00 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 10:59 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2317 to wikikube-worker2082 - hnowlan@cumin1002"
• 10:56 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from mw2318 to wikikube-worker2083
• 10:56 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2316 to wikikube-worker2081
• 10:55 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade/restart of Apache Traffic Server on A:cp-upload_eqiad for 9.2.5-1wm2
• 10:55 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2081
• 10:55 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2081
• 10:55 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:55 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2316 to wikikube-worker2081 - hnowlan@cumin1002"
• 10:55 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 10:54 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2316 to wikikube-worker2081 - hnowlan@cumin1002"
• 10:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P68641 and previous config saved to /var/cache/conftool/dbconfig/20240904-105452-arnaudb.json
• 10:53 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from mw2319 to wikikube-worker2084
• 10:50 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from mw2317 to wikikube-worker2082
• 10:50 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 10:50 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from mw2316 to wikikube-worker2081
• 10:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218 (T367781)', diff saved to https://phabricator.wikimedia.org/P68639 and previous config saved to /var/cache/conftool/dbconfig/20240904-103945-arnaudb.json
• 10:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2218 (T367781)', diff saved to https://phabricator.wikimedia.org/P68638 and previous config saved to /var/cache/conftool/dbconfig/20240904-103718-arnaudb.json
• 10:37 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2218.codfw.wmnet with reason: Maintenance
• 10:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2218.codfw.wmnet with reason: Maintenance
• 10:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1181 (T367781)', diff saved to https://phabricator.wikimedia.org/P68637 and previous config saved to /var/cache/conftool/dbconfig/20240904-103656-arnaudb.json
• 10:34 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2079.codfw.wmnet with reason: host reimage
• 10:33 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2319.codfw.wmnet
• 10:32 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2319.codfw.wmnet
• 10:32 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2318.codfw.wmnet
• 10:32 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2079.codfw.wmnet with reason: host reimage
• 10:31 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2318.codfw.wmnet
• 10:31 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2317.codfw.wmnet
• 10:31 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2317.codfw.wmnet
• 10:31 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2316.codfw.wmnet
• 10:30 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2316.codfw.wmnet
• 10:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P68636 and previous config saved to /var/cache/conftool/dbconfig/20240904-102148-arnaudb.json
• 10:19 claime: homer cr*codfw* commit 'T372878'
• 10:18 hnowlan: temporarily adding own-work licenses to test2wiki MediaWiki:Licenses to test uploads
• 10:15 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2079
• 10:15 cgoubert@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2079
• 10:15 cgoubert@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2079
• 10:15 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2079.codfw.wmnet 181.16.192.10.in-addr.arpa 1.8.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 10:15 cgoubert@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2079.codfw.wmnet 181.16.192.10.in-addr.arpa 1.8.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 10:15 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:15 cgoubert@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2079 - cgoubert@cumin1002"
• 10:15 cgoubert@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2079 - cgoubert@cumin1002"
• 10:11 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 10:11 btullis@cumin1002: END (PASS) - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash (exit_code=0) rolling restart_daemons on A:apifeatureusage
• 10:11 cgoubert@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2079
• 10:11 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2079.codfw.wmnet with OS bullseye
• 10:10 cgoubert@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2079.codfw.wmnet
• 10:08 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2325 to wikikube-worker2079
• 10:08 btullis@cumin1002: START - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash rolling restart_daemons on A:apifeatureusage
• 10:08 cgoubert@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2079
• 10:08 cgoubert@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2079
• 10:08 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:08 cgoubert@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2325 to wikikube-worker2079 - cgoubert@cumin1002"
• 10:07 cgoubert@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2325 to wikikube-worker2079 - cgoubert@cumin1002"
• 10:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P68635 and previous config saved to /var/cache/conftool/dbconfig/20240904-100641-arnaudb.json
• 10:04 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 10:04 cgoubert@cumin1002: START - Cookbook sre.hosts.rename from mw2325 to wikikube-worker2079
• 10:03 aqu: Deployed refinery-source using jenkins
• 10:02 cgoubert@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host mw2325.codfw.wmnet
• 10:01 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2325.codfw.wmnet
• 09:58 aqu: Deployed refinery using scap, then deployed onto hdfs
• 09:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1181 (T367781)', diff saved to https://phabricator.wikimedia.org/P68633 and previous config saved to /var/cache/conftool/dbconfig/20240904-095134-arnaudb.json
• 09:50 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2080.codfw.wmnet
• 09:50 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2080.codfw.wmnet
• 09:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1181 (T367781)', diff saved to https://phabricator.wikimedia.org/P68632 and previous config saved to /var/cache/conftool/dbconfig/20240904-094824-arnaudb.json
• 09:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance
• 09:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance
• 09:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T367781)', diff saved to https://phabricator.wikimedia.org/P68631 and previous config saved to /var/cache/conftool/dbconfig/20240904-094802-arnaudb.json
• 09:41 aqu@deploy1003: Finished deploy [analytics/refinery@07fd127] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@07fd1275] (duration: 04m 18s)
• 09:37 aqu@deploy1003: Started deploy [analytics/refinery@07fd127] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@07fd1275]
• 09:34 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade/restart of Apache Traffic Server on A:cp-text_drmrs for 9.2.5-1wm2
• 09:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P68630 and previous config saved to /var/cache/conftool/dbconfig/20240904-093255-arnaudb.json
• 09:23 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:23 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns entries for drmrs switch irb interface IPs - cmooney@cumin1002"
• 09:23 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns entries for drmrs switch irb interface IPs - cmooney@cumin1002"
• 09:19 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 7:00:00 on db2198.codfw.wmnet with reason: Maintenance
• 09:19 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 7:00:00 on db2198.codfw.wmnet with reason: Maintenance
• 09:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T367856)', diff saved to https://phabricator.wikimedia.org/P68629 and previous config saved to /var/cache/conftool/dbconfig/20240904-091937-marostegui.json
• 09:18 cmooney@cumin1002: START - Cookbook sre.dns.netbox
• 09:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P68628 and previous config saved to /var/cache/conftool/dbconfig/20240904-091748-arnaudb.json
• 09:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 09:17 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 09:10 isaranto@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' .
• 09:09 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' .
• 09:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P68627 and previous config saved to /var/cache/conftool/dbconfig/20240904-090429-marostegui.json
• 09:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T367781)', diff saved to https://phabricator.wikimedia.org/P68626 and previous config saved to /var/cache/conftool/dbconfig/20240904-090240-arnaudb.json
• 08:59 dcausse@deploy1003: Finished scap sync-world: Backport for Checker: ensure all labels are set (T373086) (duration: 06m 48s)
• 08:57 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 08:57 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 08:55 dcausse@deploy1003: dcausse: Continuing with sync
• 08:54 dcausse@deploy1003: dcausse: Backport for Checker: ensure all labels are set (T373086) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 08:52 dcausse@deploy1003: Started scap sync-world: Backport for Checker: ensure all labels are set (T373086)
• 08:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P68624 and previous config saved to /var/cache/conftool/dbconfig/20240904-084922-marostegui.json
• 08:48 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2080.codfw.wmnet
• 08:47 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2080.codfw.wmnet
• 08:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T367781)', diff saved to https://phabricator.wikimedia.org/P68623 and previous config saved to /var/cache/conftool/dbconfig/20240904-084435-arnaudb.json
• 08:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1170 (T367781)', diff saved to https://phabricator.wikimedia.org/P68622 and previous config saved to /var/cache/conftool/dbconfig/20240904-083622-arnaudb.json
• 08:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1170.eqiad.wmnet with reason: Maintenance
• 08:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1170.eqiad.wmnet with reason: Maintenance
• 08:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T367856)', diff saved to https://phabricator.wikimedia.org/P68621 and previous config saved to /var/cache/conftool/dbconfig/20240904-083415-marostegui.json
• 08:33 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade/restart of Apache Traffic Server on A:cp-text_drmrs for 9.2.5-1wm2
• 08:32 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade/restart of Apache Traffic Server on A:cp-upload_drmrs for 9.2.5-1wm2
• 08:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P68620 and previous config saved to /var/cache/conftool/dbconfig/20240904-082928-arnaudb.json
• 08:19 elukey: upgrade ruby-nokogiri on serpens and seaborgium for security upgrades
• 08:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P68619 and previous config saved to /var/cache/conftool/dbconfig/20240904-081420-arnaudb.json
• 07:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T367781)', diff saved to https://phabricator.wikimedia.org/P68618 and previous config saved to /var/cache/conftool/dbconfig/20240904-075913-arnaudb.json
• 07:59 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 07:58 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 07:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2214 (T367781)', diff saved to https://phabricator.wikimedia.org/P68617 and previous config saved to /var/cache/conftool/dbconfig/20240904-075659-arnaudb.json
• 07:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2214.codfw.wmnet with reason: Maintenance
• 07:56 kartik@deploy1003: Finished scap sync-world: Backport for TTMServerAid: Tell PHP that we're OK with $services starting out null (T373921) (duration: 09m 23s)
• 07:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2214.codfw.wmnet with reason: Maintenance
• 07:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1173 (T367781)', diff saved to https://phabricator.wikimedia.org/P68616 and previous config saved to /var/cache/conftool/dbconfig/20240904-075637-arnaudb.json
• 07:54 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 07:54 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 07:52 kartik@deploy1003: kartik, abi: Continuing with sync
• 07:49 kartik@deploy1003: kartik, abi: Backport for TTMServerAid: Tell PHP that we're OK with $services starting out null (T373921) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:47 kartik@deploy1003: Started scap sync-world: Backport for TTMServerAid: Tell PHP that we're OK with $services starting out null (T373921)
• 07:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P68615 and previous config saved to /var/cache/conftool/dbconfig/20240904-074130-arnaudb.json
• 07:38 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 07:34 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 07:33 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade/restart of Apache Traffic Server on A:cp-upload_drmrs for 9.2.5-1wm2
• 07:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P68614 and previous config saved to /var/cache/conftool/dbconfig/20240904-072623-arnaudb.json
• 07:14 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2437.codfw.wmnet
• 07:14 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2437.codfw.wmnet
• 07:14 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2436.codfw.wmnet
• 07:13 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2436.codfw.wmnet
• 07:13 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2038.codfw.wmnet
• 07:12 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2038.codfw.wmnet
• 07:12 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2037.codfw.wmnet
• 07:12 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2037.codfw.wmnet
• 07:12 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2037.codfw.wmnet
• 07:11 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2037.codfw.wmnet
• 07:11 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2036.codfw.wmnet
• 07:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1173 (T367781)', diff saved to https://phabricator.wikimedia.org/P68613 and previous config saved to /var/cache/conftool/dbconfig/20240904-071115-arnaudb.json
• 07:10 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2036.codfw.wmnet
• 07:10 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2012.codfw.wmnet
• 07:10 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2012.codfw.wmnet
• 07:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1173 (T367781)', diff saved to https://phabricator.wikimedia.org/P68612 and previous config saved to /var/cache/conftool/dbconfig/20240904-071007-arnaudb.json
• 07:10 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1173.eqiad.wmnet with reason: Maintenance
• 07:09 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1173.eqiad.wmnet with reason: Maintenance
• 07:09 akosiaris: T373095 depool kubernetes2011, kubernetes2012, kubernetes2036, kubernetes2037, wikikube-worker2037, wikikube-worker2038, mw2436, mw2437
• 07:08 akosiaris@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2011.codfw.wmnet
• 07:07 akosiaris@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2011.codfw.wmnet
• 06:11 aqu@deploy1003: Finished deploy [analytics/refinery@07fd127] (thin): Regular analytics weekly train THIN [analytics/refinery@07fd1275] (duration: 04m 55s)
• 06:06 aqu@deploy1003: Started deploy [analytics/refinery@07fd127] (thin): Regular analytics weekly train THIN [analytics/refinery@07fd1275]

2024-09-03

• 22:04 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host phab1005.eqiad.wmnet with OS bookworm
• 21:42 swfrench-wmf: running homer 'cr*codfw*' commit 'T372878'
• 21:34 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2080.codfw.wmnet
• 21:34 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2080.codfw.wmnet
• 21:31 swfrench-wmf: running homer 'lsw1-b3-codfw*' commit 'T372878'
• 21:13 jdrewniak@deploy1003: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 02m 18s)
• 21:11 jdrewniak@deploy1003: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 06m 25s)
• 20:59 jdrewniak@deploy1003: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 02m 19s)
• 20:56 jdrewniak@deploy1003: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 02m 21s)
• 20:56 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host phab1005
• 20:56 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host phab1005
• 20:45 toyofuku@deploy1003: Finished scap sync-world: Backport for Turn on donate link in beta (T372757), Disable lead paragraph transform on Wikivoyages (duration: 09m 43s)
• 20:44 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host phab1005.eqiad.wmnet with OS bookworm
• 20:44 swfrench-wmf: running homer 'lsw1-b3-codfw*' commit 'T372878'
• 20:44 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host phab1005.mgmt.eqiad.wmnet with reboot policy FORCED
• 20:41 toyofuku@deploy1003: jdlrobson, toyofuku: Continuing with sync
• 20:39 jclark@cumin1002: START - Cookbook sre.hosts.provision for host phab1005.mgmt.eqiad.wmnet with reboot policy FORCED
• 20:38 toyofuku@deploy1003: jdlrobson, toyofuku: Backport for Turn on donate link in beta (T372757), Disable lead paragraph transform on Wikivoyages synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:37 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host phab1005.mgmt.eqiad.wmnet with reboot policy FORCED
• 20:36 toyofuku@deploy1003: Started scap sync-world: Backport for Turn on donate link in beta (T372757), Disable lead paragraph transform on Wikivoyages
• 20:31 cjming@deploy1003: Finished scap sync-world: Backport for cirrus: Introduce an expensive query pool counter (T369808) (duration: 06m 47s)
• 20:26 cjming@deploy1003: ebernhardson, cjming: Continuing with sync
• 20:26 cjming@deploy1003: ebernhardson, cjming: Backport for cirrus: Introduce an expensive query pool counter (T369808) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:24 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2080.codfw.wmnet with OS bullseye
• 20:24 cjming@deploy1003: Started scap sync-world: Backport for cirrus: Introduce an expensive query pool counter (T369808)
• 20:08 jclark@cumin1002: START - Cookbook sre.hosts.provision for host phab1005.mgmt.eqiad.wmnet with reboot policy FORCED
• 20:07 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host phab1005
• 20:07 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host phab1005
• 20:07 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 20:07 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt phab1005 - jclark@cumin1002"
• 20:07 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt phab1005 - jclark@cumin1002"
• 20:04 jclark@cumin1002: START - Cookbook sre.dns.netbox
• 20:03 jclark@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 20:01 jclark@cumin1002: START - Cookbook sre.dns.netbox
• 19:58 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host phab1005.mgmt.eqiad.wmnet with reboot policy FORCED
• 19:57 jclark@cumin1002: START - Cookbook sre.hosts.provision for host phab1005.mgmt.eqiad.wmnet with reboot policy FORCED
• 19:56 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 19:56 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt phab1005 - jclark@cumin1002"
• 19:56 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt phab1005 - jclark@cumin1002"
• 19:55 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2080.codfw.wmnet with reason: host reimage
• 19:53 jclark@cumin1002: START - Cookbook sre.dns.netbox
• 19:52 swfrench@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2080.codfw.wmnet with reason: host reimage
• 19:52 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host gerrit1004.mgmt.eqiad.wmnet with reboot policy FORCED
• 19:48 jclark@cumin1002: START - Cookbook sre.hosts.provision for host gerrit1004.mgmt.eqiad.wmnet with reboot policy FORCED
• 19:46 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host gerrit1004.mgmt.eqiad.wmnet with reboot policy FORCED
• 19:44 jclark@cumin1002: START - Cookbook sre.hosts.provision for host gerrit1004.mgmt.eqiad.wmnet with reboot policy FORCED
• 19:42 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host gerrit1004.mgmt.eqiad.wmnet with reboot policy FORCED
• 19:42 jclark@cumin1002: START - Cookbook sre.hosts.provision for host gerrit1004.mgmt.eqiad.wmnet with reboot policy FORCED
• 19:35 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2080
• 19:35 swfrench@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2080
• 19:35 swfrench@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2080
• 19:35 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2080.codfw.wmnet 159.16.192.10.in-addr.arpa 9.5.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 19:35 swfrench@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2080.codfw.wmnet 159.16.192.10.in-addr.arpa 9.5.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 19:35 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 19:33 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host phab1005
• 19:32 swfrench@cumin2002: START - Cookbook sre.dns.netbox
• 19:31 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host phab1005
• 19:31 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 19:31 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt phab1005 - jclark@cumin1002"
• 19:30 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt phab1005 - jclark@cumin1002"
• 19:27 jclark@cumin1002: START - Cookbook sre.dns.netbox
• 19:23 swfrench@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 19:22 ladsgroup@deploy1003: Finished scap sync-world: Backport for Revert "Set wgFlaggedRevsHandleIncludes to FR_INCLUDES_CURRENT on ruwiki" (T359529) (duration: 10m 50s)
• 19:21 swfrench@cumin2002: START - Cookbook sre.dns.netbox
• 19:21 swfrench@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2080
• 19:21 swfrench@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2080.codfw.wmnet with OS bullseye
• 19:20 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2080.codfw.wmnet on all recursors
• 19:20 swfrench@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2080.codfw.wmnet on all recursors
• 19:19 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2312 to wikikube-worker2080
• 19:18 swfrench@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2080
• 19:18 swfrench@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2080
• 19:18 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 19:18 swfrench@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2312 to wikikube-worker2080 - swfrench@cumin2002"
• 19:17 swfrench@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2312 to wikikube-worker2080 - swfrench@cumin2002"
• 19:17 ladsgroup@deploy1003: ladsgroup: Continuing with sync
• 19:16 ladsgroup@deploy1003: ladsgroup: Backport for Revert "Set wgFlaggedRevsHandleIncludes to FR_INCLUDES_CURRENT on ruwiki" (T359529) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 19:13 swfrench@cumin2002: START - Cookbook sre.dns.netbox
• 19:13 swfrench@cumin2002: START - Cookbook sre.hosts.rename from mw2312 to wikikube-worker2080
• 19:11 ladsgroup@deploy1003: Started scap sync-world: Backport for Revert "Set wgFlaggedRevsHandleIncludes to FR_INCLUDES_CURRENT on ruwiki" (T359529)
• 19:08 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2078.codfw.wmnet
• 19:08 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2078.codfw.wmnet
• 19:07 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2077.codfw.wmnet
• 19:07 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2077.codfw.wmnet
• 19:07 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2076.codfw.wmnet
• 19:07 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2076.codfw.wmnet
• 19:06 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2312.codfw.wmnet
• 19:06 dancy@deploy1003: Finished scap sync-world: testwikis to 1.43.0-wmf.21 refs T373640 (duration: 08m 11s)
• 19:06 kamila_: ran homer on lsw1-a5-codfw for T372878
• 19:05 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2312.codfw.wmnet
• 19:05 swfrench@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2260.codfw.wmnet
• 19:05 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2076.codfw.wmnet with OS bullseye
• 19:04 swfrench@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2260.codfw.wmnet
• 18:59 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2077.codfw.wmnet with OS bullseye
• 18:59 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2078.codfw.wmnet with OS bullseye
• 18:58 dancy@deploy1003: Started scap sync-world: testwikis to 1.43.0-wmf.21 refs T373640
• 18:13 dancy@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.21 refs T373640
• 17:53 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2075.codfw.wmnet
• 17:53 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2075.codfw.wmnet
• 17:53 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2074.codfw.wmnet
• 17:53 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2074.codfw.wmnet
• 17:53 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2073.codfw.wmnet
• 17:53 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2073.codfw.wmnet
• 17:53 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2072.codfw.wmnet
• 17:53 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2072.codfw.wmnet
• 17:53 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2075.codfw.wmnet with OS bullseye
• 17:48 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2074.codfw.wmnet with OS bullseye
• 17:45 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2078.codfw.wmnet with reason: host reimage
• 17:42 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2078.codfw.wmnet with reason: host reimage
• 17:40 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2076.codfw.wmnet with reason: host reimage
• 17:38 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2076.codfw.wmnet with reason: host reimage
• 17:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2077.codfw.wmnet with reason: host reimage
• 17:35 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
• 17:33 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2075.codfw.wmnet with reason: host reimage
• 17:33 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2077.codfw.wmnet with reason: host reimage
• 17:31 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
• 17:29 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2075.codfw.wmnet with reason: host reimage
• 17:26 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2078
• 17:26 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2078
• 17:26 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2078
• 17:26 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2078.codfw.wmnet 75.0.192.10.in-addr.arpa 5.7.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 17:26 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2078.codfw.wmnet 75.0.192.10.in-addr.arpa 5.7.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 17:26 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:26 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2078 - kamila@cumin1002"
• 17:26 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2078 - kamila@cumin1002"
• 17:25 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 17:23 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2074.codfw.wmnet with reason: host reimage
• 17:22 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 17:22 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2078
• 17:21 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2076
• 17:21 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2076
• 17:21 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2076
• 17:21 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2076.codfw.wmnet 66.0.192.10.in-addr.arpa 6.6.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 17:21 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2076.codfw.wmnet 66.0.192.10.in-addr.arpa 6.6.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 17:21 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:21 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2076 - kamila@cumin1002"
• 17:21 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2076 - kamila@cumin1002"
• 17:21 hnowlan: homer 'cr*codfw*' commit
• 17:21 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 17:19 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2074.codfw.wmnet with reason: host reimage
• 17:19 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2078.codfw.wmnet with OS bullseye
• 17:17 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 17:17 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2076
• 17:17 hnowlan: homer 'lsw1-a5-codfw*' && homer 'lsw1-a6-codfw*' commit
• 17:16 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2077
• 17:16 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2077
• 17:16 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2077
• 17:16 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2077.codfw.wmnet 71.0.192.10.in-addr.arpa 1.7.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 17:16 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2077.codfw.wmnet 71.0.192.10.in-addr.arpa 1.7.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 17:16 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:16 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2077 - kamila@cumin1002"
• 17:16 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2077 - kamila@cumin1002"
• 17:12 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 17:12 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2077
• 17:12 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2075
• 17:12 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2075
• 17:11 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2075
• 17:11 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2075.codfw.wmnet 80.0.192.10.in-addr.arpa 0.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 17:11 hnowlan@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2075.codfw.wmnet 80.0.192.10.in-addr.arpa 0.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 17:11 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:09 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2407 to wikikube-worker2078
• 17:08 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 17:08 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2078
• 17:08 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2078
• 17:08 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:08 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2407 to wikikube-worker2078 - kamila@cumin1002"
• 17:08 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2407 to wikikube-worker2078 - kamila@cumin1002"
• 17:06 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2077.codfw.wmnet with OS bullseye
• 17:06 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2406 to wikikube-worker2077
• 17:05 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2077
• 17:05 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 17:04 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2077
• 17:04 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:04 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2406 to wikikube-worker2077 - kamila@cumin1002"
• 17:04 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2076.codfw.wmnet with OS bullseye
• 17:04 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2406 to wikikube-worker2077 - kamila@cumin1002"
• 17:03 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2402 to wikikube-worker2076
• 17:03 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw2407 to wikikube-worker2078
• 17:03 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2076
• 17:00 hnowlan@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2075
• 17:00 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2074
• 17:00 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2074
• 17:00 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 17:00 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2076
• 17:00 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:00 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2402 to wikikube-worker2076 - kamila@cumin1002"
• 17:00 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2402 to wikikube-worker2076 - kamila@cumin1002"
• 17:00 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw2406 to wikikube-worker2077
• 16:59 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2074
• 16:59 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2074.codfw.wmnet 78.0.192.10.in-addr.arpa 8.7.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 16:59 hnowlan@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2074.codfw.wmnet 78.0.192.10.in-addr.arpa 8.7.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 16:59 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:59 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2074 - hnowlan@cumin1002"
• 16:57 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (None, T373791) xfer wikidata_main from wdqs2022.codfw.wmnet -> wdqs2021.codfw.wmnet w/ force delete existing files, repooling neither afterwards
• 16:56 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 16:56 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2074 - hnowlan@cumin1002"
• 16:52 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw2402 to wikikube-worker2076
• 16:51 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 16:49 hnowlan@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2074
• 16:49 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2075.codfw.wmnet with OS bullseye
• 16:49 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2074.codfw.wmnet with OS bullseye
• 16:46 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2073.codfw.wmnet with OS bullseye
• 16:45 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2072.codfw.wmnet with OS bullseye
• 16:45 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2407.codfw.wmnet
• 16:45 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2407.codfw.wmnet
• 16:45 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2406.codfw.wmnet
• 16:44 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2402.codfw.wmnet
• 16:44 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2406.codfw.wmnet
• 16:44 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2402.codfw.wmnet
• 16:35 aqu@deploy1003: Finished deploy [analytics/refinery@07fd127]: Regular analytics weekly train [analytics/refinery@07fd1275] (duration: 08m 09s)
• 16:27 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2072.codfw.wmnet with reason: host reimage
• 16:27 aqu@deploy1003: Started deploy [analytics/refinery@07fd127]: Regular analytics weekly train [analytics/refinery@07fd1275]
• 16:27 aqu: About to deploy analytics/refinery
• 16:26 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply heap settings - bking@cumin2002 - T373895
• 16:24 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2072.codfw.wmnet with reason: host reimage
• 16:23 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2073.codfw.wmnet with reason: host reimage
• 16:21 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2073.codfw.wmnet with reason: host reimage
• 16:17 mutante: contint - installed java jdk 17 packages - just installed, in parallel to existing jdk 11, no change to java_home / what is used by CI yet. T359795
• 16:09 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (None, T373791) xfer wikidata_main from wdqs2022.codfw.wmnet -> wdqs2021.codfw.wmnet w/ force delete existing files, repooling neither afterwards
• 16:07 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 16:03 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp7015.magru.wmnet with reason: T371554
• 16:03 brett@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on cp7015.magru.wmnet with reason: T371554
• 16:01 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply heap settings - bking@cumin2002 - T373895
• 15:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:59 denisse: Enabling meta monitoring for alert[12]001 - T372418
• 15:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:58 denisse: Reverting back to alert1001 - T372418
• 15:58 bking@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=wdqs-main,name=codfw
• 15:45 godog: bounce icinga on alert2002
• 15:11 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2073
• 15:11 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2073
• 15:10 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2073
• 15:10 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2073.codfw.wmnet 25.0.192.10.in-addr.arpa 5.2.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 15:10 hnowlan@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2073.codfw.wmnet 25.0.192.10.in-addr.arpa 5.2.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 15:10 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:10 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2073 - hnowlan@cumin1002"
• 15:10 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2073 - hnowlan@cumin1002"
• 15:07 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 15:07 hnowlan@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2073
• 15:07 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2072
• 15:07 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2072
• 15:06 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2072
• 15:06 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2072.codfw.wmnet 89.0.192.10.in-addr.arpa 9.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 15:06 hnowlan@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2072.codfw.wmnet 89.0.192.10.in-addr.arpa 9.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 15:06 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:06 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2072 - hnowlan@cumin1002"
• 15:06 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2072 - hnowlan@cumin1002"
• 15:02 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2073.codfw.wmnet with OS bullseye
• 15:01 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 15:01 hnowlan@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2072
• 15:01 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2072.codfw.wmnet with OS bullseye
• 15:00 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2422 to wikikube-worker2074
• 15:00 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2074
• 14:59 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2074
• 14:59 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:59 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2423 to wikikube-worker2075
• 14:58 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2075
• 14:57 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2075
• 14:57 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:57 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2423 to wikikube-worker2075 - hnowlan@cumin1002"
• 14:57 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 14:56 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2423 to wikikube-worker2075 - hnowlan@cumin1002"
• 14:51 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 14:51 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from mw2423 to wikikube-worker2075
• 14:51 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from mw2422 to wikikube-worker2074
• 14:50 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2055 to wikikube-worker2073
• 14:49 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2073
• 14:49 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2073
• 14:49 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:49 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2055 to wikikube-worker2073 - hnowlan@cumin1002"
• 14:48 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2055 to wikikube-worker2073 - hnowlan@cumin1002"
• 14:46 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2028 to wikikube-worker2072
• 14:45 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2072
• 14:44 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 14:44 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2072
• 14:44 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:44 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2028 to wikikube-worker2072 - hnowlan@cumin1002"
• 14:44 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2028 to wikikube-worker2072 - hnowlan@cumin1002"
• 14:44 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2055 to wikikube-worker2073
• 14:42 hashar: Restarting CI Jenkins
• 14:40 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 14:40 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2028 to wikikube-worker2072
• 14:26 hnowlan@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host mw2423.codfw.wmnet
• 14:26 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2423.codfw.wmnet
• 14:25 hnowlan@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host mw2423.codfw.wmnet
• 14:25 hnowlan@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host mw2422.codfw.wmnet
• 14:25 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2422.codfw.wmnet
• 14:25 hnowlan@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host mw2422.codfw.wmnet
• 14:25 hnowlan@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host kubernetes2055.codfw.wmnet
• 14:25 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2423.codfw.wmnet
• 14:25 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2422.codfw.wmnet
• 14:24 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2055.codfw.wmnet
• 14:24 hnowlan@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host kubernetes2028.codfw.wmnet
• 14:10 denisse: Resolve DNS queries to alert2002 - T372418
• 14:06 denisse: Failing over to alert2002 - T372418
• 14:03 denisse: Stopping services in the alert1001 host - T372418
• 14:02 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T371742)', diff saved to https://phabricator.wikimedia.org/P68609 and previous config saved to /var/cache/conftool/dbconfig/20240903-140226-ladsgroup.json
• 14:00 denisse: Disabling meta-monitoring for the alert hosts - T372418
• 14:00 denisse: Disabling meta-monitoring for the alert hosts
• 14:00 jgleeson: smashpig updated from e7c7d116 to e625eef2
• 13:59 ejegg: payments-wiki upgraded from 54988ad9 to e47e61cb
• 13:58 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:55 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:54 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:51 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:49 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: apply
• 13:49 jayme@deploy1003: helmfile [eqiad] START helmfile.d/services/eventgate-main: apply
• 13:48 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:47 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P68606 and previous config saved to /var/cache/conftool/dbconfig/20240903-134719-ladsgroup.json
• 13:45 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:41 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: apply
• 13:41 jayme@deploy1003: helmfile [codfw] START helmfile.d/services/eventgate-main: apply
• 13:37 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:34 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:32 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P68605 and previous config saved to /var/cache/conftool/dbconfig/20240903-133211-ladsgroup.json
• 13:30 jayme@deploy1003: helmfile [staging] DONE helmfile.d/services/eventgate-main: apply
• 13:29 jayme@deploy1003: helmfile [staging] START helmfile.d/services/eventgate-main: apply
• 13:29 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:25 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:20 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:17 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T371742)', diff saved to https://phabricator.wikimedia.org/P68604 and previous config saved to /var/cache/conftool/dbconfig/20240903-131704-ladsgroup.json
• 13:16 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:10 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 13:07 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 12:52 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:51 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:43 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:43 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:26 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2207 (T371742)', diff saved to https://phabricator.wikimedia.org/P68602 and previous config saved to /var/cache/conftool/dbconfig/20240903-122647-ladsgroup.json
• 12:26 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2207.codfw.wmnet with reason: Maintenance
• 12:26 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2207.codfw.wmnet with reason: Maintenance
• 12:24 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:24 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:20 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 11:42 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2197.codfw.wmnet with reason: Maintenance
• 11:42 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2197.codfw.wmnet with reason: Maintenance
• 11:42 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T371742)', diff saved to https://phabricator.wikimedia.org/P68601 and previous config saved to /var/cache/conftool/dbconfig/20240903-114232-ladsgroup.json
• 11:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade/restart of Apache Traffic Server on A:cp-magru for 9.2.5-1wm2
• 11:27 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P68600 and previous config saved to /var/cache/conftool/dbconfig/20240903-112725-ladsgroup.json
• 11:12 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P68599 and previous config saved to /var/cache/conftool/dbconfig/20240903-111218-ladsgroup.json
• 10:57 moritzm: installing amd64-microcode security updates
• 10:57 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T371742)', diff saved to https://phabricator.wikimedia.org/P68598 and previous config saved to /var/cache/conftool/dbconfig/20240903-105710-ladsgroup.json
• 10:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-lab1002.eqiad.wmnet
• 10:29 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 10:29 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 10:27 klausman@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 10:26 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ml-lab1002.eqiad.wmnet
• 10:12 klausman@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
• 10:11 klausman@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
• 10:09 klausman@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
• 10:09 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2066.codfw.wmnet with OS bullseye
• 10:08 klausman@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
• 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2189 (T371742)', diff saved to https://phabricator.wikimedia.org/P68597 and previous config saved to /var/cache/conftool/dbconfig/20240903-100713-ladsgroup.json
• 10:07 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2189.codfw.wmnet with reason: Maintenance
• 10:06 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2189.codfw.wmnet with reason: Maintenance
• 10:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T371742)', diff saved to https://phabricator.wikimedia.org/P68596 and previous config saved to /var/cache/conftool/dbconfig/20240903-100651-ladsgroup.json
• 10:02 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 10:02 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 09:56 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker2001.codfw.wmnet
• 09:56 cgoubert@cumin1002: START - Cookbook sre.hosts.remove-downtime for wikikube-worker2001.codfw.wmnet
• 09:55 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2001.codfw.wmnet
• 09:55 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2001.codfw.wmnet
• 09:53 cgoubert@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2001.codfw.wmnet
• 09:53 cgoubert@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2001.codfw.wmnet with OS bullseye
• 09:52 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2001.codfw.wmnet with OS bullseye
• 09:52 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet
• 09:52 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet
• 09:51 cgoubert@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2001.codfw.wmnet
• 09:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P68594 and previous config saved to /var/cache/conftool/dbconfig/20240903-095144-ladsgroup.json
• 09:50 cgoubert@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet
• 09:50 cgoubert@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet
• 09:48 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 09:47 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
• 09:47 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 09:47 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 09:47 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
• 09:47 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
• 09:47 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
• 09:46 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
• 09:46 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 09:46 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
• 09:46 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 09:46 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 09:44 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
• 09:44 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
• 09:44 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 09:43 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 09:43 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 09:43 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 09:42 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 09:41 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 09:41 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 09:41 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 09:40 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 09:40 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 09:40 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 09:40 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 09:36 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P68592 and previous config saved to /var/cache/conftool/dbconfig/20240903-093637-ladsgroup.json
• 09:36 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2066.codfw.wmnet with reason: host reimage
• 09:33 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2066.codfw.wmnet with reason: host reimage
• 09:24 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade/restart of Apache Traffic Server on A:cp-magru for 9.2.5-1wm2
• 09:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade/restart of Apache Traffic Server on A:cp-text_codfw for 9.2.5-1wm2
• 09:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T371742)', diff saved to https://phabricator.wikimedia.org/P68591 and previous config saved to /var/cache/conftool/dbconfig/20240903-092129-ladsgroup.json
• 09:21 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:21 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:15 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2066.codfw.wmnet with OS bullseye
• 09:14 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2066.codfw.wmnet
• 09:14 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2066.codfw.wmnet
• 09:14 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2066.codfw.wmnet
• 09:14 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2066.codfw.wmnet
• 08:57 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 08:56 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 08:56 jnuche@deploy1003: Finished scap sync-world: testwikis to 1.43.0-wmf.21 refs T373640 (duration: 45m 42s)
• 08:50 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet
• 08:45 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 08:45 elukey@cumin2002: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 08:44 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet
• 08:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1001.eqiad.wmnet
• 08:37 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host sretest1001.eqiad.wmnet
• 08:36 moritzm: remove intel-microcode 3.20240312.1~deb11u1 from apt.wikimedia.org (this was a temporary import for the last round of Bullseye reboots, not superceded by 3.20240813.1~deb11u1 from the 11.1 point release) T373795
• 08:29 kevinbazira@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 08:28 moritzm: installing intel-microcode security updates
• 08:26 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 08:26 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2175 (T371742)', diff saved to https://phabricator.wikimedia.org/P68589 and previous config saved to /var/cache/conftool/dbconfig/20240903-082639-ladsgroup.json
• 08:26 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2175.codfw.wmnet with reason: Maintenance
• 08:26 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2175.codfw.wmnet with reason: Maintenance
• 08:26 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T371742)', diff saved to https://phabricator.wikimedia.org/P68588 and previous config saved to /var/cache/conftool/dbconfig/20240903-082617-ladsgroup.json
• 08:23 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade/restart of Apache Traffic Server on A:cp-text_codfw for 9.2.5-1wm2
• 08:18 elukey: upgrade spicerack to 8.12.0 on cumin1002
• 08:13 elukey: upgrade python3-nbconvert on various DE hosts for security upgrades
• 08:11 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P68586 and previous config saved to /var/cache/conftool/dbconfig/20240903-081110-ladsgroup.json
• 08:10 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 08:10 jnuche@deploy1003: Started scap sync-world: testwikis to 1.43.0-wmf.21 refs T373640
• 08:06 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade/restart of Apache Traffic Server on A:cp-upload_codfw for 9.2.5-1wm2
• 07:57 moritzm: move LDAP user cn=ncreasy from cn=nda to cn=wmf
• 07:56 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P68585 and previous config saved to /var/cache/conftool/dbconfig/20240903-075602-ladsgroup.json
• 07:40 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T371742)', diff saved to https://phabricator.wikimedia.org/P68584 and previous config saved to /var/cache/conftool/dbconfig/20240903-074055-ladsgroup.json
• 07:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 07:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 07:16 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@4ca4744] (releasing): (no justification provided) (duration: 00m 39s)
• 07:15 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@4ca4744] (releasing): (no justification provided)
• 07:11 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade/restart of Apache Traffic Server on A:cp-upload_codfw for 9.2.5-1wm2
• 06:29 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138', diff saved to https://phabricator.wikimedia.org/P68581 and previous config saved to /var/cache/conftool/dbconfig/20240903-062909-ladsgroup.json
• 06:14 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138', diff saved to https://phabricator.wikimedia.org/P68580 and previous config saved to /var/cache/conftool/dbconfig/20240903-061402-ladsgroup.json
• 05:58 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138 (T371742)', diff saved to https://phabricator.wikimedia.org/P68579 and previous config saved to /var/cache/conftool/dbconfig/20240903-055855-ladsgroup.json
• 05:04 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2138 (T371742)', diff saved to https://phabricator.wikimedia.org/P68578 and previous config saved to /var/cache/conftool/dbconfig/20240903-050400-ladsgroup.json
• 05:03 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance
• 05:03 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance
• 05:03 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T371742)', diff saved to https://phabricator.wikimedia.org/P68577 and previous config saved to /var/cache/conftool/dbconfig/20240903-050338-ladsgroup.json
• 04:48 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P68576 and previous config saved to /var/cache/conftool/dbconfig/20240903-044830-ladsgroup.json
• 04:33 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P68575 and previous config saved to /var/cache/conftool/dbconfig/20240903-043323-ladsgroup.json
• 04:18 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T371742)', diff saved to https://phabricator.wikimedia.org/P68574 and previous config saved to /var/cache/conftool/dbconfig/20240903-041816-ladsgroup.json
• 04:00 mwpresync@deploy1003: Pruned MediaWiki: 1.43.0-wmf.18 (duration: 00m 47s)
• 03:56 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2126 (T371742)', diff saved to https://phabricator.wikimedia.org/P68573 and previous config saved to /var/cache/conftool/dbconfig/20240903-035610-ladsgroup.json
• 03:56 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
• 03:55 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
• 03:55 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2126.codfw.wmnet with reason: Maintenance
• 03:55 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2126.codfw.wmnet with reason: Maintenance
• 03:55 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T371742)', diff saved to https://phabricator.wikimedia.org/P68572 and previous config saved to /var/cache/conftool/dbconfig/20240903-035534-ladsgroup.json
• 03:40 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P68571 and previous config saved to /var/cache/conftool/dbconfig/20240903-034026-ladsgroup.json
• 03:25 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P68570 and previous config saved to /var/cache/conftool/dbconfig/20240903-032519-ladsgroup.json
• 03:10 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T371742)', diff saved to https://phabricator.wikimedia.org/P68569 and previous config saved to /var/cache/conftool/dbconfig/20240903-031012-ladsgroup.json
• 02:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2125 (T371742)', diff saved to https://phabricator.wikimedia.org/P68568 and previous config saved to /var/cache/conftool/dbconfig/20240903-020730-ladsgroup.json
• 02:07 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2125.codfw.wmnet with reason: Maintenance
• 02:07 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2125.codfw.wmnet with reason: Maintenance
• 01:20 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 01:20 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 01:20 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246 (T371742)', diff saved to https://phabricator.wikimedia.org/P68567 and previous config saved to /var/cache/conftool/dbconfig/20240903-012013-ladsgroup.json
• 01:05 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246', diff saved to https://phabricator.wikimedia.org/P68566 and previous config saved to /var/cache/conftool/dbconfig/20240903-010506-ladsgroup.json
• 00:49 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246', diff saved to https://phabricator.wikimedia.org/P68565 and previous config saved to /var/cache/conftool/dbconfig/20240903-004959-ladsgroup.json
• 00:34 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246 (T371742)', diff saved to https://phabricator.wikimedia.org/P68564 and previous config saved to /var/cache/conftool/dbconfig/20240903-003452-ladsgroup.json

2024-09-02

• 23:22 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1246 (T371742)', diff saved to https://phabricator.wikimedia.org/P68563 and previous config saved to /var/cache/conftool/dbconfig/20240902-232222-ladsgroup.json
• 23:22 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1246.eqiad.wmnet with reason: Maintenance
• 23:22 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1246.eqiad.wmnet with reason: Maintenance
• 22:33 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 22:33 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 22:33 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T371742)', diff saved to https://phabricator.wikimedia.org/P68562 and previous config saved to /var/cache/conftool/dbconfig/20240902-223324-ladsgroup.json
• 22:18 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P68561 and previous config saved to /var/cache/conftool/dbconfig/20240902-221817-ladsgroup.json
• 22:03 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P68560 and previous config saved to /var/cache/conftool/dbconfig/20240902-220310-ladsgroup.json
• 21:48 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T371742)', diff saved to https://phabricator.wikimedia.org/P68559 and previous config saved to /var/cache/conftool/dbconfig/20240902-214802-ladsgroup.json
• 21:02 catrope@deploy1003: Finished scap sync-world: Backport for CodexModule: Fix double-flipping in RTL (T373676) (duration: 11m 31s)
• 20:58 catrope@deploy1003: catrope: Continuing with sync
• 20:55 catrope@deploy1003: catrope: Backport for CodexModule: Fix double-flipping in RTL (T373676) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1233 (T371742)', diff saved to https://phabricator.wikimedia.org/P68558 and previous config saved to /var/cache/conftool/dbconfig/20240902-205149-ladsgroup.json
• 20:51 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1233.eqiad.wmnet with reason: Maintenance
• 20:51 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1233.eqiad.wmnet with reason: Maintenance
• 20:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T371742)', diff saved to https://phabricator.wikimedia.org/P68557 and previous config saved to /var/cache/conftool/dbconfig/20240902-205128-ladsgroup.json
• 20:51 catrope@deploy1003: Started scap sync-world: Backport for CodexModule: Fix double-flipping in RTL (T373676)
• 20:36 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P68556 and previous config saved to /var/cache/conftool/dbconfig/20240902-203620-ladsgroup.json
• 20:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P68555 and previous config saved to /var/cache/conftool/dbconfig/20240902-202113-ladsgroup.json
• 20:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T371742)', diff saved to https://phabricator.wikimedia.org/P68554 and previous config saved to /var/cache/conftool/dbconfig/20240902-200606-ladsgroup.json
• 19:45 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1229 (T371742)', diff saved to https://phabricator.wikimedia.org/P68553 and previous config saved to /var/cache/conftool/dbconfig/20240902-194545-ladsgroup.json
• 19:45 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1229.eqiad.wmnet with reason: Maintenance
• 19:45 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1229.eqiad.wmnet with reason: Maintenance
• 19:42 aqu@deploy1003: Finished deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow (duration: 00m 09s)
• 19:42 aqu@deploy1003: Started deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow
• 19:35 aqu@deploy1003: Finished deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow (duration: 00m 09s)
• 19:34 aqu@deploy1003: Started deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow
• 19:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 19:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 18:59 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 18:59 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 18:59 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T371742)', diff saved to https://phabricator.wikimedia.org/P68552 and previous config saved to /var/cache/conftool/dbconfig/20240902-185906-ladsgroup.json
• 18:46 aqu@deploy1003: Finished deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow (duration: 00m 09s)
• 18:45 aqu@deploy1003: Started deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow
• 18:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P68551 and previous config saved to /var/cache/conftool/dbconfig/20240902-184359-ladsgroup.json
• 18:28 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P68550 and previous config saved to /var/cache/conftool/dbconfig/20240902-182852-ladsgroup.json
• 18:18 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 18:18 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 18:13 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T371742)', diff saved to https://phabricator.wikimedia.org/P68549 and previous config saved to /var/cache/conftool/dbconfig/20240902-181345-ladsgroup.json
• 17:53 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1206.eqiad.wmnet with reason: Dumps causing issues
• 17:52 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db1206.eqiad.wmnet with reason: Dumps causing issues
• 17:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1197 (T371742)', diff saved to https://phabricator.wikimedia.org/P68548 and previous config saved to /var/cache/conftool/dbconfig/20240902-175238-ladsgroup.json
• 17:52 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1197.eqiad.wmnet with reason: Maintenance
• 17:52 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1197.eqiad.wmnet with reason: Maintenance
• 17:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T371742)', diff saved to https://phabricator.wikimedia.org/P68547 and previous config saved to /var/cache/conftool/dbconfig/20240902-175216-ladsgroup.json
• 17:37 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P68545 and previous config saved to /var/cache/conftool/dbconfig/20240902-173709-ladsgroup.json
• 17:22 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P68544 and previous config saved to /var/cache/conftool/dbconfig/20240902-172202-ladsgroup.json
• 17:21 hnowlan: homer 'cr*codfw*' commit
• 17:20 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2071.codfw.wmnet
• 17:20 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2071.codfw.wmnet
• 17:19 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2070.codfw.wmnet
• 17:19 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2070.codfw.wmnet
• 17:19 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2069.codfw.wmnet
• 17:19 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2069.codfw.wmnet
• 17:19 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2068.codfw.wmnet
• 17:19 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2068.codfw.wmnet
• 17:19 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2068.codfw.wmnet with OS bullseye
• 17:16 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2070.codfw.wmnet with OS bullseye
• 17:15 hnowlan: homer 'lsw1-a3-codfw*' commit
• 17:14 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2071.codfw.wmnet with OS bullseye
• 17:14 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2195 (T367856)', diff saved to https://phabricator.wikimedia.org/P68543 and previous config saved to /var/cache/conftool/dbconfig/20240902-171418-marostegui.json
• 17:14 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 7:00:00 on db2195.codfw.wmnet with reason: Maintenance
• 17:14 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 7:00:00 on db2195.codfw.wmnet with reason: Maintenance
• 17:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T367856)', diff saved to https://phabricator.wikimedia.org/P68542 and previous config saved to /var/cache/conftool/dbconfig/20240902-171356-marostegui.json
• 17:13 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2069.codfw.wmnet with OS bullseye
• 17:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T371742)', diff saved to https://phabricator.wikimedia.org/P68541 and previous config saved to /var/cache/conftool/dbconfig/20240902-170654-ladsgroup.json
• 16:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P68540 and previous config saved to /var/cache/conftool/dbconfig/20240902-165848-marostegui.json
• 16:45 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1188 (T371742)', diff saved to https://phabricator.wikimedia.org/P68539 and previous config saved to /var/cache/conftool/dbconfig/20240902-164530-ladsgroup.json
• 16:45 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1188.eqiad.wmnet with reason: Maintenance
• 16:45 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1188.eqiad.wmnet with reason: Maintenance
• 16:45 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T371742)', diff saved to https://phabricator.wikimedia.org/P68538 and previous config saved to /var/cache/conftool/dbconfig/20240902-164508-ladsgroup.json
• 16:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P68537 and previous config saved to /var/cache/conftool/dbconfig/20240902-164343-marostegui.json
• 16:33 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2071.codfw.wmnet with reason: host reimage
• 16:31 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2071.codfw.wmnet with reason: host reimage
• 16:30 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P68536 and previous config saved to /var/cache/conftool/dbconfig/20240902-163001-ladsgroup.json
• 16:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T367856)', diff saved to https://phabricator.wikimedia.org/P68535 and previous config saved to /var/cache/conftool/dbconfig/20240902-162836-marostegui.json
• 16:15 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2071
• 16:15 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2071
• 16:14 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P68534 and previous config saved to /var/cache/conftool/dbconfig/20240902-161454-ladsgroup.json
• 16:14 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2071
• 16:14 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2071.codfw.wmnet 52.0.192.10.in-addr.arpa 2.5.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 16:14 hnowlan@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2071.codfw.wmnet 52.0.192.10.in-addr.arpa 2.5.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 16:14 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:14 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2071 - hnowlan@cumin1002"
• 16:14 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2071 - hnowlan@cumin1002"
• 16:11 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 16:08 hnowlan@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2071
• 16:08 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2071.codfw.wmnet with OS bullseye
• 15:59 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T371742)', diff saved to https://phabricator.wikimedia.org/P68533 and previous config saved to /var/cache/conftool/dbconfig/20240902-155947-ladsgroup.json
• 15:55 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2068.codfw.wmnet with reason: host reimage
• 15:51 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2068.codfw.wmnet with reason: host reimage
• 15:51 elukey: spicerack 8.12.0 installed on cumin2002
• 15:50 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2069.codfw.wmnet with reason: host reimage
• 15:50 elukey: uploaded spicerack_8.12.0 to apt.wikimedia.org bullseye-wikimedia
• 15:46 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2069.codfw.wmnet with reason: host reimage
• 15:44 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2070.codfw.wmnet with reason: host reimage
• 15:40 hnowlan@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2070.codfw.wmnet with reason: host reimage
• 15:39 claime: Enabling puppet on O:cache::text for 1070032 - T364400
• 15:36 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2068
• 15:36 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2068
• 15:19 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 15:18 hnowlan@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2070
• 15:18 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2068.codfw.wmnet with OS bullseye
• 15:18 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2069.codfw.wmnet with OS bullseye
• 15:18 hnowlan@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2070.codfw.wmnet with OS bullseye
• 15:12 jayme: running homer 'cr*codfw*' commit 'T372878'
• 15:11 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2067.codfw.wmnet
• 15:11 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2067.codfw.wmnet
• 15:11 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2066.codfw.wmnet
• 15:11 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2066.codfw.wmnet
• 15:09 jayme: running homer 'lsw1-a6-codfw*' commit 'T372878'
• 15:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:04 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:04 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:04 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:03 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:02 claime: Enabling puppet on cp2027.codfw.wmnet to test 1070032 - T364400
• 14:58 claime: Disabling puppet on O:cache::text to merge 1070032 - T364400
• 14:47 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1182 (T371742)', diff saved to https://phabricator.wikimedia.org/P68531 and previous config saved to /var/cache/conftool/dbconfig/20240902-144751-ladsgroup.json
• 14:47 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2066.codfw.wmnet with OS bullseye
• 14:47 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1182.eqiad.wmnet with reason: Maintenance
• 14:47 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1182.eqiad.wmnet with reason: Maintenance
• 14:47 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T371742)', diff saved to https://phabricator.wikimedia.org/P68530 and previous config saved to /var/cache/conftool/dbconfig/20240902-144729-ladsgroup.json
• 14:45 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 14:44 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 14:32 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P68529 and previous config saved to /var/cache/conftool/dbconfig/20240902-143222-ladsgroup.json
• 14:22 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) check for host wikikube-ctrl2003.codfw.wmnet
• 14:22 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node check for host wikikube-ctrl2003.codfw.wmnet
• 14:22 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2001.codfw.wmnet
• 14:22 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2001.codfw.wmnet
• 14:17 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P68528 and previous config saved to /var/cache/conftool/dbconfig/20240902-141715-ladsgroup.json
• 14:04 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2066.codfw.wmnet with reason: host reimage
• 14:02 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T371742)', diff saved to https://phabricator.wikimedia.org/P68527 and previous config saved to /var/cache/conftool/dbconfig/20240902-140208-ladsgroup.json
• 14:00 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2066.codfw.wmnet with reason: host reimage
• 14:00 aqu@deploy1003: Finished deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow (duration: 00m 10s)
• 14:00 aqu@deploy1003: Started deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow
• 13:51 aqu@deploy1003: Finished deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow (duration: 00m 10s)
• 13:51 aqu@deploy1003: Started deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow
• 13:43 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2066.codfw.wmnet with OS bullseye
• 13:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1162 (T371742)', diff saved to https://phabricator.wikimedia.org/P68525 and previous config saved to /var/cache/conftool/dbconfig/20240902-133950-ladsgroup.json
• 13:39 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1162.eqiad.wmnet with reason: Maintenance
• 13:39 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1162.eqiad.wmnet with reason: Maintenance
• 13:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T371742)', diff saved to https://phabricator.wikimedia.org/P68524 and previous config saved to /var/cache/conftool/dbconfig/20240902-133928-ladsgroup.json
• 13:24 TheresNoTime: done UTC afternoon backport window
• 13:24 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P68523 and previous config saved to /var/cache/conftool/dbconfig/20240902-132421-ladsgroup.json
• 13:21 samtar@deploy1003: Finished scap sync-world: Backport for IS: Add CommunityRequests to InitialiseSettings (T372527) (duration: 06m 43s)
• 13:16 samtar@deploy1003: samtar: Continuing with sync
• 13:16 samtar@deploy1003: samtar: Backport for IS: Add CommunityRequests to InitialiseSettings (T372527) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:14 samtar@deploy1003: Started scap sync-world: Backport for IS: Add CommunityRequests to InitialiseSettings (T372527)
• 13:09 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P68522 and previous config saved to /var/cache/conftool/dbconfig/20240902-130914-ladsgroup.json
• 13:07 dreamyjazz@deploy1003: Finished scap sync-world: Backport for Remove wgCheckUserPurgeOldClientHintsData (T359560) (duration: 08m 47s)
• 13:06 TheresNoTime: `[samtar@mwmaint1002 ~]$ mwscript maintenance/cleanupTitles.php --wiki=ptwiki --prefix=T195546 2>&1 | tee ~/T195546-ptwiki.log` for T195546
• 13:01 dreamyjazz@deploy1003: dreamyjazz: Continuing with sync
• 13:01 dreamyjazz@deploy1003: dreamyjazz: Backport for Remove wgCheckUserPurgeOldClientHintsData (T359560) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 12:58 topranks: apply qos classifers and schedulers to server interfaces on asw-d-codfw T339850
• 12:58 dreamyjazz@deploy1003: Started scap sync-world: Backport for Remove wgCheckUserPurgeOldClientHintsData (T359560)
• 12:56 Dreamy_Jazz: Restarted MediaModeration scanning script - https://wikitech.wikimedia.org/wiki/MediaModeration
• 12:54 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T371742)', diff saved to https://phabricator.wikimedia.org/P68520 and previous config saved to /var/cache/conftool/dbconfig/20240902-125406-ladsgroup.json
• 12:51 ladsgroup@deploy1003: Finished scap sync-world: Backport for Remove the "powered by mediawiki" override (duration: 09m 05s)
• 12:49 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1206.eqiad.wmnet with reason: dump replag
• 12:49 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1206.eqiad.wmnet with reason: dump replag
• 12:47 ladsgroup@deploy1003: ladsgroup: Continuing with sync
• 12:44 ladsgroup@deploy1003: ladsgroup: Backport for Remove the "powered by mediawiki" override synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 12:42 ladsgroup@deploy1003: Started scap sync-world: Backport for Remove the "powered by mediawiki" override
• 12:38 ladsgroup@deploy1003: Finished scap sync-world: Backport for Enable dark mode for Creator: namespace in Commons (duration: 09m 34s)
• 12:34 ladsgroup@deploy1003: ladsgroup, ebrahim: Continuing with sync
• 12:32 klausman@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1009.eqiad.wmnet
• 12:30 ladsgroup@deploy1003: ladsgroup, ebrahim: Backport for Enable dark mode for Creator: namespace in Commons synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 12:28 ladsgroup@deploy1003: Started scap sync-world: Backport for Enable dark mode for Creator: namespace in Commons
• 12:28 klausman@cumin1002: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1009.eqiad.wmnet
• 12:24 godog: enable oidc for prometheus public web interface - T326657
• 12:23 klausman@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2011.codfw.wmnet
• 12:22 samtar@deploy1003: Finished scap sync-world: Backport for Add CommunityRequests (T372527) (duration: 28m 14s)
• 12:21 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2066.codfw.wmnet with OS bullseye
• 12:19 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw[2271-2277].codfw.wmnet
• 12:19 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:19 cgoubert@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[2271-2277].codfw.wmnet decommissioned, removing all IPs except the asset tag one - cgoubert@cumin1002"
• 12:19 cgoubert@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[2271-2277].codfw.wmnet decommissioned, removing all IPs except the asset tag one - cgoubert@cumin1002"
• 12:17 klausman@cumin1002: START - Cookbook sre.hosts.reboot-single for host ml-serve2011.codfw.wmnet
• 12:17 klausman@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2009.codfw.wmnet
• 12:14 samtar@deploy1003: samtar: Continuing with sync
• 12:13 samtar@deploy1003: samtar: Backport for Add CommunityRequests (T372527) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 12:10 klausman@cumin1002: START - Cookbook sre.hosts.reboot-single for host ml-serve2009.codfw.wmnet
• 12:06 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 12:05 klausman@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1011.eqiad.wmnet
• 12:04 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2386 to wikikube-worker2068
• 12:04 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2068
• 12:03 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2068
• 12:03 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:03 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2386 to wikikube-worker2068 - hnowlan@cumin1002"
• 12:03 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2067.codfw.wmnet with OS bullseye
• 12:01 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2386 to wikikube-worker2068 - hnowlan@cumin1002"
• 11:59 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1156 (T371742)', diff saved to https://phabricator.wikimedia.org/P68519 and previous config saved to /var/cache/conftool/dbconfig/20240902-115944-ladsgroup.json
• 11:59 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 11:59 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 11:59 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1156.eqiad.wmnet with reason: Maintenance
• 11:59 klausman@cumin1002: START - Cookbook sre.hosts.reboot-single for host ml-serve1011.eqiad.wmnet
• 11:58 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1156.eqiad.wmnet with reason: Maintenance
• 11:57 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 11:56 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from mw2386 to wikikube-worker2068
• 11:55 klausman@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1010.eqiad.wmnet
• 11:54 samtar@deploy1003: Started scap sync-world: Backport for Add CommunityRequests (T372527)
• 11:50 klausman@cumin1002: START - Cookbook sre.hosts.reboot-single for host ml-serve1010.eqiad.wmnet
• 11:49 klausman@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1009.eqiad.wmnet
• 11:43 klausman@cumin1002: START - Cookbook sre.hosts.reboot-single for host ml-serve1009.eqiad.wmnet
• 11:43 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2067.codfw.wmnet with reason: host reimage
• 11:41 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2067.codfw.wmnet with reason: host reimage
• 11:38 hnowlan@cumin1002: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from mw2386 to wikikube-worker2068
• 11:38 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from mw2386 to wikikube-worker2068
• 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 11:36 cgoubert@cumin1002: START - Cookbook sre.hosts.decommission for hosts mw[2271-2277].codfw.wmnet
• 11:36 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 11:35 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw[2261-2262,2268-2270].codfw.wmnet
• 11:35 cgoubert@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:32 cgoubert@cumin1002: START - Cookbook sre.dns.netbox
• 11:32 hnowlan@cumin1002: END (FAIL) - Cookbook sre.hosts.rename (exit_code=99) from mw2386 to wikikube-worker2068
• 11:32 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:31 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2389 to wikikube-worker2071
• 11:30 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2388 to wikikube-worker2070
• 11:30 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2071
• 11:30 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 11:30 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2071
• 11:30 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:30 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2389 to wikikube-worker2071 - hnowlan@cumin1002"
• 11:30 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2389 to wikikube-worker2071 - hnowlan@cumin1002"
• 11:30 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2070
• 11:30 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2070
• 11:29 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:29 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2388 to wikikube-worker2070 - hnowlan@cumin1002"
• 11:28 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mw2386
• 11:27 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2388 to wikikube-worker2070 - hnowlan@cumin1002"
• 11:25 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 11:23 hnowlan@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2387 to wikikube-worker2069
• 11:22 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2069
• 11:22 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2069
• 11:22 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:22 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2387 to wikikube-worker2069 - hnowlan@cumin1002"
• 11:22 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2067.codfw.wmnet with OS bullseye
• 11:21 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host mw2386
• 11:21 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2067.codfw.wmnet with OS bullseye
• 11:20 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2387 to wikikube-worker2069 - hnowlan@cumin1002"
• 11:20 hnowlan@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 11:20 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 11:20 hnowlan@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 11:20 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2067.codfw.wmnet with OS bullseye
• 11:20 hnowlan@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2068
• 11:19 hnowlan@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2068
• 11:19 hnowlan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:19 hnowlan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2386 to wikikube-worker2068 - hnowlan@cumin1002"
• 11:19 cgoubert@cumin1002: START - Cookbook sre.hosts.decommission for hosts mw[2261-2262,2268-2270].codfw.wmnet
• 11:13 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 11:11 claime: Manual puppet node deactivate for mw2295 mw2296 mw2377 mw2378 mw2385 - T372878
• 11:08 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 11:03 hnowlan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2386 to wikikube-worker2068 - hnowlan@cumin1002"
• 11:02 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 11:02 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2066.codfw.wmnet with OS bullseye
• 11:02 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from mw2389 to wikikube-worker2071
• 11:02 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from mw2388 to wikikube-worker2070
• 11:01 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2066.codfw.wmnet with OS bullseye
• 11:01 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2066.codfw.wmnet with OS bullseye
• 10:59 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from mw2387 to wikikube-worker2069
• 10:57 hnowlan@cumin1002: START - Cookbook sre.dns.netbox
• 10:57 hnowlan@cumin1002: START - Cookbook sre.hosts.rename from mw2386 to wikikube-worker2068
• 10:50 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2389.codfw.wmnet
• 10:49 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2389.codfw.wmnet
• 10:44 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2388.codfw.wmnet
• 10:44 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2388.codfw.wmnet
• 10:43 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2387.codfw.wmnet
• 10:42 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2387.codfw.wmnet
• 10:34 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:34 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:31 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:29 hnowlan@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw2386.codfw.wmnet
• 10:28 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:27 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:26 hnowlan@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2386.codfw.wmnet
• 10:25 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:21 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2067.codfw.wmnet with OS bullseye
• 10:15 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1008.eqiad.wmnet
• 10:14 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1011.eqiad.wmnet
• 10:13 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1010.eqiad.wmnet
• 10:12 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1009.eqiad.wmnet
• 10:08 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on an-worker1127.eqiad.wmnet with reason: Cold booting due to RAID controller battery issue
• 10:08 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on an-worker1127.eqiad.wmnet with reason: Cold booting due to RAID controller battery issue
• 10:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply
• 10:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply
• 10:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply
• 10:06 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2066.codfw.wmnet with OS bullseye
• 10:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply
• 10:04 btullis@cumin1002: START - Cookbook sre.hosts.reboot-single for host stat1011.eqiad.wmnet
• 10:04 btullis@cumin1002: START - Cookbook sre.hosts.reboot-single for host stat1010.eqiad.wmnet
• 10:04 btullis@cumin1002: START - Cookbook sre.hosts.reboot-single for host stat1009.eqiad.wmnet
• 10:04 btullis@cumin1002: START - Cookbook sre.hosts.reboot-single for host stat1008.eqiad.wmnet
• 10:03 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 10:03 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 10:01 isaranto@deploy1003: helmfile [eqiad] DONE helmfile.d/services/api-gateway: sync
• 10:00 isaranto@deploy1003: helmfile [eqiad] START helmfile.d/services/api-gateway: sync
• 10:00 isaranto@deploy1003: helmfile [codfw] DONE helmfile.d/services/api-gateway: sync
• 09:59 isaranto@deploy1003: helmfile [codfw] START helmfile.d/services/api-gateway: sync
• 09:55 isaranto@deploy1003: helmfile [staging] DONE helmfile.d/services/api-gateway: sync
• 09:55 isaranto@deploy1003: helmfile [staging] START helmfile.d/services/api-gateway: sync
• 09:54 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:53 stevemunene@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 09:30 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2067
• 09:30 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2067
• 09:30 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2067
• 09:30 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2067.codfw.wmnet 88.0.192.10.in-addr.arpa 8.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 09:30 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2067.codfw.wmnet 88.0.192.10.in-addr.arpa 8.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 09:30 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:30 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2067 - jayme@cumin1002"
• 09:30 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2067 - jayme@cumin1002"
• 09:21 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 09:18 arnaudb@cumin1002: dbctl commit (dc=all): 'db1206 (re)pooling @ 100%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68517 and previous config saved to /var/cache/conftool/dbconfig/20240902-091844-arnaudb.json
• 09:17 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2067
• 09:16 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2066
• 09:16 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2066
• 09:16 zabe@deploy1003: Finished scap sync-world: Backport for Do not log failed autocreations on closed wikis as diagnostic errors (T373650) (duration: 06m 34s)
• 09:14 elukey: update netboot images for Bullseye and Bookworm point releases (11.11 and 12.7) following https://wikitech.wikimedia.org/wiki/SRE/Infrastructure_Foundations/Debian-installer
• 09:12 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2066
• 09:12 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2066.codfw.wmnet 197.0.192.10.in-addr.arpa 7.9.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 09:12 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2066.codfw.wmnet 197.0.192.10.in-addr.arpa 7.9.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 09:12 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:12 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2066 - jayme@cumin1002"
• 09:12 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2066 - jayme@cumin1002"
• 09:11 zabe@deploy1003: zabe: Continuing with sync
• 09:11 zabe@deploy1003: zabe: Backport for Do not log failed autocreations on closed wikis as diagnostic errors (T373650) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 09:09 zabe@deploy1003: Started scap sync-world: Backport for Do not log failed autocreations on closed wikis as diagnostic errors (T373650)
• 09:03 arnaudb@cumin1002: dbctl commit (dc=all): 'db1206 (re)pooling @ 75%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68516 and previous config saved to /var/cache/conftool/dbconfig/20240902-090339-arnaudb.json
• 09:03 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2067.codfw.wmnet with OS bullseye
• 09:02 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 09:02 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2066
• 09:02 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2066.codfw.wmnet with OS bullseye
• 09:01 elukey: restart puppetserver on puppetserver1002 to pick up new JVM settings - T373527
• 08:50 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2027 to wikikube-worker2067
• 08:49 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2067
• 08:49 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2008 to wikikube-worker2066
• 08:49 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2067
• 08:49 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:48 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2066
• 08:48 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2066
• 08:48 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:48 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2008 to wikikube-worker2066 - jayme@cumin1002"
• 08:48 arnaudb@cumin1002: dbctl commit (dc=all): 'db1206 (re)pooling @ 50%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68515 and previous config saved to /var/cache/conftool/dbconfig/20240902-084833-arnaudb.json
• 08:48 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2008 to wikikube-worker2066 - jayme@cumin1002"
• 08:47 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 08:41 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 08:41 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2027 to wikikube-worker2067
• 08:41 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2008 to wikikube-worker2066
• 08:33 arnaudb@cumin1002: dbctl commit (dc=all): 'db1206 (re)pooling @ 25%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68514 and previous config saved to /var/cache/conftool/dbconfig/20240902-083328-arnaudb.json
• 08:21 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2027.codfw.wmnet
• 08:18 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2027.codfw.wmnet
• 08:18 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes2008.codfw.wmnet
• 08:18 arnaudb@cumin1002: dbctl commit (dc=all): 'db1206 (re)pooling @ 15%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68513 and previous config saved to /var/cache/conftool/dbconfig/20240902-081822-arnaudb.json
• 08:17 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes2008.codfw.wmnet
• 08:09 moritzm: installing Linux 6.1.106 on Bookworm hosts
• 08:03 arnaudb@cumin1002: dbctl commit (dc=all): 'db1206 (re)pooling @ 6%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68512 and previous config saved to /var/cache/conftool/dbconfig/20240902-080317-arnaudb.json
• 07:51 Emperor: restart swift-proxy on ms-fe2012 T360913
• 07:48 arnaudb@cumin1002: dbctl commit (dc=all): 'db1206 (re)pooling @ 5%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68511 and previous config saved to /var/cache/conftool/dbconfig/20240902-074811-arnaudb.json
• 07:43 kartik@deploy1003: Finished scap sync-world: Backport for Enable Section Translation in bdr, btm, and dtp Wikpedias (T371420) (duration: 07m 35s)
• 07:39 kartik@deploy1003: kartik: Continuing with sync
• 07:38 kartik@deploy1003: kartik: Backport for Enable Section Translation in bdr, btm, and dtp Wikpedias (T371420) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:36 kartik@deploy1003: Started scap sync-world: Backport for Enable Section Translation in bdr, btm, and dtp Wikpedias (T371420)
• 07:33 arnaudb@cumin1002: dbctl commit (dc=all): 'db1206 (re)pooling @ 4%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68510 and previous config saved to /var/cache/conftool/dbconfig/20240902-073306-arnaudb.json
• 07:32 kartik@deploy1003: Finished scap sync-world: Backport for Enable EditCheck references on plwiki (T373079) (duration: 28m 41s)
• 07:22 kartik@deploy1003: msz2001, kartik: Continuing with sync
• 07:19 kartik@deploy1003: msz2001, kartik: Backport for Enable EditCheck references on plwiki (T373079) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 07:18 arnaudb@cumin1002: dbctl commit (dc=all): 'db1206 (re)pooling @ 3%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68509 and previous config saved to /var/cache/conftool/dbconfig/20240902-071800-arnaudb.json
• 07:04 kartik@deploy1003: Started scap sync-world: Backport for Enable EditCheck references on plwiki (T373079)
• 07:02 arnaudb@cumin1002: dbctl commit (dc=all): 'db1206 (re)pooling @ 2%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68508 and previous config saved to /var/cache/conftool/dbconfig/20240902-070255-arnaudb.json
• 06:47 arnaudb@cumin1002: dbctl commit (dc=all): 'db1206 (re)pooling @ 1%: post replag depool', diff saved to https://phabricator.wikimedia.org/P68507 and previous config saved to /var/cache/conftool/dbconfig/20240902-064749-arnaudb.json
• 06:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1206.eqiad.wmnet with reason: replag
• 06:35 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1206.eqiad.wmnet with reason: replag
• 06:34 arnaudb@cumin1002: dbctl commit (dc=all): 'depool db1206 replag', diff saved to https://phabricator.wikimedia.org/P68506 and previous config saved to /var/cache/conftool/dbconfig/20240902-063432-arnaudb.json

2024-09-01

• 16:50 oblivian@cumin1002: dbctl commit (dc=all): 'db1221 (re)pooling @ 100%: Repooling after resolving replag', diff saved to https://phabricator.wikimedia.org/P68505 and previous config saved to /var/cache/conftool/dbconfig/20240901-165009-root.json
• 16:35 oblivian@cumin1002: dbctl commit (dc=all): 'db1221 (re)pooling @ 75%: Repooling after resolving replag', diff saved to https://phabricator.wikimedia.org/P68504 and previous config saved to /var/cache/conftool/dbconfig/20240901-163504-root.json
• 16:19 oblivian@cumin1002: dbctl commit (dc=all): 'db1221 (re)pooling @ 50%: Repooling after resolving replag', diff saved to https://phabricator.wikimedia.org/P68503 and previous config saved to /var/cache/conftool/dbconfig/20240901-161958-root.json
• 16:04 oblivian@cumin1002: dbctl commit (dc=all): 'db1221 (re)pooling @ 25%: Repooling after resolving replag', diff saved to https://phabricator.wikimedia.org/P68502 and previous config saved to /var/cache/conftool/dbconfig/20240901-160453-root.json
• 15:50 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1221.eqiad.wmnet
• 15:50 mvernon@cumin2002: START - Cookbook sre.hosts.remove-downtime for db1221.eqiad.wmnet
• 15:49 oblivian@cumin1002: dbctl commit (dc=all): 'db1221 (re)pooling @ 10%: Repooling after resolving replag', diff saved to https://phabricator.wikimedia.org/P68501 and previous config saved to /var/cache/conftool/dbconfig/20240901-154948-root.json
• 15:25 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db1221.eqiad.wmnet with reason: replag
• 15:24 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 16:00:00 on db1221.eqiad.wmnet with reason: replag
• 15:15 oblivian@cumin1002: dbctl commit (dc=all): 'depool db1221, lag', diff saved to https://phabricator.wikimedia.org/P68499 and previous config saved to /var/cache/conftool/dbconfig/20240901-151530-oblivian.json

Other archives

2000s

• Archive 1: 2004 Jun - 2004 Sep
• Archive 2: 2004 Oct - 2004 Nov
• Archive 3: 2004 Dec - 2005 Mar
• Archive 4: 2005 Apr - 2005 Jul
• Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
• Archive 6: 2005 Nov - 2006 Feb
• Archive 7: 2006 Mar - 2006 Jun
• Archive 8: 2006 Jul - 2006 Sep
• Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
• Archive 10: 2007 Feb - 2007 Jun
• Archive 11: 2007 Jul - 2007 Dec
• Archive 12: 2008 Jan - 2008 Jul
• Archive 12a: 2008 Aug
• Archive 12b: 2008 Sept
• Archive 13: 2008 Oct - 2009 Jun
• Archive 14: 2009 Jun - 2009 Dec

2010s

• Archive 15: 2010 Jan - 2010 Jun
• Archive 16: 2010 Jul - 2010 Oct
• Archive 17: 2010 Nov - 2010 Dec
• Archive 18: 2011 Jan - 2011 Jun
• Archive 19: 2011 Jul - 2011 Dec
• Archive 20: 2011 Dec - 2012 Jun, with revision history 2007-02-21 to 2012-03-27
• Archive 21: 2012 Jul - 2013 Jan
• Archive 22: 2013 Jan - 2013 Jul
• Archive 23: 2013 Aug - 2013 Dec
• Archive 24: 2014 Jan - 2014 Mar
• Archive 25: 2014 April - 2014 September
• Archive 26: 2014 October - 2014 December
• Archive 27: 2015 January - 2015 July
• Archive 28: 2015 August - 2015 December
• Archive 29: 2016 January - 2016 May
• Archive 30: 2016 June - 2016 August
• Archive 31: 2016 September - 2016 December
• Archive 32: 2017 January - 2017 July
• Archive 33: 2017 August - 2017 December
• Archive 34: 2018 January - 2018 April
• Archive 35: 2018 May - 2018 August
• Archive 36: 2018 September - 2018 December
• Archive 37: 2019 January - 2019 April
• Archive 38: 2019 May - 2019 August
• Archive 39: 2019 September - 2019 December

2020s

• Archive 40: 2020 January - 2020 April
• Archive 41: 2020 May - 2020 July
• Archive 42: 2020 August - 2020 November
• Archive 43: 2020 December
• Archive 44: 2021 January - 2021 April
• Archive 45: 2021 May - 2021 July
• Archive 46: 2021 August - 2021 October
• Archive 47: 2021 November - 2021 December
• Archive 48: 2022 January
• Archive 49: 2022 February
• Archive 50: 2022 March
• Archive 51: 2022 April 1-15
• Archive 52: 2022 April 16-30
• Archive 53: 2022 May
• Archive 54: 2022 June
• Archive 55: 2022 July
• Archive 56: 2022 August
• Archive 57: 2022 September
• Archive 58: 2022 October
• Archive 59: 2022 November 1-15
• Archive 60: 2022 November 16-30
• Archive 61: 2022 December
• Archive 62: 2023 January
• Archive 63: 2023 February
• Archive 64: 2023 March
• Archive 65: 2023 April
• Archive 66: 2023 May
• Archive 67: 2023 June
• Archive 68: 2023 July
• Archive 69: 2023 August 1-15
• Archive 70: 2023 August 16-31
• Archive 71: 2023 September
• Archive 72: 2023 October
• Archive 73: 2023 November
• Archive 74: 2023 December
• Archive 75: 2024 January
• Archive 76: 2024 February
• Archive 77: 2024 March
• Archive 78: 2024 April
• Archive 79: 2024 May 1-15
• Archive 80: 2024 May 16-31
• Archive 81: 2024 June 1-15
• Archive 82: 2024 June 16-30
• Archive 83: 2024 July
• Archive 84: 2024 August
• Archive 85: 2024 September
• Archive 86: 2024 October