18:42 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: test operations in mixed opensearch/elasticsearch cluster - bking@cumin2002 - T380752:
18:42 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: test operations in mixed opensearch/elasticsearch cluster - bking@cumin2002 - T380752:
17:10 rzl@deploy2002: Started scap sync-world: T385520
17:08 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
17:08 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
16:50 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
16:45 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
16:35 dancy@deploy2002: Installation of scap version "4.137.0" completed for 204 hosts
16:31 dancy@deploy2002: Installing scap version "4.137.0" for 204 host(s)
16:27 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
16:27 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
15:51 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: test operations in mixed opensearch/elasticsearch cluster - bking@cumin2002 - T380752
15:51 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: test operations in mixed opensearch/elasticsearch cluster - bking@cumin2002 - T380752
15:49 lucaswerkmeister-wmde@deploy2002: arthurtaylor, lucaswerkmeister-wmde: Continuing with sync
10:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1026.eqiad.wmnet
10:54 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on netflow1002.eqiad.wmnet with reason: keeping gnmic running in debug mode to observe performance change
10:34 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti1026.eqiad.wmnet with reason: remove from cluster for reimage
10:27 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow1002.eqiad.wmnet with reason: disabling gnmic in systemd
10:24 vgutierrez: restarting pybal on lvs2013, effectively enabling IPIP encapsulation for swift-fe@codfw - T385564
10:23 aborrero@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) virt.cloudgw.eqiad1.wikimediacloud.org on all recursors
10:22 aborrero@cumin1002: START - Cookbook sre.dns.wipe-cache virt.cloudgw.eqiad1.wikimediacloud.org on all recursors
10:22 urbanecm@deploy2002: urbanecm: Continuing with sync
07:05 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti1036.eqiad.wmnet with reason: remove from cluster for reimage
05:34 kart_: Updated cxserver to 2025-02-20-032928-production (T386677, T386464)
13:41 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts cloudgw1002.eqiad.wmnet
13:41 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:41 andrew@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudgw1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
13:41 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudgw1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
13:32 andrew@cumin1002: START - Cookbook sre.hosts.decommission for hosts cloudgw1002.eqiad.wmnet
13:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1025.eqiad.wmnet
13:31 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudgw1001.eqiad.wmnet
13:31 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:31 andrew@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudgw1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
13:30 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudgw1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
11:29 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti1025.eqiad.wmnet with reason: remove from cluster for reimage
11:28 fabfur@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp
11:27 ladsgroup@deploy2002: ladsgroup: Continuing with sync
08:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts moscovium.eqiad.wmnet
08:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
08:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: moscovium.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
08:46 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: moscovium.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
08:45 fabfur: upgrading haproxykafka to 0.3.5 on cp4037 to test new feature (T374128)
08:45 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and A:cp
08:19 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and A:cp
08:19 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and A:cp
08:15 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo and not P{cp4044.*} and A:cp
08:11 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_ulsfo and not P{cp4052.*} and A:cp
08:08 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D
08:04 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D
08:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1023.eqiad.wmnet to cluster eqiad and group A
08:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti1023.eqiad.wmnet to cluster eqiad and group A
07:59 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1023.eqiad.wmnet
07:54 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_ulsfo and not P{cp4044.*} and A:cp
07:53 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_ulsfo and not P{cp4052.*} and A:cp
07:51 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1023.eqiad.wmnet
07:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1033.eqiad.wmnet
07:39 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet
07:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1023.eqiad.wmnet with OS bookworm
07:34 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp4052.ulsfo.wmnet,cp4044.ulsfo.wmnet} and A:cp
07:29 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp4052.ulsfo.wmnet,cp4044.ulsfo.wmnet} and A:cp
07:24 vgutierrez: upload haproxy 2.8.14 to apt.wm.o (bullseye-wikimedia) - T386751
07:17 kart_: Updated MinT to 2025-02-05-115716-production (T383750, T385552)
07:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1023.eqiad.wmnet with reason: host reimage
07:12 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1023.eqiad.wmnet with reason: host reimage
19:12 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host db2243.codfw.wmnet with OS bookworm
19:10 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
19:01 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
19:00 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
19:00 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
18:59 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
18:59 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
18:14 eevans@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-eqiad: Apply JDK 11 update - eevans@cumin1002
18:11 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
18:11 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
18:05 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
18:05 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
18:04 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
17:36 mutante: LDAP/mwmaint1002: changed email address for LDAP user jonkolbert (T386473)
17:21 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
17:08 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
17:08 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
16:57 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
16:55 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
16:18 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
16:12 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
14:02 tappof@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on grafana1002.eqiad.wmnet with reason: expand the root partition and fs on grafana1002
12:41 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti1023.eqiad.wmnet with reason: remove from cluster for reimage
12:40 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti1033.eqiad.wmnet with reason: remove from cluster for reimage
10:50 tappof@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on grafana1002.eqiad.wmnet with reason: expand the root partition and fs on grafana1002
14:26 tappof@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on grafana2001.codfw.wmnet with reason: expand the root partition and fs on grafana2001
03:39 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on cr2-magru with reason: IBGP instability from cr1 to cr2 in magru causing ping faulures from alert1002
2025-02-14
18:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab
21:01 inflatador: bking@cephosd1001:~$ sudo radosgw-admin quota set --quota-scope=user --uid=research --max-size=4T
20:58 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:57 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:56 inflatador: bking@cephosd1001:~$ sudo radosgw-admin user create --uid=research --display-name="research"
20:51 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:50 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:49 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:49 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:36 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:36 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:33 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:33 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:32 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:32 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:31 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:31 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:24 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:24 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:23 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:23 xcollazo@deploy2002: Started deploy [airflow-dags/analytics@092b9d3]: deploy latest DAGs to analytics Airflow instance. T386114.
20:23 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:23 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:09 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:03 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:03 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:03 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
19:59 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
19:59 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
19:58 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
19:58 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
19:56 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
19:56 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
19:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2146 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73475 and previous config saved to /var/cache/conftool/dbconfig/20250213-195454-root.json
18:22 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1009.eqiad.wmnet with reason: host reimage
17:34 bking@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host relforge1005.eqiad.wmnet with OS bullseye
17:34 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1008.eqiad.wmnet with reason: host reimage
17:31 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1008.eqiad.wmnet with reason: host reimage
17:13 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1008.eqiad.wmnet with OS bookworm
17:01 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1007.eqiad.wmnet with OS bookworm
16:58 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
16:55 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1153.eqiad.wmnet with reason: maintenance
16:53 cmooney@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw2-d5-eqiad
16:53 cmooney@cumin1002: START - Cookbook sre.network.tls for network device cloudsw2-d5-eqiad
16:42 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1007.eqiad.wmnet with reason: host reimage
16:39 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1007.eqiad.wmnet with reason: host reimage
15:40 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
15:35 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
15:28 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
15:23 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
15:22 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde: Continuing with sync
15:17 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
15:15 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1006.eqiad.wmnet with OS bookworm
15:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1219.eqiad.wmnet with reason: Index rebuild
15:13 marostegui@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1219.eqiad.wmnet
15:12 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
15:12 marostegui@cumin1002: START - Cookbook sre.mysql.upgrade for db2146.codfw.wmnet
15:11 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2146', diff saved to https://phabricator.wikimedia.org/P73463 and previous config saved to /var/cache/conftool/dbconfig/20250213-151117-marostegui.json
15:10 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
15:07 marostegui@cumin1002: START - Cookbook sre.mysql.upgrade for db1219.eqiad.wmnet
15:07 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1219', diff saved to https://phabricator.wikimedia.org/P73462 and previous config saved to /var/cache/conftool/dbconfig/20250213-150715-marostegui.json
15:05 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
15:03 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic1104*,elastic1105*,elastic1106* for ban hosts prior to reimage/repurpose - bking@cumin2002 - T386357
15:03 bking@cumin2002: START - Cookbook sre.elasticsearch.ban Banning hosts: elastic1104*,elastic1105*,elastic1106* for ban hosts prior to reimage/repurpose - bking@cumin2002 - T386357
15:03 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
15:03 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
14:57 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1006.eqiad.wmnet with reason: host reimage
14:53 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1006.eqiad.wmnet with reason: host reimage
14:35 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1006.eqiad.wmnet with OS bookworm
14:16 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic1104*,elastic1005*,elastic1006* for ban hosts prior to reimage/repurpose - bking@cumin2002 - T386357
14:16 bking@cumin2002: START - Cookbook sre.elasticsearch.ban Banning hosts: elastic1104*,elastic1005*,elastic1006* for ban hosts prior to reimage/repurpose - bking@cumin2002 - T386357
14:06 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1005.eqiad.wmnet with OS bookworm
13:48 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1005.eqiad.wmnet with reason: host reimage
12:45 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1004.eqiad.wmnet with reason: host reimage
12:41 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1004.eqiad.wmnet with reason: host reimage
12:25 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1004.eqiad.wmnet with OS bookworm
12:18 stevemunene: draining dse-k8s-worker1004 ready for reimage to bookworm and containerd for T377875
12:04 fnegri@cumin1002: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database kncwiki (T385188)
11:38 fnegri@cumin1002: START - Cookbook sre.wikireplicas.add-wiki for database kncwiki (T385188)
11:08 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1003.eqiad.wmnet with OS bookworm
10:50 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1003.eqiad.wmnet with reason: host reimage
20:30 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:30 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
20:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2176 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73456 and previous config saved to /var/cache/conftool/dbconfig/20250212-201424-root.json
20:01 bking@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on relforge1004.eqiad.wmnet with reason: T380752
19:59 marostegui@cumin1002: dbctl commit (dc=all): 'db2176 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73455 and previous config saved to /var/cache/conftool/dbconfig/20250212-195919-root.json
19:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2176 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73454 and previous config saved to /var/cache/conftool/dbconfig/20250212-194414-root.json
19:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2176 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73453 and previous config saved to /var/cache/conftool/dbconfig/20250212-192909-root.json
19:27 marostegui@cumin1002: dbctl commit (dc=all): 'db1232 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73452 and previous config saved to /var/cache/conftool/dbconfig/20250212-192700-root.json
19:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2176 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73451 and previous config saved to /var/cache/conftool/dbconfig/20250212-191404-root.json
19:11 marostegui@cumin1002: dbctl commit (dc=all): 'db1232 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73450 and previous config saved to /var/cache/conftool/dbconfig/20250212-191155-root.json
18:56 marostegui@cumin1002: dbctl commit (dc=all): 'db1232 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73449 and previous config saved to /var/cache/conftool/dbconfig/20250212-185649-root.json
17:10 marostegui: Install 10.6.21 on db2230 T385678
17:08 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2230.codfw.wmnet with reason: maintenance
17:02 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1002.eqiad.wmnet with reason: host reimage
16:59 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1002.eqiad.wmnet with reason: host reimage
16:43 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1002.eqiad.wmnet with OS bookworm
16:34 stevemunene@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-worker1002.eqiad.wmnet with OS bookworm
16:09 claime: Deleting benthos, changeprop, changeprop-jobqueue from staging to free pod ip blocks - T386107
16:07 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cr2-magru with reason: IBGP instability from cr1 to cr2 in magru causing ping faulures from alert1002
15:37 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2176.codfw.wmnet with reason: Index rebuild
15:36 marostegui@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2176.codfw.wmnet
15:32 stevemunene@cumin1002: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1002.eqiad.wmnet with OS bookworm
15:31 stevemunene@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-worker1002.eqiad.wmnet with OS bookworm
15:30 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2176.codfw.wmnet with reason: maintenance
15:29 marostegui@cumin1002: START - Cookbook sre.mysql.upgrade for db2176.codfw.wmnet
15:27 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2176 T385561', diff saved to https://phabricator.wikimedia.org/P73446 and previous config saved to /var/cache/conftool/dbconfig/20250212-152738-marostegui.json
13:49 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1001.eqiad.wmnet with OS bookworm
13:40 aklapper@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.16 refs T382367
13:31 stevemunene@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1001.eqiad.wmnet with reason: host reimage
13:25 stevemunene@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1001.eqiad.wmnet with reason: host reimage
10:30 aklapper@deploy2002: rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.16 refs T382367
09:27 brouberol@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: test rolling-operation cookbook - brouberol@cumin2002
09:27 brouberol@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: test rolling-operation cookbook - brouberol@cumin2002
09:24 aklapper@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.16 refs T382367
09:18 brouberol@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: test rolling-operation cookbook - brouberol@cumin2002
09:18 brouberol@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: test rolling-operation cookbook - brouberol@cumin2002
09:16 brouberol@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: test rolling-operation cookbook - brouberol@cumin2002
09:16 brouberol@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: test rolling-operation cookbook - brouberol@cumin2002
08:49 dcausse: closing the UTC morning backport widow
13:21 topranks: adding BGP trace/verbose debug mode on magru CRs as advised by Juniper support T384774
13:17 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cr2-magru with reason: IBGP instability from cr1 to cr2 in magru causing ping faulures from alert1002
18:43 dancy@deploy2002: Started scap sync-world: Testing scap 4.135.0
18:43 dancy@deploy2002: Installation of scap version "4.135.0" completed for 204 hosts
18:38 dancy@deploy2002: Installing scap version "4.135.0" for 204 host(s)
18:03 bd808@mwmaint2002: Wikitech: Renamed and attached 234 of 385 accounts claimed via Bitu (T161859)
17:40 bd808@mwmaint2002: Wikitech: Renamed and attached 57 of 82 accounts claimed via Striker where the SUL account was renamed after claiming (T161859)
17:31 bd808@mwmaint2002: Wikitech: SUL attached 593 accounts with matching names claimed via Striker or Bitu (T161859)
13:52 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cr2-magru with reason: IBGP instability from cr1 to cr2 in magru causing ping faulures from alert1002
01:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2240 (T384592)', diff saved to https://phabricator.wikimedia.org/P73430 and previous config saved to /var/cache/conftool/dbconfig/20250209-013642-marostegui.json
01:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2240', diff saved to https://phabricator.wikimedia.org/P73429 and previous config saved to /var/cache/conftool/dbconfig/20250209-012135-marostegui.json
01:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2240', diff saved to https://phabricator.wikimedia.org/P73428 and previous config saved to /var/cache/conftool/dbconfig/20250209-010628-marostegui.json
00:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2240 (T384592)', diff saved to https://phabricator.wikimedia.org/P73427 and previous config saved to /var/cache/conftool/dbconfig/20250209-005121-marostegui.json
2025-02-08
19:36 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2240 (T384592)', diff saved to https://phabricator.wikimedia.org/P73426 and previous config saved to /var/cache/conftool/dbconfig/20250208-193620-marostegui.json
19:36 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2240.codfw.wmnet with reason: Maintenance
15:32 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2239.codfw.wmnet with reason: Maintenance
15:31 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2237 (T384592)', diff saved to https://phabricator.wikimedia.org/P73425 and previous config saved to /var/cache/conftool/dbconfig/20250208-153144-marostegui.json
15:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P73424 and previous config saved to /var/cache/conftool/dbconfig/20250208-151636-marostegui.json
15:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P73423 and previous config saved to /var/cache/conftool/dbconfig/20250208-150130-marostegui.json
14:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2237 (T384592)', diff saved to https://phabricator.wikimedia.org/P73422 and previous config saved to /var/cache/conftool/dbconfig/20250208-144623-marostegui.json
09:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2237 (T384592)', diff saved to https://phabricator.wikimedia.org/P73421 and previous config saved to /var/cache/conftool/dbconfig/20250208-091745-marostegui.json
09:17 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2237.codfw.wmnet with reason: Maintenance
09:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2236 (T384592)', diff saved to https://phabricator.wikimedia.org/P73420 and previous config saved to /var/cache/conftool/dbconfig/20250208-091721-marostegui.json
09:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P73419 and previous config saved to /var/cache/conftool/dbconfig/20250208-090214-marostegui.json
08:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P73418 and previous config saved to /var/cache/conftool/dbconfig/20250208-084707-marostegui.json
08:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2236 (T384592)', diff saved to https://phabricator.wikimedia.org/P73417 and previous config saved to /var/cache/conftool/dbconfig/20250208-083201-marostegui.json
03:40 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2236 (T384592)', diff saved to https://phabricator.wikimedia.org/P73416 and previous config saved to /var/cache/conftool/dbconfig/20250208-034038-marostegui.json
03:40 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2236.codfw.wmnet with reason: Maintenance
03:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T384592)', diff saved to https://phabricator.wikimedia.org/P73415 and previous config saved to /var/cache/conftool/dbconfig/20250208-034015-marostegui.json
03:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P73414 and previous config saved to /var/cache/conftool/dbconfig/20250208-032508-marostegui.json
03:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P73413 and previous config saved to /var/cache/conftool/dbconfig/20250208-031000-marostegui.json
02:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T384592)', diff saved to https://phabricator.wikimedia.org/P73412 and previous config saved to /var/cache/conftool/dbconfig/20250208-025453-marostegui.json
00:31 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs1025.eqiad.wmnet
00:30 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs1026.eqiad.wmnet
2025-02-07
23:11 bking@cumin2002: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs1025.eqiad.wmnet
23:11 bking@cumin2002: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs1026.eqiad.wmnet
23:06 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2021.codfw.wmnet
23:01 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs1022.eqiad.wmnet
23:00 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs1021.eqiad.wmnet
22:58 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2020.codfw.wmnet
22:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2219 (T384592)', diff saved to https://phabricator.wikimedia.org/P73410 and previous config saved to /var/cache/conftool/dbconfig/20250207-220433-marostegui.json
22:04 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2219.codfw.wmnet with reason: Maintenance
22:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T384592)', diff saved to https://phabricator.wikimedia.org/P73409 and previous config saved to /var/cache/conftool/dbconfig/20250207-220411-marostegui.json
21:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P73408 and previous config saved to /var/cache/conftool/dbconfig/20250207-214904-marostegui.json
21:40 bking@cumin2002: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs1022.eqiad.wmnet
21:39 bking@cumin2002: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs1021.eqiad.wmnet
21:38 bking@cumin2002: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2021.codfw.wmnet
21:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P73407 and previous config saved to /var/cache/conftool/dbconfig/20250207-213357-marostegui.json
21:33 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2019.codfw.wmnet
21:28 bking@cumin2002: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2020.codfw.wmnet
21:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T384592)', diff saved to https://phabricator.wikimedia.org/P73406 and previous config saved to /var/cache/conftool/dbconfig/20250207-211851-marostegui.json
21:16 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2018.codfw.wmnet
20:38 marostegui@cumin1002: dbctl commit (dc=all): 'db1170 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73405 and previous config saved to /var/cache/conftool/dbconfig/20250207-203816-root.json
20:23 marostegui@cumin1002: dbctl commit (dc=all): 'db1170 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73404 and previous config saved to /var/cache/conftool/dbconfig/20250207-202311-root.json
20:08 marostegui@cumin1002: dbctl commit (dc=all): 'db1170 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73403 and previous config saved to /var/cache/conftool/dbconfig/20250207-200805-root.json
20:06 bking@cumin2002: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2019.codfw.wmnet
19:53 marostegui@cumin1002: dbctl commit (dc=all): 'db1170 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73402 and previous config saved to /var/cache/conftool/dbconfig/20250207-195300-root.json
19:48 bking@cumin2002: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2018.codfw.wmnet
19:37 marostegui@cumin1002: dbctl commit (dc=all): 'db1170 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73401 and previous config saved to /var/cache/conftool/dbconfig/20250207-193754-root.json
18:33 vriley@cumin1002: START - Cookbook sre.hosts.provision for host db1256.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
18:32 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
18:14 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1256
18:12 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host db1256
18:07 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
18:07 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt db1256 - vriley@cumin1002"
18:01 vriley@cumin1002: START - Cookbook sre.hosts.provision for host db1255.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
17:59 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1255
17:59 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
17:58 rzl@deploy2002: rzl: Continuing with sync
17:58 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host db1255
17:58 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:58 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt db1255 - vriley@cumin1002"
17:38 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
17:37 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
17:36 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:36 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add mgmt dns names for test nokia switches - cmooney@cumin1002"
17:35 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add mgmt dns names for test nokia switches - cmooney@cumin1002"
16:34 cdanis@cumin1002: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "[not really into teleological thinking] - cdanis@cumin1002"
16:34 cdanis@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: [not really into teleological thinking] - cdanis@cumin1002
16:34 cdanis@cumin1002: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: [not really into teleological thinking] - cdanis@cumin1002
16:34 cdanis@cumin1002: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "[not really into teleological thinking] - cdanis@cumin1002"
16:33 cdanis@cumin1002: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "[not really into teleological thinking] - cdanis@cumin1002"
16:33 cdanis@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: [not really into teleological thinking] - cdanis@cumin1002
16:33 cdanis@cumin1002: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: [not really into teleological thinking] - cdanis@cumin1002
16:33 cdanis@cumin1002: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "[not really into teleological thinking] - cdanis@cumin1002"
16:16 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2210 (T384592)', diff saved to https://phabricator.wikimedia.org/P73400 and previous config saved to /var/cache/conftool/dbconfig/20250207-161646-marostegui.json
16:16 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2210.codfw.wmnet with reason: Maintenance
16:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T384592)', diff saved to https://phabricator.wikimedia.org/P73399 and previous config saved to /var/cache/conftool/dbconfig/20250207-161624-marostegui.json
16:16 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
16:15 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1054.eqiad.wmnet with OS bookworm
16:13 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
16:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P73398 and previous config saved to /var/cache/conftool/dbconfig/20250207-160117-marostegui.json
15:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P73397 and previous config saved to /var/cache/conftool/dbconfig/20250207-154610-marostegui.json
15:31 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T384592)', diff saved to https://phabricator.wikimedia.org/P73396 and previous config saved to /var/cache/conftool/dbconfig/20250207-153103-marostegui.json
15:04 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1020.eqiad.wmnet with reason: maintenance
15:04 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1016.eqiad.wmnet with reason: maintenance
15:03 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1170.eqiad.wmnet with reason: Index rebuild
15:02 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1170.eqiad.wmnet
14:56 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1170.eqiad.wmnet
14:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1170', diff saved to https://phabricator.wikimedia.org/P73395 and previous config saved to /var/cache/conftool/dbconfig/20250207-145547-marostegui.json
10:03 marostegui@cumin1002: dbctl commit (dc=all): 'db1174 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73378 and previous config saved to /var/cache/conftool/dbconfig/20250207-100302-root.json
09:51 marostegui@cumin1002: dbctl commit (dc=all): 'db1234 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73377 and previous config saved to /var/cache/conftool/dbconfig/20250207-095154-root.json
09:47 marostegui@cumin1002: dbctl commit (dc=all): 'db1174 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73376 and previous config saved to /var/cache/conftool/dbconfig/20250207-094756-root.json
09:36 marostegui@cumin1002: dbctl commit (dc=all): 'db1234 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73375 and previous config saved to /var/cache/conftool/dbconfig/20250207-093649-root.json
09:15 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2206 (T384592)', diff saved to https://phabricator.wikimedia.org/P73374 and previous config saved to /var/cache/conftool/dbconfig/20250207-091459-marostegui.json
09:14 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2206.codfw.wmnet with reason: Maintenance
08:06 marostegui@cumin1002: dbctl commit (dc=all): 'es1030 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P73373 and previous config saved to /var/cache/conftool/dbconfig/20250207-080638-root.json
08:02 marostegui@cumin1002: dbctl commit (dc=all): 'es1027 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P73372 and previous config saved to /var/cache/conftool/dbconfig/20250207-080218-root.json
07:51 marostegui@cumin1002: dbctl commit (dc=all): 'es1030 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P73371 and previous config saved to /var/cache/conftool/dbconfig/20250207-075132-root.json
07:47 marostegui@cumin1002: dbctl commit (dc=all): 'es1027 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P73370 and previous config saved to /var/cache/conftool/dbconfig/20250207-074712-root.json
07:36 marostegui@cumin1002: dbctl commit (dc=all): 'es1030 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P73369 and previous config saved to /var/cache/conftool/dbconfig/20250207-073627-root.json
07:32 marostegui@cumin1002: dbctl commit (dc=all): 'es1027 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P73368 and previous config saved to /var/cache/conftool/dbconfig/20250207-073207-root.json
07:21 marostegui@cumin1002: dbctl commit (dc=all): 'es1030 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P73367 and previous config saved to /var/cache/conftool/dbconfig/20250207-072122-root.json
07:17 marostegui@cumin1002: dbctl commit (dc=all): 'es1027 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P73366 and previous config saved to /var/cache/conftool/dbconfig/20250207-071702-root.json
07:13 kevinbazira@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' .
07:12 kevinbazira@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' .
07:08 kevinbazira@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' .
07:06 marostegui@cumin1002: dbctl commit (dc=all): 'es1030 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P73365 and previous config saved to /var/cache/conftool/dbconfig/20250207-070617-root.json
07:06 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for es1030.eqiad.wmnet
07:01 marostegui@cumin1002: dbctl commit (dc=all): 'es1027 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P73364 and previous config saved to /var/cache/conftool/dbconfig/20250207-070156-root.json
07:01 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for es1027.eqiad.wmnet
06:57 root@cumin1002: START - Cookbook sre.mysql.upgrade for es1030.eqiad.wmnet
06:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1030', diff saved to https://phabricator.wikimedia.org/P73363 and previous config saved to /var/cache/conftool/dbconfig/20250207-065730-marostegui.json
06:57 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es1026 to es2 master', diff saved to https://phabricator.wikimedia.org/P73362 and previous config saved to /var/cache/conftool/dbconfig/20250207-065700-root.json
06:56 root@cumin1002: START - Cookbook sre.mysql.upgrade for es1027.eqiad.wmnet
06:56 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1027', diff saved to https://phabricator.wikimedia.org/P73361 and previous config saved to /var/cache/conftool/dbconfig/20250207-065600-marostegui.json
06:55 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es1029 to es1 master', diff saved to https://phabricator.wikimedia.org/P73360 and previous config saved to /var/cache/conftool/dbconfig/20250207-065546-root.json
06:36 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1234.eqiad.wmnet with reason: Index rebuild
06:36 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2145.codfw.wmnet with reason: Index rebuild
06:36 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2145.codfw.wmnet
06:35 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1234.eqiad.wmnet
06:35 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1174.eqiad.wmnet with reason: Index rebuild
06:35 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2150.codfw.wmnet with reason: Index rebuild
06:34 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2150.codfw.wmnet
06:34 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1174.eqiad.wmnet
06:29 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1234.eqiad.wmnet
06:29 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2145.codfw.wmnet
06:28 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1234 db2145', diff saved to https://phabricator.wikimedia.org/P73359 and previous config saved to /var/cache/conftool/dbconfig/20250207-062857-marostegui.json
06:28 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1174.eqiad.wmnet
06:28 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2150.codfw.wmnet
06:27 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1174 db2150', diff saved to https://phabricator.wikimedia.org/P73358 and previous config saved to /var/cache/conftool/dbconfig/20250207-062745-marostegui.json
03:42 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2199.codfw.wmnet with reason: Maintenance
03:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T384592)', diff saved to https://phabricator.wikimedia.org/P73357 and previous config saved to /var/cache/conftool/dbconfig/20250207-034149-marostegui.json
03:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P73356 and previous config saved to /var/cache/conftool/dbconfig/20250207-032642-marostegui.json
03:14 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1053.eqiad.wmnet with OS bookworm
03:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P73355 and previous config saved to /var/cache/conftool/dbconfig/20250207-031134-marostegui.json
02:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T384592)', diff saved to https://phabricator.wikimedia.org/P73354 and previous config saved to /var/cache/conftool/dbconfig/20250207-025628-marostegui.json
02:00 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1054.eqiad.wmnet with OS bookworm
01:57 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1041.eqiad.wmnet
01:54 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ganeti1053.eqiad.wmnet with OS bookworm
01:49 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
01:49 andrew@cumin1002: START - Cookbook sre.hosts.reboot-single for host cloudvirt1041.eqiad.wmnet
01:48 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
01:42 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
01:41 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
21:27 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2172 (T384592)', diff saved to https://phabricator.wikimedia.org/P73352 and previous config saved to /var/cache/conftool/dbconfig/20250206-212719-marostegui.json
21:27 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2172.codfw.wmnet with reason: Maintenance
21:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T384592)', diff saved to https://phabricator.wikimedia.org/P73351 and previous config saved to /var/cache/conftool/dbconfig/20250206-212656-marostegui.json
21:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P73350 and previous config saved to /var/cache/conftool/dbconfig/20250206-211149-marostegui.json
20:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P73349 and previous config saved to /var/cache/conftool/dbconfig/20250206-205642-marostegui.json
20:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P73348 and previous config saved to /var/cache/conftool/dbconfig/20250206-205437-root.json
20:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T384592)', diff saved to https://phabricator.wikimedia.org/P73347 and previous config saved to /var/cache/conftool/dbconfig/20250206-204135-marostegui.json
20:39 marostegui@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P73346 and previous config saved to /var/cache/conftool/dbconfig/20250206-203932-root.json
20:27 pt1979@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
20:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P73345 and previous config saved to /var/cache/conftool/dbconfig/20250206-202426-root.json
20:21 pt1979@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
20:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P73344 and previous config saved to /var/cache/conftool/dbconfig/20250206-200921-root.json
19:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P73343 and previous config saved to /var/cache/conftool/dbconfig/20250206-195417-root.json
19:52 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2213', diff saved to https://phabricator.wikimedia.org/P73342 and previous config saved to /var/cache/conftool/dbconfig/20250206-195250-marostegui.json
19:52 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: maintenance
16:12 pt1979@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
16:11 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-be[2051-2056].codfw.wmnet
16:11 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:11 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ms-be[2051-2056].codfw.wmnet decommissioned, removing all IPs except the asset tag one - mvernon@cumin2002"
16:09 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ms-be[2051-2056].codfw.wmnet decommissioned, removing all IPs except the asset tag one - mvernon@cumin2002"
16:05 pt1979@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
15:07 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2155 (T384592)', diff saved to https://phabricator.wikimedia.org/P73319 and previous config saved to /var/cache/conftool/dbconfig/20250206-150702-marostegui.json
15:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
15:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2155.codfw.wmnet with reason: Maintenance
15:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T384592)', diff saved to https://phabricator.wikimedia.org/P73318 and previous config saved to /var/cache/conftool/dbconfig/20250206-150624-marostegui.json
15:01 Lucas_WMDE: elukey@deploy2002 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [re-log due to stashbot issue, originally logged 14:58 UTC]
14:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P73317 and previous config saved to /var/cache/conftool/dbconfig/20250206-145117-marostegui.json
14:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P73316 and previous config saved to /var/cache/conftool/dbconfig/20250206-143609-marostegui.json
14:30 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster1003.eqiad.wmnet to drbd
14:28 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1033.eqiad.wmnet
14:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T384592)', diff saved to https://phabricator.wikimedia.org/P73314 and previous config saved to /var/cache/conftool/dbconfig/20250206-142102-marostegui.json
14:16 urbanecm@deploy2002: pmiazga, urbanecm: Continuing with sync
12:43 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti1038.eqiad.wmnet with reason: remove from cluster for reimage
12:41 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1038.eqiad.wmnet
07:19 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db2192 to s5 primary and set section read-write T385148', diff saved to https://phabricator.wikimedia.org/P73278 and previous config saved to /var/cache/conftool/dbconfig/20250206-071902-root.json
07:18 marostegui@cumin1002: dbctl commit (dc=all): 'Set s5 codfw as read-only for maintenance - T385148', diff saved to https://phabricator.wikimedia.org/P73277 and previous config saved to /var/cache/conftool/dbconfig/20250206-071836-root.json
07:18 marostegui: Starting s5 codfw failover from db2213 to db2192 - T385148
07:07 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1194.eqiad.wmnet with reason: Index rebuild
07:07 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2208.codfw.wmnet with reason: Index rebuild
07:05 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1194.eqiad.wmnet
07:02 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2208.codfw.wmnet
06:59 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T385148
06:59 marostegui@cumin1002: dbctl commit (dc=all): 'Set db2192 with weight 0 T385148', diff saved to https://phabricator.wikimedia.org/P73276 and previous config saved to /var/cache/conftool/dbconfig/20250206-065925-root.json
06:58 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2208.codfw.wmnet
06:58 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1194.eqiad.wmnet
06:58 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2208 db1194 T385550', diff saved to https://phabricator.wikimedia.org/P73275 and previous config saved to /var/cache/conftool/dbconfig/20250206-065759-marostegui.json
04:55 ejegg: payments-wiki upgraded from MW 1.39 to MW 1.43 (needs db update)
04:01 ejegg: upgraded payments-wiki-staging from 7eeb643 to 4cdd67b
03:22 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
03:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T384592)', diff saved to https://phabricator.wikimedia.org/P73274 and previous config saved to /var/cache/conftool/dbconfig/20250206-032148-marostegui.json
03:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P73273 and previous config saved to /var/cache/conftool/dbconfig/20250206-030641-marostegui.json
02:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P73272 and previous config saved to /var/cache/conftool/dbconfig/20250206-025134-marostegui.json
02:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T384592)', diff saved to https://phabricator.wikimedia.org/P73271 and previous config saved to /var/cache/conftool/dbconfig/20250206-023626-marostegui.json
23:35 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts logstash2026.codfw.wmnet
23:35 cwhite@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
23:35 cwhite@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2026.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002"
23:35 cwhite@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2026.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002"
23:19 cwhite@cumin2002: START - Cookbook sre.hosts.decommission for hosts logstash2026.codfw.wmnet
23:07 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts logstash2027.codfw.wmnet
23:07 cwhite@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
23:07 cwhite@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2027.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002"
23:06 cwhite@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2027.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002"
22:55 cwhite@cumin2002: START - Cookbook sre.hosts.decommission for hosts logstash2027.codfw.wmnet
22:54 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts logstash2028.codfw.wmnet
22:54 cwhite@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
22:54 cwhite@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2028.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002"
22:51 cwhite@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2028.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002"
22:41 cwhite@cumin2002: START - Cookbook sre.hosts.decommission for hosts logstash2028.codfw.wmnet
22:40 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts logstash2029.codfw.wmnet
22:40 cwhite@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
22:40 cwhite@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2029.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002"
22:40 cwhite@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2029.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002"
21:27 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1249 (T384592)', diff saved to https://phabricator.wikimedia.org/P73270 and previous config saved to /var/cache/conftool/dbconfig/20250205-212751-marostegui.json
21:27 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1249.eqiad.wmnet with reason: Maintenance
21:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T384592)', diff saved to https://phabricator.wikimedia.org/P73269 and previous config saved to /var/cache/conftool/dbconfig/20250205-212729-marostegui.json
21:21 cdanis: upgraded python3-conftool-requestctl and friends on puppetservers/puppetmasters
21:14 cdanis: released new conftool 5.0.2 for all distros to apt.wm.o
21:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P73268 and previous config saved to /var/cache/conftool/dbconfig/20250205-211222-marostegui.json
20:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P73267 and previous config saved to /var/cache/conftool/dbconfig/20250205-205715-marostegui.json
20:55 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
20:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T384592)', diff saved to https://phabricator.wikimedia.org/P73266 and previous config saved to /var/cache/conftool/dbconfig/20250205-204208-marostegui.json
20:10 sukhe: granting brett member,reader role on beta
18:33 marostegui@cumin1002: dbctl commit (dc=all): 'db1237 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P73264 and previous config saved to /var/cache/conftool/dbconfig/20250205-183318-root.json
18:33 marostegui@cumin1002: dbctl commit (dc=all): 'db1179 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P73263 and previous config saved to /var/cache/conftool/dbconfig/20250205-183306-root.json
17:48 marostegui@cumin1002: dbctl commit (dc=all): 'db1237 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P73257 and previous config saved to /var/cache/conftool/dbconfig/20250205-174802-root.json
17:47 marostegui@cumin1002: dbctl commit (dc=all): 'db1179 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P73256 and previous config saved to /var/cache/conftool/dbconfig/20250205-174750-root.json
17:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
17:45 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
17:43 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
17:32 marostegui@cumin1002: dbctl commit (dc=all): 'db1237 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P73255 and previous config saved to /var/cache/conftool/dbconfig/20250205-173257-root.json
17:32 marostegui@cumin1002: dbctl commit (dc=all): 'db1179 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P73254 and previous config saved to /var/cache/conftool/dbconfig/20250205-173245-root.json
17:30 mutante: phab1004 - rm /lib/systemd/system/phabricator_stats_job_mfa_check.* for gerrit:1117489 T299403
14:46 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cr2-magru with reason: IBGP instability from cr1 to cr2 in magru causing ping faulures from alert1002
14:46 lucaswerkmeister-wmde@deploy2002: anzx, lucaswerkmeister-wmde: Continuing with sync
14:00 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1248 (T384592)', diff saved to https://phabricator.wikimedia.org/P73241 and previous config saved to /var/cache/conftool/dbconfig/20250205-140039-marostegui.json
14:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1248.eqiad.wmnet with reason: Maintenance
14:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T384592)', diff saved to https://phabricator.wikimedia.org/P73240 and previous config saved to /var/cache/conftool/dbconfig/20250205-140017-marostegui.json
13:53 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73238 and previous config saved to /var/cache/conftool/dbconfig/20250205-135320-root.json
13:49 jynus: deploy removal of old hosts for the m1 dbbackups backup user T383871
13:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P73237 and previous config saved to /var/cache/conftool/dbconfig/20250205-134510-marostegui.json
13:38 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73236 and previous config saved to /var/cache/conftool/dbconfig/20250205-133815-root.json
13:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P73235 and previous config saved to /var/cache/conftool/dbconfig/20250205-133003-marostegui.json
13:23 fceratto@cumin1002: dbctl commit (dc=all): 'db1251 (re)pooling @ 100%: Pooling in', diff saved to https://phabricator.wikimedia.org/P73234 and previous config saved to /var/cache/conftool/dbconfig/20250205-132319-fceratto.json
13:23 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73233 and previous config saved to /var/cache/conftool/dbconfig/20250205-132309-root.json
13:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T384592)', diff saved to https://phabricator.wikimedia.org/P73232 and previous config saved to /var/cache/conftool/dbconfig/20250205-131456-marostegui.json
13:08 fceratto@cumin1002: dbctl commit (dc=all): 'db1251 (re)pooling @ 75%: Pooling in', diff saved to https://phabricator.wikimedia.org/P73231 and previous config saved to /var/cache/conftool/dbconfig/20250205-130813-fceratto.json
13:08 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73230 and previous config saved to /var/cache/conftool/dbconfig/20250205-130804-root.json
12:53 fceratto@cumin1002: dbctl commit (dc=all): 'db1251 (re)pooling @ 50%: Pooling in', diff saved to https://phabricator.wikimedia.org/P73228 and previous config saved to /var/cache/conftool/dbconfig/20250205-125308-fceratto.json
12:53 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73227 and previous config saved to /var/cache/conftool/dbconfig/20250205-125259-root.json
12:50 marostegui@cumin1002: START - Cookbook sre.mysql.clone of db1237.eqiad.wmnet onto db1179.eqiad.wmnet
12:46 marostegui@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1237.eqiad.wmnet onto db1179.eqiad.wmnet
12:38 fceratto@cumin1002: dbctl commit (dc=all): 'db1251 (re)pooling @ 35%: Pooling in', diff saved to https://phabricator.wikimedia.org/P73226 and previous config saved to /var/cache/conftool/dbconfig/20250205-123803-fceratto.json
12:22 fceratto@cumin1002: dbctl commit (dc=all): 'db1251 (re)pooling @ 30%: Pooling in', diff saved to https://phabricator.wikimedia.org/P73225 and previous config saved to /var/cache/conftool/dbconfig/20250205-122257-fceratto.json
10:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T385645)', diff saved to https://phabricator.wikimedia.org/P73211 and previous config saved to /var/cache/conftool/dbconfig/20250205-102758-marostegui.json
10:27 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1202.eqiad.wmnet
10:26 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1179 (T385645)', diff saved to https://phabricator.wikimedia.org/P73210 and previous config saved to /var/cache/conftool/dbconfig/20250205-102650-marostegui.json
10:26 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1179.eqiad.wmnet with reason: Maintenance
10:26 fceratto@cumin1002: dbctl commit (dc=all): 'db1251 (re)pooling @ 1%: Pooling in new host', diff saved to https://phabricator.wikimedia.org/P73209 and previous config saved to /var/cache/conftool/dbconfig/20250205-102614-fceratto.json
10:25 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2221.codfw.wmnet
10:20 fceratto@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1251.eqiad.wmnet
10:20 fceratto@cumin1002: START - Cookbook sre.hosts.remove-downtime for db1251.eqiad.wmnet
10:20 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1202.eqiad.wmnet
10:20 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2221.codfw.wmnet
10:20 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1202, db2221 for index rebuild', diff saved to https://phabricator.wikimedia.org/P73208 and previous config saved to /var/cache/conftool/dbconfig/20250205-102012-marostegui.json
10:17 fceratto@cumin1002: dbctl commit (dc=all): 'db1169 (re)pooling @ 50%: Repooling after cloning db1251', diff saved to https://phabricator.wikimedia.org/P73207 and previous config saved to /var/cache/conftool/dbconfig/20250205-101721-fceratto.json
10:14 dcausse: restarting blazegraph on wdqs1012 (BlazegraphFreeAllocatorsDecreasingRapidly)
10:02 fceratto@cumin1002: dbctl commit (dc=all): 'db1169 (re)pooling @ 25%: Repooling after cloning db1251', diff saved to https://phabricator.wikimedia.org/P73205 and previous config saved to /var/cache/conftool/dbconfig/20250205-100216-fceratto.json
09:58 mvernon@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ms-be2051.codfw.wmnet with reason: disk failed, due decom soon
09:56 mvernon@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on ms-be2075.codfw.wmnet with reason: hardware broken awaiting vendor action
09:55 kevinbazira@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' .
09:52 kevinbazira@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' .
09:47 fceratto@cumin1002: dbctl commit (dc=all): 'db1169 (re)pooling @ 10%: Repooling after cloning db1251', diff saved to https://phabricator.wikimedia.org/P73203 and previous config saved to /var/cache/conftool/dbconfig/20250205-094711-fceratto.json
09:46 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1018.eqiad.wmnet with reason: Rebuild tables
09:39 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1156.eqiad.wmnet with reason: Index rebuild
09:38 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1156.eqiad.wmnet
09:32 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-redacteddb1001.eqiad.wmnet with reason: Rebuild tables
09:32 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1014.eqiad.wmnet with reason: Rebuild tables
09:32 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1156.eqiad.wmnet
09:31 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1156 for index rebuild', diff saved to https://phabricator.wikimedia.org/P73202 and previous config saved to /var/cache/conftool/dbconfig/20250205-093152-marostegui.json
09:31 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db[1155-1156].eqiad.wmnet with reason: Rebuild tables
09:13 jnuche@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.15 refs T382366
02:34 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
02:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T384592)', diff saved to https://phabricator.wikimedia.org/P73200 and previous config saved to /var/cache/conftool/dbconfig/20250205-023428-marostegui.json
02:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P73199 and previous config saved to /var/cache/conftool/dbconfig/20250205-021921-marostegui.json
02:12 wfan: donorwiki revision changed from a039cd50 to 98027151
02:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P73198 and previous config saved to /var/cache/conftool/dbconfig/20250205-020414-marostegui.json
01:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T384592)', diff saved to https://phabricator.wikimedia.org/P73197 and previous config saved to /var/cache/conftool/dbconfig/20250205-014907-marostegui.json
00:13 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1222 (T371742)', diff saved to https://phabricator.wikimedia.org/P73196 and previous config saved to /var/cache/conftool/dbconfig/20250205-001309-ladsgroup.json
2025-02-04
23:58 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P73195 and previous config saved to /var/cache/conftool/dbconfig/20250204-235802-ladsgroup.json
23:42 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P73194 and previous config saved to /var/cache/conftool/dbconfig/20250204-234255-ladsgroup.json
23:27 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1222 (T371742)', diff saved to https://phabricator.wikimedia.org/P73193 and previous config saved to /var/cache/conftool/dbconfig/20250204-232748-ladsgroup.json
22:37 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1222 (T371742)', diff saved to https://phabricator.wikimedia.org/P73192 and previous config saved to /var/cache/conftool/dbconfig/20250204-223744-ladsgroup.json
22:37 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1222.eqiad.wmnet with reason: Maintenance
22:35 ladsgroup@deploy2002: Synchronized portals: Bump portals to HEAD (duration: 03m 12s)
22:32 ladsgroup@deploy2002: Synchronized portals/wikipedia.org/assets: Bump portals to HEAD (T368221T373204) (duration: 09m 30s)
20:37 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1244 (T384592)', diff saved to https://phabricator.wikimedia.org/P73191 and previous config saved to /var/cache/conftool/dbconfig/20250204-203754-marostegui.json
20:37 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
20:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T384592)', diff saved to https://phabricator.wikimedia.org/P73190 and previous config saved to /var/cache/conftool/dbconfig/20250204-203732-marostegui.json
20:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P73189 and previous config saved to /var/cache/conftool/dbconfig/20250204-202225-marostegui.json
20:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P73188 and previous config saved to /var/cache/conftool/dbconfig/20250204-200718-marostegui.json
19:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T384592)', diff saved to https://phabricator.wikimedia.org/P73187 and previous config saved to /var/cache/conftool/dbconfig/20250204-195211-marostegui.json
18:42 swfrench-wmf: mw-api-int to ~ 2% of traffic on PHP 8.1 - T383845
17:47 mutante: codesearch.wmflabs.org - hard reboot instance for needed mass reboots in cloud VPS
17:47 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
17:42 elukey@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
17:40 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4008.ulsfo.wmnet with OS bookworm
17:18 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
17:17 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
17:16 vgutierrez@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
17:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2222 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73186 and previous config saved to /var/cache/conftool/dbconfig/20250204-171415-root.json
17:12 elukey@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
17:11 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
17:01 elukey@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
16:59 marostegui@cumin1002: dbctl commit (dc=all): 'db2222 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73185 and previous config saved to /var/cache/conftool/dbconfig/20250204-165909-root.json
16:58 vgutierrez@cumin1002: START - Cookbook sre.hosts.reimage for host lvs4008.ulsfo.wmnet with OS bookworm
16:56 vgutierrez@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs4008.ulsfo.wmnet with OS bookworm
16:51 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
16:50 elukey@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
16:49 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
16:48 elukey@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
16:48 marostegui@cumin1002: dbctl commit (dc=all): 'db1227 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73184 and previous config saved to /var/cache/conftool/dbconfig/20250204-164802-root.json
16:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2222 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73183 and previous config saved to /var/cache/conftool/dbconfig/20250204-164405-root.json
16:32 marostegui@cumin1002: dbctl commit (dc=all): 'db1227 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73182 and previous config saved to /var/cache/conftool/dbconfig/20250204-163256-root.json
16:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2222 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73181 and previous config saved to /var/cache/conftool/dbconfig/20250204-162900-root.json
16:17 marostegui@cumin1002: dbctl commit (dc=all): 'db1227 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73180 and previous config saved to /var/cache/conftool/dbconfig/20250204-161751-root.json
16:17 topranks: disable et-0/0/0 on cr3-ulsfo to prep for optic replacement T384288
16:17 topranks: disable et-0/0/0 on cr3-ulsfo to prep for optic replacement
16:16 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: replace faulty optic et-0/0/0
16:13 marostegui@cumin1002: dbctl commit (dc=all): 'db2222 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73179 and previous config saved to /var/cache/conftool/dbconfig/20250204-161355-root.json
16:02 marostegui@cumin1002: dbctl commit (dc=all): 'db1227 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73178 and previous config saved to /var/cache/conftool/dbconfig/20250204-160246-root.json
16:00 herron@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd-server-ssl._tcp.aux-k8s-etcd.codfw.wmnet on all recursors
16:00 herron@cumin1002: START - Cookbook sre.dns.wipe-cache _etcd-server-ssl._tcp.aux-k8s-etcd.codfw.wmnet on all recursors
15:56 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
15:53 vgutierrez@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4008.ulsfo.wmnet with reason: host reimage
15:51 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
15:50 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
15:47 marostegui@cumin1002: dbctl commit (dc=all): 'db1227 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73177 and previous config saved to /var/cache/conftool/dbconfig/20250204-154740-root.json
15:44 herron@dns1004: END - running authdns-update
13:46 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73176 and previous config saved to /var/cache/conftool/dbconfig/20250204-134646-root.json
13:44 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw1004.eqiad.wmnet with OS bullseye
13:35 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw1002.eqiad.wmnet with OS bookworm
13:31 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73175 and previous config saved to /var/cache/conftool/dbconfig/20250204-133141-root.json
13:27 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw1004.eqiad.wmnet with reason: host reimage
13:23 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw1004.eqiad.wmnet with reason: host reimage
13:17 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw1002.eqiad.wmnet with reason: host reimage
13:16 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73174 and previous config saved to /var/cache/conftool/dbconfig/20250204-131636-root.json
13:14 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw1002.eqiad.wmnet with reason: host reimage
13:11 marostegui@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73173 and previous config saved to /var/cache/conftool/dbconfig/20250204-131118-root.json
13:09 godog: upgrade poolcounter-prometheus-exporter to 0.1.2 - T333947
13:07 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudgw1004.eqiad.wmnet with OS bullseye
13:04 aborrero@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudgw1004.eqiad.wmnet with OS bookworm
12:46 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73168 and previous config saved to /var/cache/conftool/dbconfig/20250204-124625-root.json
12:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P73167 and previous config saved to /var/cache/conftool/dbconfig/20250204-124345-marostegui.json
12:41 marostegui@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73166 and previous config saved to /var/cache/conftool/dbconfig/20250204-124107-root.json
12:28 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs2011.codfw.wmnet,lvs6001.drmrs.wmnet,lvs1017.eqiad.wmnet,lvs3008.esams.wmnet,lvs7001.magru.wmnet} and A:lvs (T373027)
12:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P73165 and previous config saved to /var/cache/conftool/dbconfig/20250204-122838-marostegui.json
12:27 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs2011.codfw.wmnet,lvs6001.drmrs.wmnet,lvs1017.eqiad.wmnet,lvs3008.esams.wmnet,lvs7001.magru.wmnet} and A:lvs (T373027)
12:26 vgutierrez: upgrading pybal on high-traffic1 load balancers - T373027
12:26 marostegui@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73164 and previous config saved to /var/cache/conftool/dbconfig/20250204-122602-root.json
12:25 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs2012.codfw.wmnet,lvs6002.drmrs.wmnet,lvs1018.eqiad.wmnet,lvs3009.esams.wmnet,lvs7002.magru.wmnet} and A:lvs (T373027)
12:24 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs2012.codfw.wmnet,lvs6002.drmrs.wmnet,lvs1018.eqiad.wmnet,lvs3009.esams.wmnet,lvs7002.magru.wmnet} and A:lvs (T373027)
12:23 vgutierrez: upgrading pybal on high-traffic2 load balancers - T373027
12:23 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2222.codfw.wmnet with reason: Index rebuild
12:21 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-low-traffic (T373027)
12:20 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2222.codfw.wmnet
12:20 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-low-traffic (T373027)
12:18 vgutierrez: upgrading pybal on low-traffic load balancers - T373027
12:17 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs2014.codfw.wmnet,lvs6003.drmrs.wmnet,lvs1020.eqiad.wmnet,lvs3010.esams.wmnet,lvs7003.magru.wmnet} and A:lvs (T373027)
12:15 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs2014.codfw.wmnet,lvs6003.drmrs.wmnet,lvs1020.eqiad.wmnet,lvs3010.esams.wmnet,lvs7003.magru.wmnet} and A:lvs (T373027)
12:15 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2222.codfw.wmnet
12:15 vgutierrez: upgrading pybal on secondary load balancers - T373027
12:14 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2222 for index rebuild', diff saved to https://phabricator.wikimedia.org/P73163 and previous config saved to /var/cache/conftool/dbconfig/20250204-121450-marostegui.json
12:14 marostegui@cumin1002: dbctl commit (dc=all): 'es2040 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P73162 and previous config saved to /var/cache/conftool/dbconfig/20250204-121400-root.json
12:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T384592)', diff saved to https://phabricator.wikimedia.org/P73161 and previous config saved to /var/cache/conftool/dbconfig/20250204-121331-marostegui.json
12:11 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs500[4-5]*} and A:lvs (T373027)
12:10 marostegui@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73160 and previous config saved to /var/cache/conftool/dbconfig/20250204-121056-root.json
12:10 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs500[4-5]*} and A:lvs (T373027)
12:07 elukey: manually executed docker-system-prune-dangling.service on build2001
12:04 elukey: manually dropped 2.5.1rocm6.2-1-20250202 on build2001 - T385531
12:03 vgutierrez: upgrading pybal on eqsin - T373027
11:58 marostegui@cumin1002: dbctl commit (dc=all): 'es2040 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P73158 and previous config saved to /var/cache/conftool/dbconfig/20250204-115855-root.json
11:54 vgutierrez: uploaded pybal 1.15.15 to apt.wm.o (bullseye-wikimedia) T373027
11:54 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1227.eqiad.wmnet with reason: Index rebuild
11:54 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1227.eqiad.wmnet
11:53 marostegui@cumin1002: dbctl commit (dc=all): 'db1236 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73157 and previous config saved to /var/cache/conftool/dbconfig/20250204-115323-root.json
11:48 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1227.eqiad.wmnet
11:48 jynus: deploying new backup grants for matomo and analytics_meta T383902
11:48 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1227 for index rebuild', diff saved to https://phabricator.wikimedia.org/P73156 and previous config saved to /var/cache/conftool/dbconfig/20250204-114808-marostegui.json
11:43 marostegui@cumin1002: dbctl commit (dc=all): 'es2040 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P73155 and previous config saved to /var/cache/conftool/dbconfig/20250204-114350-root.json
11:13 marostegui@cumin1002: dbctl commit (dc=all): 'es2040 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P73151 and previous config saved to /var/cache/conftool/dbconfig/20250204-111337-root.json
11:08 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73150 and previous config saved to /var/cache/conftool/dbconfig/20250204-110830-root.json
11:08 marostegui@cumin1002: dbctl commit (dc=all): 'db1236 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73149 and previous config saved to /var/cache/conftool/dbconfig/20250204-110808-root.json
11:03 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1229.eqiad.wmnet with reason: Index rebuild
11:01 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1229.eqiad.wmnet
10:59 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for es2040.codfw.wmnet
10:56 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1229.eqiad.wmnet
10:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1229 for index rebuild', diff saved to https://phabricator.wikimedia.org/P73148 and previous config saved to /var/cache/conftool/dbconfig/20250204-105546-marostegui.json
10:54 root@cumin1002: START - Cookbook sre.mysql.upgrade for es2040.codfw.wmnet
10:54 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2040 for kernel reboot', diff saved to https://phabricator.wikimedia.org/P73147 and previous config saved to /var/cache/conftool/dbconfig/20250204-105411-marostegui.json
10:53 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73146 and previous config saved to /var/cache/conftool/dbconfig/20250204-105323-root.json
10:53 marostegui@cumin1002: dbctl commit (dc=all): 'db1236 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73145 and previous config saved to /var/cache/conftool/dbconfig/20250204-105302-root.json
09:03 fceratto@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1169.eqiad.wmnet
09:03 fceratto@cumin1002: START - Cookbook sre.hosts.remove-downtime for db1169.eqiad.wmnet
08:58 marostegui@cumin1002: dbctl commit (dc=all): 'es2039 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P73132 and previous config saved to /var/cache/conftool/dbconfig/20250204-085828-root.json
08:57 marostegui@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P73131 and previous config saved to /var/cache/conftool/dbconfig/20250204-085749-root.json
08:57 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1169.eqiad.wmnet with reason: Repooling after clone - T383760
08:56 urbanecm@deploy2002: urbanecm: Continuing with sync
08:29 root@cumin1002: START - Cookbook sre.mysql.upgrade for es2039.codfw.wmnet
08:29 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2039 for kernel reboot', diff saved to https://phabricator.wikimedia.org/P73125 and previous config saved to /var/cache/conftool/dbconfig/20250204-082912-marostegui.json
08:27 marostegui@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P73124 and previous config saved to /var/cache/conftool/dbconfig/20250204-082738-root.json
08:22 marostegui@cumin1002: dbctl commit (dc=all): 'db2209 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P73123 and previous config saved to /var/cache/conftool/dbconfig/20250204-082210-root.json
08:19 urbanecm@deploy2002: urbanecm, cyndywikime: Continuing with sync
08:18 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1236.eqiad.wmnet with reason: Index rebuild
08:18 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2220.codfw.wmnet with reason: Index rebuild
08:18 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2220.codfw.wmnet
08:17 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1236.eqiad.wmnet
08:15 marostegui@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2227.codfw.wmnet onto db2209.codfw.wmnet
08:02 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1197.eqiad.wmnet with reason: Index rebuild
08:01 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1197.eqiad.wmnet
07:54 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1197.eqiad.wmnet
07:54 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1197', diff saved to https://phabricator.wikimedia.org/P73120 and previous config saved to /var/cache/conftool/dbconfig/20250204-075440-marostegui.json
07:49 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet with reason: Rebuild tables
07:48 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1013.eqiad.wmnet with reason: Rebuild tables
06:45 marostegui@cumin1002: START - Cookbook sre.mysql.clone of db2227.codfw.wmnet onto db2209.codfw.wmnet
06:44 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2227 to clone db2209', diff saved to https://phabricator.wikimedia.org/P73119 and previous config saved to /var/cache/conftool/dbconfig/20250204-064425-marostegui.json
06:26 kartik@deploy2002: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
05:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1242 (T384592)', diff saved to https://phabricator.wikimedia.org/P73118 and previous config saved to /var/cache/conftool/dbconfig/20250204-054505-marostegui.json
05:44 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
05:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T384592)', diff saved to https://phabricator.wikimedia.org/P73117 and previous config saved to /var/cache/conftool/dbconfig/20250204-054443-marostegui.json
05:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P73116 and previous config saved to /var/cache/conftool/dbconfig/20250204-052936-marostegui.json
05:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P73115 and previous config saved to /var/cache/conftool/dbconfig/20250204-051429-marostegui.json
04:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T384592)', diff saved to https://phabricator.wikimedia.org/P73114 and previous config saved to /var/cache/conftool/dbconfig/20250204-045922-marostegui.json
04:02 mwpresync@deploy2002: Started scap sync-world: testwikis to 1.44.0-wmf.15 refs T382366
03:19 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1054
03:16 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1054
03:04 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1053
02:55 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti1053
02:52 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
01:12 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
01:11 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add dns entries for new frack nodes - pt1979@cumin2002"
01:11 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add dns entries for new frack nodes - pt1979@cumin2002"
00:00 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1241 (T384592)', diff saved to https://phabricator.wikimedia.org/P73112 and previous config saved to /var/cache/conftool/dbconfig/20250204-000010-marostegui.json
00:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1241.eqiad.wmnet with reason: Maintenance
2025-02-03
23:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1238 (T384592)', diff saved to https://phabricator.wikimedia.org/P73111 and previous config saved to /var/cache/conftool/dbconfig/20250203-235947-marostegui.json
23:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P73110 and previous config saved to /var/cache/conftool/dbconfig/20250203-234440-marostegui.json
23:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P73109 and previous config saved to /var/cache/conftool/dbconfig/20250203-232933-marostegui.json
23:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1238 (T384592)', diff saved to https://phabricator.wikimedia.org/P73108 and previous config saved to /var/cache/conftool/dbconfig/20250203-231428-marostegui.json
23:10 dwisehaupt@dns1004: END - running authdns-update
23:09 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
23:09 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt franio1002 - vriley@cumin1002"
17:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1238 (T384592)', diff saved to https://phabricator.wikimedia.org/P73107 and previous config saved to /var/cache/conftool/dbconfig/20250203-175904-marostegui.json
17:58 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1238.eqiad.wmnet with reason: Maintenance
17:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T384592)', diff saved to https://phabricator.wikimedia.org/P73106 and previous config saved to /var/cache/conftool/dbconfig/20250203-175843-marostegui.json
17:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P73105 and previous config saved to /var/cache/conftool/dbconfig/20250203-174336-marostegui.json
17:37 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73104 and previous config saved to /var/cache/conftool/dbconfig/20250203-173748-root.json
17:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P73103 and previous config saved to /var/cache/conftool/dbconfig/20250203-172829-marostegui.json
17:22 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73102 and previous config saved to /var/cache/conftool/dbconfig/20250203-172243-root.json
17:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T384592)', diff saved to https://phabricator.wikimedia.org/P73101 and previous config saved to /var/cache/conftool/dbconfig/20250203-171322-marostegui.json
17:07 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73100 and previous config saved to /var/cache/conftool/dbconfig/20250203-170737-root.json
13:16 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2209 T385457', diff saved to https://phabricator.wikimedia.org/P73084 and previous config saved to /var/cache/conftool/dbconfig/20250203-131631-marostegui.json
13:15 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db2205 to s3 primary and set section read-write T385457', diff saved to https://phabricator.wikimedia.org/P73083 and previous config saved to /var/cache/conftool/dbconfig/20250203-131542-root.json
13:14 marostegui@cumin1002: dbctl commit (dc=all): 'Set s3 codfw as read-only for maintenance - T385457', diff saved to https://phabricator.wikimedia.org/P73082 and previous config saved to /var/cache/conftool/dbconfig/20250203-131452-root.json
13:14 jebe@deploy2002: Started deploy [airflow-dags/analytics_product@ce1f0f6]: (no justification provided)
13:09 cgoubert@deploy2002: Locking from deployment [MediaWiki]: Emergency s3 switchover T385457
11:00 root@cumin1002: START - Cookbook sre.mysql.upgrade for es2027.codfw.wmnet
10:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2027 for kernel reboot', diff saved to https://phabricator.wikimedia.org/P73073 and previous config saved to /var/cache/conftool/dbconfig/20250203-105935-marostegui.json
10:59 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es2034 to es3 codfw master dbtmaint T376905', diff saved to https://phabricator.wikimedia.org/P73072 and previous config saved to /var/cache/conftool/dbconfig/20250203-105915-root.json
10:36 marostegui@cumin1002: dbctl commit (dc=all): 'es2026 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P73071 and previous config saved to /var/cache/conftool/dbconfig/20250203-103649-root.json
10:36 marostegui@cumin1002: dbctl commit (dc=all): 'es2037 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P73070 and previous config saved to /var/cache/conftool/dbconfig/20250203-103634-root.json
10:21 marostegui@cumin1002: dbctl commit (dc=all): 'es2026 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P73069 and previous config saved to /var/cache/conftool/dbconfig/20250203-102144-root.json
10:21 marostegui@cumin1002: dbctl commit (dc=all): 'es2037 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P73068 and previous config saved to /var/cache/conftool/dbconfig/20250203-102129-root.json
10:06 marostegui@cumin1002: dbctl commit (dc=all): 'es2026 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P73067 and previous config saved to /var/cache/conftool/dbconfig/20250203-100638-root.json
10:06 marostegui@cumin1002: dbctl commit (dc=all): 'es2037 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P73066 and previous config saved to /var/cache/conftool/dbconfig/20250203-100623-root.json
10:03 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1221 (T384592)', diff saved to https://phabricator.wikimedia.org/P73065 and previous config saved to /var/cache/conftool/dbconfig/20250203-100300-marostegui.json
10:02 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
10:02 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1221.eqiad.wmnet with reason: Maintenance
10:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T384592)', diff saved to https://phabricator.wikimedia.org/P73064 and previous config saved to /var/cache/conftool/dbconfig/20250203-100221-marostegui.json
09:51 marostegui@cumin1002: dbctl commit (dc=all): 'es2026 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P73063 and previous config saved to /var/cache/conftool/dbconfig/20250203-095133-root.json
09:51 marostegui@cumin1002: dbctl commit (dc=all): 'es2037 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P73062 and previous config saved to /var/cache/conftool/dbconfig/20250203-095118-root.json
09:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P73061 and previous config saved to /var/cache/conftool/dbconfig/20250203-094714-marostegui.json
09:36 marostegui@cumin1002: dbctl commit (dc=all): 'es2026 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P73060 and previous config saved to /var/cache/conftool/dbconfig/20250203-093628-root.json
09:36 marostegui@cumin1002: dbctl commit (dc=all): 'es2037 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P73059 and previous config saved to /var/cache/conftool/dbconfig/20250203-093613-root.json
09:36 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for es2026.codfw.wmnet
09:35 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for es2037.codfw.wmnet
09:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P73058 and previous config saved to /var/cache/conftool/dbconfig/20250203-093207-marostegui.json
09:14 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es2031 to es2 codfw master dbtmaint T376905', diff saved to https://phabricator.wikimedia.org/P73053 and previous config saved to /var/cache/conftool/dbconfig/20250203-091450-root.json
09:13 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1182.eqiad.wmnet with reason: Index rebuild
09:12 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1182.eqiad.wmnet
09:07 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Index rebuild + upgrade
09:06 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1182.eqiad.wmnet
09:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1182 T385084', diff saved to https://phabricator.wikimedia.org/P73052 and previous config saved to /var/cache/conftool/dbconfig/20250203-090558-marostegui.json
02:54 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1199 (T384592)', diff saved to https://phabricator.wikimedia.org/P73051 and previous config saved to /var/cache/conftool/dbconfig/20250203-025443-marostegui.json
02:54 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1199.eqiad.wmnet with reason: Maintenance
02:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T384592)', diff saved to https://phabricator.wikimedia.org/P73050 and previous config saved to /var/cache/conftool/dbconfig/20250203-025421-marostegui.json
02:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P73049 and previous config saved to /var/cache/conftool/dbconfig/20250203-023914-marostegui.json
02:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P73048 and previous config saved to /var/cache/conftool/dbconfig/20250203-022407-marostegui.json
02:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T384592)', diff saved to https://phabricator.wikimedia.org/P73047 and previous config saved to /var/cache/conftool/dbconfig/20250203-020900-marostegui.json
2025-02-02
20:07 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1190 (T384592)', diff saved to https://phabricator.wikimedia.org/P73046 and previous config saved to /var/cache/conftool/dbconfig/20250202-200724-marostegui.json
20:07 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1190.eqiad.wmnet with reason: Maintenance
15:51 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
14:47 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db2212.codfw.wmnet with reason: Maintenance
11:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1184.eqiad.wmnet with reason: Maintenance
08:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T384592)', diff saved to https://phabricator.wikimedia.org/P73045 and previous config saved to /var/cache/conftool/dbconfig/20250202-085551-marostegui.json
08:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P73044 and previous config saved to /var/cache/conftool/dbconfig/20250202-084044-marostegui.json
08:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P73043 and previous config saved to /var/cache/conftool/dbconfig/20250202-082537-marostegui.json
08:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T384592)', diff saved to https://phabricator.wikimedia.org/P73042 and previous config saved to /var/cache/conftool/dbconfig/20250202-081030-marostegui.json
07:11 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2216 (T384592)', diff saved to https://phabricator.wikimedia.org/P73041 and previous config saved to /var/cache/conftool/dbconfig/20250202-071137-marostegui.json
07:11 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db2216.codfw.wmnet with reason: Maintenance
07:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2203 (T384592)', diff saved to https://phabricator.wikimedia.org/P73040 and previous config saved to /var/cache/conftool/dbconfig/20250202-071115-marostegui.json
06:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2203', diff saved to https://phabricator.wikimedia.org/P73039 and previous config saved to /var/cache/conftool/dbconfig/20250202-065608-marostegui.json
06:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2203', diff saved to https://phabricator.wikimedia.org/P73038 and previous config saved to /var/cache/conftool/dbconfig/20250202-064101-marostegui.json
06:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2203 (T384592)', diff saved to https://phabricator.wikimedia.org/P73037 and previous config saved to /var/cache/conftool/dbconfig/20250202-062554-marostegui.json
05:27 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2203 (T384592)', diff saved to https://phabricator.wikimedia.org/P73036 and previous config saved to /var/cache/conftool/dbconfig/20250202-052741-marostegui.json
05:27 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db2203.codfw.wmnet with reason: Maintenance
04:37 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db2202.codfw.wmnet with reason: Maintenance
04:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T384592)', diff saved to https://phabricator.wikimedia.org/P73035 and previous config saved to /var/cache/conftool/dbconfig/20250202-043646-marostegui.json
04:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P73034 and previous config saved to /var/cache/conftool/dbconfig/20250202-042139-marostegui.json
04:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P73033 and previous config saved to /var/cache/conftool/dbconfig/20250202-040632-marostegui.json
03:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T384592)', diff saved to https://phabricator.wikimedia.org/P73032 and previous config saved to /var/cache/conftool/dbconfig/20250202-035125-marostegui.json
02:52 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2188 (T384592)', diff saved to https://phabricator.wikimedia.org/P73031 and previous config saved to /var/cache/conftool/dbconfig/20250202-025237-marostegui.json
02:52 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db2188.codfw.wmnet with reason: Maintenance
02:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T384592)', diff saved to https://phabricator.wikimedia.org/P73030 and previous config saved to /var/cache/conftool/dbconfig/20250202-025215-marostegui.json
02:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P73029 and previous config saved to /var/cache/conftool/dbconfig/20250202-023708-marostegui.json
02:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P73028 and previous config saved to /var/cache/conftool/dbconfig/20250202-022201-marostegui.json
02:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T384592)', diff saved to https://phabricator.wikimedia.org/P73027 and previous config saved to /var/cache/conftool/dbconfig/20250202-020654-marostegui.json
00:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2176 (T384592)', diff saved to https://phabricator.wikimedia.org/P73026 and previous config saved to /var/cache/conftool/dbconfig/20250202-005259-marostegui.json
00:52 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db2176.codfw.wmnet with reason: Maintenance
00:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T384592)', diff saved to https://phabricator.wikimedia.org/P73025 and previous config saved to /var/cache/conftool/dbconfig/20250202-005236-marostegui.json
00:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P73024 and previous config saved to /var/cache/conftool/dbconfig/20250202-003730-marostegui.json
00:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P73023 and previous config saved to /var/cache/conftool/dbconfig/20250202-002223-marostegui.json
00:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T384592)', diff saved to https://phabricator.wikimedia.org/P73022 and previous config saved to /var/cache/conftool/dbconfig/20250202-000716-marostegui.json
2025-02-01
22:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2174 (T384592)', diff saved to https://phabricator.wikimedia.org/P73021 and previous config saved to /var/cache/conftool/dbconfig/20250201-225519-marostegui.json
22:55 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db2174.codfw.wmnet with reason: Maintenance
22:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T384592)', diff saved to https://phabricator.wikimedia.org/P73020 and previous config saved to /var/cache/conftool/dbconfig/20250201-225456-marostegui.json
22:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P73019 and previous config saved to /var/cache/conftool/dbconfig/20250201-223949-marostegui.json
22:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P73018 and previous config saved to /var/cache/conftool/dbconfig/20250201-222442-marostegui.json
20:56 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2173 (T384592)', diff saved to https://phabricator.wikimedia.org/P73016 and previous config saved to /var/cache/conftool/dbconfig/20250201-205602-marostegui.json
20:55 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14:00:00 on db2186.codfw.wmnet with reason: Maintenance
20:55 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db2173.codfw.wmnet with reason: Maintenance
20:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T384592)', diff saved to https://phabricator.wikimedia.org/P73015 and previous config saved to /var/cache/conftool/dbconfig/20250201-205525-marostegui.json
20:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P73014 and previous config saved to /var/cache/conftool/dbconfig/20250201-204018-marostegui.json
20:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P73013 and previous config saved to /var/cache/conftool/dbconfig/20250201-202511-marostegui.json
20:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T384592)', diff saved to https://phabricator.wikimedia.org/P73012 and previous config saved to /var/cache/conftool/dbconfig/20250201-201004-marostegui.json
19:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2170 (T384592)', diff saved to https://phabricator.wikimedia.org/P73011 and previous config saved to /var/cache/conftool/dbconfig/20250201-190526-marostegui.json
19:05 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db2170.codfw.wmnet with reason: Maintenance
19:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T384592)', diff saved to https://phabricator.wikimedia.org/P73010 and previous config saved to /var/cache/conftool/dbconfig/20250201-190504-marostegui.json
18:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P73009 and previous config saved to /var/cache/conftool/dbconfig/20250201-184957-marostegui.json
18:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P73008 and previous config saved to /var/cache/conftool/dbconfig/20250201-183450-marostegui.json
18:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T384592)', diff saved to https://phabricator.wikimedia.org/P73007 and previous config saved to /var/cache/conftool/dbconfig/20250201-181943-marostegui.json
17:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2153 (T384592)', diff saved to https://phabricator.wikimedia.org/P73006 and previous config saved to /var/cache/conftool/dbconfig/20250201-170624-marostegui.json
17:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db2153.codfw.wmnet with reason: Maintenance
17:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T384592)', diff saved to https://phabricator.wikimedia.org/P73005 and previous config saved to /var/cache/conftool/dbconfig/20250201-170602-marostegui.json
16:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P73004 and previous config saved to /var/cache/conftool/dbconfig/20250201-165055-marostegui.json
16:41 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cr2-magru with reason: IBGP instability from cr1 to cr2 in magru causing ping faulures from alert1002
16:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P73003 and previous config saved to /var/cache/conftool/dbconfig/20250201-163548-marostegui.json
16:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T384592)', diff saved to https://phabricator.wikimedia.org/P73002 and previous config saved to /var/cache/conftool/dbconfig/20250201-162041-marostegui.json
15:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2146 (T384592)', diff saved to https://phabricator.wikimedia.org/P73001 and previous config saved to /var/cache/conftool/dbconfig/20250201-151709-marostegui.json
15:17 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db2146.codfw.wmnet with reason: Maintenance
15:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T384592)', diff saved to https://phabricator.wikimedia.org/P73000 and previous config saved to /var/cache/conftool/dbconfig/20250201-151646-marostegui.json
15:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P72999 and previous config saved to /var/cache/conftool/dbconfig/20250201-150139-marostegui.json
14:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P72998 and previous config saved to /var/cache/conftool/dbconfig/20250201-144632-marostegui.json
14:31 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T384592)', diff saved to https://phabricator.wikimedia.org/P72997 and previous config saved to /var/cache/conftool/dbconfig/20250201-143125-marostegui.json
13:19 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2145 (T384592)', diff saved to https://phabricator.wikimedia.org/P72996 and previous config saved to /var/cache/conftool/dbconfig/20250201-131925-marostegui.json
13:19 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db2145.codfw.wmnet with reason: Maintenance
12:22 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db2141.codfw.wmnet with reason: Maintenance
11:18 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
10:22 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1240.eqiad.wmnet with reason: Maintenance
09:24 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1239.eqiad.wmnet with reason: Maintenance
09:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T384592)', diff saved to https://phabricator.wikimedia.org/P72995 and previous config saved to /var/cache/conftool/dbconfig/20250201-092349-marostegui.json
09:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P72994 and previous config saved to /var/cache/conftool/dbconfig/20250201-090842-marostegui.json
08:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P72993 and previous config saved to /var/cache/conftool/dbconfig/20250201-085335-marostegui.json
08:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T384592)', diff saved to https://phabricator.wikimedia.org/P72992 and previous config saved to /var/cache/conftool/dbconfig/20250201-083827-marostegui.json
07:31 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1235 (T384592)', diff saved to https://phabricator.wikimedia.org/P72991 and previous config saved to /var/cache/conftool/dbconfig/20250201-073139-marostegui.json
07:31 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1235.eqiad.wmnet with reason: Maintenance
07:31 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T384592)', diff saved to https://phabricator.wikimedia.org/P72990 and previous config saved to /var/cache/conftool/dbconfig/20250201-073116-marostegui.json
07:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P72989 and previous config saved to /var/cache/conftool/dbconfig/20250201-071609-marostegui.json
07:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P72988 and previous config saved to /var/cache/conftool/dbconfig/20250201-070103-marostegui.json
06:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T384592)', diff saved to https://phabricator.wikimedia.org/P72987 and previous config saved to /var/cache/conftool/dbconfig/20250201-064555-marostegui.json
05:30 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1234 (T384592)', diff saved to https://phabricator.wikimedia.org/P72986 and previous config saved to /var/cache/conftool/dbconfig/20250201-053027-marostegui.json
05:30 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1234.eqiad.wmnet with reason: Maintenance
05:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T384592)', diff saved to https://phabricator.wikimedia.org/P72985 and previous config saved to /var/cache/conftool/dbconfig/20250201-053005-marostegui.json
05:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P72984 and previous config saved to /var/cache/conftool/dbconfig/20250201-051458-marostegui.json
04:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P72983 and previous config saved to /var/cache/conftool/dbconfig/20250201-045951-marostegui.json
04:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T384592)', diff saved to https://phabricator.wikimedia.org/P72982 and previous config saved to /var/cache/conftool/dbconfig/20250201-044444-marostegui.json
03:34 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1232 (T384592)', diff saved to https://phabricator.wikimedia.org/P72981 and previous config saved to /var/cache/conftool/dbconfig/20250201-033412-marostegui.json
03:34 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1232.eqiad.wmnet with reason: Maintenance
03:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T384592)', diff saved to https://phabricator.wikimedia.org/P72980 and previous config saved to /var/cache/conftool/dbconfig/20250201-033350-marostegui.json
03:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P72979 and previous config saved to /var/cache/conftool/dbconfig/20250201-031843-marostegui.json
03:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P72978 and previous config saved to /var/cache/conftool/dbconfig/20250201-030337-marostegui.json
02:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T384592)', diff saved to https://phabricator.wikimedia.org/P72977 and previous config saved to /var/cache/conftool/dbconfig/20250201-024829-marostegui.json
01:37 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1219 (T384592)', diff saved to https://phabricator.wikimedia.org/P72976 and previous config saved to /var/cache/conftool/dbconfig/20250201-013748-marostegui.json
01:37 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1219.eqiad.wmnet with reason: Maintenance
01:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T384592)', diff saved to https://phabricator.wikimedia.org/P72975 and previous config saved to /var/cache/conftool/dbconfig/20250201-013726-marostegui.json
01:25 brett: import ncmonitor 1.3.1 into bookworm-wikimedia
01:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P72974 and previous config saved to /var/cache/conftool/dbconfig/20250201-012219-marostegui.json
01:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P72973 and previous config saved to /var/cache/conftool/dbconfig/20250201-010712-marostegui.json
00:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T384592)', diff saved to https://phabricator.wikimedia.org/P72971 and previous config saved to /var/cache/conftool/dbconfig/20250201-005205-marostegui.json
