Nova Resource:Tools/SAL/Archive 5

2023-12-30

• 12:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
• 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

2023-12-29

• 21:39 andrewbogott: rebooting tools-sgeweblight-10-28.tools.eqiad1.wikimedia.cloud because previous reset didn't get the queue out of error state
• 19:31 andrewbogott: restarting sge_execd on tools-sgeweblight-10-28.tools.eqiad1.wikimedia.cloud in response to error state alert

2023-12-28

• 21:03 andrewbogott: "docker-compose restart" on tools-harbor-1
• 19:18 andrewbogott: rebooting tools-harbor-1.tools.eqiad1.wikimedia.cloud, unresponsive

2023-12-23

• 18:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
• 18:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2023-12-21

• 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-16

2023-12-20

• 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-14, tools-sgeexec-10-15, tools-sgeweblight-10-18, tools-sgeweblight-10-24
• 10:01 taavi: rebooting tools-sgeweblight-10-18, -24, -25, to get rid of a large number of jobs in deleting status

2023-12-19

• 15:39 dhinus: restarting toolsdb to apply a config change T353093
• 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
• 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2023-12-18

• 16:15 taavi: reboot tools-sgeexec-10-15, -23 due to stuck NFS processes
• 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
• 14:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
• 14:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
• 14:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2023-12-16

• 22:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
• 22:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
• 20:54 bd808: Rebuilding all containers to pick up lighttpd config fix and normal package updates (T293552)
• 08:14 dhinus: restarting toolsdb with jemalloc
• 05:32 andrewbogott: restarting mariadb on toolsdb-1 because it's just about to go oom (or possibly just did)
• 00:21 dhinus: restarting toolsdb again as it's again low in free mem T353093

2023-12-15

• 20:26 andrewbogott: restarting toolsdb to avoid upcoming oom crash
• 16:49 dhinus: restarting toolsdb before it's about to go OOM, enabling performance_schema for debugging
• 14:40 dcaro: deploy toolforge-builds-cli 0.0.10 (T341067)
• 13:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T341067)
• 13:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T341067)
• 12:19 dhinus: restarting toolsdb again to apply a config fix T353093
• 10:48 dhinus: restarting toolsdb to apply new config T353093

2023-12-14

• 23:02 andrewbogott: rebooting tools-db-1 yet again
• 17:42 taavi: reboot tools-sgewebgen-10-3
• 02:20 andrewbogott: restarting tools-db-1, oomkiller killed mariadb

2023-12-13

• 19:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
• 18:53 andrewbogott: rebooting tools-nfs-2 server to resolve weird file locking issues
• 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.scale_grid_exec
• 14:23 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder (T352774)
• 14:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder (T352774)
• 14:22 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T352774)
• 14:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T352774)
• 13:54 dcaro: deploy toolforge-builds-cli version 0.0.9 (with envvars support)
• 13:32 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T338142)
• 13:31 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T338142)
• 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
• 11:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
• 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-16
• 10:48 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission (T338142)
• 10:48 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission (T338142)
• 09:49 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
• 09:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

2023-12-12

• 17:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
• 17:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-18
• 17:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-19
• 17:24 taavi: reboot tools-sgeexec-10-14
• 15:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
• 15:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-14, tools-sgeexec-10-8
• 15:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
• 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
• 15:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
• 13:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
• 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
• 12:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T352774)
• 12:16 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T352774)

2023-12-11

• 15:36 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T352774)
• 15:36 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T352774)
• 13:43 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T352774)
• 13:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T352774)
• 13:29 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission (T352774)
• 13:28 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission (T352774)

2023-12-09

• 16:45 dcaro: set toolsdb back as read-write
• 16:35 andrewbogott: rebooting tools-db-1.tools.eqiad1.wikimedia.cloud yet again
• 07:23 dcaro: set toolsdb back as read-write
• 00:54 taavi: set toolsdb back as read-write

2023-12-08

• 11:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
• 11:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2023-12-07

• 04:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-26

2023-12-05

• 21:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
• 21:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
• 19:16 andrewbogott: rebooting tools-sgeweblight-10-26.tools.eqiad1.wikimedia.cloud; can't log in even with root key
• 11:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
• 11:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 11:20 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
• 11:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 11:20 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
• 11:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 11:20 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
• 11:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 11:15 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
• 11:15 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 11:01 dcaro: rebooting tools-sgeweblight-10-25 due to memory allocation issue (T352753)
• 04:51 andrewbogott: rebooting tools-sgeweblight-10-27, tools-sgeweblight-10-17 and tools-sgeweblight-10-30; their filesystems seem locked up and I suspect NFS somehow

2023-12-04

• 09:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
• 09:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2023-12-02

• 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
• 11:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-22
• 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-13, tools-sgeweblight-10-20
• 10:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
• 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
• 00:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
• 00:08 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker role in the tools cluster
• 00:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
• 00:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
• 00:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
• 00:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
• 00:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

2023-12-01

• 23:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
• 23:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
• 23:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
• 22:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
• 21:22 andrewbogott: rebooting tools-sgeweblight-10-[18,21,32].tools.eqiad1.wikimedia.cloud to recover from nfs lockup
• 21:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
• 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
• 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

2023-11-29

• 23:11 bd808: Drained and hard rebooted tools-k8s-worker-40. K8s was showing inconsistent status of the node (offline per k8s-status tool, online per kubectl)
• 22:35 bd808: Hard reboot of tools-k8s-worker-81
• 22:33 bd808: Soft reboot of tools-k8s-worker-81
• 22:26 bd808: Cordon, drain, and restart tools-k8s-worker-81. Instance appears to have pods from tools.cluebotng that are unresponsive to kubectl commands.

2023-11-27

• 14:46 andrewbogott: shuffling toolforge etcd nodes all over the place in order to reimage cloudvirtlocal hosts
• 11:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
• 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2023-11-23

• 10:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
• 10:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2023-11-22

• 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
• 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
• 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
• 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
• 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers (T350873)
• 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers (T350873)
• 10:57 taavi: deploy maintain-kubeusers patch to manage quotas from the git config T350873
• 09:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
• 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2023-11-21

• 10:28 taavi: restart replication on tools-db-2

2023-11-20

• 15:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
• 15:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
• 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
• 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
• 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
• 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
• 13:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
• 13:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
• 10:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-cli' version '0.3.5'
• 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-cli' version '0.3.5'

2023-11-17

• 15:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.5'
• 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.5'
• 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
• 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2023-11-16

• 21:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
• 19:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
• 13:47 taavi: reboot tools-sgecron-2 with very high load average

2023-11-14

• 19:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
• 19:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
• 10:11 taavi: reboot unresponsive tools-sgeexec-10-22

2023-11-13

• 22:21 taavi: reboot! tools-sgewebgen-10-3, tools-sgeweblight-10-21, tools-sgeweblight-10-32, tools-sgeexec-10-16 due to high load average and/or stuck jobs
• 16:37 taavi: drain tools-k8s-worker-84 tools-k8s-worker-85

2023-11-09

• 11:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
• 11:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2023-11-07

• 11:45 taavi: reboot tools-sgeexec-10-8 which had high load average

2023-11-02

• 13:13 taavi: wiping data directory from tools-prometheus-7 so we have least one working server T350227

2023-11-01

• 14:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
• 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
• 09:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
• 09:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
• 08:47 taavi: restart puppetdb

2023-10-30

• 14:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
• 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

2023-10-29

• 17:46 andrewbogott: running SET GLOBAL read_only=OFF; for mariadb on tools-db-1.tools.eqiad1.wikimedia.cloud
• 17:37 andrewbogott: rebooting tools-db-1.tools.eqiad1.wikimedia.cloud to recover from the oom-killer firing

2023-10-26

• 08:29 taavi: root@tools-sgeweblight-10-21:~# sudo dpkg --configure -a
• 08:18 taavi: restart sssd on tools-nfs-2

2023-10-25

• 09:08 blancadesal: harbor up again and upgraded from 2.5 to 2.9 (T346241)
• 08:31 blancadesal: taking harbor down for upgrade (T346241)

2023-10-24

• 16:02 taavi: reboot tools-sgeweblight-10-28
• 09:49 taavi: reboot tools-sgebastion-11 due to high load
• 09:35 taavi: make ToolsDBWritableState alert paging, match icinga check removed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/956071

2023-10-23

• 15:40 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
• 15:39 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
• 14:18 dcaro: release toolforge-builds-cli 0.0.4
• 08:22 taavi: reboot tools-sgeweblight-10-14, 24 T349425

2023-10-19

• 12:48 taavi: flush queued webgrid jobs that had been waiting in the queue since the nfs issues last week

2023-10-18

• 12:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
• 12:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
• 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-70 from 1.22.17 to 1.23.17
• 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-88 from 1.22.17 to 1.23.17
• 12:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-70 from 1.22.17 to 1.23.17
• 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-69 from 1.22.17 to 1.23.17
• 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-50 from 1.22.17 to 1.23.17
• 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-88 from 1.22.17 to 1.23.17
• 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-87 from 1.22.17 to 1.23.17
• 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-69 from 1.22.17 to 1.23.17
• 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-68 from 1.22.17 to 1.23.17
• 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-50 from 1.22.17 to 1.23.17
• 12:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-49 from 1.22.17 to 1.23.17
• 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-87 from 1.22.17 to 1.23.17
• 12:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-86 from 1.22.17 to 1.23.17
• 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-68 from 1.22.17 to 1.23.17
• 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-67 from 1.22.17 to 1.23.17
• 11:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-49 from 1.22.17 to 1.23.17
• 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-48 from 1.22.17 to 1.23.17
• 11:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-86 from 1.22.17 to 1.23.17
• 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-85 from 1.22.17 to 1.23.17
• 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-67 from 1.22.17 to 1.23.17
• 11:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-66 from 1.22.17 to 1.23.17
• 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-48 from 1.22.17 to 1.23.17
• 11:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-47 from 1.22.17 to 1.23.17
• 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-85 from 1.22.17 to 1.23.17
• 11:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-84 from 1.22.17 to 1.23.17
• 11:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-66 from 1.22.17 to 1.23.17
• 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-65 from 1.22.17 to 1.23.17
• 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-47 from 1.22.17 to 1.23.17
• 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-46 from 1.22.17 to 1.23.17
• 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-84 from 1.22.17 to 1.23.17
• 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-83 from 1.22.17 to 1.23.17
• 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-6 from 1.22.17 to 1.23.17
• 11:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-65 from 1.22.17 to 1.23.17
• 11:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-64 from 1.22.17 to 1.23.17
• 11:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-83 from 1.22.17 to 1.23.17
• 11:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-82 from 1.22.17 to 1.23.17
• 11:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-46 from 1.22.17 to 1.23.17
• 11:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-45 from 1.22.17 to 1.23.17
• 11:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-6 from 1.22.17 to 1.23.17
• 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-64 from 1.22.17 to 1.23.17
• 11:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-62 from 1.22.17 to 1.23.17
• 11:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-5 from 1.22.17 to 1.23.17
• 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-82 from 1.22.17 to 1.23.17
• 11:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-81 from 1.22.17 to 1.23.17
• 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-45 from 1.22.17 to 1.23.17
• 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-44 from 1.22.17 to 1.23.17
• 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-5 from 1.22.17 to 1.23.17
• 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-81 from 1.22.17 to 1.23.17
• 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-80 from 1.22.17 to 1.23.17
• 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-62 from 1.22.17 to 1.23.17
• 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-61 from 1.22.17 to 1.23.17
• 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-44 from 1.22.17 to 1.23.17
• 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-43 from 1.22.17 to 1.23.17
• 11:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-4 from 1.22.17 to 1.23.17
• 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-80 from 1.22.17 to 1.23.17
• 11:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-79 from 1.22.17 to 1.23.17
• 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-61 from 1.22.17 to 1.23.17
• 11:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-60 from 1.22.17 to 1.23.17
• 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-43 from 1.22.17 to 1.23.17
• 11:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-42 from 1.22.17 to 1.23.17
• 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-4 from 1.22.17 to 1.23.17
• 11:49 dcaro: deploy toolforge-builds-cli 0.3.0 (T348866)
• 11:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-42 from 1.22.17 to 1.23.17
• 11:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-41 from 1.22.17 to 1.23.17
• 11:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-59 from 1.22.17 to 1.23.17
• 11:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-58 from 1.22.17 to 1.23.17
• 11:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-78 from 1.22.17 to 1.23.17
• 11:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-77 from 1.22.17 to 1.23.17
• 11:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-41 from 1.22.17 to 1.23.17
• 11:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-40 from 1.22.17 to 1.23.17
• 11:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-58 from 1.22.17 to 1.23.17
• 11:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-57 from 1.22.17 to 1.23.17
• 11:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-77 from 1.22.17 to 1.23.17
• 11:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-76 from 1.22.17 to 1.23.17
• 11:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-40 from 1.22.17 to 1.23.17
• 11:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-39 from 1.22.17 to 1.23.17
• 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-57 from 1.22.17 to 1.23.17
• 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-56 from 1.22.17 to 1.23.17
• 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-76 from 1.22.17 to 1.23.17
• 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-75 from 1.22.17 to 1.23.17
• 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-39 from 1.22.17 to 1.23.17
• 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-38 from 1.22.17 to 1.23.17
• 11:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-56 from 1.22.17 to 1.23.17
• 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-55 from 1.22.17 to 1.23.17
• 11:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-38 from 1.22.17 to 1.23.17
• 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-37 from 1.22.17 to 1.23.17
• 11:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-75 from 1.22.17 to 1.23.17
• 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-74 from 1.22.17 to 1.23.17
• 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-55 from 1.22.17 to 1.23.17
• 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-54 from 1.22.17 to 1.23.17
• 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-37 from 1.22.17 to 1.23.17
• 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-36 from 1.22.17 to 1.23.17
• 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-74 from 1.22.17 to 1.23.17
• 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-73 from 1.22.17 to 1.23.17
• 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-54 from 1.22.17 to 1.23.17
• 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-53 from 1.22.17 to 1.23.17
• 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-36 from 1.22.17 to 1.23.17
• 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-35 from 1.22.17 to 1.23.17
• 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-73 from 1.22.17 to 1.23.17
• 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-72 from 1.22.17 to 1.23.17
• 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-53 from 1.22.17 to 1.23.17
• 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-52 from 1.22.17 to 1.23.17
• 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-35 from 1.22.17 to 1.23.17
• 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-34 from 1.22.17 to 1.23.17
• 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-72 from 1.22.17 to 1.23.17
• 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-71 from 1.22.17 to 1.23.17
• 11:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-52 from 1.22.17 to 1.23.17
• 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-51 from 1.22.17 to 1.23.17
• 11:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-34 from 1.22.17 to 1.23.17
• 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-33 from 1.22.17 to 1.23.17
• 11:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-71 from 1.22.17 to 1.23.17
• 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-51 from 1.22.17 to 1.23.17
• 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-33 from 1.22.17 to 1.23.17
• 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-32 from 1.22.17 to 1.23.17
• 11:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-32 from 1.22.17 to 1.23.17
• 11:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-31 from 1.22.17 to 1.23.17
• 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-31 from 1.22.17 to 1.23.17
• 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-30 from 1.22.17 to 1.23.17
• 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-30 from 1.22.17 to 1.23.17
• 11:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-6 from 1.22.17 to 1.23.17
• 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-6 from 1.22.17 to 1.23.17
• 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-5 from 1.22.17 to 1.23.17
• 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-5 from 1.22.17 to 1.23.17
• 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-4 from 1.22.17 to 1.23.17
• 11:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-4 from 1.22.17 to 1.23.17
• 11:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.22.17 to 1.23.17 (T298005)
• 11:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.22.17 to 1.23.17 (T298005)

2023-10-16

• 09:04 dcaro: rebooting tools-k8s-worker-45 due to stuck nfs processes

2023-10-13

• 13:29 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
• 13:28 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
• 09:48 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
• 09:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
• 09:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
• 09:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
• 08:57 dcaro: rebooting tools-sgeexec-10-8 as the host is stuck/unreachable
• 07:43 dcaro: rebooting tools-sgeweblight-10-26 as it fails to allocate memory

2023-10-12

• 15:07 taavi: reboot tools-k8s-worker-70
• 14:01 taavi: deploy jobs-cli v15 T348250
• 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-jobs-framework-cli' version '15'
• 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-jobs-framework-cli' version '15'
• 12:21 dcaro: rebooting sgeexec-10-17
• 12:02 taavi: also reboot tools-sgeweblight-10-30
• 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
• 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
• 11:52 taavi: reboot tools-sgeweblight-10-22, 28

2023-10-11

• 19:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
• 17:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
• 14:41 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=99) for weblight nodes (T348634)
• 14:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.reboot_workers for weblight nodes (T348634)
• 14:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
• 14:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 14:19 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
• 14:19 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 14:16 dcaro: rebooting tools-sgeweblight-10-16 due to stuck NFS (T348634)
• 12:11 taavi: reboot k8s workers 48, 60, 65, 68, 70, 76 T348634
• 12:04 taavi: reboot k8s workers 72, 75, 82 T348634
• 12:01 taavi: reboot tools-sgecron-2 T348634
• 11:49 taavi: reboot tools-sgeexec-10-19

2023-10-10

• 08:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
• 08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2023-10-09

• 10:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.4.0'
• 10:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.4.0'
• 08:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
• 08:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
• 07:14 taavi: deploy jobs-framework-cli v14
• 07:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-jobs-framework-cli' version '14'
• 07:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-jobs-framework-cli' version '14'

2023-10-05

• 09:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-jobs-framework-cli' version '13'
• 09:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-jobs-framework-cli' version '13'
• 07:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
• 07:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

2023-10-04

• 16:54 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
• 16:54 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
• 16:20 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
• 16:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
• 13:16 taavi: rollout toolforge-weld 1.3.0
• 13:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.3.0'
• 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.3.0'
• 13:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
• 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
• 07:40 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
• 07:40 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

2023-10-03

• 13:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
• 13:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
• 12:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
• 12:10 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
• 09:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
• 09:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

2023-10-02

• 08:38 dcaro: rollout toolforge-cli 0.3.4

2023-10-01

• 14:43 andrewbogott: rebooting tools-sgegrid-shadow because it's fussing about nfs

2023-09-29

• 10:48 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers (T347665)
• 10:20 wm-bot2: taavi@runko END (PASS) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=0) for exec nodes
• 10:14 wm-bot2: taavi@runko END (PASS) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=0) for weblight nodes
• 09:59 wm-bot2: taavi@runko START - Cookbook wmcs.toolforge.grid.reboot_workers for exec nodes
• 09:58 wm-bot2: taavi@runko END (FAIL) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=99) for exec nodes
• 09:58 wm-bot2: taavi@runko START - Cookbook wmcs.toolforge.grid.reboot_workers for exec nodes
• 09:57 wm-bot2: taavi@runko END (FAIL) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=99) for exec nodes
• 09:56 wm-bot2: taavi@runko START - Cookbook wmcs.toolforge.grid.reboot_workers for exec nodes
• 09:55 wm-bot2: taavi@runko END (FAIL) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=99) for exec nodes
• 09:52 wm-bot2: taavi@runko END (PASS) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=0) for webgen nodes
• 09:51 wm-bot2: taavi@runko START - Cookbook wmcs.toolforge.grid.reboot_workers for exec nodes
• 09:51 wm-bot2: taavi@runko END (FAIL) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=99) for exec nodes
• 09:15 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-76 (T347665)
• 09:06 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-76 (T347665)
• 09:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-72 (T347665)
• 09:04 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-72 (T347665)

2023-09-27

• 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
• 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

2023-09-26

• 00:07 andrewbogott: rebooting tools-puppetdb-1 in case that straightens out the puppet failures

2023-09-25

• 09:39 dcaro: deploying builds-builder 0.0.71
• 07:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
• 07:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

2023-09-22

• 10:17 taavi: reboot tools-prometheus-6
• 10:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
• 09:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

2023-09-21

• 16:16 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' (T344717)
• 16:03 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' (T344717)
• 16:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
• 16:02 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
• 15:46 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' (T344717)
• 15:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' (T344717)
• 15:24 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' (T344717)
• 15:23 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' (T344717)

2023-09-20

• 19:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-webservice' version '0.103'
• 19:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-webservice' version '0.103'
• 11:04 taavi: deploying toolforge-webservice 0.102
• 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-webservice' version '0.102'
• 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-webservice' version '0.102'
• 06:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
• 06:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
• 06:20 taavi: reboot tools-sgebastion-11 due to stuck NFS handles

2023-09-19

• 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
• 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
• 14:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
• 14:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
• 09:54 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
• 09:54 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 09:51 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
• 09:51 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

2023-09-18

• 10:41 dhinus: restarted stuck pod (webservice stop+start) T346126
• 07:37 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-64 (T346123)
• 07:35 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-64 (T346123)

2023-09-17

• 18:12 taavi: reboot tools-sgeexec-10-22

2023-09-15

• 12:32 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
• 12:31 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
• 12:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-65 (T346123)
• 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-65 (T346123)
• 11:55 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-34 (T346123)
• 11:46 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-34 (T346123)
• 10:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-52 (T346123)
• 10:02 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-52 (T346123)
• 10:01 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-48 (T346123)
• 09:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-48 (T346123)
• 09:52 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-75 (T346123)
• 09:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-75 (T346123)
• 09:28 dcaro: rebooting tools-sge-cron-2 (T346123)
• 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-70 (T346123)
• 09:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-70 (T346123)
• 09:10 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-69 (T346123)
• 09:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-69 (T346123)
• 08:49 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-78 (T346123)
• 08:48 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-78 (T346123)
• 08:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-76 (T346126)
• 08:36 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-76 (T346126)

2023-09-14

• 16:11 dcaro: increasing secrets quota to 30 (T339916)
• 12:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
• 12:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
• 12:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-emailer
• 12:06 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-emailer
• 12:01 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission
• 12:00 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission
• 10:12 dcaro: deploy bulids-api 0.0.96
• 09:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
• 09:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
• 08:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone (T341084)
• 08:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone (T341084)

2023-09-13

• 17:14 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone (T341084)
• 17:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)
• 12:51 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusersNone (T341084)
• 12:51 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)
• 12:41 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusersNone (T341084)
• 12:41 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)
• 12:40 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusersNone (T341084)
• 12:40 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)
• 10:41 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone (T341084)
• 10:41 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)
• 10:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone (T341084)
• 10:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)
• 10:35 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone (T341084)
• 10:34 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)

2023-09-12

• 15:25 andrewbogott: rebooting tools-sgeweblight-10-26.tools.eqiad1.wikimedia.cloud, oom
• 09:02 taavi: restart a bunch of sge nodes due to NFS lockups
• 08:43 taavi: reboot tools-sgebastion-10 due to stuck NFS mounts

2023-09-11

• 12:34 dcaro: deploy kubernetes-metrics (T341084)

2023-09-05

• 13:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone (T341462)
• 13:31 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone (T341462)
• 11:00 dhinus: restarting mariadb on toolsdb-2 (replica) to test slave_parallel_threads (T345450)

2023-09-01

• 12:12 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
• 12:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 12:12 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
• 12:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 11:54 taavi: reboot unresponsible tools-sgeweblight-10-21
• 09:03 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
• 09:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

2023-08-31

• 13:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
• 13:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
• 12:52 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=0)
• 12:51 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.get_cluster_status
• 09:50 wm-bot2: deployed kubernetes component api-gateway (c0faf0f) (T341462) - cookbook ran by dcaro@urcuchillay
• 09:49 wm-bot2: deployed kubernetes component jobs-api (9c9bee0) (T341462) - cookbook ran by dcaro@urcuchillay
• 09:48 wm-bot2: deployed kubernetes component api-gateway (9c9bee0) (T341462) - cookbook ran by dcaro@urcuchillay
• 09:43 wm-bot2: deployed kubernetes component api-gateway (485046b) (T341462) - cookbook ran by dcaro@urcuchillay
• 09:41 wm-bot2: deployed kubernetes component api-gateway (c0faf0f) (T341462) - cookbook ran by dcaro@urcuchillay

2023-08-30

• 10:06 dcaro: upgrade toolforge-weld to 1.2.1 (T344155)
• 08:59 dcaro: restarting harbor to flush caches (T344435)
• 08:43 dcaro: cleaning up empty harbor projects (T344435)

2023-08-29

• 14:17 wm-bot2: deployed kubernetes component jobs-api (485046b) (T341462) - cookbook ran by dcaro@urcuchillay
• 13:06 wm-bot2: deployed kubernetes component jobs-emailer (6f9c8cf) - cookbook ran by taavi@runko

2023-08-28

• 14:58 wm-bot2: deployed kubernetes component envvars-api (90055b5) (T344502) - cookbook ran by dcaro@urcuchillay

2023-08-25

• 02:00 bd808: Reboot of login.toolforge.org hung until a hard reboot was triggered via horizon
• 01:51 bd808: Scheduled reboot of login.toolforge.org for 2023-08-25 01:56:08 UTC

2023-08-22

• 15:27 taavi: fix broken k8s config files T344289#9110359
• 14:31 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers (27328a4) (T344668) - cookbook ran by taavi@runko
• 14:17 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/maintain-kubeusers:eaeb46b from https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers (eaeb46b) (T344668) - cookbook ran by taavi@runko

2023-08-18

• 13:46 wm-bot2: deployed kubernetes component envvars-api (06c26be) (T341462) - cookbook ran by dcaro@urcuchillay
• 12:55 taavi: reboot frozen tools-sgebastion-10
• 12:48 wm-bot2: deployed kubernetes component builds-api (727e6a7) (T341462) - cookbook ran by dcaro@urcuchillay

2023-08-17

• 12:19 dcaro: deploy builds-api builds-api-0.0.85-20230817105952-25c2b55f

2023-08-15

• 23:29 bd808: Rebooted tools-db-1.tools.eqiad1.wikimedia.cloud for T344298

2023-07-26

• 09:30 wm-bot2: deployed kubernetes component image-config (06066ba) - cookbook ran by taavi@runko

2023-07-25

• 13:03 wm-bot2: deployed kubernetes component image-config (0eb287a) - cookbook ran by taavi@runko
• 13:03 taavi: add php8.2 image T335352 T335507

2023-07-24

• 21:45 bd808: Rebuilding container images for refactored config and new PHP 8.2 image (T335352)
• 17:31 taavi: hard reboot tools-harbor-1, unresponsible

2023-07-23

• 14:17 taavi: hard reboot tools-sgeexec-10-15

2023-07-20

• 15:19 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 again with newer image (T342338, T321188)
• 13:09 wm-bot2: updating docker-registry.tools.wmflabs.org/toolforge-distroless-base-debug:latest (T321188) - cookbook ran by arturo@nostromo
• 11:27 wm-bot2: updating docker-registry.tools.wmflabs.org/toolforge-distroless-base:debug (T321188) - cookbook ran by arturo@endurance
• 11:25 wm-bot2: updating docker-registry.tools.wmflabs.org/toolforge-distroless-base:latest (T321188) - cookbook ran by arturo@endurance

2023-07-19

• 16:34 wm-bot2: updating docker-registry.tools.wmflabs.org/toolforge-distroless-base@sha256:77051c1e40d180d0695b5a9ba7a15161ecac7220ea8c1ed6721bd1c8329b1b2f (T321188) - cookbook ran by arturo@nostromo
• 16:30 wm-bot2: updating docker-registry.tools.wmflabs.org/toolforge-distroless-base@sha256:eebb155bd1116e3b67e2ce43244f9c9958df0cbb75a84c231565fae2ed87c9f4 (T321188) - cookbook ran by arturo@nostromo
• 16:05 wm-bot2: updating docker-registry.tools.wmflabs.org/toolforge-distroless-base@sha256:c11cf17ee8a54dd3a44908ed3f38ffbfb41f1c8c6a2264de9b3e2f5ef4576006 (T321188) - cookbook ran by arturo@nostromo
• 15:38 arturo: root@tools-docker-registry-05:~# docker-registry garbage-collect /etc/docker/registry/config.yml (T321188)
• 15:37 arturo: root@tools-docker-registry-05:~# curl -sS -X DELETE localhost:5000/v2/toolforge-distroless-base/manifests/sha256:2d4d28e45bbe4e38177fd4fdc922dbfaf95e607b06bbc4187a90410d895b4491 (T321188)
• 15:09 arturo: try to rescue docker-registry.tools.wmflabs.org/toolforge-distroless-base@sha256:eebb155bd1116e3b67e2ce43244f9c9958df0cbb75a84c231565fae2ed87c9f4 back into the registry from a k8s worker local cache (T321188)

2023-07-18

• 11:02 arturo: redeploy jobs-emailer 0.0.41-20230718103342-3dddcfb8 into k8s (T341084)

2023-07-14

• 22:45 taavi: reboot tools-sgebastion-11 (dev.toolforge.org) to recover from stuck NFS client causing a high load average
• 09:48 dcaro: deploy builds-api 0.0.78, ci rebuild

2023-07-13

• 14:40 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers (75db740) - cookbook ran by taavi@runko
• 14:30 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/maintain-kubeusers:87c3616 from https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers (87c3616) - cookbook ran by taavi@runko
• 08:45 dcaro: rebooting tools-sgeexec-10-22 due to nfs lockup

2023-07-12

• 12:46 arturo: deployed builds-admission 0.0.63-20230712120152-2ef80a7c (T341084)
• 10:06 dcaro: deployed api-gateway 0.0.16, no changes, ci rebuild (T341084)

2023-07-11

• 10:14 dcaro: deploy ingress-admission 0.0.38, ci rebuild (T341084)

2023-07-10

• 20:39 taavi: freeing up disk space usage on tools docker-registry with `taavi@tools-docker-registry-05:~$ sudo sudo -u docker-registry docker-registry garbage-collect /etc/docker/registry/config.yml --delete-untagged`
• 13:01 dcaro: deploy envvars-api 0.0.22 (T341462)
• 09:27 dcaro: deploying calico-0.0.6-20230710081103-dcbbe692, just a rebuild (T341084)

2023-07-09

• 13:26 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by taavi@runko

2023-07-05

• 16:32 dcaro: deploy image-config 0.0.14 (no real changes, just ci rebuild)
• 07:39 taavi: deploying jobs-api 0.0.213-20230705073411-09895639

2023-07-04

• 17:06 taavi: deploy tools-webservice 0.101 for T341088
• 16:38 dcaro: deploy volume-admission 0.0.40 (no real changes, just ci rebuild)
• 11:44 dcaro: deploy jobs-api 0.0.212

2023-07-03

• 19:09 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config.git (561b4d9) - cookbook ran by taavi@runko
• 13:49 dcaro: deploy envvars-api 0.0.21 (no real changes, ci rebuild)
• 13:29 dcaro: deploy builds-api 0.0.75 (no real changes, just ci rebuild)
• 13:17 dcaro: deploy envvars-admission 0.0.8
• 12:17 wm-bot2: Copied Apt package python3-toolforge-weld 1.1.1 to the tools Apt repo on bookworm, bullseye, buster - cookbook ran by taavi@runko
• 12:16 wm-bot2: Copied Apt package python-toolforge-weld 1.1.1 to the tools Apt repo on - cookbook ran by taavi@runko
• 12:12 taavi: deploy jobs-api 0.1.5
• 12:01 dcaro: deploy builds-api 0.0.74
• 09:24 dcaro: deploy envvars-api 0.0.20

2023-06-30

• 18:21 taavi: deploy new jobs-api release to fix T340829

2023-06-29

• 10:19 dcaro: deploy toolforge-cli 0.3.2

2023-06-27

• 16:48 taavi: building initial set of bookworm based images: node18, ruby31, python311 (T335507)
• 09:01 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@endurance
• 08:54 arturo: force-reboot tools-sgeexec-10-15 (unresponsive)

2023-06-23

• 15:42 dcaro: deploy builds-api 0.3.2 (T337025)

2023-06-22

• 11:57 taavi: update toolforge-jobs-framework-cli to 12
• 09:57 dcaro: deploy builds-api 0.3.1
• 09:32 dcaro: deploy builds-api 0.3.0

2023-06-21

• 11:57 dcaro: deploy bulids-api 0.2.0 (T337025)

2023-06-20

• 14:21 taavi: fix gitlab merge settings for tools-webservice to match the agreed values (fast-forward, squash encouraged)
• 12:11 dcaro: deploy toolforge-envvars-cli (upgrades pthyon3-toolforge-weld) (T337538)
• 12:04 dcaro: deployed api-gateway with envvars endpoint support (T337538)
• 11:59 dcaro: deploy buildservice with aptfile support (T336669)

2023-06-16

• 16:26 andrewbogott: restarting apache2 on toolserver-proxy-01.tools.eqiad1.wikimedia.cloud in hopes of stopping a flapping alert
• 08:15 dcaro: deployed latest builds-api 0.1.0

2023-06-15

• 14:05 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by andrew@bullseye

2023-06-13

• 14:27 dcaro: rebooted tools-harbor-1 as it was not responding

2023-06-12

• 09:03 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@nostromo

2023-06-09

• 19:57 andrewbogott: rebooting tools-sgeweblight-10-18 to see if it helps with T338644
• 19:38 andrewbogott: rebooting tools-sgeweblight-10-28 for T337806

2023-06-08

• 20:21 bd808: Rebuilding container images (T337897)
• 14:16 dcaro: restart tools-sgeweblight-10-17.tools.eqiad1.wikimedia.cloud due to nfs hiccup
• 14:07 dcaro: restarting the tools-sgeexec-10-17 node due to nfs hiccup
• 14:00 dcaro: restarting the tools-sgegrid-master node due to nfs hiccup
• 12:00 dcaro: powering off tools-k8s-etcd-18 (T334644)
• 07:18 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (24e7828) - cookbook ran by taavi@runko

2023-06-07

• 12:45 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:a5eb7dc from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (a5eb7dc) - cookbook ran by taavi@runko

2023-06-05

• 07:53 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus

2023-06-01

• 10:07 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api (7e57832) (T337218) - cookbook ran by dcaro@vulcanus
• 09:21 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api (0f4076a) (T336130) - cookbook ran by dcaro@vulcanus
• 09:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller (ef7f103) (T336130) - cookbook ran by dcaro@vulcanus
• 07:52 dcaro: rebooted tools-package-builder-04 (stuck not letting me log in with my user)

2023-05-31

• 02:38 andrewbogott: rebooted tools-sgeweblight-10-16, T337806

2023-05-30

• 00:22 andrewbogott: rebooted tools-sgeweblight-10-30, oom
• 00:16 andrewbogott: rebooted tools-sgeweblight-10-24, seems to be oom

2023-05-26

• 13:13 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller (ef7f103) (T337218) - cookbook ran by dcaro@vulcanus
• 12:59 dcaro: rebooting tools-sgeexec-10-16.tools.eqiad1.wikimedia.cloud for stale NFS handles (D processes)

2023-05-24

• 12:28 dcaro: deploy latest buildservice (T335865)
• 12:28 dcaro: deploy latest buildservice (T336050)

2023-05-23

• 14:40 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller (0c7b25b) - cookbook ran by fran@wmf3169

2023-05-22

• 10:06 arturo: hard-reboot tools-sgeexec-10-18 (monitoring reporting it as down)

2023-05-19

• 13:38 arturo: uncordon tools-k8s-worker-47/48/64/75
• 08:46 bd808: Building new perl532-sssd/{base,web} images (T323522, T320904)

2023-05-17

• 16:05 dcaro: release toolforge-cli 0.3.0 (T336225)
• 12:48 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway (fa8ed2c) (T336225) - cookbook ran by dcaro@vulcanus
• 12:48 wm-bot2: rebooted k8s node tools-k8s-worker-71 (T316544) - cookbook ran by dcaro@vulcanus
• 12:45 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway (d1bb238) (T336225) - cookbook ran by dcaro@vulcanus
• 12:43 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api (8d21314) - cookbook ran by dcaro@vulcanus
• 10:54 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-buildpack-admission-controller:7199a9e from https://github.com/toolforge/buildpack-admission-controller (7199a9e) - cookbook ran by fran@wmf3169
• 08:49 wm-bot2: rebooted k8s node tools-k8s-worker-55 (T316544) - cookbook ran by dcaro@vulcanus
• 08:33 wm-bot2: rebooted k8s node tools-k8s-worker-64 (T316544) - cookbook ran by dcaro@vulcanus
• 08:32 wm-bot2: rebooted k8s node tools-k8s-worker-75 (T316544) - cookbook ran by dcaro@vulcanus
• 08:25 wm-bot2: rebooted k8s node tools-k8s-worker-74 (T316544) - cookbook ran by dcaro@vulcanus
• 08:17 wm-bot2: rebooted k8s node tools-k8s-worker-61 (T316544) - cookbook ran by dcaro@vulcanus
• 08:10 wm-bot2: rebooted k8s node tools-k8s-worker-70 (T316544) - cookbook ran by dcaro@vulcanus
• 08:03 wm-bot2: rebooted k8s node tools-k8s-worker-66 (T316544) - cookbook ran by dcaro@vulcanus
• 07:54 wm-bot2: rebooted k8s node tools-k8s-worker-72 (T316544) - cookbook ran by dcaro@vulcanus
• 07:46 wm-bot2: rebooted k8s node tools-k8s-worker-47 (T316544) - cookbook ran by dcaro@vulcanus
• 07:45 wm-bot2: rebooted k8s node tools-k8s-worker-48 (T316544) - cookbook ran by dcaro@vulcanus
• 07:42 wm-bot2: rebooted k8s node tools-k8s-worker-69 (T316544) - cookbook ran by dcaro@vulcanus
• 07:29 wm-bot2: rebooted k8s node tools-k8s-worker-76 (T316544) - cookbook ran by dcaro@vulcanus

2023-05-16

• 23:24 bd808: kubectl uncordon tools-k8s-worker-69
• 23:22 bd808: Force reboot tools-k8s-worker-69 via Horizon
• 23:18 bd808: kubectl drain --ignore-daemonsets --delete-emptydir-data --force tools-k8s-worker-69
• 23:17 bd808: kubectl cordon tools-k8s-worker-69
• 14:37 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/builds-api:35b57c6 from https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api.git (35b57c6) - cookbook ran by dcaro@vulcanus
• 13:05 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller (df52a39) (T334081) - cookbook ran by dcaro@vulcanus
• 12:54 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller (ad5b2b5) (T334081) - cookbook ran by dcaro@vulcanus
• 11:52 dcaro: release toolforge-weld 0.2.0 and toolforge-webservice 0.98
• 08:08 dcaro: reboot tools-mail-03 (T316544)
• 08:07 dcaro: reboot tools-sgebastion-10 (T316544)

2023-05-15

• 22:50 bd808: Rebuilding bullseye and buster docker containers to pick up make package addition (T320343)
• 22:09 wm-bot2: rebooted k8s node tools-k8s-worker-66 (T316544) - cookbook ran by andrew@bullseye
• 22:07 wm-bot2: rebooted k8s node tools-k8s-worker-65 (T316544) - cookbook ran by andrew@bullseye
• 22:06 wm-bot2: rebooted k8s node tools-k8s-worker-64 (T316544) - cookbook ran by andrew@bullseye
• 22:04 wm-bot2: rebooted k8s node tools-k8s-worker-62 (T316544) - cookbook ran by andrew@bullseye
• 22:02 wm-bot2: rebooted k8s node tools-k8s-worker-61 (T316544) - cookbook ran by andrew@bullseye
• 21:58 wm-bot2: rebooted k8s node tools-k8s-worker-60 (T316544) - cookbook ran by andrew@bullseye
• 21:56 wm-bot2: rebooted k8s node tools-k8s-worker-59 (T316544) - cookbook ran by andrew@bullseye
• 21:54 wm-bot2: rebooted k8s node tools-k8s-worker-58 (T316544) - cookbook ran by andrew@bullseye
• 21:52 wm-bot2: rebooted k8s node tools-k8s-worker-57 (T316544) - cookbook ran by andrew@bullseye
• 21:51 wm-bot2: rebooted k8s node tools-k8s-worker-56 (T316544) - cookbook ran by andrew@bullseye
• 21:50 wm-bot2: rebooted k8s node tools-k8s-worker-55 (T316544) - cookbook ran by andrew@bullseye
• 21:49 wm-bot2: rebooted k8s node tools-k8s-worker-54 (T316544) - cookbook ran by andrew@bullseye
• 21:47 wm-bot2: rebooted k8s node tools-k8s-worker-53 (T316544) - cookbook ran by andrew@bullseye
• 21:44 wm-bot2: rebooted k8s node tools-k8s-worker-52 (T316544) - cookbook ran by andrew@bullseye
• 21:42 wm-bot2: rebooted k8s node tools-k8s-worker-51 (T316544) - cookbook ran by andrew@bullseye
• 21:41 wm-bot2: rebooted k8s node tools-k8s-worker-50 (T316544) - cookbook ran by andrew@bullseye
• 21:40 wm-bot2: rebooted k8s node tools-k8s-worker-49 (T316544) - cookbook ran by andrew@bullseye
• 21:38 wm-bot2: rebooted k8s node tools-k8s-worker-48 (T316544) - cookbook ran by andrew@bullseye
• 21:37 wm-bot2: rebooted k8s node tools-k8s-worker-47 (T316544) - cookbook ran by andrew@bullseye
• 21:33 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by andrew@bullseye
• 21:16 wm-bot2: rebooted k8s node tools-k8s-worker-45 (T316544) - cookbook ran by dcaro@vulcanus
• 21:15 wm-bot2: rebooted k8s node tools-k8s-worker-44 (T316544) - cookbook ran by dcaro@vulcanus
• 21:13 wm-bot2: rebooted k8s node tools-k8s-worker-43 (T316544) - cookbook ran by dcaro@vulcanus
• 21:12 wm-bot2: rebooted k8s node tools-k8s-worker-42 (T316544) - cookbook ran by dcaro@vulcanus
• 21:09 wm-bot2: rebooted k8s node tools-k8s-worker-41 (T316544) - cookbook ran by dcaro@vulcanus
• 21:03 wm-bot2: rebooted k8s node tools-k8s-worker-40 (T316544) - cookbook ran by dcaro@vulcanus
• 20:58 wm-bot2: rebooted k8s node tools-k8s-worker-39 (T316544) - cookbook ran by dcaro@vulcanus
• 20:52 wm-bot2: rebooted k8s node tools-k8s-worker-38 (T316544) - cookbook ran by dcaro@vulcanus
• 20:50 wm-bot2: rebooted k8s node tools-k8s-worker-37 (T316544) - cookbook ran by dcaro@vulcanus
• 20:49 wm-bot2: rebooted k8s node tools-k8s-worker-36 (T316544) - cookbook ran by dcaro@vulcanus
• 20:48 wm-bot2: rebooted k8s node tools-k8s-worker-35 (T316544) - cookbook ran by dcaro@vulcanus
• 20:47 wm-bot2: rebooted k8s node tools-k8s-worker-34 (T316544) - cookbook ran by dcaro@vulcanus
• 20:42 wm-bot2: rebooted k8s node tools-k8s-worker-33 (T316544) - cookbook ran by dcaro@vulcanus
• 20:41 andrewbogott: rebooting frozen VMs: tools-k8s-worker-65, tools-sgeweblight-10-27, tools-k8s-worker-45, tools-k8s-worker-36, tools-sgewebgen-10-3 (fallout from earlier nfs outage)
• 20:36 wm-bot2: rebooted k8s node tools-k8s-worker-32 (T316544) - cookbook ran by dcaro@vulcanus
• 20:32 wm-bot2: rebooted k8s node tools-k8s-worker-31 (T316544) - cookbook ran by dcaro@vulcanus
• 20:24 wm-bot2: rebooted k8s node tools-k8s-worker-30 (T316544) - cookbook ran by dcaro@vulcanus
• 19:04 wm-bot2: rebooted k8s node tools-k8s-worker-67 (T316544) - cookbook ran by dcaro@vulcanus
• 18:56 wm-bot2: rebooted k8s node tools-k8s-worker-68 (T316544) - cookbook ran by dcaro@vulcanus
• 18:49 wm-bot2: rebooted k8s node tools-k8s-worker-69 (T316544) - cookbook ran by dcaro@vulcanus
• 18:46 bd808: Hard reboot tools-static-14 via Horizon per IRC report of unresponsive requests
• 18:44 wm-bot2: rebooted k8s node tools-k8s-worker-70 (T316544) - cookbook ran by dcaro@vulcanus
• 18:42 wm-bot2: rebooted k8s node tools-k8s-worker-71 (T316544) - cookbook ran by dcaro@vulcanus
• 18:39 wm-bot2: rebooted k8s node tools-k8s-worker-72 (T316544) - cookbook ran by dcaro@vulcanus
• 18:34 wm-bot2: rebooted k8s node tools-k8s-worker-73 (T316544) - cookbook ran by dcaro@vulcanus
• 18:28 wm-bot2: rebooted k8s node tools-k8s-worker-74 (T316544) - cookbook ran by dcaro@vulcanus
• 18:22 wm-bot2: rebooted k8s node tools-k8s-worker-75 (T316544) - cookbook ran by dcaro@vulcanus
• 18:22 taavi: clear mail queue
• 18:21 wm-bot2: rebooted k8s node tools-k8s-worker-76 (T316544) - cookbook ran by dcaro@vulcanus
• 18:15 wm-bot2: rebooted k8s node tools-k8s-worker-77 (T316544) - cookbook ran by dcaro@vulcanus
• 18:08 wm-bot2: rebooted k8s node tools-k8s-worker-80 (T316544) - cookbook ran by dcaro@vulcanus
• 18:06 wm-bot2: rebooted k8s node tools-k8s-worker-81 (T316544) - cookbook ran by dcaro@vulcanus
• 18:05 wm-bot2: rebooted k8s node tools-k8s-worker-82 (T316544) - cookbook ran by dcaro@vulcanus
• 17:57 wm-bot2: rebooted k8s node tools-k8s-worker-83 (T316544) - cookbook ran by dcaro@vulcanus
• 17:48 wm-bot2: rebooted k8s node tools-k8s-worker-84 (T316544) - cookbook ran by dcaro@vulcanus
• 17:47 wm-bot2: rebooted k8s node tools-k8s-worker-85 (T316544) - cookbook ran by dcaro@vulcanus
• 17:38 wm-bot2: rebooted k8s node tools-k8s-worker-86 (T316544) - cookbook ran by dcaro@vulcanus
• 17:37 wm-bot2: rebooted k8s node tools-k8s-worker-87 (T316544) - cookbook ran by dcaro@vulcanus
• 17:35 wm-bot2: rebooted k8s node tools-k8s-worker-88 (T316544) - cookbook ran by dcaro@vulcanus
• 17:34 wm-bot2: rebooting all the workers of tools k8s cluster (64 nodes) (T316544) - cookbook ran by dcaro@vulcanus
• 17:20 wm-bot2: rebooted k8s node tools-k8s-worker-87 (T316544) - cookbook ran by dcaro@vulcanus
• 17:19 wm-bot2: rebooted k8s node tools-k8s-worker-88 (T316544) - cookbook ran by dcaro@vulcanus
• 17:17 bd808: Rebuilding bullseye and buster docker containers to pick up openssh-client package addition (T258841)
• 17:12 wm-bot2: rebooting the whole tools k8s cluster (64 nodes) (T316544) - cookbook ran by dcaro@vulcanus
• 17:06 dcaro: rebooting tools-sgegrid-shadow (T316544)
• 17:00 dcaro: rebooting tools-sgegrid-master (T316544)
• 16:55 dcaro: rebooting tools-sgeexec-10-20 (T316544)
• 16:53 dcaro: rebooting tools-sgeweblight-10-18 (T316544)
• 16:53 dcaro: rebooting tools-sgeweblight-10-25 (T316544)
• 16:53 dcaro: rebooting tools-sgeweblight-10-20 (T316544)
• 16:52 dcaro: rebooting tools-sgeweblight-10-21 (T316544)
• 16:52 dcaro: rebooting tools-sgeexec-10-22 (T316544)
• 16:51 dcaro: rebooting tools-sgeweblight-10-28 (T316544)
• 16:50 dcaro: rebooting tools-sgeexec-10-17 (T316544)
• 16:48 dcaro: rebooting tools-sgeexec-10-21 (T316544)
• 16:47 dcaro: rebooting tools-sgeexec-10-19 (T316544)
• 16:45 dcaro: rebooting tools-sgeexec-10-8 (T316544)
• 16:45 dcaro: rebooting tools-sgeweblight-10-24 (T316544)
• 16:44 dcaro: rebooting tools-sgewebgen-10-2 (T316544)
• 16:44 dcaro: rebooting tools-sgeweblight-10-16 (T316544)
• 16:43 dcaro: rebooting tools-sgeweblight-10-30 (T316544)
• 16:43 dcaro: rebooting tools-sgeexec-10-18 (T316544)
• 16:42 dcaro: rebooting tools-sgeexec-10-16 (T316544)
• 16:42 dcaro: rebooting tools-sgeexec-10-14 (T316544)
• 16:41 dcaro: rebooting tools-sgeweblight-10-32 (T316544)
• 16:40 dcaro: rebooting tools-sgeweblight-10-22 (T316544)
• 16:39 dcaro: rebooting tools-sgeweblight-10-17 (T316544)
• 16:32 dcaro: rebooting tools-sgeexec-10-13.tools.eqiad1.wikimedia.cloud (T316544)
• 16:23 dcaro: rebooting tools-sgeweblight-10-26 (T316544)
• 16:15 bd808: Hard reboot of tools-sgebastion-11 via Horizon (done circa 16:11Z)
• 16:14 arturo: rebooted a bunch of nodes to cleanup D procs and high load avg because NFS outage (result of T316544)
• 12:36 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/builds-api:09f3b49-dev from https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api.git (32a8ae9) - cookbook ran by dcaro@vulcanus
• 09:12 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/volume-admission:c64da5a from https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller (c64da5a) - cookbook ran by dcaro@vulcanus

2023-05-13

• 09:13 taavi: reboot tools-sgeexec-10-15,17,18,21

2023-05-11

• 15:48 bd808: Rebooted tools-sgebastion-10 for T336510
• 15:31 bd808: Sent `wall` for reboot of tools-sgebastion-10 circa 15:40Z

2023-05-09

• 16:36 taavi: delegated beta.toolforge.org domain to toolsbeta per T257386
• 09:35 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (ad4fa2a) - cookbook ran by taavi@runko

2023-05-08

• 09:12 arturo: force-reboot tools-sgeexec-10-13 (reported as down by the monitoring, no SSH)

2023-05-07

• 16:06 taavi: remove inbound 25/tcp rule from the toolserver legacy server T136225

2023-05-05

• 22:21 bd808: Added "RepoLookoutBot" to hiera key "dynamicproxy::blocked_user_agent_regex" to stop unnecessary scans by https://www.repo-lookout.org/
• 22:20 bd808: Added
• 11:30 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:811164e from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (811164e) - cookbook ran by taavi@runko
• 09:13 dcaro: rebooted tools-sgeexec-10-16 as it was stuck (T335009)

2023-05-04

• 15:15 wm-bot2: removed instance tools-k8s-etcd-15 - cookbook ran by andrew@bullseye
• 14:13 wm-bot2: removed instance tools-k8s-etcd-14 - cookbook ran by andrew@bullseye

2023-05-03

• 12:41 wm-bot2: removed instance tools-k8s-etcd-13 - cookbook ran by andrew@bullseye

2023-05-02

• 00:29 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller (7199a9e) - cookbook ran by raymond@ubuntu

2023-05-01

• 23:17 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-buildpack-admission-controller:3b3803f from https://github.com/toolforge/buildpack-admission-controller (3b3803f) - cookbook ran by raymond@ubuntu

2023-04-28

• 15:01 arturo: force reboot tools-k8s-worker-79, unresponsive
• 08:27 dcaro: rebooting tools-sgeweblight-10-28 (T335336)
• 07:20 dcaro: rebooting tools-sgegrid-shadow due to stale nfs mount
• 00:09 bd808: `kubectl uncordon tools-k8s-worker-67` (T335543)
• 00:07 bd808: Hard reboot tools-k8s-worker-67.tools.eqiad1.wikimedia.cloud via horizon (T335543)
• 00:04 bd808: Rebooting tools-k8s-worker-67.tools.eqiad1.wikimedia.cloud (T335543)

2023-04-27

• 23:59 bd808: `kubectl drain --ignore-daemonsets --delete-emptydir-data --force tools-k8s-worker-67` (T335543)
• 20:50 bd808: Started process to rebuild all buster and bullseye based container images again. Prior problem seems to have been stale images in local cache on the build server.
• 20:42 bd808: Container image rebuild failed with GPG errors in buster-sssd base image. Will investigate and attempt to restart once resolved in a local dev environment.
• 20:33 bd808: Started process to rebuild all buster and bullseye based container images per https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Building_toolforge_specific_images

2023-04-18

• 16:46 dcaro: force-rebooting tools-sgeweblight-10-25/26/27 as they got stuck stopping the grid_exec process
• 16:35 dcaro: rebooting root@tools-sgeweblight-10-27 due to stuck exec daemon not releasing port 6445
• 16:35 dcaro: rebooting root@tools-sgeweblight-10-25 due to stuck exec daemon not releasing port 6445
• 16:32 dcaro: rebooting root@tools-sgeweblight-10-26 due to stuck exec daemon not releasing port 6445
• 16:26 dcaro: rebooting root@tools-sgeexec-10-14 due to stuck exec daemon not releasing port 6445

2023-04-17

• 13:10 dcaro: rebooting tools-sgegrid-master node (T334847)
• 02:43 legoktm: manual restart of apache2 on toolserver-proxy-1 to completely pick up renewed TLS cert (alert was flapping)

2023-04-11

• 16:11 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (b65439b) - cookbook ran by arturo@nostromo
• 15:46 arturo: upload toolforge-jobs-framework-cli v11 to aptly
• 14:17 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller.git (d878e49) (T324834) - cookbook ran by dcaro@vulcanus
• 13:19 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:c6c693c from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (c6c693c) - cookbook ran by arturo@nostromo
• 12:09 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/volume-admission:40bd3b3 from https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller (40bd3b3) - cookbook ran by dcaro@vulcanus
• 10:34 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx (9aed7e5) - cookbook ran by taavi@runko
• 09:15 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/calico (c6a3e29) (T329677) - cookbook ran by taavi@runko
• 08:45 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko

2023-04-10

• 10:46 taavi: patch existing PSP roles to use policy/v1beta1 T331619
• 09:16 arturo: upgrading k8s cluster to 1.22 (T286856)

2023-04-07

• 14:34 wm-bot2: drained, depooled and removed k8s control node tools-k8s-control-3 (T333929) - cookbook ran by taavi@runko
• 14:30 wm-bot2: removed instance tools-k8s-control-2 - cookbook ran by taavi@runko

2023-04-05

• 15:16 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (5ea5992) - cookbook ran by taavi@runko
• 15:10 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:3569803 from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (3569803) - cookbook ran by taavi@runko
• 14:56 wm-bot2: Added a new k8s worker tools-k8s-worker-88.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko
• 14:42 wm-bot2: Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko
• 14:42 wm-bot2: Added a new k8s worker tools-k8s-worker-87.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko
• 14:28 wm-bot2: Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko
• 14:28 wm-bot2: Added a new k8s worker tools-k8s-worker-86.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko
• 14:15 wm-bot2: Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko
• 14:15 wm-bot2: Added a new k8s worker tools-k8s-worker-85.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko
• 14:01 wm-bot2: Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko
• 14:01 wm-bot2: Added a new k8s worker tools-k8s-worker-84.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko
• 13:47 wm-bot2: Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko
• 13:47 wm-bot2: Added a new k8s worker tools-k8s-worker-83.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko
• 13:34 wm-bot2: Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko
• 13:33 wm-bot2: removed instance tools-k8s-worker-83 - cookbook ran by taavi@runko
• 13:15 wm-bot2: Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko
• 13:06 wm-bot2: removing grid node tools-sgeweblight-10-31.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko
• 13:02 wm-bot2: removing grid node tools-sgeweblight-10-29.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko
• 13:00 wm-bot2: removing grid node tools-sgeexec-10-9.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko
• 12:58 wm-bot2: removing grid node tools-sgeweblight-10-15.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko
• 12:54 wm-bot2: removing grid node tools-sgeexec-10-7.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko
• 12:52 wm-bot2: removing grid node tools-sgeweblight-10-13.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko
• 12:34 wm-bot2: drained, depooled and removed k8s control node tools-k8s-control-1 - cookbook ran by taavi@runko
• 12:07 wm-bot2: Added a new k8s control tools-k8s-control-6.tools.eqiad1.wikimedia.cloud to the cluster - cookbook ran by taavi@runko
• 11:53 wm-bot2: Adding a new k8s control node - cookbook ran by taavi@runko
• 11:51 wm-bot2: removed instance tools-k8s-control-6 - cookbook ran by taavi@runko
• 11:39 wm-bot2: Adding a new k8s control node (T333929) - cookbook ran by taavi@runko
• 11:38 wm-bot2: removed instance tools-k8s-control-6 - cookbook ran by taavi@runko
• 11:21 wm-bot2: Adding a new k8s control node (T333929) - cookbook ran by taavi@runko
• 11:21 wm-bot2: removed instance tools-k8s-control-6 - cookbook ran by taavi@runko
• 11:09 wm-bot2: Adding a new k8s control node (T333929) - cookbook ran by taavi@runko
• 10:53 wm-bot2: removed instance tools-k8s-control-6 - cookbook ran by taavi@runko
• 10:41 wm-bot2: Adding a new k8s control node (T333929) - cookbook ran by taavi@runko
• 10:41 wm-bot2: removed instance tools-k8s-control-6 - cookbook ran by taavi@runko
• 10:16 wm-bot2: Adding a new k8s control node (T333929) - cookbook ran by taavi@runko

2023-04-04

• 19:00 wm-bot2: Adding a new k8s control node (T333929) - cookbook ran by taavi@runko
• 18:59 wm-bot2: removed instance tools-k8s-control-5 - cookbook ran by taavi@runko
• 18:46 wm-bot2: Adding a new k8s control node (T333929) - cookbook ran by taavi@runko
• 18:45 wm-bot2: Adding a new k8s CONTROL node (T333929) - cookbook ran by taavi@runko
• 10:15 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@nostromo
• 09:28 arturo: hard-reboot the 3 k8s control nodes

2023-04-03

• 17:13 wm-bot2: rebooted k8s node tools-k8s-worker-31 - cookbook ran by taavi@runko
• 17:11 wm-bot2: rebooted k8s node tools-k8s-worker-32 - cookbook ran by taavi@runko
• 17:09 wm-bot2: rebooted k8s node tools-k8s-worker-33 - cookbook ran by taavi@runko
• 17:07 wm-bot2: rebooted k8s node tools-k8s-worker-34 - cookbook ran by taavi@runko
• 17:05 wm-bot2: rebooted k8s node tools-k8s-worker-35 - cookbook ran by taavi@runko
• 17:04 wm-bot2: rebooted k8s node tools-k8s-worker-36 - cookbook ran by taavi@runko
• 17:02 wm-bot2: rebooted k8s node tools-k8s-worker-37 - cookbook ran by taavi@runko
• 17:00 wm-bot2: rebooted k8s node tools-k8s-worker-38 - cookbook ran by taavi@runko
• 16:58 wm-bot2: rebooted k8s node tools-k8s-worker-39 - cookbook ran by taavi@runko
• 16:56 wm-bot2: rebooted k8s node tools-k8s-worker-40 - cookbook ran by taavi@runko
• 16:55 wm-bot2: rebooted k8s node tools-k8s-worker-41 - cookbook ran by taavi@runko
• 16:53 wm-bot2: rebooted k8s node tools-k8s-worker-42 - cookbook ran by taavi@runko
• 16:51 wm-bot2: rebooted k8s node tools-k8s-worker-43 - cookbook ran by taavi@runko
• 16:49 wm-bot2: rebooted k8s node tools-k8s-worker-44 - cookbook ran by taavi@runko
• 16:45 wm-bot2: rebooted k8s node tools-k8s-worker-45 - cookbook ran by taavi@runko
• 16:43 wm-bot2: rebooted k8s node tools-k8s-worker-46 - cookbook ran by taavi@runko
• 16:41 wm-bot2: rebooted k8s node tools-k8s-worker-47 - cookbook ran by taavi@runko
• 16:40 wm-bot2: rebooted k8s node tools-k8s-worker-48 - cookbook ran by taavi@runko
• 16:38 wm-bot2: rebooted k8s node tools-k8s-worker-49 - cookbook ran by taavi@runko
• 16:36 wm-bot2: rebooted k8s node tools-k8s-worker-50 - cookbook ran by taavi@runko
• 16:35 wm-bot2: rebooted k8s node tools-k8s-worker-51 - cookbook ran by taavi@runko
• 16:33 wm-bot2: rebooted k8s node tools-k8s-worker-52 - cookbook ran by taavi@runko
• 16:31 wm-bot2: rebooted k8s node tools-k8s-worker-53 - cookbook ran by taavi@runko
• 16:28 wm-bot2: rebooted k8s node tools-k8s-worker-54 - cookbook ran by taavi@runko
• 16:27 wm-bot2: rebooted k8s node tools-k8s-worker-55 - cookbook ran by taavi@runko
• 16:25 wm-bot2: rebooted k8s node tools-k8s-worker-56 - cookbook ran by taavi@runko
• 16:23 wm-bot2: rebooted k8s node tools-k8s-worker-57 - cookbook ran by taavi@runko
• 16:21 wm-bot2: rebooted k8s node tools-k8s-worker-58 - cookbook ran by taavi@runko
• 16:20 wm-bot2: rebooted k8s node tools-k8s-worker-59 - cookbook ran by taavi@runko
• 16:18 wm-bot2: rebooted k8s node tools-k8s-worker-60 - cookbook ran by taavi@runko
• 16:09 wm-bot2: rebooted k8s node tools-k8s-worker-61 - cookbook ran by taavi@runko
• 16:07 wm-bot2: rebooted k8s node tools-k8s-worker-62 - cookbook ran by taavi@runko
• 16:01 wm-bot2: rebooted k8s node tools-k8s-worker-64 - cookbook ran by taavi@runko
• 16:00 wm-bot2: rebooting the whole tools k8s cluster (58 nodes) - cookbook ran by taavi@runko
• 15:58 wm-bot2: rebooted k8s node tools-k8s-worker-65 - cookbook ran by taavi@runko
• 15:56 wm-bot2: rebooted k8s node tools-k8s-worker-66 - cookbook ran by taavi@runko
• 15:48 wm-bot2: rebooted k8s node tools-k8s-worker-67 - cookbook ran by taavi@runko
• 15:38 wm-bot2: rebooted k8s node tools-k8s-worker-68 - cookbook ran by taavi@runko
• 15:36 wm-bot2: rebooted k8s node tools-k8s-worker-69 - cookbook ran by taavi@runko
• 15:34 wm-bot2: rebooted k8s node tools-k8s-worker-70 - cookbook ran by taavi@runko
• 15:32 wm-bot2: rebooted k8s node tools-k8s-worker-71 - cookbook ran by taavi@runko
• 15:30 wm-bot2: rebooted k8s node tools-k8s-worker-72 - cookbook ran by taavi@runko
• 15:28 wm-bot2: rebooted k8s node tools-k8s-worker-73 - cookbook ran by taavi@runko
• 15:26 wm-bot2: rebooted k8s node tools-k8s-worker-74 - cookbook ran by taavi@runko
• 15:24 wm-bot2: rebooted k8s node tools-k8s-worker-75 - cookbook ran by taavi@runko
• 15:22 wm-bot2: rebooting the whole tools k8s cluster (58 nodes) - cookbook ran by taavi@runko
• 15:17 wm-bot2: rebooted k8s node tools-k8s-worker-75 - cookbook ran by taavi@runko
• 15:14 wm-bot2: rebooted k8s node tools-k8s-worker-76 - cookbook ran by taavi@runko
• 15:12 wm-bot2: rebooted k8s node tools-k8s-worker-77 - cookbook ran by taavi@runko
• 15:10 wm-bot2: rebooted k8s node tools-k8s-worker-78 - cookbook ran by taavi@runko
• 15:08 wm-bot2: rebooted k8s node tools-k8s-worker-79 - cookbook ran by taavi@runko
• 15:06 wm-bot2: rebooted k8s node tools-k8s-worker-80 - cookbook ran by taavi@runko
• 14:59 wm-bot2: rebooted k8s node tools-k8s-worker-81 - cookbook ran by taavi@runko
• 14:41 wm-bot2: rebooted k8s node tools-k8s-worker-82 - cookbook ran by taavi@runko
• 14:38 wm-bot2: rebooting the whole tools k8s cluster (58 nodes) - cookbook ran by taavi@runko
• 14:13 andrewbogott: test log to see if stashbot is back working
• 13:19 andrewbogott: forcing puppet run on all toolforge VMs
• 08:28 taavi: stop exim4.service on tools-sgecron-2 T333477
• 06:52 taavi: stop jobs-framework-emailer to prevent spam due to NFS being read-only T333477

2023-03-29

• 16:07 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook (dc26f52) - cookbook ran by raymond@ubuntu
• 15:21 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/registry-admission:24115c7 from https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook (24115c7) - cookbook ran by raymond@ubuntu

2023-03-28

• 19:43 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller (e1b9815) - cookbook ran by raymond@ubuntu

2023-03-27

• 22:51 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-buildpack-admission-controller:70d550a from https://github.com/toolforge/buildpack-admission-controller (70d550a) - cookbook ran by raymond@ubuntu

2023-03-26

• 20:28 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by taavi@runko

2023-03-24

• 14:13 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@endurance

2023-03-21

• 08:11 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by taavi@runko

2023-03-20

• 13:39 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by taavi@runko
• 10:57 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@endurance

2023-03-19

• 09:32 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by taavi@runko

2023-03-17

• 15:56 andrewbogott: truncating .out, .err, and .log files to 10MB in anticipation of moving the NFS volumes

2023-03-13

• 09:50 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-buildpack-admission-controller:f90bd8f from https://github.com/toolforge/buildpack-admission-controller (f90bd8f) - cookbook ran by dcaro@vulcanus

2023-03-12

• 13:40 taavi: restart haproxy on tools-k8s-haproxy-3

2023-03-11

• 18:38 wm-bot2: removing grid node tools-sgeexec-10-11.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 18:36 wm-bot2: removing grid node tools-sgeexec-10-11.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 18:34 wm-bot2: removing grid node tools-sgeexec-10-11.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 18:31 taavi: reboot misbehaving tools-sgeexec-10-11

2023-03-10

• 16:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers (8b42b15) - cookbook ran by taavi@runko

2023-03-09

• 10:13 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers (53e7f81) - cookbook ran by taavi@runko
• 10:04 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/maintain-kubeusers:834807c from https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers (834807c) - cookbook ran by taavi@runko

2023-03-08

• 22:31 bd808: Live hacked user-maintainer clusterrole to work around breakage in T331572

2023-03-07

• 11:34 wm-bot2: Increased quotas by 2 volumes - cookbook ran by fran@wmf3169
• 11:09 wm-bot2: Increased quotas by 6 snapshots - cookbook ran by fran@wmf3169
• 11:07 wm-bot2: Increased quotas by 4000 gigabytes - cookbook ran by fran@wmf3169

2023-03-06

• 12:51 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook (6688477) - cookbook ran by taavi@runko
• 12:33 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/registry-admission:e916fee from https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook (e916fee) - cookbook ran by taavi@runko
• 12:16 arturo: delete calico deployment, redeploy from https://gitlab.wikimedia.org/repos/cloud/toolforge/calico (T328539)

2023-03-05

• 15:43 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers (3e04025) - cookbook ran by taavi@runko

2023-03-02

• 11:32 arturo: aborrero@tools-k8s-control-2:~$ sudo -i kubectl apply -f /etc/kubernetes/toolforge-tool-roles.yaml (https://gerrit.wikimedia.org/r/c/operations/puppet/+/889836)

2023-03-01

• 13:18 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (13eda9d) - cookbook ran by taavi@runko

2023-02-28

• 17:19 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway (9252af7) - cookbook ran by taavi@runko
• 17:04 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (e46da83) - cookbook ran by taavi@runko

2023-02-23

• 18:07 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway (efb60b3) - cookbook ran by taavi@runko
• 09:33 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/buildpack-admission:b34e2f8 from https://github.com/toolforge/buildpack-admission-controller.git (b34e2f8) - cookbook ran by taavi@runko

2023-02-21

• 09:37 arturo: hard-reboot tools-sgeexec-10-11 (unresponsive to ssh)

2023-02-20

• 11:24 taavi: redeploy volume-admission with helm and cert-manager certificates T329530 T292238
• 11:15 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/volume-admission:7fd13ac from https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller (ede8bd0) - cookbook ran by taavi@runko
• 11:05 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-volume-admission-controller:7fd13ac from https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller (7fd13ac) - cookbook ran by taavi@runko
• 10:39 wm-bot2: Increased quotas by 4000 gigabytes - cookbook ran by fran@wmf3169
• 09:20 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@nostromo

2023-02-19

• 09:16 taavi: uncordon tools-k8s-worker-[80-82] after fixing security groups T329378

2023-02-17

• 11:32 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (eeeea4c) - cookbook ran by arturo@endurance
• 11:31 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config (7729b18) (T254636) - cookbook ran by arturo@endurance
• 11:26 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:8a9b97e from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (eeeea4c) - cookbook ran by arturo@endurance
• 11:24 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:8a9b97e from https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-framework-api (618ab29) - cookbook ran by arturo@endurance
• 10:25 arturo: build and push mariadb-sssd/base docker image for Toolforge (T320178, T254636)

2023-02-16

• 15:58 wm-bot2: Increased quotas by 4000 gigabytes - cookbook ran by fran@wmf3169
• 15:30 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager (d71994e) - cookbook ran by arturo@nostromo
• 13:52 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller (7191997) - cookbook ran by taavi@runko
• 13:44 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/ingress-admission:1fe8ec4 from https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller (1fe8ec4) - cookbook ran by taavi@runko
• 12:47 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/ingress-admission:e9b9920 from https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller (e9b9920) - cookbook ran by taavi@runko
• 10:35 arturo: aborrero@tools-k8s-control-1:~$ sudo -i kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml
• 09:48 arturo: grid engine was failed over to shadow server, manually put it back into normal https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Grid#GridEngine_Master
• 09:39 arturo: aborrero@tools-sgegrid-shadow:~$ sudo truncate -s 1G /var/log/syslog (was 17G, full root disk)

2023-02-15

• 18:03 taavi: deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/889585/ to increase amount of haproxy max connections
• 15:19 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@nostromo
• 09:50 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager.git (e3f3ce1) (T329453) - cookbook ran by taavi@runko
• 09:30 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@nostromo

2023-02-14

• 15:07 taavi: import cert-manager components to local docker registry T329453
• 12:12 arturo: the fixed webservicemonitor is starting a bunch of grid webservices (T329611)
• 12:10 arturo: included tools-manifests 0.25 in tools-buster aptly repo, deploying it now! (T329611, T329467, T244809)

2023-02-13

• 16:05 wm-bot2: Increased quotas by 4000 gigabytes - cookbook ran by fran@wmf3169
• 16:03 taavi: update maintain-kubeusers deployment to use helm
• 15:05 taavi: deploy jobs-api updates, improving some status messages
• 15:04 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (13d87c4) - cookbook ran by taavi@runko
• 15:00 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:390ed64 from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (390ed64) - cookbook ran by taavi@runko
• 13:14 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/maintain-kubeusers:aac195b from https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers (aac195b) - cookbook ran by taavi@runko

2023-02-10

• 15:45 taavi: reboot tools-k8s-worker-82 to troubleshoot network issues
• 12:44 wm-bot2: Added a new k8s worker tools-k8s-worker-82.tools.eqiad1.wikimedia.cloud to the worker pool (T329357) - cookbook ran by taavi@runko
• 12:31 wm-bot2: Adding a new k8s worker node (T329357) - cookbook ran by taavi@runko
• 12:29 wm-bot2: Added a new k8s worker tools-k8s-worker-81.tools.eqiad1.wikimedia.cloud to the worker pool (T329357) - cookbook ran by taavi@runko
• 12:15 wm-bot2: Adding a new k8s worker node (T329357) - cookbook ran by taavi@runko
• 11:53 wm-bot2: Adding a new k8s worker node (T329357) - cookbook ran by taavi@runko
• 11:44 wm-bot2: removing grid node tools-sgeweblight-10-23.tools.eqiad1.wikimedia.cloud (T329357) - cookbook ran by taavi@runko
• 11:42 wm-bot2: removing grid node tools-sgeexec-10-5.tools.eqiad1.wikimedia.cloud (T329357) - cookbook ran by taavi@runko
• 11:39 wm-bot2: removing grid node tools-sgeweblight-10-19.tools.eqiad1.wikimedia.cloud (T329357) - cookbook ran by taavi@runko
• 11:26 wm-bot2: removing grid node tools-sgeweblight-10-12.tools.eqiad1.wikimedia.cloud (T329357) - cookbook ran by taavi@runko
• 11:24 wm-bot2: removing grid node tools-sgeexec-10-1.tools.eqiad1.wikimedia.cloud (T329357) - cookbook ran by taavi@runko

2023-02-01

• 16:03 taavi: deployed tools-webservice 0.89
• 15:43 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config (372037f) - cookbook ran by taavi@runko

2023-01-26

• 15:05 taavi: drain and reboot tools-k8s-worker-74 which seems to have some issues with nfs
• 14:37 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (307f302) - cookbook ran by taavi@runko
• 14:30 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:05966c6 from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (05966c6) - cookbook ran by taavi@runko

2023-01-24

• 12:04 taavi: deploying toolforge-jobs-framework-cli v10 T327775
• 10:07 taavi: publish toolforge-jobs-framework-cli v9

2023-01-23

• 11:31 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (d5ae229) - cookbook ran by taavi@runko
• 11:23 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:d085c50 from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (d085c50) - cookbook ran by taavi@runko
• 11:17 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config (864171a) - cookbook ran by taavi@runko

2023-01-20

• 23:24 andrewbogott: truncating logfiles with find . -name '*.err' -size +1G -exec truncate --size=100M {} \;
• 21:24 andrewbogott: truncating logfiles with find . -name '*.out' -size +1G -exec truncate --size=100M {} \;
• 01:06 andrewbogott: truncating logfiles with find . -name '*.log' -size +1G -exec truncate --size=100M {} \;

2023-01-19

• 11:46 arturo: `aborrero@tools-k8s-control-1:~$ sudo -i kubectl delete clusterrolebinding jobs-api-psp` (cleanup unused stuff)

2023-01-18

• 15:42 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (0ad4c66) - cookbook ran by arturo@nostromo
• 15:29 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:54cc15e from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (54cc15e) - cookbook ran by arturo@nostromo

2023-01-17

• 13:55 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (8cf38a1) - cookbook ran by arturo@endurance
• 13:51 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (0d0a882) - cookbook ran by arturo@endurance
• 13:34 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:3a58c1d from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (3a58c1d) - cookbook ran by arturo@endurance

2023-01-10

• 11:55 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (8e0a2f9) - cookbook ran by arturo@endurance
• 11:52 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:9514b00 from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (8e0a2f9) - cookbook ran by arturo@endurance
• 11:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (0243967) - cookbook ran by arturo@endurance

2023-01-03

• 17:17 andrewbogott: find -name '*.log' -size +1G -exec truncate --size=1G {} \;

2022-12-20

• 09:07 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@nostromo

2022-12-12

• 14:36 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus

2022-12-09

• 07:20 taavi: change the canonical tools-mail external hostname to use mail.tools.wmcloud.org and add valid spf to toolforge.org T324809

2022-12-05

• 11:06 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus

2022-11-30

• 10:39 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (bc3529d) - cookbook ran by arturo@nostromo
• 10:17 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:c360d54 from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (c360d54) - cookbook ran by arturo@nostromo

2022-11-29

• 19:52 taavi: clear puppet failure emails from exim queues

2022-11-09

• 08:58 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@nostromo

2022-11-05

• 19:28 andrewbogott: cleaning up nfs share with root@labstore1004:/srv/tools/shared/tools# find -name '*.err' -size +1G -exec truncate --size=1G {} \;
• 13:26 andrewbogott: cleaning up nfs share with root@labstore1004:/srv/tools/shared/tools# find -name '*.log' -size +1G -exec truncate --size=1G {} \;

2022-11-04

• 20:41 andrewbogott: cleaning up nfs share with root@labstore1004:/srv/tools/shared/tools# find -name '*.err' -not -newermt "Nov 1, 2021" -exec rm {} \;
• 14:02 andrewbogott: cleaning up nfs share with root@labstore1004:/srv/tools/shared/tools# find -name '*.log' -not -newermt "Nov 1, 2021" -exec rm {} \;
• 12:20 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (d464be4) (T304900) - cookbook ran by arturo@nostromo
• 12:12 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:2b800f5 from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (2b800f5) (T304900) - cookbook ran by arturo@nostromo

2022-11-01

• 09:37 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master (T322110) - cookbook ran by dcaro@vulcanus

2022-10-26

• 08:45 dcaro: depooling and rebooting tools-sgeexec-10-22 to get nfs scratch working again

2022-10-25

• 16:14 wm-bot2: Increased quotas by 5120 gigabytes - cookbook ran by fran@wmf3169
• 15:26 dcaro: pushed a newer docker-registry.tools.wmflabs.org/python:3.9-slim-bullseye (from upstream pthyon:3.9-slim-bullseye)

2022-10-20

• 16:54 andrewbogott: rebooting tools-package-builder-04
• 16:49 andrewbogott: rebooting redis nodes (one at a time)
• 10:54 taavi: rebuild mono68-sssd image with the expired DST Root CA X3 removed T311466

2022-10-18

• 11:52 taavi: deploy toolforge-jobs-framework-cli deb v8
• 10:30 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer (64385e9) (T320405) - cookbook ran by arturo@nostromo
• 10:27 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:9be2272 from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (9be2272) - cookbook ran by taavi@runko
• 10:18 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-emailer:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer (64385e9) (T320405) - cookbook ran by arturo@nostromo

2022-10-17

• 07:25 taavi: push updated perl532 images T320824

2022-10-14

• 07:54 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (0cc020e) (T311466) - cookbook ran by taavi@runko

2022-10-13

• 15:10 arturo: restart jobs-emailer pod

2022-10-12

• 23:25 bd808: Rebuilding all Toolforge docker images (T278436, T311466, T293552)
• 20:43 bd808: Rebuilding all Toolforge docker images to pick up bug and security fix packages. Third try seems to be working. (T316554)
• 20:31 bd808: Rebuilding all Toolforge docker images to pick up bug and security fix packages after fixing bug in building the bullseye base image. (T316554)
• 16:26 dcaro: deploy the latest registry admission webhook, now for real (image tag 07bc7db)
• 12:48 dcaro: deploy the latest registry admission webhook (image tag 07bc7db)
• 09:26 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus
• 09:19 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus

2022-10-11

• 13:52 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:8574c36 from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (8574c36) - cookbook ran by taavi@runko

2022-10-10

• 19:30 taavi: rebooting all k8s worker nodes to clean up labstore1006/7 remains
• 16:51 taavi: clean up labstore1006/7 mounts from k8s control nodes T320425
• 11:35 arturo: aborrero@tools-k8s-control-1:~$ sudo -i kubectl -n jobs-emailer rollout restart deployment/jobs-emailer (T317998)
• 08:44 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (afa90ed) (T320284) - cookbook ran by taavi@runko
• 08:39 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (afa90ed) - cookbook ran by taavi@runko

2022-10-09

• 17:29 taavi: kill 10 idle tmux sessions of user 'hoi' on tools-sgebastion-10 T320352

2022-10-07

• 13:02 taavi: taavi@cloudcontrol1005 ~ $ sudo mark_tool --disable oncall # T320240

2022-10-06

• 00:39 bd808: Image rebuild failing with debian apt repo signature issue. Will investigate tomorrow. (T316554)
• 00:36 bd808: Rebuilding all Toolforge docker images to pick up bug and security fix packages. (T316554)
• 00:04 bd808: Building new php74-sssd-base & web images (T310435)

2022-10-03

• 14:36 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/volume-admission:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller (8da432b) - cookbook ran by taavi@runko

2022-09-28

• 21:23 lucaswerkmeister: on tools-sgebastion-10: run-puppet-agent # T318858
• 21:22 lucaswerkmeister: on tools-sgebastion-10: apt remove emacs-common emacs-bin-common # fix package conflict, T318858
• 21:15 lucaswerkmeister: added root SSH key for myself, manually ran puppet on tools-sgebastion-10 to apply it (seemingly successfully)

2022-09-22

• 12:30 taavi: add TheresNoTime to the 'toollabs-trusted' gerrit group T317438
• 12:27 taavi: add TheresNoTime as a project admin and to the roots sudo policy T317438

2022-09-10

• 07:39 wm-bot2: removing instance tools-prometheus-03 - cookbook ran by taavi@runko

2022-09-07

• 10:22 dcaro: Pushing the new toolforge builder image based on the new 0.8 buildpacks (T316854)

2022-09-06

• 08:06 dcaro_away: Published new toolforge-bullseye0-run and toolforge-bullseye0-build images for the toolforge buildpack builder (T316854)

2022-08-25

• 10:40 taavi: tagged new version of the python39-web container with a shell implementation of webservice-runner T293552

2022-08-24

• 12:20 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx (eba66bc) - cookbook ran by taavi@runko
• 12:20 taavi: upgrading ingress-nginx to v1.3

2022-08-20

• 07:44 dcaro_away: all k8s nodes ready now \o/ (T315718)
• 07:43 dcaro_away: rebooted tools-k8s-control-2, seemed stuck trying to wait for tools home (nfs?), after reboot came back up (T315718)
• 07:41 dcaro_away: cloudvirt1023 down took out 3 workers, 1 control, and a grid exec and a weblight, they are taking long to restart, looking (T315718)

2022-08-18

• 14:45 andrewbogott: adding lucaswerkmeister as projectadmin (T314527)
• 14:43 andrewbogott: removing some inactive projectadmins: rush, petrb, mdipietro, jeh, krenair

2022-08-17

• 16:34 taavi: kubectl sudo delete cm -n tool-wdml maintain-kubeusers # T315459
• 08:30 taavi: failing the grid from the shadow back to the master, some disruption expected

2022-08-16

• 17:28 taavi: fail over docker-registry, tools-docker-registry-06->docker-registry-05

2022-08-11

• 16:57 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by taavi@runko
• 16:55 taavi: restart puppetdb on tools-puppetdb-1, crashed during the ceph issues

2022-08-05

• 15:08 wm-bot2: removing grid node tools-sgewebgen-10-1.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 15:05 wm-bot2: removing grid node tools-sgeexec-10-12.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 15:00 wm-bot2: created node tools-sgewebgen-10-3.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko

2022-08-03

• 15:51 dhinus: recreated jobs-api pods to pick up new ConfigMap
• 15:02 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (c47ac41) - cookbook ran by fran@MacBook-Pro.station

2022-07-20

• 19:31 taavi: reboot toolserver-proxy-01 to free up disk space probably held by stale file handles
• 08:06 wm-bot2: removing grid node tools-sgeexec-10-6.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

2022-07-19

• 17:53 wm-bot2: created node tools-sgeexec-10-21.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko
• 17:00 wm-bot2: removing grid node tools-sgeexec-10-3.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 16:58 wm-bot2: removing grid node tools-sgeexec-10-4.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 16:24 wm-bot2: created node tools-sgeexec-10-20.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko
• 15:59 taavi: tag current maintain-kubernetes :beta image as: :latest

2022-07-17

• 15:52 wm-bot2: removing grid node tools-sgeexec-10-10.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 15:43 wm-bot2: removing grid node tools-sgeexec-10-2.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 13:26 wm-bot2: created node tools-sgeexec-10-16.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko

2022-07-14

• 13:48 taavi: rebooting tools-sgeexec-10-2

2022-07-13

• 12:09 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus

2022-07-11

• 16:06 wm-bot2: Increased quotas by {self.increases} (T312692) - cookbook ran by nskaggs@x1carbon

2022-07-07

• 07:34 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus

2022-06-28

• 17:34 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master (T311538) - cookbook ran by dcaro@vulcanus
• 15:51 taavi: add 4096G cinder quota T311509

2022-06-27

• 18:14 taavi: restart calico, appears to have got stuck after the ca replacement operation
• 18:02 taavi: switchover active cron server to tools-sgecron-2 T284767
• 17:54 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0915.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 17:52 wm-bot2: removing grid node tools-sgewebgrid-generic-0902.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 17:49 wm-bot2: removing grid node tools-sgeexec-0942.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 17:15 taavi: T311412 updating ca used by k8s-apiserver->etcd communication, breakage may happen
• 14:58 taavi: renew puppet ca cert and certificate for tools-puppetmaster-02 T311412
• 14:50 taavi: backup /var/lib/puppet/server to /root/puppet-ca-backup-2022-06-27.tar.gz on tools-puppetmaster-02 before we do anything else to it T311412

2022-06-23

• 17:51 wm-bot2: removing grid node tools-sgeexec-0941.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 17:49 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0916.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 17:46 wm-bot2: removing grid node tools-sgewebgrid-generic-0901.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 17:32 wm-bot2: removing grid node tools-sgeexec-0939.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 17:30 wm-bot2: removing grid node tools-sgeexec-0938.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 17:27 wm-bot2: removing grid node tools-sgeexec-0937.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 17:22 wm-bot2: removing grid node tools-sgeexec-0936.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 17:19 wm-bot2: removing grid node tools-sgeexec-0935.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 17:17 wm-bot2: removing grid node tools-sgeexec-0934.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 17:14 wm-bot2: removing grid node tools-sgeexec-0933.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 17:11 wm-bot2: removing grid node tools-sgeexec-0932.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 17:09 wm-bot2: removing grid node tools-sgeexec-0920.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 15:30 wm-bot2: removing grid node tools-sgeexec-0947.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 13:59 taavi: removing remaining continuous jobs from the stretch grid T277653

2022-06-22

• 15:54 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0917.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 15:51 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0918.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 15:47 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0919.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 15:45 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0920.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

2022-06-21

• 15:23 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0914.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 15:20 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0914.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 15:18 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0913.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
• 15:07 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0912.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

2022-06-03

• 20:07 wm-bot2: created node tools-sgeweblight-10-26.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
• 19:51 balloons: Scaling webservice nodes to 20, using new 8G swap flavor T309821
• 19:35 wm-bot2: created node tools-sgeweblight-10-25.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
• 19:03 wm-bot2: created node tools-sgeweblight-10-20.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
• 19:01 wm-bot2: created node tools-sgeweblight-10-19.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
• 19:00 balloons: depooled old nodes, bringing entirely new grid of nodes online T309821
• 18:22 wm-bot2: created node tools-sgeweblight-10-17.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
• 17:54 wm-bot2: created node tools-sgeweblight-10-16.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
• 17:52 wm-bot2: created node tools-sgeweblight-10-15.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
• 16:59 andrewbogott: building a bunch of new lighttpd nodes (beginning with tools-sgeweblight-10-12) using a flavor with more swap space
• 16:56 wm-bot2: created node tools-sgeweblight-10-12.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
• 15:50 balloons: fix fix g3.cores4.ram8.disk20.swap24.ephem20 flavor to include swap. Convert to fix g3.cores4.ram8.disk20.swap8.ephem20 flavor T309821
• 15:50 balloons: temp add 1.0G swap to sgeweblight hosts T309821
• 15:50 balloons: fix fix g3.cores4.ram8.disk20.swap24.ephem20 flavor to include swap. Convert to fix g3.cores4.ram8.disk20.swap8.ephem20 flavor t309821
• 15:49 balloons: temp add 1.0G swap to sgeweblight hosts t309821
• 13:25 bd808: Upgrading fleet to tools-webservice 0.86 (T309821)
• 13:20 bd808: publish tools-webservice 0.86 (T309821)
• 12:46 taavi: start webservicemonitor on tools-sgecron-01 T309821
• 10:36 taavi: draining each sgeweblight node one by one, and removing the jobs stuck in 'deleting' too
• 05:05 taavi: removing duplicate (there should be only one per tool) web service jobs from the grid T309821
• 04:52 taavi: revert bd808's changes to profile::toolforge::active_proxy_host
• 03:21 bd808: Cleared queue error states after deploying new toolforge-webservice package (T309821)
• 03:10 bd808: publish tools-webservice 0.85 with hack for T309821

2022-06-02

• 22:26 bd808: Rebooting tools-sgeweblight-10-1.tools.eqiad1.wikimedia.cloud. Node is full of jobs that are not tracked by grid master and failing to spawn new jobs sent by the scheduler
• 21:56 bd808: Removed legacy "active_proxy_host" hiera setting
• 21:55 bd808: Updated hiera to use fqdn of 'tools-proxy-06.tools.eqiad1.wikimedia.cloud' for profile::toolforge::active_proxy_host key
• 21:41 bd808: Updated hiera to use fqdn of 'tools-proxy-06.tools.eqiad1.wikimedia.cloud' for active_redis key
• 21:23 wm-bot2: created node tools-sgeweblight-10-8.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko
• 12:42 wm-bot2: rebooting stretch exec grid workers - cookbook ran by taavi@runko
• 12:13 wm-bot2: created node tools-sgeweblight-10-7.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko
• 12:03 dcaro: refresh prometheus certs (T308402)
• 11:47 dcaro: refresh registry-admission-controller certs (T308402)
• 11:42 dcaro: refresh ingress-admission-controller certs (T308402)
• 11:36 dcaro: refresh volume-admission-controller certs (T308402)
• 11:24 wm-bot2: created node tools-sgeweblight-10-6.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko
• 11:17 taavi: publish jobutils 1.44 that updates the grid default from stretch to buster T277653
• 10:16 taavi: publish tools-webservice 0.84 that updates the grid default from stretch to buster T277653
• 09:54 wm-bot2: created node tools-sgeexec-10-14.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko

2022-06-01

• 11:18 taavi: depool and remove tools-sgeexec-09[07-14]

2022-05-31

• 16:51 taavi: delete tools-sgeexec-0904 for T309525 experimentation

2022-05-30

• 08:24 taavi: depool tools-sgeexec-[0901-0909] (7 nodes total) T277653

2022-05-26

• 15:39 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (e6fa299) (T309146) - cookbook ran by taavi@runko

2022-05-22

• 17:04 taavi: failover tools-redis to the updated cluster T278541
• 16:42 wm-bot2: removing grid node tools-sgeexec-0940.tools.eqiad1.wikimedia.cloud (T308982) - cookbook ran by taavi@runko

2022-05-16

• 14:02 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx (7037eca) - cookbook ran by taavi@runko

2022-05-14

• 10:47 taavi: hard reboot unresponsible tools-sgeexec-0940

2022-05-12

• 12:36 taavi: re-enable CronJobControllerV2 T308205
• 09:28 taavi: deploy jobs-api update T308204
• 09:15 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (e6fa299) (T308204) - cookbook ran by taavi@runko

2022-05-10

• 15:18 taavi: depool tools-k8s-worker-42 for experiments
• 13:54 taavi: enable distro-wikimedia unattended upgrades T290494

2022-05-06

• 19:46 bd808: Rebuilt toolforge-perl532-sssd-base & toolforge-perl532-sssd-web to add liblocale-codes-perl (T307812)

2022-05-05

• 17:28 taavi: deploy tools-webservice 0.83 T307693

2022-05-03

• 08:20 taavi: redis: start replication from the old cluster to the new one (T278541)

2022-05-02

• 08:54 taavi: restart acme-chief.service T307333

2022-04-25

• 14:56 bd808: Rebuilding all docker images to pick up toolforge-webservice v0.82 (T214343)
• 14:46 bd808: Building toolforge-webservice v0.82

2022-04-23

• 16:51 bd808: Built new perl532-sssd/{base,web} images and pushed to registry (T214343)

2022-04-20

• 16:58 taavi: reboot toolserver-proxy-01 to free up disk space from stale file handles(?)
• 07:51 wm-bot: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (8f37a04) - cookbook ran by taavi@runko

2022-04-16

• 18:53 wm-bot: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics (2c485e9) - cookbook ran by taavi@runko

2022-04-12

• 21:32 bd808: Added komla to Gerrit group 'toollabs-trusted' (T305986)
• 21:27 bd808: Added komla to 'roots' sudoers policy (T305986)
• 21:24 bd808: Add komla as projectadmin (T305986)

2022-04-10

• 18:43 taavi: deleted `/tmp/dwl02.out-20210915` on tools-sgebastion-07 (not touched since september, taking up 1.3G of disk space)

2022-04-09

• 15:30 taavi: manually prune user.log on tools-prometheus-03 to free up some space on /

2022-04-08

• 10:44 arturo: disabled debug mode on the k8s jobs-emailer component

2022-04-05

• 07:52 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (d7d3463) - cookbook ran by arturo@nostromo
• 07:44 wm-bot: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (d7d3463) - cookbook ran by arturo@nostromo
• 07:21 arturo: deploying toolforge-jobs-framework-cli v7

2022-04-04

• 17:05 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (cbcfc47) - cookbook ran by arturo@nostromo
• 16:56 wm-bot: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (cbcfc47) - cookbook ran by arturo@nostromo
• 09:28 arturo: deployed toolforge-jobs-framework-cli v6 into aptly and installed it on buster bastions

2022-03-28

• 09:32 wm-bot: cleaned up grid queue errors on tools-sgegrid-master.tools.eqiad1.wikimedia.cloud (T304816) - cookbook ran by arturo@nostromo

2022-03-15

• 16:57 wm-bot: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-emailer:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer (084ee51) - cookbook ran by arturo@nostromo
• 11:24 arturo: cleared error state on queue continuous@tools-sgeexec-0939.tools.eqiad.wmflabs (a job took a very long time to be scheduled...)

2022-03-14

• 11:44 arturo: deploy jobs-framework-emailer 9470a5f (T286135)
• 10:48 dcaro: pushed v0.33.2 tekton control and webhook images, and bashA5.1.4 to the local repo (T297090)

2022-03-10

• 09:42 arturo: cleaned grid queue error state @ tools-sgewebgrid-generic-0902

2022-03-01

• 13:41 dcaro: rebooting tools-sgeexec-0916 to clear any state (T302702)
• 12:11 dcaro: Cleared error state queues for sgeexec-0916 (T302702)
• 10:23 arturo: tools-sgeeex-0913/0916 are depooled, queue errors. Reboot them and clean errors by hand

2022-02-28

• 08:02 taavi: reboot sgeexec-0916
• 07:49 taavi: depool tools-sgeexec-0916.tools as it is out of disk space on /

2022-02-17

• 08:23 taavi: deleted tools-clushmaster-02
• 08:14 taavi: made tools-puppetmaster-02 its own client to fix `puppet node deactivate` puppetdb access

2022-02-16

• 00:12 bd808: Image builds completed.

2022-02-15

• 23:17 bd808: Image builds failed in buster php image with an apt error. The error looks transient, so starting builds over.
• 23:06 bd808: Started full rebuild of Toolforge containers to pick up webservice 0.81 and other package updates in tmux session on tools-docker-imagebuilder-01
• 22:58 bd808: `sudo apt-get update && sudo apt-get install toolforge-webservice` on all bastions to pick up 0.81
• 22:50 bd808: Built new toollabs-webservice 0.81
• 18:43 bd808: Enabled puppet on tools-proxy-05
• 18:38 bd808: Disabled puppet on tools-proxy-05 for manual testing of nginx config changes
• 18:21 taavi: delete tools-package-builder-03
• 11:49 arturo: invalidate sssd cache in all bastions to debug T301736
• 11:16 arturo: purge debian package `unscd` on tools-sgebastion-10/11 for T301736
• 11:15 arturo: reboot tools-sgebastion-10 for T301736

2022-02-10

• 15:07 taavi: shutdown tools-clushmaster-02 T298191
• 13:25 wm-bot: trying to join node tools-sgewebgen-10-2 to the grid cluster in tools. - cookbook ran by arturo@nostromo
• 13:24 wm-bot: trying to join node tools-sgewebgen-10-1 to the grid cluster in tools. - cookbook ran by arturo@nostromo
• 13:07 wm-bot: trying to join node tools-sgeweblight-10-5 to the grid cluster in tools. - cookbook ran by arturo@nostromo
• 13:06 wm-bot: trying to join node tools-sgeweblight-10-4 to the grid cluster in tools. - cookbook ran by arturo@nostromo
• 13:05 wm-bot: trying to join node tools-sgeweblight-10-3 to the grid cluster in tools. - cookbook ran by arturo@nostromo
• 13:03 wm-bot: trying to join node tools-sgeweblight-10-2 to the grid cluster in tools. - cookbook ran by arturo@nostromo
• 12:54 wm-bot: trying to join node tools-sgeweblight-10-1.tools.eqiad1.wikimedia.cloud to the grid cluster in tools. - cookbook ran by arturo@nostromo
• 08:45 taavi: set `profile::base::manage_ssh_keys: true` globally T214427
• 08:16 taavi: enable puppetdb and re-enable puppet with puppetdb ssh key management disabled (profile::base::manage_ssh_keys: false) - T214427
• 08:06 taavi: disable puppet globally for enabling puppetdb T214427

2022-02-09

• 19:29 taavi: installed tools-puppetdb-1, not configured on puppetmaster side yet T214427
• 18:56 wm-bot: pooled 10 grid nodes tools-sgeweblight-10-[1-5],tools-sgewebgen-10-[1,2],tools-sgeexec-10-[1-10] (T277653) - cookbook ran by arturo@nostromo
• 18:30 wm-bot: pooled 9 grid nodes tools-sgeexec-10-[2-10],tools-sgewebgen-[3,15] - cookbook ran by arturo@nostromo
• 18:25 arturo: ignore last message
• 18:24 wm-bot: pooled 9 grid nodes tools-sgeexec-10-[2-10],tools-sgewebgen-[3,15] - cookbook ran by arturo@nostromo
• 14:04 taavi: created tools-cumin-1/toolsbeta-cumin-1 T298191

2022-02-07

• 17:37 taavi: generated authdns_acmechief ssh key and stored password in a text file in local labs/private repository (T288406)
• 12:52 taavi: updated maintain-kubeusers for T301081

2022-02-04

• 22:33 taavi: `root@tools-sgebastion-10:/data/project/ru_monuments/.kube# mv config old_config` # experimenting with T301015
• 21:36 taavi: clear error state from some webgrid nodes

2022-02-03

• 09:06 taavi: run `sudo apt-get clean` on login-buster/dev-buster to clean up disk space
• 08:01 taavi: restart acme-chief to force renewal of toolserver.org certificate

2022-01-30

• 14:41 taavi: created a neutron port with ip 172.16.2.46 for a service ip for toolforge redis automatic failover T278541
• 14:22 taavi: creating a cluster of 3 bullseye redis hosts for T278541

2022-01-26

• 18:33 wm-bot: depooled grid node tools-sgeexec-10-10 - cookbook ran by arturo@nostromo
• 18:33 wm-bot: depooled grid node tools-sgeexec-10-9 - cookbook ran by arturo@nostromo
• 18:33 wm-bot: depooled grid node tools-sgeexec-10-8 - cookbook ran by arturo@nostromo
• 18:32 wm-bot: depooled grid node tools-sgeexec-10-7 - cookbook ran by arturo@nostromo
• 18:32 wm-bot: depooled grid node tools-sgeexec-10-6 - cookbook ran by arturo@nostromo
• 18:31 wm-bot: depooled grid node tools-sgeexec-10-5 - cookbook ran by arturo@nostromo
• 18:30 wm-bot: depooled grid node tools-sgeexec-10-4 - cookbook ran by arturo@nostromo
• 18:28 wm-bot: depooled grid node tools-sgeexec-10-3 - cookbook ran by arturo@nostromo
• 18:27 wm-bot: depooled grid node tools-sgeexec-10-2 - cookbook ran by arturo@nostromo
• 18:27 wm-bot: depooled grid node tools-sgeexec-10-1 - cookbook ran by arturo@nostromo
• 13:55 arturo: scaling up the buster web grid with 5 lighttd and 2 generic nodes (T277653)

2022-01-25

• 11:50 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo
• 11:44 arturo: rebooting buster exec nodes
• 08:34 taavi: sign puppet certificate for tools-sgeexec-10-4

2022-01-24

• 17:44 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo
• 15:23 arturo: scaling up the grid with 10 buster exec nodes (T277653)

2022-01-20

• 17:05 arturo: drop 9 of the 10 buster exec nodes created earlier. They didn't get DNS records
• 12:56 arturo: scaling up the grid with 10 buster exec nodes (T277653)

2022-01-19

• 17:34 andrewbogott: rebooting tools-sgeexec-0913.tools.eqiad1.wikimedia.cloud to recover from (presumed) fallout from the scratch/nfs move

2022-01-14

• 19:09 taavi: set /var/run/lighttpd as world-writable on all lighttpd webgrid nodes, T299243

2022-01-12

• 11:27 arturo: created puppet prefix `tools-sgeweblight`, drop `tools-sgeweblig`
• 11:03 arturo: created puppet prefix 'tools-sgeweblig'
• 11:02 arturo: created puppet prefix 'toolsbeta-sgeweblig'

2022-01-04

• 17:18 bd808: tools-acme-chief-01: sudo service acme-chief restart
• 08:12 taavi: disable puppet & exim4 on T298501