Jump to content
Nova Resource : Codesearch/SAL
2025-03-28
19:39 Krinkle: Reboot codesearch9, web service was down and ssh not responding/timing out. Alert fired 41min ago about instance being perceiced as down.
2024-12-11
08:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T381911 )
08:12 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase (T381911 )
2024-10-15
18:10 mutante: - re-enabled puppet on codesearch9
2024-10-10
19:36 mutante: stopping and restarting all hound services
17:24 mutante: rebooting codesearch9 again
16:08 mutante: running commands listed in commit message of gerrit:920243
16:02 mutante: adding new project member eoghan and granting member role, also granting member role to aokoth
2024-10-09
16:52 mutante: soft rebooting instance codesearch 9 which was unreachable again
2024-10-03
17:16 mutante: - service not reachable - cant ssh to instance codesearch9, attempting to soft reboot VM
2024-06-18
14:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T367878 )
14:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase (T367878 )
2024-06-14
16:59 mutante: createing instance codesearch9 with bookworm to replace codesearch8 (T367479 )
2024-04-19
21:22 Krinkle: Attempt to fix puppet failure since 48h `codesearch8 puppet-agent[21081]: (/Stage[main]/Codesearch/Git::Clone[labs/codesearch]/Exec[git_pull_labs/codesearch]/returns) error: insufficient permission for adding an object to repository database .git/objects`
2024-04-03
2024-03-15
03:53 Krinkle: Deploying fix for T321347
2024-03-01
11:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
11:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
11:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
11:38 dcaro: reboot codesearch8 as it's locked out of resources
11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
2023-07-12
17:48 mutante: added new member aokoth
2023-05-24
15:11 legoktm: temporarily taking down to free up disk space (T337263 )
2023-04-21
14:57 legoktm: rebooting codesearch8 to see if that fixes networking issues (T335197 )
2023-03-13
01:12 Krinkle: Change codesearch.wmcloud.org proxy from port 3002 (Hound UI) to 3003 (new Codesearch frontend), ref T263354
2023-03-12
2022-12-02
19:41 Krinkle: codesearch8 instance unresponsive to http and ssh, rebooted from horizon
2022-11-01
04:35 legoktm: temporarily stopping to fix full disk
2022-06-17
20:04 legoktm: added Krinkle as project admin, Gerrit +2
2022-05-20
05:16 legoktm: restarting to pick up label changes
2022-04-09
03:55 legoktm: restarting everything to add new "wmcs" instance
2022-03-15
02:09 legoktm: forcing manual run of codesearch-write-config for T303641
2021-11-03
16:06 legoktm: manually triggered codesearch-write-config job to pick up fix for T294915 immediately
2021-10-24
22:39 legoktm: one more restart of everything, switched hound container over to bullseye for newer git
22:20 legoktm: restarted everything to add new prometheus metrics endpoint: https://codesearch.wmcloud.org/_metrics
2021-10-23
16:07 majavah: hard reboot codesearch8 after OOM crash
2021-05-21
16:08 legoktm: restarting everything for T283317
2021-05-20
23:45 legoktm: restarting to switch docker images
2021-04-14
01:38 legoktm: deleting codesearch6 instance (T279763 )
2021-04-13
20:53 legoktm: switched codesearch.wmcloud.org and codesearch-beta.wmcloud.org to point to codesearch8 (T279763 )
2021-04-09
08:57 legoktm: deleting codesearch7, creating codesearch8 as the correct size (T279763 )
08:52 legoktm: creating codesearch7 instance (T279763 )
08:48 legoktm: moved role::labs::lvm::srv from project puppet to codesearch06's puppet with the goal of moving to cinder
2021-03-05
23:45 legoktm: restarting for new shouthow profile (T253597 )
2021-01-26
02:01 legoktm: restarting for new puppet profile (T272947 )
2021-01-09
00:52 legoktm: bunch of restarts, repos with non-master default branches should be properly supported
2020-11-25
19:03 legoktm: fixed permissions on /srv/codesearch/.git/objects, forcing puppet run which is triggering restarts
2020-11-18
22:45 Amir1: restart codesearch to pick up new config (T268187 )
2020-11-09
11:05 Amir1: restart codesearch
2020-10-30
22:43 GarbageMonster: restarting codesearch to pick up new config (T266909 )
2020-10-05
04:57 legoktm: restarting to deploy T264557
2020-09-22
20:33 legoktm: (re)starting all hound instances now that gerrit-replica is back up
2020-08-28
19:17 Amir1: restarting codesearch to pick up the new config (T261517 )
2020-08-12
08:57 legoktm: restarting proxy to implement etag changes
2020-08-11
23:48 legoktm: restarting hound-deployed for the same reason
21:30 Amir1: restarting hound-search to pick up the new config for services (T260124 )
2020-07-26
2020-07-24
13:39 Amir1: restarting codesearch to pick up the new config (T231572 )
2020-07-22
14:03 Amir1: restart codesearch to pick up new config (adding mediawiki/vagrant)
2020-07-13
21:04 Amir1: create codesearch.wmcloud.org dns proxy (T256206 )
2020-06-28
09:31 legoktm: root@codesearch6:~# chown codesearch:codesearch -R /srv/codesearch
2020-06-11
11:04 legoktm: restarting everything after gerrit-replica 502s fixed T255094 T255125
2020-03-06
18:03 bd808: Forced Puppet run on codesearch6 to apply Puppet fix for T246017
2020-02-25
21:52 Amir1: restart hound proxy
21:50 Amir1: sudo service hound-libraries restart (T246120 T246121 )
2020-02-24
17:51 bd808: `systemctl restart docker` on codesearch6 (T246017 )
17:37 bd808: `systemctl restart hound-search.service` on codesearch6 (T246017 )
2020-02-22
16:54 Amir1: hard reboot of codesearch6 (T245920 )
2020-02-19
14:48 Amir1: restarting hound_proxy to pick up the new changes
2020-02-13
19:02 Reedy: is down "Hound is still starting up, please wait a few minutes for the initial indexing to complete."
2020-01-25
04:14 legoktm: restarted for new libraries search profile
2020-01-23
02:00 legoktm: shutting down and deleting temporary codesearch5 instance
01:59 legoktm: shutting down old codesearch4 instance (but not deleting yet)
01:57 legoktm: deleting temporary codesearch6.wmflabs.org domain
01:56 legoktm: retargetting codesearch.wmflabs.org to codesearch6 (T242319 )
2020-01-19
06:08 legoktm: temporarily taking everything down to free up disk space (T243121 )
2020-01-17
07:37 legoktm: rolling restart of hound instances to test new bootstrapping overload protection
2020-01-16
20:17 legoktm: legoktm@codesearch5:~$ sudo crontab -u codesearch -r
2020-01-11
03:57 legoktm: added mutante as project admin to help with puppetization efforts
2019-12-22
12:29 Amir1: ladsgroup@codesearch4:~$ sudo service hound-armchairgm restart (T241319 )
2019-12-15
01:30 legoktm: rolling restart of all hound instances because of gerrit-replica outage (T240776 , T240763 )
00:40 Amir1: sudo service hound-search restart (T240776 )
2019-10-16
10:17 Amir1: ladsgroup@codesearch4:~$ sudo service hound-services restart (T235596 )
2019-09-30
16:14 Amir1: soft-rebooting codesearch4
15:49 Amir1: ladsgroup@codesearch4:~$ sudo MODE=restart /srv/codesearch/manage.sh (T234211 )
2019-09-29
21:49 Amir1: git pull for puppet (T219248 )
2019-09-18
00:06 Amir1: another restart
2019-09-17
23:48 Amir1: restart to pick up the shiny new "services" section
2019-09-16
15:56 Amir1: ladsgroup@codesearch4:~$ sudo MODE=restart /srv/codesearch/manage.sh (It seems it's down, forcing a restart)
2019-09-03
22:39 Amir1: "sudo MODE=restart /srv/codesearch/manage.sh"
2019-08-07
20:10 legoktm: deleting now unneeded ggmirror-02 instance
20:04 legoktm: switched over to using gerrit-replica
2019-07-17
00:53 legoktm: manual rolling restart of everything to start using new gerrit mirror
2019-07-16
23:04 legoktm: deleted medium sized ggmirror-01 instance, it ran out of disk space. recreated as ggmirror-02 with size large
2019-07-11
15:01 legoktm: legoktm@codesearch4:/srv/puppet$ git pull
2019-06-06
18:15 legoktm: force updated config and doing a rolling restart (T223771 )
2019-05-24
21:54 legoktm: maintenance to reduce disk space
2019-03-28
08:13 legoktm: legoktm@codesearch4:/srv/puppet$ git pull # T219248
2019-01-03
00:35 legoktm: restarting hound instances for T211999
2018-12-03
06:17 legoktm: deleted codesearch3 instance (T210372 )
2018-11-26
18:33 duh: codesearch4: deleted all of /srv/hound/hound-core/data/ and restarted the hound-core service for a fresh clone (T210409 )
06:36 legoktm: retarget codesearch.wmflabs.org to codesearch4 instance
06:14 legoktm: created and setting up codesearch4 image based on Debian Stretch
06:03 legoktm: deleted codesearch-sourcegraph1 instance
2018-11-16
13:51 andrewbogott: migrating project to eqiad1
2018-11-05
07:46 legoktm: Implemented automatic restarts of hound to regularly pick up new config changes
2018-09-27
2018-09-21
07:36 legoktm: restarted to pick up new config / repository lists
2018-09-05
05:25 legoktm: restarted all hound backends after deleting all of their data
05:16 legoktm: stopping hound backends to clean up disk space
04:52 legoktm: restarted hound for gerrit:458105
2018-08-17
18:17 legoktm: rolling restart to pick up new config
2018-07-09
13:07 legoktm: restarting everything for config update
2018-04-13
17:41 legoktm: booted up codesearch3, it was down for some reason
2018-03-02
02:12 legoktm: deleted codesearch2 instance
02:09 legoktm: pointed codesearch.wmflabs.org at codesearch3:3002
02:05 legoktm: ran `mkfs.ext4 -T news /dev/mapper/vd-second--local--disk` (h/t bd808) to quintuple the number of inodes
01:52 legoktm: applied role::labs::lvm::srv role to codesearch3
01:52 legoktm: created codesearch3 instance (jessie, medium flavor)
2018-02-28
04:44 legoktm: restarted hound-search service (T188477 ), also I restarted hound-extensions earlier (T188450 )
2018-02-24
08:47 legoktm: added MaxSem as a project admin
2018-02-23
22:23 legoktm: added d3r1ck's correct account as project admin
2018-02-14
20:19 legoktm: added Addshore, D3r1ck01, Ladsgroup, and Prtksxna to Gerrit labs-codesearch project
20:12 legoktm: added Addshore, D3r1ck01, Ladsgroup, and Prtksxna as project admins of codesearch
2018-01-18
13:32 legoktm: rebooting codesearch2 to see how well it recovers
11:36 legoktm: legoktm@codesearch2:~$ sudo systemctl enable hound_proxy
2018-01-16
22:49 chasemp: restarted everything I could find in the stack :)
2017-12-21
23:28 legoktm: restarting hound instances to pick up config change
