Jump to content
Nova Resource : Clouddb-services/SAL
2024-06-14
19:08 andrewbogott: shutting down and deleting clouddb-services project for T359810
2024-01-09
12:01 taavi: delete old neutron ports used for old wiki replica VIPs T346947
2024-01-08
12:57 taavi: shut off old wiki replica proxies in cloud vps T346947
2023-11-29
13:28 taavi: restarting haproxy on clouddb-wikireplicas-proxy to kill long-lived mariadb connections
2023-06-05
18:36 taavi: add btullis to the project
2023-04-14
09:21 dhinus: shutdown old ToolsDB VMs (clouddb1001, clouddb1002) T301949
2023-04-04
08:35 taavi: revoke puppet certs for now-deleted clouddb1003/4 (osmdb replicas) T323159
2023-03-28
20:19 andrewbogott: deleted old osmdb servers clouddb1003 and clouddb1004. Both have been shut down for some time.
2023-03-06
15:38 dhinus: mariadb is up again on clouddb1001 T329970
15:33 dhinus: stopping mariadb in clouddb1001 for a few minutes T329970
2023-02-21
17:18 andrewbogott: shutting down postgres on clouddb1004/1003, then shutting down the vms
2023-02-20
20:19 taavi: manually start prometheus-mysql-exporter on clouddb1001
17:11 dhinus: mariadb on clouddb1001 is running again
17:07 dhinus: stopping mariadb on clouddb1001 for about 5 minutes (T329970 )
11:23 dhinus: mariadb on clouddb1001 is running again
11:16 dhinus: stopping mariadb on clouddb1001 for about 2 minutes (T329970 )
2023-02-17
16:13 dhinus: drop unused database s51203__baglama2_p T323502
16:12 dhinus: drop unused database s52467__new_hashtags T329461
2023-01-30
11:02 dhinus: 'SET GLOBAL read_only = 0;' after restarting mariadb in clouddb1001 (T328273 )
10:53 dhinus: restarting mariadb in clouddb1001 to apply minor version upgrade (T328273 )
2023-01-27
15:52 taavi: repaired wmf-mariadb101 installation on clouddb1001
10:35 dhinus: clouddb1001 unlock tables; (T301949 )
10:33 dhinus: clouddb1001 flush tables with read lock; (T301949 )
2023-01-25
14:37 dhinus: clouddb1001 unlock tables; (T301949 )
14:34 dhinus: clouddb1001 flush tables with read lock; (T301949 )
2023-01-24
16:01 dhinus: clouddb1001 unlock tables; (T301949 )
15:46 dhinus: clouddb1001 flush tables with read lock; (T301949 )
2023-01-05
09:12 dhinus: disabled puppet checks on clouddb100[3-4] (touch /.no-puppet-checks) T323159
2023-01-04
14:12 dhinus: deleted CNAME wikilabels.db.svc.eqiad.wmflabs (T307389 )
2022-12-09
17:20 wm-bot2: Increased quotas by 4000 gigabytes (T301949 ) - cookbook ran by fran@wmf3169
13:50 dhinus: clouddb100 set innodb_file_format=Barracuda and innodb_large_prefix=1 (those values were lost after a reboot)
2022-12-07
22:51 dhinus: remove read_only mode on clouddb1001 (SET GLOBAL read_only = 0;)
18:05 wm-bot2: Increased quotas by 100 gigabytes (T301949 ) - cookbook ran by fran@wmf3169
2022-11-11
17:37 dhinus: mariadb restarted on clouddb1002 and again in sync with primary (took a bit to catch up) T301949
16:14 dhinus: rsyncing a single database from clouddb1002 to tools-db-2 (as a test for T301949 )
16:12 dhinus: stopping mariadb for a few minutes on replica server clouddb1002 (T301949 )
2022-11-08
08:38 taavi: delete clouddb-wikilabels-01,02 T307389
2022-11-03
14:39 dhinus: depooling dbproxy1019 'confctl select "service=wikireplicas-b,name=dbproxy1019" set/pooled=no' T313445
13:58 dhinus: shutting down dbproxy1019 T313445
2022-11-02
09:37 taavi: shut down the wikilabels servers T307389
2022-07-02
05:56 taavi: toolsdb: add s54518__mw to list of replication ignored databases, data mismatch between primary and replica
2022-06-28
16:30 taavi: stopped mariadb on clouddb1002, starting rsync clouddb1002->tools-db-1 on a root screen session on tools-db-1 T301949
2022-05-17
14:12 taavi: enable gtid on toolsdb T301993
2022-02-17
12:03 taavi: add myself as a project member so I don't need to ssh in as root@
2021-09-02
18:52 bstorm: removed strange old duplicate cron for osmupdater T285668
2021-08-31
20:19 bstorm: attempting to resync OSMDB back to Feb 21st 2019 T285668
2021-08-30
22:07 bstorm: restarting osmdb on clouddb1003 to try to capture enough connections T285668
21:53 bstorm: disable puppet and osm updater script on clouddb1003 T285668
2021-05-26
16:53 bstorm: restarting postgresql since T220164 was closed. Hoping all connections don't get used up again.
2021-05-25
14:35 dcaro: taking down clouddb1002 replica for reboot of cloudvirt1020 (T275893 )
2021-05-04
22:57 bstorm: manually added cnames for toolsdb, osmdb and wikilabelsdb in db.svc.wikimedia.cloud zone T278252
2021-04-05
09:56 arturo: make jhernandez (IRC joakino) projectadmin (T278975 )
2021-03-10
12:44 arturo: briefly stopping VM clouddb-wikireplicas-proxy-2 to migrate hypervisor
10:57 arturo: briefly stopped VM clouddb-wikireplicas-proxy-1 to disable VMX cpu flag
2021-03-02
23:29 bstorm: bringing toolsdb back up 😟
2021-02-26
23:20 bstorm: rebooting clouddb-wikilabels-02 for patches
22:55 bstorm: rebooting clouddb-wikireplicas-proxy-1 and clouddb-wikireplicas-proxy-2 before (hopefully) many people are using them
2021-01-29
18:21 bstorm: deleting clouddb-toolsdb-03 as it isn't used
2020-12-23
19:20 bstorm: created clouddb-wikireplicas-proxy-1 and clouddb-wikireplicas-proxy-2 as well as the 16 neutron ports for wikireplicas proxying
2020-12-17
02:14 bstorm: toolsdb is back and so is the replica T266587
01:10 bstorm: the sync is done and we have a good copy of the toolsdb data, proceeding with the upgrades and stuff to that hypervisor while configuring replication to work again T266587
2020-12-16
18:34 bstorm: restarted sync from toolsdb to its replica server after cleanup to prevent disk filling T266587
17:31 bstorm: sync started from toolsdb to its replica server T266587
17:29 bstorm: stopped mariadb on the replica T266587
17:28 bstorm: shutdown toolsdb T266587
17:24 bstorm: settings toolsdb to readonly to prepare for shutdown T266587
17:06 bstorm: switching the secondary config back to clouddb1002 in order to minimize concerns about affecting ceph performance T266587
2020-11-15
19:45 bstorm: restarting the import to clouddb-toolsdb-03 with --max-allowed-packet=1G to rule out that as a problem entirely T266587
19:36 bstorm: set max_allowed_package to 64MB on clouddb-toolsdb-03 T266587
2020-11-11
08:20 bstorm: loading dump on the replica T266587
06:06 bstorm: dump completed, transferring to replica to start things up again T266587
2020-11-10
16:24 bstorm: set toolsdb to read-only T266587
2020-10-29
23:49 bstorm: launching clouddb-toolsdb-03 (trying to make a better naming convention) to replace failed replica T266587
2020-10-27
18:04 bstorm: set clouddb1002 to read-only since it was not supposed to be writeable anyway T266587
2020-10-20
18:36 bstorm: brought up mariadb and replication on clouddb1002 T263677
17:14 bstorm: shutting down clouddb1003 T263677
17:13 bstorm: stopping postgresql on clouddb1003 T263677
17:08 bstorm: poweroff clouddb1002 T263677
17:08 bstorm: stopping mariadb on clouddb1002 T263677
17:07 bstorm: shut down replication on clouddb1002 (now with task) T263677
17:05 bstorm: shut down replication on clouddb1002
2020-09-29
23:48 bstorm: restarting postgresql on clouddb1003 to clear the number of connections
2020-09-08
18:21 bstorm: copied the profile::mariadb::section_ports key into prefix puppet to fix puppet after the refactor for wmfmariadbpy
2020-07-16
17:56 bstorm: Significantly lifted traffic shaping limits on clouddb1001/toolsdb to improve network performance T257884
2020-06-09
22:55 bstorm_: Reset the passwords for T254931
2020-06-04
17:41 bd808: Restarting mariadb after hand applying {{gerrit|602433}} (T253738 )
2020-06-01
18:51 bstorm_: restarting mariadb on clouddb1002 T253738
2020-05-29
23:42 bstorm_: stopped puppet and restarting mariadb on clouddb1002 after filtering out a table T253738
2020-03-27
12:26 arturo: add myself as project admin
2020-03-20
13:43 jeh: upgrade puppetmaster to v5 T241719
2020-03-18
17:49 bstorm_: updated the tools-prometheus security group with the current IP addresses for the prometheus servers
2019-10-25
2019-10-24
18:34 bstorm_: downgraded clouddb1001 to 10.1.39 T236420 T236384
18:19 bstorm_: stopped puppet on clouddb1001 and removed unattended-upgrades for now T236384
16:03 bstorm_: stopped puppet on clouddb1002 and removed unattended-upgrades for now T236384
13:27 phamhi: add phamhi as user and projectadmin
12:34 arturo: start both clouddb1001 and clouddb1004 (T227540 )
11:15 arturo: poweroff VM clouddb1001, hypervisor will be powered off (T227540 )
11:13 arturo: poweroff VM clouddb1004 (T227540 )
11:09 phamhi: stopped postgresl in clouddb1004 (T227540 )
11:04 arturo: stopped mariadb in clouddb1001 (T227540 )
2019-09-30
17:15 bstorm_: restart wikilabels postgresql
17:15 bstorm_: restart osmdb on clouddb1003
2019-09-16
19:41 jeh: add openstack security group rules for Tools prometheus scrapes to mariadb exporter
19:28 jeh: add openstack security group rules for labmon prometheus scraps to mariadb exporter
2019-05-24
23:38 bstorm_: T224062 clouddb1002 is now free of postgresql services and the volume is reclaimed for toolsdb use
23:06 bstorm_: T224318 cloudb-wikilabels-02 is now a replica of clouddb-wikilabels-01!
21:00 bstorm_: T224062 Moved wikilabels postgres db to clouddb-wikilabels-01
2019-05-22
00:14 bstorm_: T224062 wikilabels postgres is now replicating to clouddb-wikilabels-01
2019-04-05
00:35 bstorm_: added the rsyncd ports to security groups for the osmdb cluster
2019-04-04
23:58 bstorm_: pg_basebackup finished on clouddb1004 and streaming replication is flowing
18:46 bstorm_: T219652 pg_basebackup started on clouddb1004
18:35 bstorm_: T219652 postgresql on clouddb1003 had to be restarted to get it to accept WAL connections
18:00 bstorm_: T219652 clouddb1003 is now the OSMdb primary
2019-03-28
23:17 bstorm_: clouddb1003 is now a full osmdb replica T193264
19:32 bstorm_: pg_basebackup started on clouddb1003 for osmdb T193264
2019-02-28
02:09 bstorm_: clouddb1002 is now the replica for clouddb1001/toolsdb and is catching up slowly
2019-02-18
19:01 bstorm_: labsdb1004 is now a replica of clouddb1001, which is toolsdb now.
18:26 arturo: (jaime T193264 ) setting clouddb1001 in read_write mode
18:12 arturo: T193264 pointing tools.db.svc.eqiad.wmflabs to clouddb1001
17:55 arturo: (jaime T193264 ) set clouddb1001 in read_only=1
16:11 andrewbogott: deleting clouddb-utils-01 VM and associated puppet prefix; we aren't going to run maintain_dbusers here after all
15:51 andrewbogott: removing dns and public IP from clouddb1001
2019-02-17
21:21 bstorm_: The slave of labsdb1005.eqiad.wmnet is now clouddb1001.clouddb-services.eqiad.wmflabs
19:16 arturo: T193264 delete VM clouddb-services-01
18:54 arturo: T193264 create VM clouddb-services-01 for PoC of running maintain-dbusers from here
2019-02-16
13:59 arturo: T193264 switched clouddb1001/1004 to the new project local puppetmaster
13:54 arturo: T193264 create 'clouddb10' puppet prefix to store puppet/hiera config for database servers in this project
13:47 arturo: T193264 create 'clouddb-services-puppetmaster' puppet prefix to store puppet/hiera config for this project puppetmaster
13:43 arturo: T193264 create 'clouddb-services-puppetmaster-01' instance
13:33 arturo: add myself as user and projectadmin
2019-02-15
23:42 bd808: Added BryanDavis (self), Arturo Borrero Gonzalez, Marostegui, and Jcrespo as admins in project
22:49 bstorm_: created mariadb security group and lvs for a new database T193264
