Jump to content

Release Engineering/SAL

From Wikitech

2025-04-26

  • 00:05 bd808: Punched a hole in the beta cluster network blocks to allow 38.242.176.0/22 through.

2025-04-24

2025-04-23

  • 22:59 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534)
  • 22:43 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534)
  • 22:15 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up a huge pile of new blocks (T392534)
  • 22:11 James_F: Zuul: [mediawiki/services/parsoid/testreduce] Switch Node 20 CI on, for T382177
  • 21:47 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534)
  • 21:29 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534)
  • 20:47 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534)
  • 17:43 James_F: Zuul: [mediawiki/services/parsoid/testreduce] Disable CI for now, for T382177
  • 16:57 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/commit/a80e5211100f1cc42e4ae020d4266ea22938eb5a (T383097)
  • 14:25 James_F: Zuul: [wikimedia/portals] Switch to Node 20, for T382179

2025-04-17

  • 10:15 hashar: gerrit: reparented apps.git to All-Archived-Projects.git in order to BLOCK `mediawiki-replication`. I have also archived all subprojects # T392198

2025-04-16

  • 20:59 bd808: Blocked 193.43.72.0/24 and 14.160.0.0/11 because beta was very, very sad
  • 16:02 James_F: Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst non-voting for now
  • 09:20 hashar: integration: restarted integration-puppetserver-01

2025-04-15

  • 22:02 James_F: Zuul: [mediawiki/extensions/WikiLambda] Make Catalyst job voting, for T368002
  • 19:40 bd808: Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392003)
  • 18:11 bd808: `bd808@deployment-cache-text08:~$ sudo service varnish-frontend restart` (T392003)
  • 18:06 bd808: `sudo puppet agent -tv` on deployment-cache-text08 to update varnish deny list (T392003)
  • 17:30 bd808: `shutdown -r now` on deployment-mediawiki14. Load has been growing for ~2 days.

2025-04-11

  • 19:53 James_F: Zuul: [oojs/router] Mark as archived, for T391709
  • 14:00 hashar: restarted integration-puppetserver: jvm went out of memory

2025-04-10

2025-04-08

2025-04-07

  • 17:20 bd808: `service navtiming stop` to halt "Unhandled exception in main loop, restarting consumer" crash loop (T391272)
  • 17:15 bd808: Reboot deployment-webperf21 (T391272)
  • 16:58 bd808: `puppet agent -tv` to catch up with missed puppet runs on deployment-webperf21 (T391272)
  • 16:56 bd808: `rm /var/log/user.log.1` on deployment-webperf21 (T391272)
  • 16:47 bd808: `sudo /usr/local/sbin/clean-stale-puppet-certs --clean` on deployment-puppetserver-1 to clean up dangling certs for deployment-elastic{09,10,11}

2025-04-04

2025-04-03

  • 15:06 James_F: Zuul: Configure the REL1_44 test and gate pipelines, for T390695
  • 13:33 James_F: Docker: [quibble-bullseye] Revert MardiaDB to 10.5, for (against) T366646
  • 13:08 James_F: Zuul: [mediawiki/extensions/MetricsPlatform] Publish JS docs

2025-04-02

2025-04-01

2025-03-31

2025-03-28

2025-03-27

  • 16:28 bd808: Moved Puppet configuration from deployment-cache-text08 to deployment-cache-text prefix Puppet
  • 16:05 bd808: `sudo systemctl restart varnish-frontend` on deployment-cache-text08 (T390209)
  • 15:05 bd808: Moved role::acme_chief::cloud from individual instance config to deployment-acme-chief Puppet prefix.
  • 00:55 bd808: Removed prefix puppet classes for deployment-acme-chief (T390128)

2025-03-26

2025-03-25

2025-03-24

2025-03-23

  • 18:41 James_F: Zuul: Add 0xDeadbeef to CI allowlist
  • 18:34 James_F: Zuul: [operations/debs/bdsync] Mark as archived, for T377882
  • 18:31 James_F: Zuul: [mediawiki/extensions/CheckUser] Add GrowthExperiments dependency, for T386435
  • 18:29 James_F: Zuul: [mediawiki/extensions/CategoryWatch] Add Echo CI dependency

2025-03-20

  • 23:31 bd808: integration: thcipriani added integration-agent-docker-106{0,1,2} earlier today (T389554)
  • 22:50 brennen: integration: added jenkins nodes for integration-agent-docker-106{3,4,5} with 3 executors per each (T389554)
  • 21:41 brennen: integration: launched integration-agent-docker-106{3,4,5} (T389554)
  • 21:25 eileen: civicrm upgraded from 7b532ad7 to fba4c3d6
  • 15:13 dancy: Rebooting integration-agent-docker-1046 (Seems to be be inaccessible since February)
  • 08:28 taavi: reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1129765

2025-03-19

2025-03-18

  • 20:25 bd808: Rebooting deployment-jobrunner05 because things just seem weird (T387631, T387276)
  • 15:18 sergi0: run CommunityUpdates config schema migration `foreachwikiindblist growthexperiments extensions/CommunityConfiguration/maintenance/migrateConfig.php CommunityUpdates` (T387737)

2025-03-14

2025-03-13

  • 21:29 dancy: Finished gitlab cloud runners k8s production cluster upgrade (T388836)
  • 20:42 dancy: Finished gitlab cloud runners k8s staging cluster upgrade (T388836)
  • 20:09 dancy: Starting gitlab cloud runners k8s production cluster upgrade (T388836)
  • 19:26 dancy: Starting gitlab cloud runners k8s staging cluster upgrade (T388836)

2025-03-11

  • 22:54 bd808: Deleted unattached volumes: alert01, db09, deploy03, mwmaint, ores02, parsoid14-srv, prometheus05
  • 22:39 bd808: Released unused floating IPs 185.15.56.9 and 185.15.56.97 back to global pool
  • 22:08 bd808: Updated mail.beta.wmflabs.org service name to point to 185.15.56.115
  • 22:04 bd808: Deleted orphan parsoid-external-ci-access.beta.wmflabs.org. DNS record
  • 21:53 bd808: Deleted dangling prometheus-beta.wmcloud.org web proxy
  • 21:50 bd808: Deleted dangling w-beta.wmflabs.org web proxy
  • 21:42 bd808: Deleted unused "deployment-parsoid" Prefix Puppet configuration
  • 20:48 James_F: Docker: [quibble-bullseye-php81 & php81] Use PCRE2 backport from component/php81, for T386006
  • 13:19 James_F: Zuul: [mediawiki/extensions/ActiveAbstract] Mark as archived, for T382069
  • 03:54 eileen: civicrm upgraded from f2222fcd to ec20a105

2025-03-10

  • 15:20 James_F: Zuul: [mediawiki/services/servicelib-node] Mark as archived, for T388424
  • 13:47 hashar: gerrit: removed leftover empty directory `/srv/gerrit/plugins/lfs`. Data have been migrated to `/srv/gerrit/plugins/lfs` as part of moving Gerrit data out of `/`. See T333143

2025-03-08

  • 01:22 James_F: Zuul: [php-session-serializer] Enable PHP 8.4 as voting, for T368270

2025-03-07

  • 21:00 James_F: Zuul: [mediawiki/libs/Shellbox] Enable PHP 8.4 as voting, for T386570
  • 20:53 James_F: Zuul: [wikipeg] Enable PHP 8.4 as voting, for T386570
  • 20:07 James_F: Zuul: [mediawiki/libs/Equivset] Enable PHP 8.4 as voting, for T387806

2025-03-05

  • 00:21 dancy: Reeanbled beta-scap-sync-world (T166010)

2025-03-04

  • 23:26 dancy: Disabling beta-scap-sync-world for noise reduction while dealing with T166010
  • 22:10 James_F: Zuul: [mediawiki/services/example-node-api] Mark as archived, for T387933
  • 01:42 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Disable on PHP 8.4, for T386570
  • 01:13 James_F: Zuul: Add WgevaertWikiBase to CI allowlist
  • 01:03 James_F: Zuul: Start testing in PHP 8.4 for 'mediawiki-php-library' repos, for T386108

2025-02-28

  • 18:20 dancy: Upgrading gitlab-runner to v17.7.1 in production gitlab-cloud-runners (T386297)
  • 18:12 dancy: Upgrading gitlab-runner to v17.7.1 in staging gitlab-cloud-runners (T386297)
  • 17:52 dancy: Upgraded scap to 4.138.0 in beta cluster
  • 16:43 bd808: Deleted now dangling parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud. DNS record (T385849)
  • 16:40 bd808: Deleted deployment-parsoid14.deployment-prep.eqiad1.wikimedia.cloud (T385849)
  • 16:39 bd808: Deleted parsoid-external-ci-access.wmcloud.org proxy (T385849)
  • 16:37 bd808: Deleted deployment-alert01.deployment-prep.eqiad1.wikimedia.cloud (T385849)
  • 16:36 bd808: Deleted deployment-bastion.deployment-prep.eqiad1.wikimedia.cloud (T385849)

2025-02-27

2025-02-26

  • 20:21 James_F: jforrester@doc1003:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/LdapAuthentication/ #T376097
  • 20:18 James_F: Zuul: [mediawiki/extensions/LdapAuthentication] Mark as archived, for T376097
  • 13:20 hashar: Updating Quibble jobs to 1.13.0. "Skip execution upon a success cache hit" which would make some jobs to skip tests entirely when a set of commits/image is known to have previously passed # T383243 | dduvall
  • 11:06 hashar: Tag Quibble 1.13.0 @ 0ac128f # T383243
  • 09:11 hashar: deployment-prep: cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/1122901 "php: use component/pcre2 when using Php 8.1" to fix php # T387276
  • 01:55 bd808: `./jjb-update 'integration-quibble-fullrun-*-php81' '*-php81-phan' '*php81*'`
  • 01:16 Reedy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1122700 T386006

2025-02-25

  • 20:25 James_F: Docker: [php81] Update PHP to 8.1.31-1+wmf11u4, for T386006
  • 14:07 James_F: Docker: [php81] Upgrade Wikimedia's PHP to 8.1.31-1+wmf11u3 & PCRE to 10.42 for T386006

2025-02-24

2025-02-22

  • 11:27 taavi: rebooting integration-agent-docker-1047 which thinks it is gerrit

2025-02-21

  • 22:54 brennen: gitlab: removing expiration date for 14 tokens expiring in 2025 (T385930)
  • 22:36 brennen: gitlab: set require_personal_access_token_expiry and service_access_tokens_expiration_enforced to false

2025-02-20

  • 20:15 dancy: Updated buildkitd to v0.20.0 in gitlab-cloud-runners (T386955)
  • 20:15 dancy: Updated buildkitd to v0.20.0 in gitlab-cloud-runners

2025-02-19

2025-02-18

2025-02-13

  • 13:57 James_F: Zuul: [mediawiki/extensions/CirrusSearch] Drop WikibaseCirrusSearch dep, for T386015

2025-02-12

  • 17:22 James_F: Zuul: Add User:Michi j to CI allowlist
  • 17:21 James_F: Zuul: Add Dragoniez to CI allowlist

2025-02-11

  • 15:43 James_F: Zuul: Make PHP 8.4 voting on lib repos where it already passes, for T386108

2025-02-10

  • 14:27 James_F: Zuul: Add Bunnypranav to CI allowlist

2025-02-08

  • 00:07 bd808: Added `profile::maps::osm_master::disable_waterlines_import_timer: false` to deployment-maps prefix hiera (T385921)

2025-02-07

2025-02-06

  • 23:53 bd808: Updated citoid-beta.wmflabs.org to point to deployment-docker-citoid02
  • 23:50 bd808: Deleted beta-prometheus.wmflabs.org; it was pointed to an IP now owned by the mdwikioffline project.
  • 23:43 bd808: Deleted recently orphaned spiderpig.wmcloud.org proxy after discussion with dancy
  • 16:20 bd808: Rebooted deployment-sessionstore06 (T385803)
  • 12:07 andrewbogott: rebooting all servers for T385264

2025-02-05

  • 19:17 James_F: Zuul: [mediawiki/extensions/DonationInterface] Switch CI from PHP74 to PHP82
  • 18:23 James_F: Zuul: [mediawiki/extensions/cldr] Raise FR-special job to REL1_43
  • 18:22 James_F: Zuul: [mediawiki/extensions/DonationInterface] Raise FR-special job to REL1_43
  • 18:11 James_F: Zuul: [labs/tools/heritage] Fold template into this, only user
  • 18:08 James_F: Zuul: [mediawiki/extensions/FundraisingEmailUnsubscribe] Test in PHP 8.2+ only
  • 17:29 James_F: Zuul: [mediawiki/core] Test fundraising branches against PHP 8.2
  • 17:19 James_F: Zuul: [mediawiki/extensions/FundraisingEmailUnsubscribe] Mark as non-prod

2025-02-03

2025-01-30

  • 15:12 James_F: Zuul: [mediawiki/extensions/Wikibase] Only inject EntitySchema on 1.43+, for T385175
  • 01:39 James_F: Zuul: [mediawiki/core] Remove composer variant from wmf branches
  • 00:42 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1115131

2025-01-29

  • 18:03 James_F: Zuul: Make FR REL1_43-php82 voting for cldr and FEU
  • 17:54 James_F: Zuul: Add FR REL1_43-php82 as experimental to other extensions
  • 17:40 James_F: Zuul: [mediawiki/extensions/cldr] Add FR REL1_43-php82 as experimental
  • 17:40 James_F: Zuul: [mediawiki/extensions/cldr] Re-enable FR-tech job as voting, passes fine
  • 16:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1115064
  • 16:33 hashar: gerrit: marked all legacy Puppet modules as read-only ( https://gerrit.wikimedia.org/r/admin/repos/q/filter:operations/puppet/ ) and removed the associated GitHub mirrors that existed for some of them

2025-01-28

2025-01-27

  • 18:17 hashar: Cleaned disk on integration-agent-docker-1051

2025-01-25

2025-01-24

  • 21:44 James_F: Revert "Zuul: Switch Fundraising jobs to REL1_43"

2025-01-23

  • 16:31 dancy: Updating production gitlab-cloud-runners to v17.6.1
  • 16:23 dancy: Updating staging gitlab-cloud-runners to v17.6.1

2025-01-22

  • 18:14 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add Wikibase as a phan dependency

2025-01-20

  • 09:55 hashar: Updating Quibble jobs to enable success cache experiment - T383243
  • 08:20 hashar: Updating all Jenkins jobs to update Quibble to 1.12.0

2025-01-17

  • 16:59 dduvall: Building Docker images for Quibble 1.12.0
  • 15:00 hashar: Building Docker images for Quibble 1.12.0
  • 12:56 hashar: Tag Quibble 1.12.0 @ 633099e # T365978, T383243

2025-01-14

  • 17:14 brennen: integration project: create integration-agent-docker-1059 for T383254
  • 16:50 brennen: integration project: create integration-agent-docker-1058 for T383254

2025-01-10

  • 15:55 dancy: Updating gitlab-cloud-runners (prod) to v17.5.5 (T383263)
  • 15:49 dancy: Updating gitlab-cloud-runners (staging) to v17.5.5

2025-01-09

2025-01-08

  • 22:07 hashar: castor: deleting potentially corrupted npm cache. On integration-castor05: sudo rm -fR /srv/castor/castor-mw-ext-and-skins/master/{wmf-quibble-selenium-php74,quibble-vendor-mysql-php74-selenium}/npm # T383237

2025-01-07

  • 22:07 hashar: Deleted /srv/zuul/git/operations/dumps/dcat on both contint1002 and contint2002 # T157818
  • 19:00 bd808: `/usr/local/sbin/clean-stale-puppet-certs --clean` (T383153)
  • 18:53 taavi: taavi@deployment-puppetserver-1:~$ sudo puppetserver ca clean --certname maps-master01.maps-experiments.eqiad1.wikimedia.cloud # T383153
  • 18:50 taavi: taavi@deployment-puppetserver-1:~$ sudo puppet node clean geoshapes.maps-experiments.eqiad1.wikimedia.cloud # T383153
  • 18:30 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance deployment-etcd04
  • 18:30 bd808@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance deployment-etcd04
  • 14:48 hashar: Manually renamed wikibase-daily-npm-audit-daily-node18-npmaudit to node20 variant and refresh the config with JJB
  • 14:33 James_F: Zuul: [mediawiki/extensions/WikiLambda] Only run standalone jobs in master

2025-01-06

  • 20:16 andrewbogott: removed the (non-existent?) role::mw_rc_irc from puppet config for deployment-ircd03.deployment-prep.eqiad1.wikimedia.cloud
  • 19:35 bd808: Manually generated missing en_US.UTF-8 locale on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud (T361381)
  • 19:32 bd808: Added `postgresql::postgis::postgresql_postgis_package: postgresql-15-postgis-3` to deployment-maps Prefix Puppet to work around default parameter problem (T361381)
  • 19:31 bd808: Issued new Puppet cert for deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud (T361381)
  • 19:27 bd808: Added `postgresql::postgis::postgresql_postgis_package: ignored` to deployment-maps Prefix Puppet to work around default parameter problem (T361381)
  • 19:15 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/71 (T382709)
  • 19:11 bd808: Added placeholders for `graphite_host` and `statsd` to deployment-webperf Prefix Puppet
  • 18:53 bd808: Fixed missing profile::swift::global_account_keys::{codfw, eqiad} placeholders breaking deployment-ms-* puppet runs
  • 18:38 bd808: Fixed incorrect deployment-restbase prefix puppet setting that was causing puppet run failures
  • 18:19 bd808: Issued a new Puppet client cert for traindev01.deployment-prep.eqiad1.wikimedia.cloud
  • 14:58 James_F: Zuul: Drop CI for REL1_41 branch, now EOL per T376550
  • 09:03 hashar: gerrit: flushed diff_intraline, diff_summary, gerrit_file_diff and git_file_diff caches after having turned on diff3 style # T359821

2025-01-02


Archives

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Release_Engineering/SAL&oldid=2296165"