Server Admin Log/Archive 18

This page archive server admin logs from January 1st 2011 till June 30th 2011

June 30

20:43 logmsgbot: awjrichards synchronized php-1.17/extensions/CentralNotice/special/SpecialBannerLoader.php 'r91218'
20:42 logmsgbot: awjrichards synchronized php-1.17/extensions/CentralNotice/special/SpecialBannerController.php 'r91218'
18:39 RoanKattouw: Running extensions/Vector/switchExperimentPrefs.php on enwiki to disable WikiLove for those with the "Exclude from experiments" preference
18:38 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove by default on enwiki'
18:28 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove on enwiki, non-default initially'
18:18 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 91190:
18:17 RoanKattouw: Deploying updates for ArticleFeedback and WikiLove
18:09 RobH: srv169 still offline, do not repool (set to false in pybal on lvs3)
18:00 RobH: doing reboot testing on srv169 and puppet testing. server is depooled, disregard notices.
17:56 RobH: depooled srv169, its not updating its /var/local/apache via puppet for some reason.
17:53 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Add code enabling WikiLove by default if $wmgWikiLoveDefault is true'
17:52 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add wmgWikiLoveDefault variable, set to false'
17:48 RobH: blog.wikimedia.org updated to wp3.1.4
16:48 RobH: srv169 reinstalled and returned to service. (lvs3 is not repooling, citing a key error that has since been removed, perhaps pybal nees to be refreshed?)
16:25 RobH: reinstalling srv169
10:57 mark: Changed preferred path for AS14907 (pmtpa/eqiad) - AS43821 (esams) from 14907 2828 6908 43821 to 14907 3257 1299 43821 (bidirectionally)

June 29

23:39 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 91090:
23:38 awjr: running scap to deploy CentralNotice changes (r91118)
23:25 logmsgbot: awjrichards synchronized php-1.17/extensions/CentralNotice/centralnotice.css 'r91118'
23:24 logmsgbot: awjrichards synchronized php-1.17/extensions/CentralNotice/centralnotice.js 'r91118'
23:03 RobH: db1001-db1048 (minus a few hosts mentioned in rt#1105) are installed, puppetized, and ready to be placed into replication service.
22:53 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Making CentralNotice config changes for test wiki (page path and CentralDBname)'
21:03 RobH: OS install running on db1033-db1048
20:53 RobH: db1033-db1048 raid setup, ready for OS installs
20:44 mark: Started rsync of ms5 commons thumb dir '1' to ms1004
20:23 RobH: working on db1033-db1048 (setting up hardware raid)
20:10 RobH: db1017-db1031 os installed, puppet updated, ready for replication setup (db1023 & db1032 need further troubleshooting)
19:54 RobH: os install on db1017-db1031 done, running puppet updates
19:45 mark: Started rsync of ms5 commons thumb dir '0' to ms1004
19:36 mark: Deployed ms1004 (eqiad) as caching thumbs server in a similar configuration as ms6.esams; requesting thumbs from ms5 and caching (and purging) them locally
19:35 RobH: db1023 and db1032 drac unresponsive, rt# 1111 abd 1112 to troubleshoot next onsite
19:23 RobH: rob is working installs on db1017-db1032
19:08 RobH: updated netboot.cfg for private1-b-eqiad broadcast
18:46 RobH: raid10 setup on db1017-db1027
18:30 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 91090:
18:27 RoanKattouw: Deploying r91089 (jQuery in the head and position=top support) with scap
18:11 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove on testwiki'
17:58 mark: OS-installed ms1004
17:33 RobH: os install running on db1017-db1021
17:22 mark: Deployed DNS entries for ms1001-1004
17:20 RobH: db1015 and db1016 os installed, puppet updated, ready to go into replication service
17:20 RobH: corrected dns, pushed db1017 and upwards into eqiad-b-private ip range
17:18 mark: Caught ms5 doing RAID10 resync check, stopped it using echo idle > /sys/block/md1/md/sync_action
16:43 RobH: updating dns for the all eqiad dbs
16:36 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'In the spirit of bug 29277: also set scriptDirUrl for $wgLocalFileRepo'
15:49 RobH: starting install on db1015-db1018
15:45 RobH: db1011, db1012, db1013, db1014 os installed, puppet updated, ready to go into replication service
14:49 RobH: installing db1011, db1012, db1013, & db1014
14:40 RobH: db1007, db1008, db1009, & db1010 os installed, puppet updated, ready to go into replication service
14:32 RobH: stat1.wikimedia.org server online for Erik Z
13:46 RobH: db1007, db1008, db1009, db1010 OS installed, needs puppet run
13:42 RobH: starting os install on stat1, db1010 and db1009. mislogged before, was finishing install on db1007 and db1008
13:39 RobH: updated dns with stat1 server info
13:31 RobH: db1008 & db1009 finishing OS install
01:50 RobH: db1005 & db1006 os installed, puppet updated, ready to go into replication service
00:42 logmsgbot: demon synchronizing Wikimedia installation... Revision: 91036: Pushing r91036 for Filepage.css fixes

June 28

23:29 RobH: db1003 and db1004 os installed, puppet updated, ready to be put into replication service
22:46 RobH: db1001 installed and updated with puppet, ready to be put into replication service
22:45 RobH: db1002 installed and updated with puppet, ready to be put into replication service
22:44 RobH: db1001 is installed with OS and up (has been for awhile), db1002 and db1003 finishing installs
22:24 RobH: carbon host online as install tftp host in eqiad.
22:10 RobH: syncing docroot and gracefully restarting apaches to take the update
22:01 RobH: carbon rebooting
21:43 RobH: testing tftp function of carbon from cp1001 as its on the same public vlan network
21:13 RobH: updated puppet to point tftp hosts in eqiad to carbon
21:13 RobH: rsyncing over the tftpboot directory from brewster to carbon
21:00 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28781. Change requested by jamesofur on irc'
20:51 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28781. This time without breaking the world'
20:33 RobH: reinstalling carbon
20:27 RobH: updated misc::install-server and brewster/carbon entires in puppet, install server roles are now called on a service by service bais. brewster also needs reboot for kernel update, so doing that now as well.
17:47 RobH: carbon host powered down, the misc::install-server class in puppet needs to be broken out into more specialized roles, carbon reinstalled and updated to just host the tftp roles of the install server
17:36 RobH: new install host carbon os installed, running puppet updates
17:11 RobH: copied db41 sudoers file to db40 since it was overwritten by old raid utils package. now maybe the log spam will stop
15:20 RoanKattouw: NTP has synced clock in prototype, it's got the correct time now
15:18 RoanKattouw: Installed ntp on prototype, its clock was 10 minutes behind. It seems to be slowly catching up now
14:58 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling while my eyes are on other stuff'
14:52 RobH: stat1 racked and remote accessible. has public vlan setup, needs IP allocated and os installed. (also needs to be added to racktables)
13:59 mark: Partly implemented European routing policy on AS14907 as well, and tuned a bit for routing southern routes via Tampa
13:27 logmsgbot midom cleared profiling data
13:27 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php
13:25 logmsgbot midom cleared profiling data
13:24 logmsgbot midom cleared profiling data
13:20 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round'
13:19 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round'
13:13 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round ended'
13:12 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round'
11:35 Nemo_bis: !Wikipedia and other !Wikimedia sites problems at 6:15 and 9:30 UTC due to database overload fixed in 30 min, config now tweaked.
10:41 mark: Set 'send-community' on the iBGP peer groups on csw1-sdtpa and csw5-pmtpa
10:41 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php
10:39 domas: reenabled full mysql pcache, watching with half of my right eye
10:39 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php
10:38 domas: reduced max_connections on db40 to 1000, enabled "connection recycling" (disconnects longest idling connections when max_connections gets reached), set wait_timeout=2
10:30 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'adding enwiki to the mix'
10:23 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling db-pcache on eswiki for transaction analysis'
09:52 logmsgbot: mark synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable MySQL parser cache'
09:22 mark: Activated BGP sessions to TiNet / AS3257
06:49 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled SubPageList3 on wikiversity'
06:45 logmsgbot: tstarling synchronized php-1.17/extensions/SubPageList3/SubPageList3.php 'r90659, fix fatal error'
06:31 logmsgbot: tstarling synchronized php-1.17/includes/objectcache/SqlBagOStuff.php 'r90942, proper fix'
06:21 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabling MySQL parser cache'
06:20 logmsgbot: tstarling synchronized php-1.17/includes/objectcache/SqlBagOStuff.php
06:16 Tim: temporarily disabled db40 parser cache due to connection errors
06:15 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php
06:14 logmsgbot: root restarted all apaches
06:14 logmsgbot: root is doing a hard restart of all apaches (apaches overload)
00:31 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling MySQL parser cache on all wikis'
00:09 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling MySQL parser cache on enwiki'

June 27

23:22 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'no automatic purging'
23:15 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling MySQL parser cache on en.wikiquote'
21:54 Ryan_Lane: repooling sq69
21:09 RobH: grosley back online
21:00 RobH: grosley reboot in process, disk checking.
20:57 RobH: powercycling grosley
20:56 RobH: grossley entirely unresponsive from serial console
20:51 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 29603 - enabled abusefilter for iawiki. Bug 29279 - turned off Devanagari digits for hiwiki'
20:36 logmsgbot: pdhanda ran sync-common-all 'Enabled abusefilter for iawiki. Bug 29603'
20:18 RobH: allocated 11/41 on csw1-sdtpa to vlan101 for use by stat1
19:32 RobH: raised power warnign threshhold on phases on ps1-d1-sdtpa from 12 to 16
18:13 RobH: rebooting carbon for installation in eqiad.
18:06 RobH: updating dns to have carbon mgmt dns info
17:42 andrewS: 47 r410 servers now await placement @ sdtpa
17:13 andrewS: 36 new dell servers sit outside sdtpa cage and await placement
16:40 Ryan_Lane: depooling sq69 for continued work on XFF for geoiplookup
15:30 JeLuF: increased puppet's timeout for "apt-get update" from 2 to 4 minutes. Regular apt-get update on storage1 takes more than 2 minutes
11:50 mark: Changed ganglia / gmond configuration in puppet to use a different, associative hash structure
03:06 Tim: changed mailman site password

June 26

13:13 rainman-sr: built search index for nycwikimedia

June 25

03:26 andrewS: 8 new servers in sdtpa d1 awaiting labels and power/data hookups
03:25 andrewS: sdtpa feels hotter than normal. will keep an eye on cage temp next few days.
00:30 Ryan_Lane: repooling sq69

June 24

22:26 notpeter: initialized and started repl between nfs1 and 2 for opendj
22:08 Ryan_Lane: depooling sq69 to test geoiplookup changes for https
21:36 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28529 - Change move-rootuserpages right in Portuguese Wikipedia'
21:32 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28398 - Custom namespaces for ku.wiktionary and ku.wikipedia'
21:24 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29527 - Enable AbuseFilter on Catalan Wikipedia'
21:23 logmsgbot: jeluf synchronized php-1.17/wmf-config/abusefilter.php '29527 - Enable AbuseFilter on Catalan Wikipedia'
20:00 andrewS: pmtpa: we should have power and the overhead railings to run data cables by late monday, as per Matt.
19:20 Ryan_Lane: depooled srv301
12:37 Tim-away: finished fixing ExtensionDistributor, re-enabled cron job and xinetd
07:37 Tim: XO appears to be back up, after ~3 hours of downtime
07:30 Tim: fixing ExtensionDistributor snapshot permissions, was owned by mark instead of extdist
06:21 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29168 - Define namespace aliases for Author, Page, Index on all Wikisources'
06:10 Tim: configured csw5-pmtpa to send its syslogs to 10.0.5.8 instead of 10.0.0.1
00:19 logmsgbot: tstarling synchronized php-1.17/includes/db/DatabaseMysql.php 'reapplied r90644, it was fixing an actual problem'

June 23

22:36 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 14407. Added Login icon for incubatorwiki'
22:33 logmsgbot: pdhanda synchronized images/sul/incubatorwiki.png 'Bug 14407. Login icon for incubatorwiki'
22:18 logmsgbot: pdhanda synchronized php-1.17/wmf-config/CommonSettings.php 'Bug 14407. Added incubatorwiki to CentralAuthAutoLoginWikis'
21:30 logmsgbot: catrope synchronized php-1.17/maintenance/update.php 'Sync out update.php too, just because Ariel is being paranoid about it :D'
21:27 ^demon: svn up'd /h/w/c/p/m/update.php for nice warning about not running update.php on the cluster
21:06 logmsgbot: pdhanda ran sync-common-all 'Updating to r90676 of extensions/UploadWizard'
21:03 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Enabling wgNoticeInfrastructure for testwiki per request by Kaldari'
20:36 pdhanda: updated to r90667 of extensions/Narayam
20:33 logmsgbot: pdhanda ran sync-common-all
18:38 JeLuF: #21237 - removed brion's email address from apache config
18:10 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28909 - Change path to Low Saxon Wiktionary logo'
18:04 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29391 - Enable e-mail notification for watchlists on br.wikimedia'
18:03 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29348 - Email notifications for stewardwiki'
18:00 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29360 - Define $wgUploadMissingFileUrl for ptwikibooks'
17:02 binasher: sync-file r90664 includes/db/LoadMonitor.php
17:01 logmsgbot: root synchronized php-1.17/includes/db/LoadMonitor.php 'max threads logging'
16:47 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 'attempting to set max threads on testwiki only'
16:32 RoanKattouw: Site came back up instantly after Tim disabled max_threads
16:28 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 'disabled max threads'
16:28 logmsgbot: catrope synchronized php-1.17/includes/db/LoadBalancer.php 'Make the "LB failure with no last connection" DB error log message more verbose'
16:24 binasher: upping max threads in db.php
16:24 logmsgbot: root synchronized php-1.17/wmf-config/db.php 'upping max threads'
16:17 logmsgbot: laner synchronized php-1.17/includes/db/DatabaseMysql.php 'Roll back Tim's getLag() change from 03:18 today (r90644)'
15:58 Ryan_Lane: restarting pybal on lvs3
15:52 logmsgbot: catrope synchronized php-1.17/includes/db/Database.php 'Roll back Tim's getLag() change from 03:18 today (r90644)'
15:50 RoanKattouw: Going to roll this back: 03:18 logmsgbot: tstarling synchronized php-1.17/includes/db/DatabaseMysql.php 'Database::getLag() fix from r90644'
15:46 RoanKattouw: Lots of "LB failure with no last connection" errors in dberror.log, suddenly started at 10:53 UTC
15:43 RoanKattouw: Torrus broken, reporting lots of NaNs
15:40 RoanKattouw: Strike that, site not completely down. Intermittent service reported in US/Canada
15:39 RoanKattouw: AMS squids all down in Nagios. Load is approximately halved on app servers. TPA squids look like they should be fine, but the site is reported to be down worldwide
15:37 RoanKattouw: Mark not near a computer, texting Ryan instead
15:36 RoanKattouw: All wikis seem to be down worldwide. Have texted mark
09:18 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29215 - Add import source for dearbcomwiki'
09:14 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29380 - Change logo of Kannada wiktionary (kn.wiktionary.org)'
08:48 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29515 - install Narayam in all Sanskrit wiki projects'
08:42 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29474 - Search Index and Author namespaces by default on all Wikisources which don't have a different configuration'
08:27 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29172 - Add Page, Author and Index namespaces to $wgContentNamespaces on all Wikisources'
07:53 JeLuF: fixed broken ganglia installation on project2 (reported by puppetmon)
07:51 JeLuF: added new class mobile::disabled in puppet that has apache disabled, assigned mobile1 to it.
07:19 logmsgbot: tstarling synchronized php-1.17/extensions/SubPageList3/SubPageList3.php 'r90654'
07:12 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'enabling SubPageList3 on testwiki'
07:11 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php
07:07 logmsgbot: root synchronizing Wikimedia installation... Revision: 90493:
07:05 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 90493:
07:04 Tim: running scap to push out SubPageList3 source directory (not enabled)
06:36 Tim: killed puppet-monitor on fenari, I'm pretty sure that's the thing that was spamming #wikimedia-tech every 3 minutes
05:34 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled mysql parser cache on testwiki'
05:33 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'mysql parser cache pre-deploy changes'
05:32 Tim: created "parsercache" database on db40 and added suitable wikiuser/wikiadmin grants
04:46 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'put db33 back'
04:46 Tim: restarted the parser cache hit rate monitor
03:20 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 'restored max lag to 30s, now that r90644 is deployed it shouldn't break anything'
03:18 logmsgbot: tstarling synchronized php-1.17/includes/db/DatabaseMysql.php 'Database::getLag() fix from r90644'
03:03 Tim: started ntpd on db36, not sure why it was stopped, it has an init script. Clock skew was 1.7 seconds.
02:55 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 'increasing max lag to 3000'
00:43 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'removed db33 for TS dump'

June 22

23:08 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabled extension NewUserMessage on rowikisource. Bug 28307.'
22:47 Ryan_Lane: repooling sq69
22:07 Ryan_Lane: depooling sq69 to test vcl changes for https geoiplookup
21:23 notpeter: resized db10:/a to 220G
19:10 Reedy: Wikimedia repo post commit hook fixed for CR
18:43 Reedy: Wikimedia repo set up as a CodeReview repo on mw.org
18:04 JeLuF: fixed puppetmon, log format has changed
13:29 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'put db21 back'
10:41 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'put db34 back'
02:14 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'remove db34 to dump for TS'
02:08 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'remove db21 to dump s6 for TS'
01:59 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'put db18 back'

June 21

23:33 logmsgbot: pdhanda ran sync-common-all 'Creating nyc.wikimedia.org Bug 29273'
23:29 pdhanda: creating nyc.wikimedia.org
22:41 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/SpecialArticleFeedback.php 'Attempt to fix fatal'
22:14 binasher: pulling srv301 from api lvs pool to prep mobile testing
21:40 binasher: returned mobile4 to lvs
20:32 logmsgbot: preilly synchronized php-1.17/extensions/WikimediaMobile/redirector_test.js 'r90541'
20:32 logmsgbot: preilly synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 'r90541'
20:31 logmsgbot: preilly synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.i18n.php 'r90541'
20:31 logmsgbot: preilly synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js 'r90541'
19:25 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'taking db18 out of rotation to dump s7 for TS'
19:26 river: taking db18 out of rotation to dump s7 for TS
18:56 Ryan_Lane: moving preilly from restricted to mortals, for deploy access
15:01 apergos: deleted the old ssh key for snap2.wm.o from the puppet db also, as fenari was whining about it similarly, hope that gets is. spence puppet run *finally* done, sheesh
13:17 apergos: after changing snapshot2 from external to internal subnet (oops forgot to log that huh), puppet on spence was complaining about the external resource Nagios_host[snapshot2] not being able to override the local resource, so in puppet db on db9 I deleted the old snapshot2 nagios resources, leaving only the new ones, puppet run is in process now on spence and is bleeping sloooow.
09:14 apergos: taking down snapshot2 in prep for os upgrade (reinstall)
01:52 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Fix checkuserwiki permissions back to how they were'
00:49 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Permission changes for checkuserwiki. Bug 28781.'

June 20

23:57 binasher: reconfiguring mobile1 for testing the new php mobile mediawiki extension
23:03 notpeter: added checks for google safe browsing for lots of MWF sites to nagios
22:53 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Missed Comment 1 in bug 28785.'
22:42 logmsgbot: reedy synchronized php-1.17/includes/api/ApiEditPage.php 'bug 29278 r90943'
21:57 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Changed language, sitename and namespace for checkuserwiki. Bug 28781.'
21:33 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Changed icons on checkuserwiki. Bug 28785.'
19:20 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r90480'
18:51 binasher: drop db29 (s6 master) weight to 0
18:51 logmsgbot: root synchronized php-1.17/wmf-config/db.php 'drop db29 weight to 0'
18:21 logmsgbot: root synchronized php-1.17/wmf-config/db.php 'adding db43 to s6 with a weight of 20'
18:21 binasher: increased db43 weight to 1500
18:07 binasher: just put db43 into rotation in the s6 cluster
18:04 logmsgbot: root synchronized php-1.17/wmf-config/db.php 'adding db43 to s6 with a weight of 20'
16:53 pdhanda: Upgrading wikitech.wikimedia.org to 1.17
15:43 apergos: started en pedia dumps run on snapshot 4, from root screen session as backup user. This will run as 32 jobs in parallel; we should keep a half an eye on the db in about 16-17 hours (previous runs had only 15 jobs going at once).
14:18 mark: Configured TiNet transit on cr2-eqiad (BGP sessions deactivated)
14:18 mark: Configured border ACL filtering on the eqiad border
00:46 logmsgbot: tstarling synchronized php-1.17/includes/db/LoadBalancer.php 'r90423'

June 17

20:39 RobH: email is working again, but address in question is not filtered
20:38 RobH: pushed a small exim change to try to sort out a sender address, but it didnt work, rolled back changes
16:44 RobH: upgraded blog.wikimedia.org to wordpress 3.1.3
16:44 RobH: restarted morebots.
00:33 Ryan_Lane: restarting pybal on amslvs4
00:32 Ryan_Lane: restarting pybal on amslvs2
00:31 Ryan_Lane: restarting pybal on amslvs3
00:30 Ryan_Lane: restarting pybal on amslvs1

June 16

23:51 Ryan_Lane: restarting pybal on amslvs3
23:51 Ryan_Lane: restarting pybal on amslvs1
23:50 Ryan_Lane: adding uploadsvc and uploadsecure services to amslvs1 and amslvs3
22:25 Ryan_Lane: restarting pybal on lvs1
21:47 RobH: puppet now works on eqiad hosts.
21:38 Ryan_Lane: enabled https for bits. still needs varnish changes.
21:38 RobH: updating dns with puppet cname in eqiad to sockpuppet
21:37 RobH: dispatched gilman to peter for puppetize testing of grosley
21:21 RobH: dns update for puppetmaster cname in eqiad
21:13 RobH: updated dns with carbon install server and cp server ranges
21:10 mark: Added all eqiad subnets to dhcpd.conf
21:10 mark: Prepared puppet's base.pp for eqiad
21:05 Ryan_Lane: restarting pybal on amslvs3
21:03 Ryan_Lane: killing and starting pybal on amslvs1
21:03 Ryan_Lane: restarting pybal on amslvs1
21:01 Ryan_Lane: adding bitssvc address to amslvs systems
20:52 notpeter: restarting drbd replication on nfs2
20:52 RobH: db1001 is our first wmf proper installed host in eqiad, huzzah
20:44 mark: Added static route for 10.2.1.23/32 to lvs2 on csw1-sdtpa
20:09 mark: Added 10.64.0.0/12 to squid http_access on brewster
18:51 Ryan_Lane: bound bitssvc address to lvs2, restarting pybal
18:33 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php
18:24 Ryan_Lane: restarting pybal on lvs2
18:24 Ryan_Lane: adding bitssvc and bitssecure services to pmtpa lvs
18:01 notpeter: reimaging nfs2
16:36 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php
14:54 andrewS: srv169 plugged in and powered on
14:26 RobH: updating again, there are db1001-1016 in a single rack, doh
14:24 RobH: updating dns with new dbs in eqiad
12:00 RoanKattouw: Set $wgInstantCommons back to true on prototype
11:59 RoanKattouw: Enabled WikiLove on prototype
11:15 andrewS: will be working on srv278 later today
11:12 logmsgbot: root synchronized php-1.17/wmf-config/mc.php
09:50 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'raise account creation throttle for el pedia workshop'
01:38 Tim: success, my SIP connection now works again
01:38 Tim: attempting to remove the SIP connection rate limit from sfoasterisk

June 15

21:32 awjr: killing epicly long running query in civicrm db on db9
21:28 logmsgbot: reedy synchronized php-1.17/wmf-config/codereview.php 'Couple more auto defers'
19:07 logmsgbot: catrope synchronized php-1.17/extensions/WikiLove/WikiLove.api.php 'r90138'
18:47 logmsgbot: catrope synchronized php-1.17/extensions/WikiLove/modules/ext.wikiLove/ext.wikiLove.core.js 'r90135'
18:45 logmsgbot: catrope synchronized php-1.17/extensions/WikiLove/WikiLove.hooks.php 'r90135'
18:18 logmsgbot: catrope synchronized php-1.17/resources/mediawiki.util/mediawiki.util.js 'r90128'
18:03 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove on officewiki'
18:02 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Add switch for WikiLove'
18:01 mark: Upgraded msw1-eqiad to JUNOS 10.4R3.4
17:59 RoanKattouw: Adding WikiLove table on officewiki (extensions/WikiLove/patches/WikiLoveLog.sql)
17:58 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add $wmgUseWikiLove, false by default'
17:53 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 90124:
17:52 RoanKattouw: Syncing WikiLove code. Code sync only, not enabled anywhere yet
16:19 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Revert raised account creation throttle for Lennart on testwiki'
12:37 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29351 Set timezone to America New_York for pcdwiki'
05:17 apergos: phase 2 of integrity checks on dump files (bz2s) running from screen on sn1 as root

June 14

19:41 RobH: chad is crazy it seems.
19:41 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29215: import sources for arbcom_dewiki'
19:40 logmsgbot: catrope synchronized php-1.17/README 'Test sync to confirm whether Chad is crazy'
18:29 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'bug 29331 - Set e-mail capture sender to improve@wikimedia.org'
17:50 notpeter: restarting udp2log on locke after upgrade
15:10 mark: Upgraded asw-a-eqiad to JUNOS 10.4R3.4
14:42 mark: Upgraded asw-b-eqiad to JUNOS 10.4R3.4 (jtac recommended version)
14:26 apergos: running a bunch of rudimentary integrity checks on xml dumps, out of screen as root on snapshot1
06:09 logmsgbot: tstarling synchronized php-1.17/includes/HistoryBlob.php 'r90036'

June 13

23:40 notpeter: taking down nfs-home on nfs2
23:03 Ryan_Lane: ran sync-docroot - how does this not get logged?
19:16 RobH: snapshot4 properly boots to serial console display on bootloader and os
18:31 RobH: snapshot4 rebooting to troubleshoot console redirection
13:27 apergos: snapshot4 installed and *almost* correct (stray apache started up but the rest looks ok)

June 12

19:47 JeLuF: Software RAID on fenari was broken, mirror sdb1 was failed. Re-added sdb1. If the error is persistent, sdb needs to be replaced.
19:19 JeLuF: cron.{hourly,daily,...} on hume was not running due to a broken record in /etc/crontab, so logrotate was not running
12:58 logmsgbot: ariel synchronized php-1.17/includes/HistoryBlob.php 'check for existence of mhash function, not extension (need for lucid php build) (take 2, after svn up :-P)'
12:51 logmsgbot: ariel synchronized php-1.17/includes/HistoryBlob.php 'check for existence of mhash function, not extension (need for lucid php build)'

June 11

07:37 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Move section disabling minordefault out of the if($wmgUseUsabilityInitiative) conditional and document it'
07:12 RoanKattouw: Running script to fix preferences of users that can't turn off minordefault on enwiki. See bug 24313 comment 55. Script is in maintenance/fixMinorPrefs.php

June 10

21:26 binasher: mobile2 goes natty
18:08 Ryan_Lane: added ip and dns entry for internproxy.wikimedia.org in the sandbox vlan
17:28 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'See whether this fixes bug 29314'
15:21 RobH: copying old data from ms4 for solaris specific info. do not disrupt ms4.
02:24 RobH: ms4 repaired and back online, ready for reinstall per ticket 885
01:15 Ryan_Lane: starting services on virt1, since iptables rules are in

June 9

23:54 binasher: upgrading mobile2 to maverick (pre natty)
23:04 Ryan_Lane: added preilly account as a restricted user. Added him to mobile1-5
22:30 binasher: upgrading mobile3 to natty ice
21:38 binasher: upgrading mobile3 to maverick (then natty)
20:02 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 89789:
20:02 RoanKattouw: Running scap to deploy ArticleFeedback changes
18:59 binasher: mobile5 natty
18:42 binasher: mobile5 upgrade to maverick on the way to natty
15:28 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Raise account creation throttle for Lennart's IP on testwiki'
15:10 RobH: dns updated, all nameservers dig correctly.
15:10 RobH: pushing dns updates to add mgmt of all eqiad devices
01:22 Ryan_Lane: shutting off services on virt1 (including puppet) until I get a chance to firewall them properly

June 8

22:35 Ryan_Lane: rebooting virt1
21:58 hcatlin: deploying to mobile cluster
21:34 Ryan_Lane: changing controller.labs.wikimedia.org to a CNAME for virt1.wikimedia.org
21:27 mark: Started ircd on ekrem
21:20 mark: Moved routing of vlan 101 from csw1-sdtpa to csw5-pmtpa
21:06 binasher: changing pybal healthcheck url for mobile
20:47 Ryan_Lane: rebooting controller.labs.wikimedia.org
20:44 Ryan_Lane: restarting pdns on ns0, it's locked up
20:33 Ryan_Lane: make that deleting virt1.pmtpa.wmnet, and changing virt1.mgmt.pmtpa.wmnet to controller.labs.mgmt.pmtpa.wmnet
20:32 Ryan_Lane: changing virt1.pmtpa.wmnet to controller.labs.pmtpa.wmnet
20:27 Ryan_Lane: changing ip of controller.labs.wikimedia.org
20:17 mark: Rebooting ekrem
20:16 mark: Changed ekrem's ip
20:10 mark: Added 2nd ip to ekrem (208.80.152.178)
19:56 binasher: upgrade mobile4 to natty
19:53 mark: Shutdown ekrem's switch port and null routed the ip for a while
19:53 Ryan_Lane: deleting vmdev1 and vmdev2, and moving the controller.labs.wikimedia.org entry into the public services subnet
19:45 binasher: upgrading mobile4 to maverick on its way to natty
19:31 Ryan_Lane: reclaiming sandbox ips for owa.tesla, owa1.tesla, owa2.tesla, lvs1.tesla, and grid.tesla
18:10 Ryan_Lane: depooling mobile5
18:05 Ryan_Lane: changing weight for mobile5
17:42 Ryan_Lane: upping the weight of mobile5 to 24
17:27 Ryan_Lane: upping weight of mobile5 to 16
17:18 Ryan_Lane: depooling mobile4
17:16 Ryan_Lane: reducing weight of mobile4/5 to 8
17:13 Ryan_Lane: repooling mobile4/5 with a weight of 16
16:29 ^demon|away: irc feeds are down, somebody who has access should investigate.
15:39 mark: Rebooting sq71 to remove old temp eqiad tunnel config
14:49 mark: Setup PIM-SM on eqiad core routers and joined the pmtpa domain
14:49 mark: Changed pmtpa PIM-DM to PIM-SM
11:31 mark: Added A records for cr1-eqiad and cr2-eqiad loopback interfaces
10:47 mark: Set local-pref value for private peers on cr1-eqiad and cr2-eqiad
10:43 mark: Removed cr1-eqiad:ae1.666 configuration to remove the temporary tunneled link
10:39 mark: Configured OSPF and OSPFv3 on csw1-sdtpa:e16/1 and cr2-eqiad:xe-5/2/1
10:39 mark: Assigned IP addresses to csw1-sdtpa:e16/1 and cr2-eqiad:xe-5/2/1 to bring up the second 10G wave
00:37 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling CustomUserSignup on testwiki'

June 7

23:51 Ryan_Lane: rebooting virt1/2
23:21 Ryan_Lane: rebooting virt1
23:21 Ryan_Lane: moving nova-network to virt2, as it was causing issues accessing the public IP on virt1
23:12 Ryan_Lane: rebooting virt1
22:33 Ryan_Lane: removed nova-network from virt2-4, since multiple network nodes isn't supported. Added to virt1. Rebooting virt2-4.
22:11 hcatlin: deploying to mobile cluster
21:55 Ryan_Lane: restarting opendj on nfs2
19:56 logmsgbot: tfinc synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling collections extension on wikinews'
19:39 andrewS: ms4 ticket 729 updated, system is in testing
17:46 Ryan_Lane: rebooting virt1 to enable virtualization in the bios
17:42 Ryan_Lane: rebooting virt2 to enable virtualization in the bios
17:39 Ryan_Lane: rebooting virt3 to enable virtualization in the bios
17:35 Ryan_Lane: rebooting virt4 to enable virtualization in the bios
17:01 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Disabling Tidy on collabwiki per bug 29295'
16:50 mark: Adjusted BGP filters on cr1-eqiad to allow advertisement of pmtpa aggregates to BGP peers
16:29 mark: Removed static routes from cr1-eqiad and cr2-eqiad
16:18 mark: Setup full mesh of iBGP sessions between csw5-pmtpa, csw1-sdtpa, cr1-eqiad, cr2-eqiad (separate v4/v6 sessions)
15:17 mark: Removed IPv6-only iBGP session between cr1-eqiad and cr2-eqiad and converted the v4 iBGP session into multi-address-family
14:22 mark: Setup OSPFv3 between csw1-sdtpa and csw5-pmtpa
14:08 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28292'
14:04 logmsgbot: reedy synchronized php-1.17/includes/api/ApiPageSet.php 'r89646'
13:58 mark: Setup OSPF and OSPFv3 on the first 10G wave circuit between csw1-sdtpa and cr1-eqiad
13:39 RobH: updaed puppet giving andrew shields basic restricted shell access to fenari
11:08 mark: Setup quick log rotation of ipv6and4 on maerlant
04:51 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'Let normal users test/signoff on revisions, just like we let them post comments. Restricting it to "coders" kind of defeats the purpose'

June 6

20:22 logmsgbot: hashar: ci.tesla: cleaned up backlog and /tmp (I really need to fix this)
17:00 logmsgbot: hashar: ran namespaceDupes.php frwikiversity --fix (bug 29015)
15:45 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaIncubator/IncubatorTest.php 'r89571'
15:45 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaIncubator/SpecialViewUserLang.php 'r89571'
14:54 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaIncubator/SpecialRandomByTest.php 'r89565'
12:23 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaIncubator/IncubatorTest.php 'r89561'
11:41 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaIncubator/SpecialViewUserLang.php 'Attempted fix for fatal error'
11:36 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikimediaIncubator extension on incubatorwiki'
11:28 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Add incubator loading code'
11:24 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add $wmgUseIncubator, set to false'
11:14 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 89556:
11:13 RoanKattouw: Running scap to deploy WikimediaIncubator extension (code push only, not yet enabled anywhere)
09:46 logmsgbot: catrope synchronized php-1.17/includes/json/Services_JSON.php 'r89555'
06:18 Ryan_Lane: rebooted virt2/3, so that nova-network would reconfigure itself properly
05:26 Ryan_Lane: depooled mobile4/5

June 5

23:22 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29269 Enable wgRawHtml on collabwiki - it is locked down wiki'
18:06 logmsgbot: reedy synchronized php-1.17/includes/api/ApiPageSet.php 'r89514 for bug 25734'
15:32 logmsgbot: reedy synchronized php-1.17/wmf-config/codereview.php 'Reduce semantic autodefer regex'
09:18 mark: killed old stray lsearchd process, started new one on search7 (search pool 3)

June 4

11:17 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29270'

June 3

22:44 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php
22:43 logmsgbot: pdhanda synchronized closed.dblist
22:10 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 29007. Permission changed per comment 2'
21:50 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 29007. Added autopatrol to fawiki'
21:34 Ryan_Lane: upping weights of mobile4/5 to 24
21:28 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.i18n.php 'r89444, r89445'
21:28 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'r89444, r89445'
21:27 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.css 'r89444, r89445'
21:27 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js 'r89444, r89445'
21:27 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r89444, r89445'
20:43 Ryan_Lane: added temporary dns entry controller.labs.wikimedia.org until we figure out a proper DNS scheme for labs
19:43 Ryan_Lane: repooling mobile4/5 with a weight of 8
19:09 notpeter: pushing mible 4 and 5 into pool
19:00 Ryan_Lane: reinstalling opendj on virt1 with a new domain; using labs instead of tesla
17:59 awjr: killing eternally running query on civicrm db on db9
17:28 pdhanda: running maintenance/namespaceDupes.php on frwikiversity

June 2

20:00 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'Live hack to disable join CTA'
19:03 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump AFT tracking version to 8'
19:02 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r89355'
19:02 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.startup.js 'r89355'
19:01 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.css

June 1

21:46 logmsgbot: reedy synchronized php-1.17/includes/User.php 'bug 27514 Make unblockself available as a global permission'
21:24 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 26319 Enable collection on fawiki. Testing on rtl'
21:13 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 27323 WMF Logo/Button not served via Bits'
19:07 Ryan_Lane: switching test.wikipedia.org back to text cname, since it is using a fake cert. (testing was successful)
18:47 Ryan_Lane: changing cname for test.wikipedia.org to wikipedia-lb, to test https
18:41 Ryan_Lane: changing scheduler for https services from wrr to sh on pmtpa lvs servers. restarting pybal on lvs1
18:38 Ryan_Lane: make that ns1
18:38 Ryan_Lane: restarting pdns on ns2, it locked up after authdns-update
18:36 Ryan_Lane: adding wikipedia-lb geodns entry
18:30 Ryan_Lane: restarting pybal on amslvs1
18:29 Ryan_Lane: adding wikipedia-lb service to amslvs1
16:11 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29228'
15:43 logmsgbot: reedy synchronized php-1.17/extensions/CentralAuth/SpecialCentralAuth.php 'bug 28767'
15:40 logmsgbot: reedy synchronized php-1.17/extensions/CentralAuth/SpecialCentralAuth.php 'bug 28767'
14:39 Reedy: Poking at CentralAuth (bug 28767) on testwiki. If things get more broken, it's probably me
09:32 apergos: there's a hung tcpdump on amssq37 tht won't die. it doesn't seem to be doing much but it won't die either (that was me looking at purge issues)
08:18 thedj: Cache invalidations issues continue, but squid source unknown atm.

May 31

23:28 logmsgbot: asher synchronized php-1.17/wmf-config/db.php 'removing db7 from rotation'
23:19 Ryan_Lane: restarting pybal on lvs1
23:07 Ryan_Lane: adding forward entry for wikipedia-lb, for https testing
22:44 Ryan_Lane: restarting pdns on ns0, it's locked up
22:43 Ryan_Lane: changing cname for wikimania2005 to wikimedia-lb, for testing https
22:37 Ryan_Lane: adding geodns entry for wikimedia-lb
22:26 Ryan_Lane: adding wikimedia-lb forward entry for esams
21:56 Ryan_Lane: switching wikimania2005 back to text
21:45 Ryan_Lane: changing cname of wikimania2005 from text to wikimedia-lb to test https
20:51 hashar: wikitech 'ci.tesla: setup a cron for user ci to delete /tmp/mwParser* directories'
20:26 mark: Extended LVS prefix lists on csw1-esams and csw2-esams with 91.198.174.224/28
20:25 Ryan_Lane: restarting pybal on lvs1
19:52 Ryan_Lane: updating yvon and gurvin in dns, moving them to the squid-lvs address range
19:40 Ryan_Lane: adding wikimedia https service to pybal, and restarting pybal
18:47 mark: Disabled (redundant) PIM on csw5-pmtpa:ve3, so only csw1-sdtpa is running PIM now
18:20 Ryan_Lane: and restarting apache on it
18:20 Ryan_Lane: running sync common on srv186
16:07 logmsgbot: hashar: ci.tesla: was actually out of inodes on /. /tmp filled with mwParser-...-images directories
16:02 logmsgbot: hashar: ci.tesla: restarted cruisecontrol (no space left on device) error. It still has space though!

May 29

21:19 logmsgbot: demon synchronized php-1.17/extensions/LiquidThreads/classes/View.php 'r89136'
19:17 logmsgbot: hashar: ci.tesla installed imagemagick php-gd php5-gd
18:42 logmsgbot: hashar: resumed continuous integration on ci.tesla
18:37 logmsgbot: hashar: ci.tesla out of disk, cleaning up /dev/sda1 wich is / (a good idea would be to use separate partitions
17:12 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29119 Allow nlwiki crats to remove rollbacker group'
05:57 Ryan_Lane: restarted pdns on dobson, it wasn't responding to requests
03:07 Ryan_Lane: killed some long running civi queries on db9 that have been running for over 200,000 seconds.
02:54 mark: Killed oprofile daemon and unloaded oprofile module on db32
02:38 mark: Restarted squid-frontend on sq76

May 28

19:41 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'Set wgCodeReviewMaxDiffPaths = 30'
17:52 Andrew: Loaded board vote configuration from extensions/SecurePoll/cli/wm-setup/bv2011-*-final3-jump.xml
17:48 logmsgbot: andrew synchronized php-1.17/wmf-config/CommonSettings.php 'Hooks for SecurePoll'
15:01 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29119'
08:15 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/includes/user/Auth.php 'deploy r89026'
07:49 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 89008:
07:48 Andrew: Running scap to deploy SecurePoll updates

May 27

22:17 logmsgbot: demon synchronized closed.dblist 'bug 29064: cleanup wikimania wiki closures (2005-2010)'
22:17 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29064: cleanup wikimania wiki closures (2005-2010)'
22:00 logmsgbot: catrope synchronized php-1.17/includes/GlobalFunctions.php 'Remove wfMsgExt profiling again'
21:53 logmsgbot: catrope synchronized php-1.17/includes/MessageBlobStore.php 'Prospective fix for the UW bug'
21:38 logmsgbot: catrope synchronized php-1.17/includes/MessageBlobStore.php 'More profiling'
21:35 logmsgbot: catrope synchronized php-1.17/includes/GlobalFunctions.php 'Add profiling points in wfMsgExt()'
21:29 logmsgbot: catrope synchronized php-1.17/includes/MessageBlobStore.php 'Add more profiling'
21:15 logmsgbot: catrope synchronized php-1.17/StartProfiler.php 'Live hack for debugging UploadWizard RL weirdness'
21:13 logmsgbot: catrope synchronized php-1.17/load.php 'Revert live hack'
21:04 logmsgbot: catrope synchronized php-1.17/load.php 'Add live hack for profiling'
20:48 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 29129. Added the new namespace to be search by default for metawiki.'
20:30 logmsgbot: pdhanda synchronized php/wmf-config/InitialiseSettings.php
12:10 Andrew: Installing SecurePoll DB tables on wikis in ~/centralauth_wikis_missing_securepoll
07:57 apergos: the images Rl2-quality-full-003.jpg and Picture_602.11.85.5.jpg have been removed from our image server permanently, office action (legal request).
06:37 apergos: thanks tim
06:37 apergos: restarted irc bot on ekrem
06:09 apergos: er... on ekrem :-P
06:09 apergos: restarted irc daemon, hopefully properly
06:02 apergos: attempting to reboot ekrem (was not pingable)

May 26

23:20 notpeter: powercycling knsq5
21:34 Ryan_Lane: adding en.prototype, de.prototype and test.prototype cnames for prototype.wikimedia.org
18:58 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Raise AFT tracking percentage from 2% to 10% (restores pre-ramp-up volume) and put all users in the show-expertise bucket. Also bump tracking version to 7'
18:54 Ryan_Lane: adding wikimedialbsecure service to amslvs1, for testing https
18:52 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js 'r88909'
18:47 Ryan_Lane: adding wikimedia-lb.esams address to ams and kn text squids, also adding lvs configuration for this address
18:31 Ryan_Lane: removing mediawiki-lb.esams and foundation-lb.esams reverse addresses, they were assigned to the same ips as text.esams and bits.esams respectively
17:48 Ryan_Lane: adding forward entry for wikimedia-lb.pmtpa.wikimedia.org
15:34 mark: Moved AS13030 connection configuration from port 2/11 to port 2/11 on br1-knams

May 25

22:39 awjr: svn up'd on civicrm.wikimedia.org to r189 - deploying further optimizations to contribution searching
21:22 Ryan_Lane: adding wikimedia-lb svc address to squids
21:22 Ryan_Lane: adding wikimedialbsecure service to lvs2, for testing https access to wikimedia.org sites.
21:21 Ryan_Lane: modified pybal configuration to add wikimedia-lb ip to text backend
19:52 logmsgbot: demon synchronized php-1.17/extensions/CentralAuth/SpecialWikiSets.php 'r88825'
18:32 Ryan_Lane: turning off the selenium grid virtual machines on tesla
17:04 RobH: updated bugzilla per rt 764 and 768. whine feature should work now
16:44 awjr: deployed patch to civicrm.wikimedia.org to optimize contribution search query (r185, wikimedia repo)
14:57 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Don't allow users to create accounts on closed wikis either'
14:54 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Cleanup wikimania200[58] entries: bug 29064'
14:53 logmsgbot: demon synchronized closed.dblist 'Add wikimania200[58] to closed.dblist: bug 29064'
14:45 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29064: close wikimania2009wiki'
14:43 logmsgbot: demon synchronized closed.dblist 'bug 29064: close wikimania2009wiki'
12:26 Tim: marked the deletion jobs in OTRS as "invalid" to avoid confusing error messages
12:25 Tim: re-enabled anti-deletion security patch in OTRS, disabled "temporarily" by Fred on June 4, 2009
11:58 domas: rebooting db9, looks unhealthy
11:06 apergos: so that was me on srv225: stopped apache, waited for puppet restart: it now syncs, then does the restart. oh btw srv225 isn't in the pybal apache conf file, weirdly... anyone know why?
09:15 apergos: restarted front and backend squids on sq75
08:47 apergos: restarted front and back end squids on sq81, front end was 20gb and back end was out to lunch
08:35 apergos: ekrem's root partition was full; cleared out some older logs from /var/log/apache2, compressed the most recently rotated (it had failed from disk full), reloaded puppet
07:20 Tim: set read_only=1 on db10, it's a slave so it should be read only. Nagios says it was broken for 6 months.
07:07 logmsgbot: midom synchronized php-1.17/wmf-config/db.php
05:09 Tim: restarted nagios IRC bot
01:05 Tim: created an account for myself in watchmouse
00:48 Andrew: Running populateBv2011EditCount.php in a screen on hume
00:32 Ryan_Lane: pruning binlogs on db9

May 24

23:12 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 29129. Added a namespace alias for metawiki.'
20:36 notpeter_: resyncing drbd nfs1 to nfs2
20:28 RobH: mobile4 awaiting final configurations for service, mobile5 installation done, not yet run puppet updates
20:20 RobH: mobile4 is running puppet updates, mobile5 is mid-install
19:46 RobH: mobile4 & mobile5 racked and accessible for mgmt. installing OS on mobile4 now
18:00 Ryan_Lane: added new reverse zone for svc ips in esams; added entries for bits.svc.esams.wmnet and upload.svc.esams.wmnet
17:31 Ryan_Lane: removing bits-lb and upload-lb public ip enties, and adding bits.svc.pmtpa.wmnet and upload.svc.pmtpa.wmnet entries for https testing
16:46 mark: Upgraded cr2-eqiad to JUNOS 10.4R4.5
16:44 mark: PIM configuration was missing on some virtual interfaces on csw1-sdtpa, which broke purging to esams. Put back in place
16:09 mark: Reloading re1.cr2-eqiad
16:01 mark: Fixed tunnel hack between pmtpa and eqiad
15:01 RobH: db20 networking restarted, back online
14:58 RobH: Miguel is rebooting db18
13:55 mark: reloading csw5
13:52 domas: ns2 has been down for 3 hours, will try restarting pdns
13:47 mark: reloading asw-a4
13:45 mark: reloading asw-d3
13:39 domas: db18/20/28 are still down, may need power cycle. Took them out of rotation.
13:37 RobH: restarted networking on db33-39.
13:30 mark: restarted networking on db30-32.
13:25 domas: DB server issue appears to be a linux kernel issue, forcedeth-related, can be fixed with /etc/init.d/networking restart. Doing so on db10-30, asking mark to do the rest.
13:23 mark: restarted slot 3 since that's where the problematic database servers are
13:12 mark: csw1-sdtpa back up, but a whole lot of database servers are unreachable
13:07 mark: upgraded and reloaded csw1-sdtpa
12:05 mark: Upgraded bootrom and software on asw-d1-sdtpa and asw-a5-sdtpa
10:53 mark: Copied existing 2.7.0.2 image to secondary flash, and installed 2.7.0.3a image on primary flash on csw5-pmtpa
10:42 mark: Copied existing 2.6.0 image to secondary flash, and installed 2.7.0.3a image on primary flash on csw1-sdtpa
04:20 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php 'r88707'
04:03 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php
04:03 Tim: enabling GoogleNewsSitemap on enwikinews

May 23

21:18 Ryan_Lane: added upload-lb.pmtpa.wm.o and bits-lb.pmtpa.wm.o addresses to reverse dns for https preparation
19:32 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29003'
18:45 apergos: reloading puppet on all hosts, with a minute sleep in between, running in screen on fenari as root. needed for enabling pluginsync on clients
18:20 Ryan_Lane: moved ipv6 proxy from iris to maerlant
17:35 RobH: db20 back online, needs mysql setup for replication on another cluster. created rt#836
17:29 RobH: db20 was offline from earlier hardware issues that appear to have been resolved. it is in no current deployment group. booting it online to see if it requires reinstallation
14:31 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/SitemapFeed.php
14:31 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/FeedSMItem.php 'r88640'
14:30 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php 'r88640'
14:23 hcatlin: Deploying code update to Mobile wikipedia
05:08 Tim: in watchmouse: updated subversion monitoring, was broken since May 7

May 22

17:47 logmsgbot: reedy synchronized php-1.17/extensions/CodeReview/ui/CodeStatusChangeListView.php 'r88585'
17:46 logmsgbot: reedy synchronized php-1.17/extensions/CodeReview/ui/CodeCommentsListView.php 'r88585'
00:09 Reedy: Commons is back up and editable

May 21

23:38 Reedy: Intermittant MySQL errors occuring on hits to 10.0.6.41, it is being looked into
23:35 Ryan_Lane: powercycling db31, it's dead
10:27 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '25963 - Enable of Recent changes patrol for Serbian Wikisource (sr.wikisource.org)'

May 20

22:30 Reedy: StewardWiki security issues fixed
22:22 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Stewardwiki'
21:23 awjr: running contribution auditing scripts on civicrm.wikimedia.org
21:21 awjr: enabling log audit module on civicrm.wikimedia.org
21:18 awjr: svn up'ing to r183 of wikimedia branch on production instance of civicrm on grosley
20:57 logmsgbot: pdhanda ran sync-common-all 'Bug 28717. Group changes and flagged revs enabled for bnwiki'
20:50 Ryan_Lane: maerlant too
20:50 Ryan_Lane: rebooting maerlent
20:13 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28773. more rights and namespaces for stewardwiki'
17:16 RobH: updated dns for new mobile servers mgmt
17:12 RoanKattouw: Applying index changes on click_tracking_user_properties from r88462 to the cluster. Using /home/catrope/patch-click_tracking_user_properties-index.sql
15:50 logmsgbot: pdhanda synchronized php-1.17/wmf-config/abusefilter.php 'Bug 28461. Custom abuse filter settings for be_x_oldwiki'
15:50 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28461. Custom abuse filter settings for be_x_oldwiki'
14:41 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28614. Added suppressredirect to autopatrolled on svwikisource'
14:33 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28576. Changed upload settings for huwiki'
03:12 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Touch config'
03:12 logmsgbot: demon synchronized closed.dblist 'bug 29044: closure of tkwikibooks'
02:48 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29058: import sources for arwikinews'
02:04 logmsgbot: LocalisationUpdate failed
00:06 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29057'

May 19

22:26 ^demon: add new 'tools' group to public svn authz
21:02 logmsgbot: hashar: cruisecontrol: tests should now run in roughly 1min30! Still need attention though
20:16 logmsgbot: hashar: cruisecontrol: archived build logs for 2010 and 2011 from 01 till 04
18:48 logmsgbot: hashar: relaunched CruiseControl service on ci
02:01 logmsgbot: LocalisationUpdate failed

May 18

19:44 Ryan_Lane: restarting squid processes on sq71
18:11 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Turning GNSM back off for now -- apache spike'
17:50 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Deploying GNSM for enwikinews'
17:29 logmsgbot: demon synchronizing Wikimedia installation... Revision: 88303:
16:06 ^demon: didn't work, had errors with ArticleAssessmentPilot, FlaggedRevs, LQT_alpha, and MessagesTp. Plus lots of rsync permission errors. Saved output to /home/demon/logs/l10nupdatefail.log
15:53 ^demon: doing manual run of l10nupdate
15:24 logmsgbot: demon synchronized php-1.17/wmf-config/extension-list
02:05 logmsgbot: LocalisationUpdate failed

May 17

18:33 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'fix capitalisation fail'
18:30 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 29017 again doesn't seem to have pushed out'
13:24 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Re-enable CLDR'
13:23 logmsgbot: catrope synchronized php-1.17/languages/Language.php 'r88303'
13:09 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable CLDR because of fatal'
13:03 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable CLDR extension on all wikis'
12:52 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 88299:
12:51 RoanKattouw: Running scap to deploy CLDR extension. Only enabled on test for now, will go site-wide in a minute
11:57 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesEn.php 'r88295'
11:34 logmsgbot: catrope synchronized php-1.17/extensions/UploadWizard/resources/jquery/jquery.validate.wmCommonsBlacklist.js 'r88293'
09:52 Tim: creating an account on wikitech for priyanka
09:51 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Re-enable UploadWizard on Commons, adding note about how to switch it back off if it explodes'
09:39 logmsgbot: catrope synchronized php-1.17/extensions/UploadWizard/SpecialUploadWizard.php 'Fix fatal in live hack'
09:37 logmsgbot: catrope synchronized php-1.17/extensions/UploadWizard/SpecialUploadWizard.php 'Live hack to debug UploadWizard issue'
09:17 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Re-enabling UploadWizard on test to hopefully reproduce ResourceLoader issue'
05:21 mark: Reduced passenger pool size to 200 on all mobile servers
04:46 mark: Depooled mobile2 and rebalanced LVS weights on m.wikipedia

May 16

21:56 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28504 - Need to remove all tokipona settings from InitialiseSettings.php'
21:52 logmsgbot: hashar: cleaning up settings files for tokipona stuff (bug 28504)
19:03 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 29017'
09:37 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'raise account throttle for el wiki for workshop (maybe I can talk the community into upping the threshhold generally??)'

May 15

21:13 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Closing 4 wikis: thwikinews (bug 28341), huwikinews (bug 28342), cowikibooks (bug 28644), astwikiquote (bug 28964)'
21:13 logmsgbot: demon synchronized closed.dblist 'Closing 4 wikis: thwikinews (bug 28341), huwikinews (bug 28342), cowikibooks (bug 28644), astwikiquote (bug 28964)'
21:02 logmsgbot: demon synchronized php-1.17/includes/DefaultSettings.php 'r88203'
21:01 logmsgbot: demon synchronized php-1.17/includes/User.php 'r88203'
12:27 logmsgbot: pdhanda ran sync-common-all
12:12 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 28141'
11:48 logmsgbot: robh synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28987'
11:32 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28727 - please create Portal namespace on am.wikipedia'
11:05 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '25172 - Enable DynamicPageList extension on it.wikisource'
10:35 logmsgbot: hashar: had to manually delete a row from the ptwiki.langlink table (ll_from: 162503, ll_lang: zh-classic, ll_title: Template:Link A (bug 28805)
10:27 logmsgbot: hashar: running update ignore query on ptwiki for bug28805
10:23 logmsgbot: hashar: batch for bug 28805 finished.
10:17 logmsgbot: hashar: running batch queries on all wikis for bug 28805
08:44 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28609 - Disable uploads for eu.wikipedia.org'
01:14 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28914 - Adding 'ipblock-exempt' userright to the Bots usergroup in enwiki'
00:52 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28959 - si.wikipedia $wgSitename'

May 14

22:46 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27434'
22:08 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'wgMetanamespace for amwiki (bug 28727)'
22:05 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28727 - please create Portal namespace on am.wikipedia'
17:11 mark: Fixed IPv6 routing on csw5-pmtpa
16:34 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 28153'
16:30 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 28153'
16:16 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 28141'
16:16 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 28141'
16:16 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28141'
16:14 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28141'
15:57 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27078'
15:52 RobH: added dns entry for wikimania2012.w.o
15:23 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28773'
15:22 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php
14:50 awjr: cleaned out hudson job files for 'Donations queue consume' on grosley and archived them to storage3
14:22 mark: Allocated service IPs for all projects for HTTPS in DNS
14:06 logmsgbot: tstarling synchronized php-1.17/lib/geshi
13:53 mark: Upgraded kernel on streber, rebooting it
13:41 logmsgbot: reedy synchronized php-1.17/lib/geshi 'bug 24383'
13:31 logmsgbot: reedy synchronized php-1.17/lib/geshi 'bug 24383'
13:17 logmsgbot: reedy synchronized php-1.17/lib/geshi 'bug 24383'
13:13 logmsgbot: reedy synchronized php-1.17/lib/geshi 'bug 24383'
13:11 logmsgbot: reedy synchronized php-1.17/lib/geshinew 'Test pushing symlink'
12:35 Ryan_Lane: shutting down owa1-3.tesla.usability.wikimedia.org
12:24 Ryan_Lane: adding maerlant.esams.wikimedia.org as a https/ipv6 proxy node
10:06 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled enotif for user talk pages on all wikis'
09:54 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28895'
09:43 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28612. Added the autopatrolled group to elwiktionary'
09:28 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'Fill in template for bug 28727 (Portal namespace creation for am.wikipedia)'
09:16 logmsgbot: tstarling synchronized php-1.17/includes/MessageCache.php 'removed message stats patch'
08:51 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28479. Added the patroller group to plwikiquote'

May 13

17:33 Ryan_Lane: yvon and gurvin configured as https servers
17:26 RobH: gurvin os installed pupppet not run
17:14 RobH: moved gurvin vlan form 2 to 101
17:06 RoanKattouw: Uncommented srv196 in the node list and ran sync-common on it. It was giving me colorless diffs on CodeReview
17:03 RobH: yvon reinstalled, wont run puppet initial run due to error from new https service entry
16:51 RobH: updating dns for hosts that didnt have public ips (gurvin, some corrections for yvon)
16:28 logmsgbot: catrope ran sync-common-all
16:22 RoanKattouw: Running sync-common-all because sync-docroot didn't work
16:12 RoanKattouw: Syncing docroot
16:12 RoanKattouw: Adding xml/api directory to mediawiki.org's docroot, with an index.html similar to the one in the export-0.4 directory
16:02 logmsgbot: reedy synchronized php-1.17/RELEASE-NOTES
12:14 logmsgbot: robh synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28347'
10:48 logmsgbot: robh synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 26037'
10:31 Tim: deployed a patch to gather i18n usage stats for siebrand

May 11

22:09 mark: Reloaded csw5-pmtpa on existing code base
01:16 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'Temporarily disabling UploadWizard per Neil K while a bug is being worked out.'
01:15 logmsgbot: demon synchronized php-1.17/extensions/UploadWizard/SpecialUploadWizard.php 'r87865'
01:14 logmsgbot: demon synchronized php-1.17/extensions/UploadWizard/UploadWizard.i18n.php 'r87865'
00:10 notpeter: puppet back to normal on all boxes that are up except for knsq5.

May 10

23:36 Ryan_Lane: fixing torrus. why oh why must you corrupt yourself constantly, torrus?
23:06 notpeter: distributing new puppet certs to all boxes managed by puppet
22:51 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'ED 1.17'
21:34 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling Article Feedback dashboard on enwiki'
21:30 logmsgbot: awjrichards synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.php 'r87844'
21:29 logmsgbot: awjrichards synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.i18n.php 'r87844'
21:28 logmsgbot: awjrichards synchronized php-1.17/extensions/ArticleFeedback/populateAFStatistics.php 'r87844'
21:28 logmsgbot: awjrichards synchronized php-1.17/extensions/ArticleFeedback/SpecialArticleFeedback.php 'r87844'
21:02 Ryan_Lane: powercycling srv281 since it's dead
21:01 Ryan_Lane: powercycling srv247 since it's dead
20:56 Ryan_Lane: mobile has stablized, moving weights back to normal
20:50 Ryan_Lane: lowering weight on mobile1 and mobile2
20:45 Ryan_Lane: restarted apache on mobile2.
19:16 mark: Moved all inter-subnet routing from csw5-pmtpa to csw1-pmtpa
19:16 mark: Reconfigured pybal on lvs3 to BGP connect to csw1-sdtpa instead of csw5-pmtpa, and restarted it
17:53 mark: Moved vlan 2 routing from csw5-pmtpa:ve2 to csw1-pmtpa:ve2
16:27 apergos: or at least srv254.
16:25 apergos: nagios is whining about some services that are actually in operation (eg srv250)
16:07 Ryan_Lane: or not, mgmt interface isn't letting me in
16:06 Ryan_Lane: powercycling srv250
13:56 Tim: removed myself from SMS notification in nagios, since the configuration is so broken that it sends me a text every 10 minutes day or night
13:49 RobH: nagios sms pages seem to be delayed from the actual service outage by a long, long delay
13:10 Tim: did the above in both puppet and nagios directly since puppet seems to be still broken
13:07 Tim: increased the number of retries of payments.wikimedia.org before sending an SMS from 1 to 20
02:02 notpeter: accidentally deleted sockpuppet:/var/lib/puppet/ssl am regnerating keys and will begin signing soon

May 9

23:36 notpeter: so annoying. doubling puppet freshness check timing to 4 hours.
22:01 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disabling dashboard on enwiki pending fixes to data collection script'
21:20 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87774:
21:20 RoanKattouw: Running scap to deploy UploadWizard changes
20:54 RoanKattouw: Running extensions/ArticleFeedback/populateAFStatistics.php on enwiki
20:49 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/populateAFStatistics.php 'r87770'
20:49 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r87770'
20:30 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling ArticleFeedback dashboard on testwiki'
20:28 RoanKattouw: Running populateAFStatistics.php on testwiki
19:39 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'Dummy sync to push config change through in ResourceLoader'
19:39 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Re-enable AFT click tracking at 2%, and bump tracking version'
19:32 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'Force config changes to update in startup module'
19:21 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to the target value of 2.7%'
19:03 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 2%'
18:41 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 1%'
18:39 RobH: upgraded Reedy's cluster access to allow him to deploy and do shell requests, will take awhile to push out to the cluster
18:25 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 0.5% for real this time'
18:21 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump AFT tracking version, needed to actually change the click tracking percentage'
18:16 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28495'
18:14 Ryan_Lane: adding a shell account for asher (Asher Feldman), and adding him to roots
18:13 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Temporarily disable click tracking for AFT'
18:11 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bump AFT lottery odds to 1%'
18:10 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable AFT dashboard on enwiki'
18:09 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump ArticleFeedback tracking version'
18:01 RoanKattouw: Running schema change (adding index of aa_timestamp field on enwiki.article_feedback table) on s1 by directly running it on the master. Table size is ~275k rows
17:56 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Set $wmgArticleFeedbackLotteryOdds to 0.005 on enwiki. This enables AFT on 0.5% of all enwiki main namespace pages'
17:44 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87742:
17:43 RoanKattouw: Running scap to deploy ArticleFeedback changes
16:49 RoanKattouw: Deploying ArticleFeedback changes to testwiki
14:21 apergos: restarting amssq39 squid back-end (memory hog)
13:41 apergos: restarting squid back-end on amssq40 (memory hog)
13:27 logmsgbot: catrope synchronized php-1.17/resources/mediawiki/mediawiki.js 'r87712'
13:27 logmsgbot: catrope synchronized php-1.17/includes/OutputPage.php 'r87712'
13:26 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderContext.php 'r87712'
13:26 RoanKattouw: Deploying r87712, which should fix the ResourceLoader breakage in IE
13:12 apergos: restarted squid back-end on amssq37 (hogging memory)
08:06 apergos: rebooting srv198 for testing
07:52 apergos: rebooting srv281 for testing

May 8

22:46 RobH: fixed deep linking issue for uploads from old techblog. merge did not copy, had to manually pull from singer and also add additional rewrite rule to apache via puppet
22:40 RobH: copied over techblog upload data, which upon further review was not copied by the merge plugin as expected
18:29 RobH: updated categories on blog to remove testblog url entry
18:20 RobH: updating dns to revert bad change on blog propogation, split wikimediafoundation template files to themselves, were soft linked to wiktionary.org
18:05 RobH: blog redirects are in place for projects, not sure why or what added them, as they do not belong. fixed hooper via puppet to handle blog.wikimediafoundation.org
15:46 RobH: added exim simple mail sender to puppet entry for hooper

May 7

20:31 RobH: hooper running back on puppet, with all config files being updated normally
18:11 RobH: tinkering with redirects on hooper. disabled puppet daemon while doing local apache config changes, once tested will push up to puppet and reenable it on hooper
18:08 RobH: blog dns updated enough for both myself and guillom to test basic blog function. all basic use seems fine. still testing more advanced use cases
16:58 RobH: updated dns for blog and techblog. may take an hour or so to propogate, cannot update new blog until it resolves for authors to it. (old blogs will slowly cycle out of dns)
16:40 RobH: blog migration moving along, all data migrated, basic rewrites and such are in place, final review and cleanup in progress before changing testblog.w.o to blog.w.o in database and updating DNS
16:03 RobH: testblog.w.o currently now running 'master' database for the wmf blog. online as testblog, can do quick db replace when its ready to go online
15:57 RobH: starting the copy of the main blog, any changes to blog.w.o after this time will be lost in the migration to the new host
15:53 RobH: updating blog software on hooper, installing all extensions before connecting to database copy of newsblog (not yet copied)
15:37 RobH: wiping out testblog on hooper and getting basic software installed for new blog deployment. database not yet copied, old blogs still 'primary'
12:55 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'raise account throttle for el wiki for workshop'
09:20 rainman-sr: deleted some old logs on search11 to free up space

May 6

22:46 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28773. Modified wgAddGroups and enabled subpages for stewardwiki'
20:39 Ryan_Lane: added apaches::service to imagescalers class in puppet to ensure that sync-common will run before apache is started on the imagescalers
18:10 awjr: enabling drupal/civicrm logging on civicrm.wikimedia.org (forgot to turn this back on post-upgrade)
17:41 notpeter: killing the shit out of puppet to clear its cache.
17:38 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28791. Fixed logo for piwiki'
17:19 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28689 - enabled NewUserMessage on thwiki'
13:44 RobH: installed and activated wordpress importer on testblog
08:58 RoanKattouw: Ran sync-common on srv281 and restarted Apache. It was complaining about missing directories in docroot
08:55 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87486:
08:55 RoanKattouw: Uncommenting srv281 in /etc/dsh/group/mediawiki-installation and scapping again
08:53 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87486:
08:52 RoanKattouw: Running scap because srv281 is out of sync. I thought we had measures to prevent out-of-sync servers from being pooled?
05:03 apergos: powercycled srv281 (again)
01:55 Tim: stopped xinetd on fenari, running svn up to add REL1_17 branch to extdist snapshot

May 5

17:32 notpeter: taking down nfs1 to upgrade to lucid and upgrade udplog
05:47 logmsgbot: tstarling synchronized php-1.17/img_auth.php 'r87486'
05:47 logmsgbot: tstarling synchronized php-1.17/includes/WebRequest.php 'r87486'
05:46 logmsgbot: tstarling synchronized php-1.17/includes/User.php 'r87486'

May 4

23:11 awjr: all CiviCRM-related cron and hudson jobs re-enabled
22:19 awjr: CiviCRM upgrade complete, putting civicrm.wikimedia.org back on-line
22:17 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php 'r87462'
22:17 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap.php 'r87462'
22:16 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap.alias.php 'r87462'
21:33 awjr: performing CiviCRM upgrade (3.1.6->3.4.0) on grosley
21:24 awjr: upgrade to drupal 6.20 complete for civicrm.wikimedia.org
21:10 awjr: replacing /srv/org.wikimedia.civicrm dir on grosley with svn co from new deployment branch in wikimedia repo (/branches/deployment/fundraising-civicrm/d620c34)
20:51 awjr: disabling civicrm and drupal-related cron/hudson jobs
20:49 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87452:
20:43 logmsgbot: catrope synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php 'Fix GNSM, r87488'
20:37 awjr: backing up 'civicrm' database from db9
20:34 awjr: backing up 'drupal' database from db9
20:30 awjr: taking civicrm.wikimedia.org offline for drupal and civicrm upgrade
20:14 RoanKattouw: Deploying new UploadWizard code to testwiki
18:42 Ryan_Lane: created a restricted account for neilk
15:33 logmsgbot: demon synchronized php-1.17/extensions/CentralAuth/SpecialWikiSets.php
14:29 hcatlin: deploying mobile language update

May 3

22:38 logmsgbot: pdhanda synchronized php-1.17/wmf-config/abusefilter.php 'Bug 28502 - custom config for thwiki'
22:37 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28502 - enabled abusefilter for thwiki'
21:21 Ryan_Lane: powercycling srv281
21:12 logmsgbot: catrope synchronized php-1.17/README 'Test sync to see which hosts are borked'
21:04 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28257. Added requested NamespaceAlias to orwiki'
20:32 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87355:
20:31 RoanKattouw: Running scap to deploy 1.17wmf1 merges (r87353)
16:24 awjr: re-upgrading dev.civicrm.wikimedia.org from civicrm 3.1.6 to 3.4.0
15:59 RobH: testblog ops work done, no other services impacted
15:13 RobH: working on testblog on hooper and db9, no other services should be affected
03:30 RobH: issue of singer network spiking and plummeting, along with swap death, at ~3am gmt
03:25 RobH: singer blocked on db9, flushed hosts to allow to reconnect
01:30 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php 'r87311'
01:29 logmsgbot: demon synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'r87311'
00:44 tomaszf: killing long running CiviReport query on db9 due to watchdog being off for upgrade

May 2

20:02 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Adding Research namespace to meta, per bug #28742'
17:52 Ryan_Lane: powercycling srv281
17:50 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabled LabeledSectionTransclusion for mediawiki - requested by guillom'
17:28 RobH: fixing issue with bad serial assignment leading to bad network config on ps1-a1/2-eqiad, restarting both power strips
15:47 mark: Powercycled amssq35
04:21 logmsgbot: tstarling synchronized php-1.17/includes/Wiki.php 'r87235'
03:58 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php
03:34 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php
03:31 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php
02:13 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27891 log'
02:00 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 'r87234'
01:22 logmsgbot: demon synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'r87233'
01:21 logmsgbot: demon synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.php 'r87233'
00:09 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'wgBlockDisablesLogin on checkuserwiki'

May 1

21:14 Ryan_Lane: ran flush host on db9 to bring blog.wikimedia.org back

April 30

11:38 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryAllUsers.php 'Attempt to fix auprop=blockinfo bug'
11:15 Ryan_Lane: (make that blog.wikimedia.org ;) )
11:14 Ryan_Lane: ran flush hosts on db9 to bring blogs.wikimedia.org back up

April 29

23:27 ^demon: all of that was to deploy GoogleNewsSitemap to testwiki for the enwikinews folks.
23:23 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php
23:22 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php
23:18 logmsgbot: demon synchronizing Wikimedia installation... Revision: 87149: r87149
19:43 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Fix config bug that caused tracking to fail for AFT, and bump tracking version'
19:18 mark: powercycled ersch
15:40 RobH: typo asw-b8
15:39 RobH: swapping msw1-eqiad fan tray with asw-a8-eqiad fan tray
05:58 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 'adding srv170 after hw reset'
05:55 logmsgbot: midom synchronized php-1.17/wmf-config/db.php

April 28

23:46 Tim: doing hard reset of alsted, it's not sending ssh version strings and not allowing logins by serial
22:17 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disabled option to allow blocked users to edit their talk pages on itwiki. See Bug 9073'
21:02 Ryan_Lane: created Ambassador-announce-l list
20:52 richcole1: replacing memory in srv281
20:49 RobH: srv281 shutdown for memory replacement
20:41 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable by default, option to enable blocked users to edit their own talk page'
20:34 richcole1: replacing bad drive in srv284
20:08 Ryan_Lane: powercycling srv281
20:03 RobH: ms4 is powered down, was already offline and unresponsive to console.
18:19 logmsgbot: demon synchronizing Wikimedia installation... Revision: 87091: r87090, r87091
18:02 mark: Restarted apache on srv235, probably corrupted apc cache
17:57 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Re-enabling EmailCapture on enwiki'
17:54 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 87084:
17:47 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Preparing conf for ArticleFeedback with way less click tracking to prevent cluster breakage'
17:40 RobH: etherpad.proxy to 000-etherpad.proxy on hooper apache2 enabled, should fix
17:34 RobH: hooper vhost for testblog back online
17:17 RobH: rolled back testblog vhost from hooper
17:01 RobH: testblog enabled for initial setup on hooper without disrupting etherpad service (on either of its url redirects)
16:56 RobH: updating etherpad.proxy apache config to include eiximenis serveralias and enabling the testblog vhost
16:48 Reedy: Manually pinged CR update script, had got stuck - 2 revisions. Let me know if it does it again
16:21 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87085:
16:21 RoanKattouw: Rolling back ArticleFeedback deployment
16:11 RoanKattouw: API cluster was briefly dead just now after enabling EmailCapture, disabling it seems to be bringing it back
16:09 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable EmailCapture, maybe this will bring our API Apaches back'
16:04 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling EmailCapture on enwiki'
16:01 RoanKattouw: (that was for the ArticleFeedback deployment)
15:58 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 87083:
14:08 nelson: Upped the max_connections for rsyncd from 2 to 5 across the test cluster.
13:56 nelson: Upped the concurrency of swift-object-replicator from default of 1 to 5
13:38 mark: Restored I/O scheduler deadline for device /dev/sda on alsted
13:34 mark: Temporarily set I/O scheduler cfq for device /dev/sda on alsted
12:29 mark: powercycled harmon
01:00 awjr: upgraded dev.civicrm.wikimedia.org to CiviCRM 3.4.0

April 27

20:24 nimish_g: nimishg enabling CustomUserSignup extension on enwiki
20:20 logmsgbot: nimishg synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling'
01:44 logmsgbot: nimishg synchronized php-1.17/extensions/ClickTracking/modules/ext.UserBuckets.js 'setting cookie expiry to 90 days'
01:44 logmsgbot: nimishg synchronized php-1.17/extensions/ClickTracking/ClickTracking.hooks.php 'setting cookie expiry to 90 days'
01:43 logmsgbot: nimishg synchronized php-1.17/extensions/CustomUserSignup/CustomUserSignup.hooks.php 'setting cooky expiry to 90 days'
01:03 Tim: updated sudoers in puppet for r83000
00:25 Tim: reniced sshd on sockpuppet to -2 so that I can log in while catalog updates are running

April 26

23:40 logmsgbot: nimishg synchronizing Wikimedia installation... Revision: 86895:
23:33 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php
23:26 logmsgbot: root synchronizing Wikimedia installation... Revision: 86895:
23:15 Tim: disabled CustomUserSignup, was causing fatal error on mediawiki.org due to incorrect path
23:15 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php
23:13 logmsgbot: nimishg synchronized php-1.17/wmf-config/InitialiseSettings.php 'turning'
23:00 logmsgbot: nimishg synchronized php-1.17/wmf-config/InitialiseSettings.php 'turning'
22:55 logmsgbot: nimishg synchronized php-1.17/extensions/CustomUserSignup/CustomUserSignup.hooks.php 'r86994'
22:49 notpeter: stopping nagios for a wee sec and then restarting.
22:09 logmsgbot: nimishg synchronizing Wikimedia installation... Revision: 86895:
22:08 nimish_g: nimishg pushing CustomUserSignup code to cluster, running on mediawikiwiki only
21:28 logmsgbot: awjrichards synchronized php-1.17/includes/CategoryPage.php 'r86988'
20:38 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled CustomUserSignup extension for testwiki'
20:37 logmsgbot: pdhanda synchronized php-1.17/wmf-config/CommonSettings.php 'enabled CustomUserSignup extension for testwiki'
20:32 awjr: disabled wmf_premiums module on civicrm.wikimedia.org - it is currently not being used and potentially causes queue consumption to choke with certain contribution messages
19:51 logmsgbot: catrope ran sync-common-all
19:50 RoanKattouw: Running sync-common-all to deploy UploadWizard changse
17:52 pdhanda: Running maintenance/populateParentId.php on all wikis
08:21 Andrew: sync-common-all worked. scap still broken
08:21 logmsgbot: andrew ran sync-common-all
08:21 Andrew: trying sync-common-all
08:19 Andrew: syncs are broken, log littered with XXX: [sudo] password for andrew:
08:12 Andrew: re-scapping, typo in extension-list
08:12 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 86895:
08:11 Andrew: Scapping to enable DisableAccount extension
08:11 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 86895:
08:02 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 86895:
08:02 Andrew: running scap to deploy the code itself
08:01 Andrew: deploying DisableAccount extension to checkuserwiki, stewardwiki, arbcom_enwiki since the special page was removed without consulting Philippe
02:15 logmsgbot: robh synchronized php-1.17/wmf-config/InitialiseSettings.php 'adding settings for checkuser and steward wikis'

April 25

23:33 Ryan_Lane: added python-mwclient to lucid repo
21:36 RobH: storage2 still offline, wont boot into os, but is remotely accessible
21:20 RobH: trying to fix storage2
20:16 notpeter: actually adding everyone on ops to watchmouse service... didn't know this had not already been done.
20:02 RobH: updated csw1 to removed labels and move to default vlan ports 11/12, 11/14, 11/19, & 11/21. old connection ports for dataset2, tridge, ms1, and ms5
19:53 RobH: the datacenter is looking awesome.
19:45 RobH: ms1 moved from temp network to permanent home, no downtime, responding fine
19:42 RobH: ms5 connection moved, no downtime, responds fine, less than 4 seconds
19:40 RobH: updated csw1-sdtpa 15/1,15/2 from vlan 105 to vlan 2, 15/3 and 15/4 from vlan 105 to 101
18:52 RobH: snapshot4 relocated to new home, ready for os install
18:42 RobH: db19 and db20 back online (not in services as they have other issues)
18:39 RobH: db19 and db20 powering back up
18:25 RobH: virt4 experienced an accidental reboot when rebalancing power in the rack, my fault, not the hardware
18:12 RobH: rack b2 power rebalanced
18:01 RobH: db19 set to slave, depooled in db.php, no other services evident, shutting down (mysql stopped cleanly)
18:00 RobH: db20 shutdown
18:00 RobH: didnt log that i setup ports 11/38-40 for db19, db20, and snapshot4 on csw1-sdtpa. tested out fine and all my major configuration changes on netowrk should be complete
17:56 RobH: ok, db20 and db19 are coming offline to relocate their rack location due to power distro issues
15:47 RobH: delay, not coming down yet, need more cables
15:46 RobH: db19 is coming down as well, it is depooled anyhow
15:46 RobH: db20 is coming down, ganglia aggregation for those hosts may be delayed until it is back online.
15:21 RobH: relocating snapshot4 into rack c2, it will be offline during this process
15:20 RobH: db43-db47 network setup, sites not down, yay me
15:10 RobH: being on csw1 makes robh nervous.
15:09 RobH: labeling and setting up ports on 11/33 through 11/37 on csw1-sdtpa for db43 through db47
14:47 RobH: fixed storage2 serial console (set it to higher rate, magically works, or it just fears me) and also confirmed its remote power control is functioning
14:42 RobH: stealing dataset1's known good scs connection to test storage2. dataset1 service will remain unaffected.

April 24

21:30 Ryan_Lane: restarting apache on mobile1
15:35 RobH: swapping bad disk in db30, hotswap, should be fine
14:36 RobH: swapping out the management switch in c1-sdtpa. msw-c1-sdtpa will be offline, so the mgmt interfaces of servers in that rack will be offline. all normal services will remain unaffected.

April 23

22:31 RobH: required even.
22:31 RobH: no drives display error leds, futher investigation requried
22:27 RobH: ms2 is having bad drive investigated. if we do this right, it wont go down. if we don't it will. is a slave es server.
22:00 RobH: singer returned to operation, blog, techblog, survey, and secure returned to normal operation
21:52 RobH: singer is once again coming back down for drive replacement. This will take offline blog.wikimedia.org, techblog.wikimedia.org, survey.wikimedia.org, and secure.wikipedia.org. Service will be returned as soon as possible.
21:19 RobH: singer back online, for awhile, will come back down for further repair shortly.
21:05 RobH: singer going down, blogs will be offline, so will secure, system will return to service as soon as possible
21:00 RobH: preparing to fix the dead drive in singer, this will offline secure, blog, techblog, and survey during the drive replacement process
19:49 mark: Upgrading mr1-pmtpa to junos 10.4R3.4
17:49 RobH: migrating searchidx1 & search1-search10 to new ports in same rack. moving one at a time and ensuring link lights between moves. (already tested with search10)
14:11 RobH: db19 is back online, seems to not have any mysql setup done.
14:02 RobH: restarting db19
14:02 RobH: arcconf checks out all drives on db19 are indeed working as rich found earlier
12:47 mark: Added (x121Address=1) condition to the LDAP query of the ldap_aliases router on mchenry's exim
00:32 hcatlin: Mobile: Deploying fix to an issue that kept the standard-style Main_Page from displaying on mobile
00:25 Ryan_Lane: restarting memcached on all of the mobile servers
00:23 Ryan_Lane: repooling mobile3, since mobile will die without it (fun!!)
00:17 Ryan_Lane: depooling mobile3
00:13 Ryan_Lane: restarting apache on mobile3
00:10 Ryan_Lane: puppet was broken on mobile1, reinstalled it

April 22

23:56 domas: detached gdb from srv193 apache, apparently it was used for something
23:14 notpeter: restarting nagios (again)wq
22:43 notpeter: restarting nagios
19:23 apergos: shot all stopped rsyncs on ms5 (that were copying from ms4 about two weeks ago), changed all perms on the directories they had reached so thumbs can be served/read from them.. oh. not me, someone else must have done it, I'm not here :-P
19:02 RobH: ms4 shutting down for memory troubleshooting
18:52 RobH: ms4 troubleshooting, disragrd bounces]
18:51 notpeter: restarting nagios
12:41 hcatlin: Restarting mobile cluster with April code update.
00:49 notpeter: restarting nagios. hopefully now with more sms!

April 21

23:32 logmsgbot: midom synchronized php-1.17/includes/ImagePage.php
20:54 logmsgbot: pdhanda synchronized live-1.5/404.php
19:02 domas: bumped up maxclients/serverlimit on singer to 350 (up from 150), set maxrequestsperchild to 30 to avoid heap blowup (down from 0), all governed via apache2/conf.d/maxrequests
18:25 Ryan_Lane: restarting apache on singer
18:19 Ryan_Lane: applying system patches to raskin
17:47 Ryan_Lane: restarting apache on singer
17:04 logmsgbot: pdhanda synchronized live-1.5/404.php
16:52 rainman-sr: copying stuff from /home/ariel/searchidx/ to searchidx2 as it didn't get copied
15:32 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Fixing typo to properly check for wgUseEmailCapture'
15:29 logmsgbot: awjrichards ran sync-common-all
15:06 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Added section for EmailCapture, enabling on testwiki'
15:05 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Added section for EmailCapture'
14:50 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 86464:
14:49 awjr: running scap on fenari to dark-deploy EmailCapture extension
03:44 Ryan_Lane: uninstalled python-json on wikitech, and installed python-simplejson
01:45 Ryan_Lane: testing statusnet library again
01:45 Ryan_Lane: testing statusnet library again
01:44 Ryan_Lane: testing statusnet library again
01:35 Ryan_Lane: testing a statusnet library for morebots that has a sane license
01:20 Ryan_Lane: installing ca-certificates on wikitech
01:11 Ryan_Lane: installing curl on wikitech
01:04 Ryan_Lane: installing python-json on wikitech
00:11 Ryan_Lane: added a sweet, special message to morebots
00:03 notpeter: moving some searchidx files into puppet's sphere of influnece.

April 20

23:10 rainman-sr: search indexing is up and running again, might take a couple of days for things to catch up though
23:07 rainman-sr: doing a slow restart of the whole search cluster (2 minute intervals between host restarts)
22:52 rainman-sr: starting search indexer on searchidx2
16:29 mark: Squids finished rebuilding, removed iptables rules
16:21 mark: Removed iptables rules on sq77/78
16:09 mark: Downgraded sq65
16:06 mark: removed iptables rule on sq66
16:05 Ryan_Lane: downgrading squid on knsq27, knsq28, knsq29, knsq30, knsq26, knsq25, knsq24
16:02 Ryan_Lane: downgrading squid on amssq31
16:01 Ryan_Lane: downgrading squid on amssq38
16:01 Ryan_Lane: downgrading squid on amssq39
16:01 Ryan_Lane: downgrading squid on amssq40
16:01 Ryan_Lane: downgrading squid on amssq41
16:00 Ryan_Lane: downgrading squid on amssq37
15:59 Ryan_Lane: downgrading squid on amssq36
15:59 Ryan_Lane: downgrading squid on amssq32
15:59 Ryan_Lane: downgrading squid on amssq33
15:58 Ryan_Lane: downgrading squid on amssq34
15:58 Ryan_Lane: downgrading squid on amssq35
15:57 mark: Downgrading Squid on sq71-78
15:57 Ryan_Lane: downgrading squid on amssq42
15:56 Ryan_Lane: downgrading squid on amssq43
15:56 Ryan_Lane: downgrading squid on amssq45
15:54 Ryan_Lane: downgrading squid onamssq45
15:53 Ryan_Lane: downgrading squid on knsq23
15:33 RobH: lowered maxclients more on singer, its going to slow down secure, but hopefully keep it online longer.
15:04 Ryan_Lane: downgraded sq71 to squid-2.7.7
14:51 mark: Pulled squid 2.7.9 packages from the lucid-wikimedia reprepro APT repository, reinstated 2.7.7
14:34 mark: Downgrading squid on sq66
14:15 mark: Stopping squid on sq72
14:14 RobH: reduced singer apache maxclients from 400 to 200, hopefully will reduce singer crashes
14:09 RobH: singer died, rebooting.
14:07 RobH: secure, blogs on singer, bouncing due to singer cpu max
13:53 RobH: singer threw invalid cert warning, cpu spike was occuring, restarting apache and load is normalizing
11:34 logmsgbot: catrope ran sync-common-all
11:33 RoanKattouw: Running sync-common-all to deploy r86464
11:11 logmsgbot: catrope synchronized php-1.17/includes/Exif.php 'Fix for bug 28615 (fatal on image description page)'
00:41 awjr: deployment of r86441 to payments servers complete
00:37 awjr: deploying changes to send full donor address information to PayflowPro (r86441) to all payments servers
00:31 awjr: re-deploying changes to send full donor address information to PayflowPro on payments2.wikimedia.org (suspected cause of previous issue: local browser cacheing)
00:14 logmsgbot: midom synchronized php-1.17/wmf-config/db.php

April 19

23:24 richcole1: rebooting db11 to check raid
22:57 awjr: reverting to r84367 of deployment branch on payments2.wikimedia.org due to bad address information appearing in civicrm
22:53 awjr: deploying changes to send full donor address information to PayflowPro payments2.wikimedia.org
22:20 Ryan_Lane: virt1-4 installed
20:54 Ryan_Lane: powercycling singer
20:53 Ryan_Lane: restarting apache on spence
20:52 mark: powercycled singer
20:35 Ryan_Lane: powercycling srv196
17:31 Ryan_Lane: finished ams and kn text squids
17:17 RobH: change live, etherpad still online and functional
17:16 RobH: pushing change to hooper so it doesnt kill etherpad when other apache vhosts exist
17:16 RobH: updated puppet entry for etherpad vhost
16:54 Ryan_Lane: upgrading squid on all squid systems, slowly, over the course of the day
15:09 mark: Killed all etherpad-user processes and started etherpad
15:09 RoanKattouw: Removing data for the section edit link experiment from the clicktracking table. Data is backed up in /home/catrope/sel . This will delete 8.4M of the 9.6M rows in the clicktracking table
15:05 RoanKattouw: Etherpad is broken, serving 500s. Have e-mailed Peter
14:06 logmsgbot: demon synchronized php-1.17/includes/diff/DifferenceEngine.php 'r86395'
12:50 mark: NICEd bayes processing cron job on mchenry
10:14 RoanKattouw: <logmsgbot> !log catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump $wgArticleFeedbackTrackingVersion to 1'
10:14 RoanKattouw: <logmsgbot> !log catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'r86384'
10:14 RoanKattouw: <logmsgbot> !log catrope synchronized php-1.17/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'r86299'
10:14 RoanKattouw: Restarted morebots
01:05 Tim: added an entry to /etc/sudoers on fenari and hume allowing wikidev users to run commands as apache

April 18

21:28 mark: Setup PIM-DM on cr1-eqiad:ae1.666 (pseudowire port to pmtpa) and csw5-pmtpa:ve10 (counterpart) to get multicast routing from tampa
21:24 mark: Setup PIM-SM on cr1-eqiad and cr2-eqiad, with static RPs and a local RP on cr1-eqiad. PIM is running on all production subnets.
16:16 pdhanda: running maintenance/CleanupImages on all.dblist
15:58 mark: Rolling restart of pmtpa text frontend squids; one every two minutes
14:27 mark: Powercycled db11
14:13 logmsgbot: mark synchronized php-1.17/wmf-config/db.php 'depool db11'
14:12 Reedy: DB11 related issues. Ops are looking at the issue
10:30 logmsgbot: root synchronizing Wikimedia installation... Revision: 86086:
10:05 logmsgbot: root synchronizing Wikimedia installation... Revision: 86086:
08:43 Tim: running /home/wikipedia/bin/l10nupdate to get r86294

April 17

20:27 mark: Setup dirty "pseudowire" between pmtpa and eqiad; a GRETAP tunnel inside a v4-in-v6 tunnel bridged to eth1/eth0.666 on 2 linux boxes (sq71 on pmtpa side). Now running OSPF on it to get full v4 interconnectivity

April 16

08:33 apergos: disabled snaps on searchidx on in rainman's crontab, moving some en.wiki.* files in /a/search/indexes/snapshot off to fenari:/home/ariel/searchidx/....

April 15

18:41 mark: Rsynced private thumbs dir from ms4 to ms5
15:45 mark: Upgraded re0.cr1-eqiad to JUNOS 10.4R3.4
15:27 mark: Switching over routing engines from re0 to re1 on cr1-eqiad
15:23 mark: Upgraded re1.cr1-eqiad to JUNOS 10.4R3.4
14:44 mark: Upgraded re0.cr2-eqiad to JUNOS 10.4R3.4
14:34 mark: Upgraded re1.cr2-eqiad to JUNOS 10.4R3.4
09:57 apergos: moving some old (from 2009) dirs from /a/search/indexes/import on searchidx1 to /home/ariel/searchidx1/... on fenari, if I'm told we can toss em outright I'll do that later.
06:01 apergos: cleared out space on searchidx1 again. we only have 4.5gb free and it's still at 100%. out of tricks...
00:54 notpeter: py making some changes to nagios. subscribing the service to lots of files. also, lots more files being pushed out by puppet.

April 14

23:41 Ryan_Lane: adding reverse mappings for virt1-4
23:28 Ryan_Lane: adding dns entries for virt1-4
22:28 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Re-enable ArticleFeedback on enwiki, upgrade is complete'
22:09 RoanKattouw: Running populateAFRevisions.php on enwiki
22:06 RoanKattouw: Creating article_feedback_revisions table on enwiki
22:04 RoanKattouw: Clearing message blobs
22:03 logmsgbot: catrope ran sync-common-all
22:03 RoanKattouw: Deploying ArticleFeedback changes to the cluster with sync-common-all
22:02 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable ArticleFeedback on enwiki for upgrade'
21:42 mark: Added vlan 103 to ports e 15/5 to 15/8 (virt1-4 eth1) (tagged) on csw1-sdtpa
21:42 mark: Put ports e 15/1 to 15/4 (virt1-4 eth0) in vlan 105 (untagged) on csw1-sdtpa
21:40 mark: Created VLAN 105 ("virt-hosts") on csw1-sdtpa, assigned virtual interface ve 9, assigned ip 10.4.16.0/24
21:39 mark: Added virtual interface ve 7 ("virt") on csw1-sdtpa, assigned ip 10.4.0.1/24
21:39 mark: Removed virtual interface ve 7 ("virt") on csw5-pmtpa
21:24 Ryan_Lane: cleaning up searchidx1
21:11 RoanKattouw: Running extensions/ArticleFeedback/populateAFRevisions.php on testwiki
21:11 RoanKattouw: Creating article_feedback_revisions table on testwiki
21:08 RoanKattouw: Updating ArticleFeedback code on test
16:18 RobH: singer back online, blogs, survey, and secure should all be back in service
16:15 RobH: blogs offline due to singer outage, as well as survey, working on resolution
16:14 RobH: powercycling singer, unresponsive to ssh and to serial console
16:13 RobH: secure issue is being investigated
16:13 pdhanda: running cleanupTitles.php for all wikis in all.dblist
16:13 RobH: investigating singer cpu spike
15:36 mark: Implemented LDAP lookups on mail relay mchenry
14:13 Reedy: Delayed message. Ran repopulateCodePaths on mediawikiwiki
14:13 Reedy: Delayed message. Ran populateFollowupRevisions on mediawikiwiki
14:04 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/modules/ext.codereview.js 'Disable comment field autofocus'
07:54 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/CodeReview.php 'Fix CodeReview fatals'
07:50 logmsgbot: catrope ran sync-common-all
07:47 RoanKattouw: Running sync-common-all and clearMessageBlobs.php to deploy UploadWizard changes
07:35 RoanKattouw: Updating MediaWiki on fenari to r86034 for UploadWizard deployment. Only pushing changes to testwiki for now
07:14 logmsgbot: tstarling synchronized php-1.17/includes/WebRequest.php 'r86031'
07:14 logmsgbot: tstarling synchronized php-1.17/img_auth.php 'r86031'

April 13

23:18 notpeter: restarting nagios
21:15 notpeter: restarting nagios
19:54 awjr: disabling drupal/civicrm cron on contacts.wikimedia.org (singer) as part of an interim solution to civimail problems in CiviCRM 3.4beta1
18:31 ^demon: ran code review updates for mediawikiwiki
12:54 mark: Restarted pdns on linne
12:54 mark: Defined /64 neighbor blocks in 1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa in DNS
12:31 mark: Defined /31 neighbor blocks in 154.80.208.in-addr.arpa in DNS
09:15 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'increase account creation throttle value for el wiki for editing workshop (soon we have a two week break from these :-P)'
01:55 Ryan_Lane: upgrading squid to 2.7.9-5 on sq58 and sq73
01:35 Ryan_Lane: upgrading squid to 2.7.9-4 on sq58 and sq73
00:42 Ryan_Lane|food: also removing sheepdog from nova-*.tesla
00:39 Ryan_Lane|food: removing natty versions of libvirt and friends and re-installing the lucid versions

April 12

20:18 RobH: killed apache vhost for outreachcivi since it appears defunct in favor of contactscivi on singer
19:08 Ryan_Lane: upgrading nova-* on nova-*.tesla
19:05 Ryan_Lane: updating squid on sq73 to test new squid and patches
05:51 Ryan_Lane: purging nova-ajax-console-proxy from nova-compute1/2.tesla and adding to nova-controller.tesla
04:57 Ryan_Lane: purging nova-ajax-console-proxy from nova-controller.tesla and adding to nova-compute1/2.tesla
02:11 logmsgbot: tstarling synchronized php-1.17/includes/Sanitizer.php 'r85859'
01:30 logmsgbot: root synchronizing Wikimedia installation... Revision: 85853:
01:28 Tim: svn up/scap to deploy r85848
00:31 Ryan_Lane: installing glance on nova-controller.tesla

April 11

22:13 Ryan_Lane: installing python-nova.adminclient on nova-controller.tesla
21:19 notpeter: added lag check to storage3. restarting nagios
20:33 Ryan_Lane: pushing squid-2.7.9-1wm1 to wikimedia-lucid repo
05:15 Tim: did squid configuration update for bug 28235
04:40 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php 'Made categories reviewable on kawiki'
03:23 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php
03:21 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'flaggedrevs for kawiki'
03:13 AaronSchulz: Enabled FlaggedRevs for Georgian Wikipedia
03:11 logmsgbot: aaron ran sync-common-all

April 10

05:22 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'increase account creation throttle value for el wiki for editing workshop (can't wait for local sysadmins to be able to do this :-P)'

April 9

17:07 logmsgbot: ariel ran sync-common-all 'sync for elwikinews round 2, let's get the import right this time folks cause this is too nerve-wracking'
16:40 logmsgbot: ariel synchronized all.dblist 'remove elwikinews, need to drop and recreate after borked import'
16:29 logmsgbot: robh synchronized php-1.17/wmf-config/InitialiseSettings.php 'logo updates for a couple wikis for phillipe'
10:10 mark: Changed dataset1's clock source to HPET, synced it with ntpdate and restarted ntpd
07:58 apergos: power cycling searchidx1, load was at 60, unresponsive to commands after login from mgmt console
02:46 RobH: troubleshooting a couple new wikis, had to sync-apaches and restart them gracefully
01:01 notpeter: changed my.cnf on storage3 to replicated-do-db= drupal,mysql,civicrm
00:50 Ryan_Lane: installing nova-ajax-console-proxy on nova-controller.tesla

April 8

23:19 logmsgbot: laner ran sync-common-all
23:18 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php
22:50 Ryan_Lane: that graceful was me
22:28 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php
22:16 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php
22:14 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php
22:07 logmsgbot: laner ran sync-common-all
22:06 Ryan_Lane: gave myself deploy access in svn
17:10 notpeter: pushing out new dns zones. forgot to change ptr record for yvon...
15:09 RobH: updating dns with testblog info
13:36 mark: Added swap on /dev/sdc1 and /dev/sdd1 on ms5
13:34 mark: Stopped RAID10 array /dev/md2 again, sync takes too long
13:30 mark: Created RAID10 array for swap across first partition of 46 drives on ms5
13:21 mark: Stopped all rsyncs to investigate ms5's sudden kswapd system cpu load
07:57 apergos: assigned snapshot1 internal ip in dns
06:11 apergos: moving snapshot1 to internal vlan etc
04:15 notpeter: pushing new dns w/ eixamanius as cname for hooper and yvon as new name for box that was previously eixamanius
04:15 notpeter: stopping etherpad

April 7

21:43 notpeter: removed a silly check for hooper that I made and restarted nagios
19:06 Ryan_Lane: switching openstack deb repo back to trunk, and upgrading packages on nova-controller, since we are likely to target cactus now
15:40 mark: Restarted rsyncs
15:26 mark: Created a test btrfs snapshot of /export on ms6
15:12 mark: Temporarily stopped the rsyncs on ms5 to test zfs send performance
13:56 mark: Reenabled ms6 as backend on esams.upload squids
13:11 apergos: replaced ms4 in fstab on fenari with ms5 so we have thumbs mounted there
12:08 mark: Restarted rsyncs on ms5
12:07 apergos: nginx conf file change to "thumb" added to puppet
12:00 mark: Removed the test snapshot on ms5
11:47 apergos: edited in place /etc/nginx/sites-available/thumbs and /export/thumbs/scripts/thumb-handler.php to make thumbs generated on the fly return 200. they were returning 404
10:25 logmsgbot: catrope synchronized php-1.17/thumb.php 'Attempted fix for wrong temp/thumb paths'
10:20 apergos: (after reports from en vp that the search index has not been updated for over 4 days)
10:18 apergos: restarting search indexer on searchidx1
09:35 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialUploadStash.php 'debugging'
09:08 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialUploadStash.php 'r85612'
08:23 logmsgbot: catrope synchronized php-1.17/extensions/UploadWizard/UploadWizard.php 'Fix fatal due to missing API module'
08:17 logmsgbot: catrope ran sync-common-all
08:16 RoanKattouw: I meant srv196, not srv193
08:15 RoanKattouw: Deploying UploadWizard for real this time, forgot to svn up first. sync-common-all then clearMessageBlobs.php
08:14 RoanKattouw: Commenting out srv193 in mediawiki-installation node list because its timeouts take forever
08:10 RoanKattouw: srv196 is not responding to SSH or syncs from fenari (they time out after a looong time) but Nagios says SSH is fine. Should be fixed or temporarily depooled
08:08 RoanKattouw: Clearing message blobs
08:07 logmsgbot: catrope ran sync-common-all
08:04 RoanKattouw: Scap broke with sudo stuff AGAIN, running sync-common-all
08:01 RoanKattouw: Running scap to deploy UploadWizard changes
07:11 apergos: turned em off again, started seeing timeouts. bah
06:39 apergos: and two more...
06:31 apergos: restarted two of the 8 rsyncs on ms5, keeping an eye on them
01:31 domas: added nobarrier to xfs mount options on db32 and db37

April 6

20:38 RobH: updated puppet with a svn::client class (rt#721)
20:18 RobH: pulled wm09schols, wm10schols, and wm10reg out of enabled sites on singer
20:05 apergos: suspended all rsyncs on ms5, we were seeing nfs timeouts on the renderers all of a sudden
18:50 apergos: killed morebots and let the restart script start it up again

April 5

23:00 Ryan_Lane: restarting search indexer on searchidx to free space held by deleted logs
22:58 Ryan_Lane: clearing up some space on searchidx1
22:20 notpeter: crammed an etherpad db into db9's mysql hole.
17:57 Ryan_Lane: restarting llsearchd on all search boxes
17:45 RoanKattouw: Restarted morebots, running on wikitech as catrope
17:45 Ryan_Lane: changing the udp log location for search to emery
12:16 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Undo $wgForceUIMsgAsContentMsg change on incubator from last night per DannyB'

April 4

23:28 Ryan_Lane: uploading ircecho package to lucid-wikimedia repo, for nagios irc bot
22:22 Ryan_Lane: upgrading wikimedia-task-appserver package on srv281
22:22 Ryan_Lane: uploading new version of wikimedia-task-appserver to lucid-wikimedia repo; merges back in 1.17 changes that were missing
22:04 RobH: updated noc robots entry in its apache config on fenari
21:58 Ryan_Lane: srv281 is acting as a temporary scaling server for testing of lucid imagescalers, and to help with thumbs load.
21:27 Ryan_Lane: depooling srv281 from appservers
21:21 Ryan_Lane: syncing apaches to get configuration pushed to srv281
21:16 Ryan_Lane: rebooting srv281
21:01 Ryan_Lane: adding srv281 to rendering cluster in pybal via fenari
20:32 Ryan_Lane: uploading a new version of wikimedia-task-appserver fixing a problem with sync-common
20:13 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add mainpage to $wgForceUIMsgAsContentMsg for incubatorwiki'
19:55 Ryan_Lane: srv281 successfully ran imagescaler puppet class. ready for testing.
19:47 Ryan_Lane: adding php5-fss to lucid-wikimedia repo
19:11 Ryan_Lane: adding wikimedia-task-appserver to lucid-wikimedia repo
18:58 RobH: bugzilla updates complete
18:50 RobH: updating bugzilla per rt#718 bz#28409 bz#28402
18:42 notpeter: added cname etherpad for hooper.wikimedia.org
18:00 Ryan_Lane: added the wikimedia-fonts package to lucid-wikimedia repo
17:29 notpeter: adding self to nagios group. rebooterizing nagios.
05:58 apergos: cleaned up perms on commons/thumb/a/af, left over from interrupted rsync test last night
05:50 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling pool counter on all wikis'
04:12 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling PoolCounter on testwiki and test2wiki'
01:22 Tim: apache CPU overload lasted ~10 mins, v. high backend request rate, don't know cause, seems to have stopped now

April 3

18:42 apergos: 8 rsyncs of ms4 thumbs restarted with better perms so scalers can write... in screen as root on ms5. If we start seeing nfs timesouts in the scaler logs please shoot a couple
17:14 mark: Deployed max-connections on all cache peers for esams.upload squids to their florida parents (current limit 200)
17:00 mark: Removed the carp weights on the esams backends again, as the weighting was completely screwed up
16:59 mark: Started knsq13 backend
14:27 logmsgbot: catrope ran sync-common-all
14:26 RoanKattouw: Running sync-common-all to deploy r85256
13:03 apergos: shot rsyncs on ms5, setting 777 dir perms on all thumbnail dirs (eg e/ef/blablah.jpg) so scalers can write into them
12:53 apergos: did same for rest of projects and subdirs (777 on hash dirs)
12:46 apergos: chmod 777 on commons/thumb/*/* on ms5 so that scalers can create directories in there (mismatch of uid apache vs www-data)
11:12 mark: Raised per-squid connection limit to ms5 of 200 to 400 connections
11:05 mark: Raised per-squid connection limit to ms5 of 100 to 200 connections
10:55 mark: Fixed squid loop, the pmtpa.upload squids were using the esams squids as "CARP parents for distant content"
10:29 mark: Fixed puppet on sq42/43
09:44 mark: Lowered FCGI thumb handlers from 90 to 60 again, to reduce concurrency
08:08 mark: Started 4 more rsyncs (8 total now)
07:49 mark: Removed mlocate from ms5, puppetising
07:42 mark: Started 4 rsyncs from ms4 to ms5 (--ignore-existing)
07:32 mark: increased thumb handler count from 60 to 90
07:11 mark: Doubled the amount of fcgi thumb handlers
07:08 mark: Turned off logging of 404s to nginx error.log
06:50 mark: Restarted Apache on the image scalers
06:49 mark: Reconfigured ms5 to use the 404 thumb handler
06:48 Ryan_Lane: disabling nfs on ms4
06:33 mark: Running puppet on all apaches to fix fstab and mount ms5.pmtpa.wmnet:/export/thumbs
06:32 mark: Unmounting /mnt/thumbs on all mediawiki-installation servers
06:30 mark: Remounted NFS /mnt/thumbs on the scalers to ms5
06:28 Ryan_Lane: bring nfs back up
06:28 Ryan_Lane: brought ms4 back up. stopping the web server service and nfs
06:20 mark: Setup NFS kernel server on ms5
06:18 Ryan_Lane: powercycling ms4
05:29 Ryan_Lane: rebooting ms4 with -d to get a coredump
05:14 apergos: reanbling webserver on ms4 for testing
04:45 apergos: stopping web service on ms4 for the moment
04:29 apergos: shot webserver again
04:26 apergos: turned off hourly snaps on ms4, turned back on webserver and nfs
04:09 apergos: rebooted ms4, shut down webserver and nfsd temporarily for testing
02:58 apergos: still looking at kernel memory issues, still rebooting, ryan should be here in a few minutes to help out
02:03 apergos: a solaris advisor... also have zfs arch cache max to 2g which is ridiculously low but wtf right?
02:02 apergos: set tcp_time_wait_interval to 10000 at suggestion of
01:37 apergos: lowered zfs arch max to 2g (someone should reset this later)... will take effect on next reboot
00:29 apergos: rebooting with the new zfs arc cache max value, which will reduce the min value as well... dunno if this will give us enough breathing room or not
00:24 apergos: set zfs arc cache to ridiculously low value of 4gb, since when it's healthy it's using much less than that (1gb), this will take effect on reboot
00:22 Reedy: Still experiencing MS4 issues, thumb service is likely to be problematic for most users

April 2

23:47 apergos: rebooting ms4 from serial console, out to lunch and took the renderers down too
18:42 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Per NeilK, change Category:Uploaded_by_UploadWizard to Category:Uploaded_with_UploadWizard'
17:59 mark: Upgrading varnish to 2.1.5
17:14 logmsgbot: demon synchronized php-1.17/includes/filerepo/LocalFile.php 'r85200'
14:19 mark: Implemented CARP weights for distant CARP parents on squid configurator (used to be all equal before)
11:36 mark: Created btrfs filesystem on ms6, striped (raid10 style) over 46 devices - very experimental
09:50 mark: Reinstalling ms6 with Ubuntu 10.04
09:50 mark: Fixed torrus again
06:02 mark: !wikipedia The image thumbnail servers appear stable now
04:59 mark: Increased nginx worker processes from 1 to 4, set file limit to 30k
04:40 mark: !wikipedia Image Thumbnail server outage, it's being worked on
04:34 mark: Power cycling ms4 again
04:06 mark: Power cycled ms4 again
04:02 mark: Removed ms4 from pmtpa.upload config, sending all thumbs to ms5
03:47 mark: Restarted rsyncs ms4->ms5
03:25 Ryan_Lane: powercycling ms4 again
02:59 Ryan_Lane: rebooting ms4
02:46 Ryan_Lane: seems ms4 is totally dead, powercycling it
01:09 Ryan_Lane: installing python-pyinotify on spence for an updated ircecho

April 1

21:35 Ryan_Lane: purging some binlogs on db9 to free up space
21:35 RobH: bugzilla now version 4
21:31 RobH: taking down bugzilla for a quick upgrade
18:48 Ryan_Lane: added ctwoo, brion, py, and reedy to the engineering alias
18:36 mark: Deployed ms5.pmtpa.wmnet as a special 'apache' for pmtpa squid uploads... now serving a small portion of commons thumbs
18:11 RobH: bugzilla back online, CRproxy was affected, and repaired
17:30 RobH: bugzilla.wikimedia.org going offline for database backup and upgrade
17:13 RobH: beginning upgrade process for bugzilla, it's availability will be in question during this time
16:59 mark: Turned off Etag in the webserver7 configuration (/opt/webserver7/https-ms4/config/obj.conf) on ms4
16:50 notpeter: rm-ing old binlogs on db9 after confirming that there is no slave lag on db10
15:53 mark: Puppetised nginx and htcp purger setup on ms5
11:36 apergos: restarted lighty on dataset2 (but why did it die?)
00:06 logmsgbot: tstarling synchronized php-1.17/includes/specials/SpecialImport.php 'r85099'

March 31

20:52 notpeter: also, restarting t3h nagioz
20:51 notpeter: added self to /etc/nagios/contacts.cfg
20:32 logmsgbot: awjrichards synchronized php-1.17/extensions/UserDailyContribs/api/ApiUserDailyContribs.php 'r85088 updates to UserDailyContribs API to retrieve past year edit count'
18:58 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialImport.php 'r85078'
14:58 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php
14:58 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'same cleanup; ptwikinews (just did sqwiki)'
14:49 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php
14:48 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'moved add/removgroups settings from flaggedrevs.php to here'
14:44 mark: power cycled singer
14:40 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'added auteditor group on bewiki'
14:36 mark: Temporarily overriding reload caching request headers on upload squids to ease ms4 load
14:35 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php 'added auteditor group on bewiki; removed redundant config settings'
14:24 mark: Remounted /export/thumbs with noatime on ms4
14:21 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php 'added stablesettings right to sysop on bewiki'
14:01 mark: Outage resolved by disabling ehcache
13:57 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php
13:51 logmsgbot: root synchronized php-1.17/wmf-config/InitialiseSettings.php
13:37 logmsgbot: mark synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable ehcache'
11:43 mark: set max_stale 1 month on pmtpa.upload squids
11:21 mark: Set refresh_pattern on pmtpa upload squids
10:47 mark: Set refresh_pattern on sq85
00:21 Ryan_Lane: added drac management script and dependencies to fenari, via puppet
00:06 Ryan_Lane: make that python-paramiko
00:06 Ryan_Lane: installing paramiko on fenari

March 30

23:15 Ryan_Lane: adding puppet configuration for the collector and adding that class to spence
23:09 Ryan_Lane: updating udpprofile package again on spence
23:02 Ryan_Lane: upgrading udpprofile package on spence to fix init script
22:48 Ryan_Lane: killing the collector on spence, and trying to run via the package
22:48 Ryan_Lane: installing udpprofile package on spence
22:44 Ryan_Lane: created a package for the udpprofile code in svn; added to the main lucid-wikimedia repo
19:33 Ryan_Lane: fixed sanger certificate check for ldaps. warning will go away on next puppet run
17:05 Ryan_Lane1: enabled a nagios certificate check on sanger for ldaps
16:57 thedj: Maintenance ongoing on image server. Thumbnails on #Wikimedia sites may appear broken; we expect to restore full service shortly. (by Danese)
15:23 apergos: s/fine/find/
15:22 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'increase account creation throttle value for el wiki for editing workshop (maybe we should fine a better solution?)'
13:03 mark: Started 8 parallel rsyncs of ms4 /export/thumbs/wikipedia/commons/thumb/{8..f} to ms5
10:50 Tim: restarted ehcache with increased cache size limits
10:49 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabling ehcache'
10:45 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling ehcache temporarily'
10:15 apergos: rebooting ms5, trying to fix up raid config
08:52 logmsgbot: catrope ran sync-common-all
08:52 RoanKattouw: Running sync-common-all to push out UploadWizard changes
08:30 RoanKattouw: Deploying UploadWizard cahnges to testwiki
05:47 Tim: restarted profile-stats-logger on fenari, it died when spence went down
05:39 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled ehcache on all wikis'
05:29 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabled ehcache'
05:27 Tim: got OOM. Restarted tomcat with substantially more than the default heap maximum of 128MB
05:19 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling ehcache, something is wrong'
05:05 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enable ehcache on enwiki'
05:00 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enable ehcache on enwikiquote'
04:59 logmsgbot: tstarling synchronized php-1.17/includes/GlobalFunctions.php 'live hack: send site notice cache to memcached not ehcache'
04:58 logmsgbot: tstarling synchronized php-1.17/includes/MessageCache.php 'another one'
04:55 logmsgbot: tstarling synchronized php-1.17/includes/Skin.php 'live hack to send sidebar cache to memcached not ehcache'
04:50 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling ehcache client on testwiki and test2wiki'
04:46 Tim: installed ehcache under tomcat on db40. Documented at Ehcache.
03:49 Tim: did apt-get upgrade on db40, now rebooting to get new kernel
01:10 mark: Reinstalled ms5 with Ubuntu 10.04, created software RAID10 array /dev/md1 out of 48 drives

March 29

20:44 mark: Moved ms5 to internal vlan 2
20:34 mark: Pushed DNS for ms5.pmtpa.wmnet
18:56 mark: Changed NFS client params to rsize=8192,wsize=8192 on srv228:/mnt/thumbs
15:37 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'increase account creation throttle value for el wiki for editing workshop'
10:41 apergos: rsync on dataset2 for White Cat one last time, please shoot in three days (check with user first)
00:42 Ryan_Lane: enabled ssl on RT, all non-http now forces to https for RT
00:42 Ryan_Lane: fixed puppet issue on erzurumi by adding the wikidev group.

March 28

23:37 Ryan_Lane: spence died; powercycling it
23:06 logmsgbot: awjrichards synchronized php-1.17/extensions/UserDailyContribs/api/ApiUserDailyContribs.php 'Deploying bug fixes to UserDailyContribs API (r84935)'
22:41 Ryan_Lane: added a certificate check for ldaps on nfs1/2
22:36 Ryan_Lane: pushed the internal CA certificate to spence, so that we can monitor the LDAP certificates
21:43 awjr: running sync-comon all to deploy api updates to extension UserDailyContrib
21:43 logmsgbot: awjrichards ran sync-common-all
21:38 logmsgbot: awjrichards synchronized php-1.17/extensions/UserDailyContribs/api 'Attempting to create UserDailyContribs/api dir using sync-file'
21:07 Ryan_Lane: pushed out a cert check for secure.wikimedia.org
19:17 Ryan_Lane: changing weight of mobile1 to 10
17:10 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable Vector/SectionEditLinks on enwiki'
16:58 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryCategoryMembers.php 'r84908'
14:58 apergos: was seeing sluggish response from scalers, ms4, running out of swap space
14:54 apergos: restarting web server on ms4...
05:58 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling ehcache'
05:57 Tim: ehcache broke, disabling client
05:39 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php
05:35 Tim: db40 CPU usage is 5% of all cores, disk util <1%, network ~1%, request rate ~58% of weekly peak. Enabling ehcache on all wikis.
05:32 Tim: fixing torrus per wikitech instructions
05:08 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling ehcache on enwiki'
05:07 Tim: resource utilisation on db40 is a nothingth of a percent, moving on to stage 4
05:04 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling ehcache for enwikiquote'
04:56 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'put ehcache first so it will get more load'
04:45 logmsgbot: tstarling synchronized php-1.17/includes/objectcache/EhcacheBagOStuff.php 'curl fix'
04:38 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enable ehcache on test and test2'
04:36 logmsgbot: tstarling synchronized php-1.17/includes/objectcache/EhcacheBagOStuff.php 'don't use CURLOPT_TIMEOUT_MS'
04:28 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'ehcache setup'
04:28 Tim: fixed sync on searchidx1
04:27 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'test sync'
04:27 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'test sync'
04:26 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'ehcache setup'

March 27

22:28 Ryan_Lane: restarting apache on singer
22:19 Ryan_Lane: pushing new star cert to singer
21:55 RoanKattouw: SSL certificate for secure.wikimedia.org expired about an hour ago, no ops around. I've just texted Ryan
19:52 logmsgbot: demon synchronized php-1.17/extensions/ProofreadPage/ProofreadPage_body.php 'r84871'
14:14 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderModule.php 'r84845, hopefully fixes catchable fatal error about makeCombinedStyles()'
10:55 logmsgbot: hashar: on searchidx1 wmf-config path does not exist triggering an error on code sync
10:54 logmsgbot: hashar: auto defers SocialProfile, editor_trends & WikiSnaps
10:54 logmsgbot: hashar synchronized php-1.17/wmf-config/codereview.php

March 26

21:41 logmsgbot: ariel synchronized php-1.17/extensions/Narayam/Narayam.i18n.php 'remove BOM from file, was being written to stdout'
19:13 apergos: running rsync for White Cat again to pick up en wiki 20110115 hist files, on dataset2 in screen session as root, please shoot after 3 days (check with user first in channel)
15:05 logmsgbot: mark synchronized php-1.17/wmf-config/db.php 'Repooled db38 with lower load (100 instead of 400)'
14:57 logmsgbot: mark synchronized php-1.17/wmf-config/db.php 'Depooling db38'
10:55 logmsgbot: midom synchronized php-1.17/wmf-config/mc.php 'srv206 to srv243 swap'
10:53 domas: I misread before. memcached check was broken for 23 days, not 23 hours. duct-taped by creating symlink php-1.17 -> wmf-deployment on spence, not sure for how long :)
10:50 domas: added 'pkill -9 -f smssend' hourly crontab on spence...

March 25

15:36 mark: Upgraded csw1-esams and br1-knams to IronWare 2.7.3a
15:31 mark: Reloading br1-knams
15:09 mark: Reloading csw1-esams
15:09 logmsgbot: demon synchronized php-1.17/includes/HTMLForm.php 'r84748'
14:53 mark: Upgraded JUNOS to 10.0S10.0 on csw2-esams
13:18 mark: Moving esams traffic to pmtpa for upcoming router firmware upgrade/reload
04:49 Tim: reniced apaches to nice level 0 again
03:53 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php
03:51 logmsgbot: aaron synchronized php-1.17/wmf-config/abusefilter.php 'Moved abusefilter rights here from InitialiseSettings.php'
03:36 logmsgbot: root synchronizing Wikimedia installation... Revision: 84728:
03:36 Tim: svn up/scap to r84727
02:45 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'Added missing autopatrolled group to siwiki'
02:29 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'Added missing rollbacker group to siwiki'
02:07 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php 'DO NOT PUT ABUSEFILTER SETTINGS IN HERE'
02:06 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'Copy abusefilter right settings from flaggedrevs.php to here, were they belong'
01:55 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php
01:24 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php 'Simple config cleanups'
00:48 hcatlin: Started rolling restarts passenger/ruby cluster on mobile2

March 24

23:50 tomaszf: lowering weight for mobile2 from 45 -> 30 on m.
22:51 logmsgbot: tfinc synchronized php-1.17/wmf-config/InitialiseSettings.php 'adding missing redirects for mobile sites'
22:33 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php
22:27 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28195 - added two namespaces two sqwikinews'
20:23 Ryan_Lane: fixing the authorized_keys file on ragweed
17:16 rainman-sr: clearing up some space on searchidx1
17:04 RobH: hooper switch port vlan updated, system ready for installation
16:54 tomaszf: rebalancing mobile1/2/3 traffic load to 15/40/60
15:54 logmsgbot: catrope synchronized php-1.17/extensions/Narayam/Narayam.i18n.php 'r84684'
15:54 logmsgbot: catrope synchronized php-1.17/extensions/Narayam/Narayam.hooks.php 'r84684'
15:54 logmsgbot: catrope synchronized php-1.17/extensions/Narayam/Narayam.php 'r84684'
15:40 logmsgbot: catrope synchronized php-1.17/extensions/Gadgets/Gadgets_body.php 'r84683'
13:37 thedj: ResourceLoader broken due to bug 28215
04:28 logmsgbot: tstarling synchronized php-1.17/skins/common/shared.css 'r84666'
02:58 Tim: fixing bug 9204: AllowEncodedSlashes On

March 23

19:37 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'Bump filter cache version'
19:37 logmsgbot: catrope synchronized php-1.17/includes/libs/JavaScriptMinifier.php 'r84628'
18:34 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderModule.php
18:34 logmsgbot: catrope synchronized php-1.17/includes/AutoLoader.php
18:32 RoanKattouw: Something went wrong with that sync, trying to fix it with individual file syncs
18:30 logmsgbot: catrope ran sync-common-all
18:29 RoanKattouw: Running sync-common-all to deploy ResourceLoader changes and some other random things (r84613, r84616, r84620)
17:49 RoanKattouw: I've staged some ResourceLoader changes on testwiki. Please DO NOT RUN sync-common-all or scap for the next hour or so
17:00 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialUpload.php 'r84605'
17:00 logmsgbot: catrope synchronized php-1.17/includes/upload/UploadBase.php 'r84605'
17:00 logmsgbot: catrope synchronized php-1.17/extensions/TitleBlacklist/TitleBlacklist.list.php 'r84605'
16:46 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesLtg.php 'r84604'
16:46 logmsgbot: catrope synchronized php-1.17/languages/Names.php 'r84604'
16:46 logmsgbot: catrope synchronized php-1.17/includes/ChangeTags.php 'r84604'
16:46 logmsgbot: catrope synchronized php-1.17/extensions/Renameuser/Renameuser.php 'r84604'
16:46 logmsgbot: catrope synchronized php-1.17/extensions/Renameuser/Renameuser_body.php 'r84604'
16:28 logmsgbot: catrope synchronized php-fatal-error.html 'Update this file from SVN /trunk/tools/php/wmerrors/error.html . This also changes its chmod so people who are not Tim can change this file'
16:14 logmsgbot: demon synchronized php-1.17/extensions/FlaggedRevs/forms/RevisionReviewForm.php 'r84599'
13:08 apergos: started shovelling some data over to the newly revived dataset1 (from dataset2, screen session as root)
12:55 mark: Configured OWA on owa{1,2,3} to send error log emails to nimish, instead of to "empty recipients", and therefore, root
12:28 apergos: yeah I wanted sync-common-file and typed all instead. grrrr but apparently no harm done
12:15 logmsgbot: ariel ran sync-common-all 'php-1.17/wmf-config/CommonSettings.php raise acct throttle for el wiki for editing workshop (3)'
00:25 notpeter: dumped bugzilla db from db9 and imported to db10, skipped some errors, slaving is moving again.

March 22

21:12 Ryan_Lane: changing donation.tesla to nomad.tesla in DNS
19:34 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/api/ApiQueryCodeRevisions.php 'r84545'
18:44 Ryan_Lane: destroying donation.tesla.usability.wikimedia.org and replacing with nomad.tesla.usability.wikimedia.org
17:26 nimish_g: changed owa sql timeouts, restarted apaches on owa machines
17:19 RobH: dataset1 back online for testing, apergos will take over later today/tomorrow, will not go into service just yet
14:47 logmsgbot: catrope ran sync-common-all
14:46 RoanKattouw: Running sync-common-all to deploy r84467, r84484, r84495 and r84530. Needed because directory additions don't work with sync-file AFAIK
06:16 logmsgbot: ariel synchronized docroot/www.wikimedia.org/google01afe16dd444713b.html 'authorize generic tech account for google webmaster tools (for use with google storage)'
00:28 Tim: removing old binlogs from db9

March 21

23:30 Ryan_Lane: upgrading nova-* packages on nova-*.tesla
23:30 Ryan_Lane: installing ajaxterm on nova-compute1/2
21:59 Reedy: dataset1 is getting another beating. You know it deserves it, right?
20:42 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28129 - Enable subpages in main namespace on sawikisource'
20:37 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28134 - Enable alias "WL" -> "Wikilajme" for Albanian Wikinews'
20:18 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27256 - Correcting content page count at en.wikibooks and pt.wikibooks'
20:12 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '26281 - Please allow AbuseFilter on Hungarian Wikisource and Wikibooks'
20:10 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28160 - AbuseFilter on Kannada Wikipedia'
03:44 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling PoolCounter due to bug 28144'
01:42 logmsgbot: demon synchronized php-1.17/extensions/FlaggedRevs/FlaggedArticleView.php 'r84437'
00:26 rainman-sr: built search indexes for new wikis: eowikisource, ltgwiki, sawikisource, sahwikisource, kbdwiki, sqwikinews

March 20

21:19 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/WikiEditor.php 'r84422'
21:19 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/WikiEditor.hooks.php 'r84422'
21:11 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/WikiEditor.php 'rm duplicate call'
21:11 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/WikiEditor.hooks.php 'Fix fatal'
21:09 logmsgbot: catrope ran sync-common-all
21:07 RoanKattouw: Deploying r84420 with sync-common-all
19:57 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27379 - Please create Wikipedia in Latgalian language'
15:18 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add oldwikisource to import sources on sawikisource'
14:50 logmsgbot: catrope synchronized php-1.17/resources/jquery/jquery.textSelection.js 'r84385'
12:35 RoanKattouw: Clearing and rebuilding LocalisationUpdate cache. Using altered version of /h/w/bin/l10nupdate which pushes updates to Apaches immediately, then clears message blobs
09:39 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add English Wikisource to import sources on sawikisource'
09:36 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Change sitename too on orwiki, was untranslated'
09:35 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Fix previous change - set $wgMetaNamespace, not $wgSitename'
09:33 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27948 - Set $wgSitename on orwiki'

March 19

14:51 RoanKattouw: Purging http://eo.wikisource.org/wiki/Speciala%C4%B5o:Preferoj
14:51 logmsgbot: hashar: running purge http://eo.wikisource.org (no trailing slash)
14:34 logmsgbot: hashar: previous sync was to fix wikisource eo logo
14:33 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php
13:01 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabling PoolCounter (was disabled during the previous breakage)'
12:59 logmsgbot: tstarling synchronized php-1.17/includes/PoolCounter.php 'r84323: do not show error to user on connection refused etc.'
12:25 Tim: attempted to restart poolcounterd with a different file descriptor limit. It didn't start up properly, causing every parse request to return an error. Fixed in about 30 seconds.
12:23 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php
12:15 logmsgbot: tstarling synchronized php-1.17/extensions/PoolCounter/PoolCounterClient_body.php 'timeout fix'
12:14 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabling PoolCounter, now with error logging enabled'
12:13 logmsgbot: tstarling synchronized php-1.17/includes/PoolCounter.php 'poolcounter error logging'
12:12 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'poolcounter error logging'
11:24 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28114 - sqwikinews logo'
07:06 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27663 - Please install Abuse Filter in Bashkir Wikipedia'
07:05 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '24466 - Enable Collection extension on hiwiki'
07:04 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27926 - Enable AbuseFilter for Outreach Wiki'
07:03 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28049 - AbuseFilter in fa.wikiquote'
07:01 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27578 - Enable Extension'
06:16 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'disable poolcounter - it should not be serving errors'

March 18

22:04 logmsgbot: jeluf synchronized php-1.17/cache/interwiki.cdb 'Updating interwiki cache'
21:55 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php 'set sqwikinews logo'
21:50 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27842 - enable blocked user to edit talk page on hewikiquote'
21:48 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27740 - give sysop the abusefilter-modify-restricted right in the Hebrew Wikipedia'
21:39 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27840 - enable transwiki import to latin wikisource'
21:34 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28092 - Add Portal namespace on Swedish Wikiversity'
21:32 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28092 - Add Portal namespace on Swedish Wikiversity'
21:28 logmsgbot: jeluf synchronized php-1.17/wmf-config/CommonSettings.php '27896 - Enable Special:Cite at en.wikiversity'
21:21 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28022 - Create the Albanian Wikinews'
21:13 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28003 - Create the Kabardian Circassian Wikipedia'
21:06 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27557 - Create Sakha Wikisource'
20:58 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27379 - Please create Wikipedia in Latgalian language'
20:41 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '26136 - Create Wikisource in Esperanto'
20:34 JeLuF: apache-graceful-all reports lots of NTP errors
20:14 logmsgbot: jeluf ran sync-common-all 'Created eowikisource, ltgwiki, sawikisource, sahwikisource, kbdwiki, sqwikinews'
19:47 domas: fixed autocommit in wmf-config by uncommenting a cron entry in my crontab \o/
19:44 logmsgbot: jeluf synchronized langlist 'Added ltg and kbd'
19:00 awjr: Archived Hudson build logs for CiviMail-related Hudson jobs to storage3 (/archive/fundraising/hudson_builds) and cleared out related log dirs on Grosley. Hudson CiviMail-related builds now back to normal.
10:13 Tim: fixed punctuation in bugzilla quip 1155 by direct DB access
03:05 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling PoolCounter on all wikis'
03:04 Tim: poolcounterd CPU usage still negligible. Enabling on all wikis.
03:00 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling PoolCounter on en.wikipedia.org'
02:59 logmsgbot: tstarling synchronized php-1.17/wmf-config/PoolCounterSettings.php 'longer connect timeout'
02:53 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling PoolCounter on en.wiktionary.org'
02:39 logmsgbot: root synchronizing Wikimedia installation... Revision: 84213:
02:37 logmsgbot: root synchronizing Wikimedia installation... Revision: 84213:
02:37 Tim: added PoolCounter to extension-list
02:36 logmsgbot: tstarling synchronized php-1.17/extensions/PoolCounter/PoolCounterClient_body.php
02:34 logmsgbot: tstarling synchronized php-1.17/extensions/Renameuser/Renameuser.php 'increase contribution limit to 50k'

March 17

22:47 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump $wgVectorSectionEditExperiment'
22:46 logmsgbot: catrope synchronized php-1.17/extensions/Vector/modules/ext.vector.sectionEditLinks.js 'r84213'
22:28 RoanKattouw: Clearing message blobs
22:27 logmsgbot: catrope ran sync-common-all
22:27 RoanKattouw: Deploying r84160 and r84210 with sync-common-all
21:03 logmsgbot: mark synchronized php-1.17/wmf-config/db.php 'Temp depool srv178-180'
20:37 mark: Shutdown srv178-180 for relocation to rack D1
19:21 tomaszf: ran purge-varnish MobileRedirect.js to clear out old redirect regex
19:06 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php
19:06 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js
18:59 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 'Bump style version
07:41 logmsgbot: ariel ran sync-common-all 'php-1.17/wmf-config/CommonSettings.php raise acct throttle for el wiki for editing workshop (2)'
02:29 mark: Set high performance http sysctl settings on mobile1 (temp, needs puppetising)
02:22 tomaszf: changing weight on mobile1 to 15 instead of 30
02:20 mark: restarted apache on mobile1

March 16

15:39 logmsgbot: ariel ran sync-common-all 'php-1.17/wmf-config/CommonSettings.php raise acct throttle for el wiki for editing workshop'
13:01 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Don't enable Narayam by default'
12:38 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable Narayam on mlwiki'
12:33 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable Narayam on mlwikisource'
12:30 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable Narayam on mlwiktionary'
12:28 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable Narayam on mlwikibooks'
12:25 RoanKattouw: I meant mlwikiquote of course
12:25 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable Narayam on nlwikiquote'
12:19 logmsgbot: catrope ran sync-common-all
12:18 RoanKattouw: Running sync-common-all to push Narayam code to the cluster (currently only enabled on testwiki)

March 15

21:18 hcatlin: restarted mobile app on mobile1 due to unexpected load increase beginning at 20:43
20:36 RoanKattouw: Clearing message blobs
20:36 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r84052'
20:35 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.i18n.php 'r84052'
19:28 Ryan_Lane: added svg to allowed upload types for wikitech
17:51 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Cleaning out cruft from CentralizeNotice config; Updating how https is handled for CentralNotice; PoolCounter settings inclusion in CommonSettings (but not checking in/enabling new Poolcounter settings in InitialiseSettings)'
17:00 mark: Replaced root's authorized_keys file
06:47 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'pool counter deployment on testwiki only'
06:46 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'pool counter deployment on testwiki only'
06:42 logmsgbot: root synchronizing Wikimedia installation... Revision: 83954:
06:42 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 83954:
06:41 Tim: running scap to get the PoolCounter extension
06:28 Tim: installed poolcounter on tarin
06:08 Tim: tarin upgraded to lucid
05:15 Tim: tarin just has an ancient test wordpress install on it which hasn't been touched since April 2010. Removing the website and mysql instance, will attempt to upgrade the distro.
03:20 logmsgbot: tstarling synchronized php-1.17/includes/HTMLCacheUpdate.php 'disable squid cache invalidation from HTMLCacheUpdate, to hopefully eliminate apache CPU spikes'
03:12 Tim: started job runners
03:09 Tim: apache CPU spike, stopped job runners
03:05 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'removed bug 27891 logging'
03:04 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 'removed bug 27891 logging'

March 14

19:57 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.css 'r83954'
19:33 logmsgbot: catrope synchronized php-1.17/extensions/ClickTracking/modules/ext.clickTracking.js 'r83952'
19:33 logmsgbot: catrope synchronized php-1.17/extensions/ClickTracking/modules/jquery.clickTracking.js 'r83952'
19:28 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable ArticleFeedback on enwiki'
19:23 logmsgbot: catrope ran sync-common-all
19:22 RoanKattouw: Got sudo weirdness on scap again, running sync-common-all
19:21 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 83949:
19:20 logmsgbot: catrope synchronized php-1.17/extensions/Collection/Collection.suggest.php 'Syntax error'
19:19 RoanKattouw: Running scap to deploy ArticleFeedback (only enabled on test for now, will enable on enwiki after this)
18:38 Ryan_Lane: installing sudo schema on nova-controller.tesla and restarting opendj
18:14 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/CodeReview.php 'Style version bump'
18:14 logmsgbot: catrope ran sync-common-all
18:12 RoanKattouw: Running sync-common-all to deploy r83934
09:55 Tim: increased memcached read timeout from 100ms to 500ms. This reduced the read timeout rate from 51/s to 3.4/s.
09:50 logmsgbot: tstarling synchronized php-1.17/wmf-config/mc.php 'increasing memcached timeout from 100ms to 500ms while monitoring timeout rate'
09:44 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'log memcached timeout errors'
09:40 logmsgbot: tstarling synchronized php-1.17/includes/memcached-client.php 'log timeout errors'
06:26 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 'more specific logging hack'
04:32 logmsgbot: tstarling synchronized php-1.17/includes/Article.php 'r83869'
03:42 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 'logging tweak'
03:36 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 'logging tweak'
03:32 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 'more logging'
03:23 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php
03:07 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'logging hack for bug 27891'
03:06 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 'logging hack for bug 27891'
01:52 logmsgbot: tstarling synchronized php-1.17/includes/HTMLCacheUpdate.php 'throttle HTML cache invaldiation'
00:27 logmsgbot: midom synchronized php-1.17/includes/Block.php
00:13 domas: reniced sshd to -10, apache2 to 0, to have better colors on graphs :)
00:03 logmsgbot: midom synchronized php-1.17/includes/Block.php 'livehack for separate connection'

March 13

19:25 logmsgbot: catrope synchronized php-1.17/skins/vector/main-rtl.css
19:24 logmsgbot: catrope synchronized php-1.17/skins/vector/main-ltr.css 'Put this back but forgot to sync, d'oh'
16:16 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '26446 - Change title of Kannada Wikipedia'
15:54 logmsgbot: hashar: Have set commonswiki user Wikinger77 email to the same as the dewiki account with the name.
15:37 logmsgbot: midom synchronized php-1.17/languages/Language.php
15:24 logmsgbot: hashar synchronized php-1.17/wmf-config/CommonSettings.php 'enable script for proofreadpage extension'
15:23 logmsgbot: hashar synchronized php-1.17/wmf-config/proofreadpage.php 'script to configure the proofreadpage extension'
14:48 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27255 - Setup Rollback group at ko.wikipedia'
14:40 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27268 - New "Rollbacker" group for zh.wikinews'
12:15 logmsgbot: hashar: ran a groupby query on db36:enwiki database by mistakes :/
11:51 logmsgbot: hashar synchronized php-1.17/wmf-config/CommonSettings.php 'bug 24313 - hide preference minordefault for enwiki'
11:29 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27651 - Create WikiProject namespace for ko.wikipedia'
11:27 logmsgbot: hashar: committed db.php
11:17 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27728 - Add WP as an alias to project namespace in Portuguese Wikipedia'
11:11 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27878 - Change project namespace on kuwiktionary'
11:02 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27898 - Create Autor namespace at ca.wikisource'
10:52 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27898 - Create Autor namespace at ca.wikisource'
10:44 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27413 - Allow eswiki bureaucrats to add/remove the confirmed usergroup'
09:56 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27998 - Add WP alias on dawiki'
01:31 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 'even out a bit'
00:07 domas: db16 write-behind cache failed yet again, changing states doesn't help much, battery seems to be dead ( FSA_EM_ENHANCED_BATTERY_CHANGE ? ) - set innodb log sync to 0

March 12

15:47 logmsgbot: midom synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.hooks.php
15:15 logmsgbot: midom synchronized php-1.17/extensions/FlaggedRevs/specialpages/ProblemChanges_body.php
15:04 logmsgbot: midom synchronized php-1.17/extensions/FlaggedRevs/specialpages/PendingChanges_body.php
14:17 logmsgbot: hashar: shell madness is over for today.
14:06 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27195 - Grant anonymous users 'createpage' right and enable Extension'
14:01 domas: fixed *-filter segfaults on locke, probably :)
13:14 logmsgbot: hashar: Cleaning bugs in the Wikimedia product
13:04 logmsgbot: hashar: Reedy made me an admin on bugzilla. Then allowed myself to edit components.
12:51 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'whitespaces'
12:37 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27954 - Change autoconfirmed parameters on pt.wiki'
12:30 logmsgbot: hashar: committed db.php
12:27 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27971 - Change the translation of "user" and "user_talk" namespaces on ka.wiki'
12:14 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27361 - Update logo for Fiji Hindi Wikipedia'
12:05 logmsgbot: hashar: ran namespaceDupes.php on cywiki : Looks good!
12:04 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27684 - Create Portal namespace on cy.wikipedia.org'
11:55 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27826 - Logo update on Tamil Wikipedia'
11:50 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27918 - Enable patrolled edits on Hebrew Wikibooks'
11:37 logmsgbot: midom synchronized php-1.17/extensions/ClickTracking/ClickTracking.hooks.php
11:37 logmsgbot: midom synchronized php-1.17/extensions/ClickTracking/ClickTracking.hooks.php

March 11

23:42 logmsgbot: catrope synchronized php-1.17/extensions/Vector/Vector.php 'r83735'
23:42 logmsgbot: catrope synchronized php-1.17/extensions/Vector/Vector.hooks.php 'r83735'
23:34 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Set $wgVectorSectionEditLinksExperiment to 1'
23:30 logmsgbot: catrope synchronized php-1.17/extensions/ClickTracking/modules/jquery.clickTracking.js
23:12 RoanKattouw: Apache CPU went back to normal at 23:10
23:07 RoanKattouw: Apache cluster CPU shot up to ~95% around 23:05 UTC
21:43 rainman-sr: freed up some space on searchidx1 which was again out of disc space
12:00 RoanKattouw: updateCollation.php finished on enwiki. The script has now run to completion on all wikis
04:39 Tim: made myself owner of the mediawiki-cvs mailing list
02:25 logmsgbot: tstarling synchronized php-1.17/StartProfiler.php 'separate CLI from other profiling'
02:12 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh 'increased number of processes per job runner from 4 to 5'
02:10 Tim: installed job runners on srv231-247
00:59 ^demon: landing page for svn.wm.o is now php, so it can read latest public dump file if available
00:29 Ryan_Lane: patching ec2 api on nova-controller.tesla to return project instead of user in the status

March 10

21:55 Ryan_Lane: updating nova-* packages on nova-*.telsa
21:36 logmsgbot: catrope synchronized php-1.17/extensions/Vector/modules/ext.vector.sectionEditLinks.js 'r83672'
21:35 logmsgbot: catrope synchronized php-1.17/extensions/Vector/Vector.hooks.php 'r83672'
21:35 logmsgbot: catrope synchronized php-1.17/extensions/Vector/Vector.php 'r83672'
21:34 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Set $wgVectorSectionEditLinksLotteryOdds to 1%'
21:18 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 27918 - Enable patrolled edits on Hebrew Wikibooks'
21:10 logmsgbot: hashar: we have pending changes in db.php and CommonSettings.php . Please commit! :p
18:52 awjr: restarting dovecot on grosley in feeble attempt to resolve civimail issue
18:22 RoanKattouw: Put the 1.16wmf4 versions of skins/vector/main-{ltr,rtl}.css in /h/w/c/php/skins/vector as a temp hack so Squid-cached pages don't point to nonexistent CSS
18:18 Ryan_Lane: removing custom built memcached from lucid-wikimedia
17:03 Ryan_Lane: adding memcached to lucid-wikimedia repo
16:17 RoanKattouw: updateCollation.php has finished on s4 (commonswiki) and s6 (fr+ja+ruwiki). Now only running on s1 (enwiki)
11:40 logmsgbot: tstarling synchronized php-1.17/includes/job/JobQueue.php
09:15 hashar: Moved server admin logs from November 1st 2010 till December 31st 2010 to Server admin log/Archive 17
05:32 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh 'de-prioritise renameUser due to slave lag'
05:28 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh
05:28 logmsgbot: tstarling synchronized php-1.17/maintenance/runJobs.php
05:13 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 'offloading db24 since it is very laggy'
04:56 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh 'prioritise renameUser'
04:43 logmsgbot: tstarling synchronized php-1.17/maintenance/nextJobDB.php 'testwiki hack'
04:24 logmsgbot: tstarling synchronized php-1.17/maintenance/nextJobDB.php 'bug fixes'
04:23 logmsgbot: tstarling synchronized php-1.17/maintenance/nextJobDB.php 'bug fixes'
04:11 logmsgbot: tstarling synchronized php-1.17/maintenance/nextJobDB.php
02:48 awjr: attempting to restart Hudson on grosley due to it crashing with Java exceptions
02:27 logmsgbot: tstarling synchronized php-1.17/includes/job/JobQueue.php
02:20 logmsgbot: tstarling synchronized php-1.17/includes/DefaultSettings.php
02:20 logmsgbot: tstarling synchronized php-1.17/includes/GlobalFunctions.php
02:20 logmsgbot: tstarling synchronized php-1.17/includes/job/JobQueue.php
02:19 Tim: svn up/scap for r83628
02:18 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'wgAggregateStatsID'

March 9

21:55 Ryan_Lane: adding imagemagick to lucid-wikimedia
20:01 logmsgbot: catrope synchronized php-1.17/extensions/Vector/Vector.php 'r83608'
19:55 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable ClickTracking on testwiki too'
19:51 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Setting $wmgClickTrackThrottle to -1 on all wikis'
19:48 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable ClickTracking on enwiki, needed for VectorSectionEditLinks'
19:41 logmsgbot: catrope ran sync-common-all
19:40 RoanKattouw: Running sync-common-all to deploy VectorSectionEditLinks for 0.1% of enwiki users
19:12 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 'studid typo'
18:57 logmsgbot: catrope synchronized php-1.17/includes/db/LoadBalancer.php 'r83591'
18:48 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php
18:47 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js
18:05 logmsgbot: hashar: running refreshLinks.php wikimania2011wiki
17:44 RobH: survey.w.o upgrade complete
17:40 Ryan_Lane: adding php5.3 packages to lucid-wikimedia main repo
17:32 RobH: upgrading survey.w.o install, so it may be inaccessible for a few minutes tops.
14:50 RoanKattouw: ETAs for still-running updateCollation.php instances: enwiki: 33 hours, commons: 14 hours, s6 (fr+ja+ruwiki): 19 hours
14:29 RoanKattouw: updateCollation.php has finished on s2
12:41 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 'Style version bump'
12:40 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js 'Fix syntax error'
12:31 RoanKattouw: updateCollation.php finished on s3
11:55 RoanKattouw: Correction, 11:00 UTC
11:55 RoanKattouw: updateCollation.php finished on s5 (dewiki) around 12:00 UTC
11:52 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 'Style version bump'
11:50 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js 'Live hack mobile redirect script to fix JS errors'
10:10 RoanKattouw: cl_collation!='uppercase' was causing a full table scan on categorylinks on dewiki and frwiki. FORCE INDEX didn't help, so I live-hacked the script to do cl_collation='' instead. This makes the script run noticeably faster across the board
10:07 RoanKattouw: updateCollation.php has finished on s7
02:36 logmsgbot: tfinc synchronized php-1.17/extensions/CentralNotice/SpecialBannerController.php
02:36 logmsgbot: tfinc synchronized php-1.17/extensions/CentralNotice/CentralNotice.i18n.php
02:35 logmsgbot: tfinc synchronized php-1.17/extensions/CentralNotice/1x1.png
01:04 logmsgbot: catrope synchronized php-1.17/includes/CategoryPage.php 'Live hack to make from=a work in category pages'
01:03 logmsgbot: catrope synchronized php-1.17/includes/CategoryPage.php 'Unbreak lowercase parameters like pagefrom=a'
00:43 logmsgbot: catrope synchronized php-1.17/extensions/CategoryTree/CategoryPageSubclass.php 'r83651'
00:14 logmsgbot: catrope synchronized php-1.17/includes/CategoryPage.php
00:09 RoanKattouw: s2 lag back to normal
00:04 RoanKattouw: This fixed s1 replag. s2 replag is still pretty bad, db24 is now at 1560s
00:02 RoanKattouw: Fixed wait for slave logic in updateCollation.php , was broken

March 8

23:17 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/backend/CodeRevision.php 'r83551'
22:41 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Adding openZIM as file format for Collection extension'
21:47 RoanKattouw: Restarted all collation update jobs with a more efficient SELECT query (with STRAIGHT_JOIN) and a smaller batch size (50 instead of 1000)
21:18 RoanKattouw: Restart updateCollation.php on s4 (commons) with a batch size of 50 to see how that plays out
21:07 RoanKattouw: Move complete. Aborted updateCollation.php on fenari, now running 7 instances of /home/catrope/collatecluster in a screen on hume
21:01 RoanKattouw: Breaking out updateCollation.php run into one process per DB cluster, and running them on hume rather than fenari
20:42 logmsgbot: catrope synchronized php-1.17/includes/CategoryPage.php 'Experimental fix by Aryeh for category sorting problems'
20:35 logmsgbot: catrope synchronized php-1.17/includes/parser/Parser.php
20:35 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryCategoryMembers.php
20:34 logmsgbot: catrope synchronized php-1.17/includes/Title.php
20:34 logmsgbot: catrope synchronized php-1.17/includes/LinksUpdate.php
20:34 logmsgbot: catrope synchronized php-1.17/includes/CategoryPage.php
20:34 logmsgbot: catrope synchronized php-1.17/extensions/intersection/DynamicPageList.php
20:34 logmsgbot: catrope synchronized php-1.17/extensions/FlaggedRevs/specialpages/UnreviewedPages_body.php
20:34 logmsgbot: catrope synchronized php-1.17/extensions/Collection/Collection.body.php
20:33 logmsgbot: catrope synchronized php-1.17/extensions/CategoryTree/CategoryTreeFunctions.php
20:32 RoanKattouw: Re-syncing a bunch of files without changes, didn't seem to have been picked up right by scap
19:38 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryCategoryMembers.php 'Sync without changes, maybe this didn't get picked up right'
19:35 logmsgbot: catrope synchronized php-1.17/includes/CategoryPage.php 'Use array_merge instead of + for paranoia'
17:46 RoanKattouw: Running updateCollation.php on all wikis
17:37 RoanKattouw: Running updateCollation.php on cswiktionary
17:36 logmsgbot: robh synchronizing Wikimedia installation... Revision: 83531:
17:24 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 83531:
17:23 RoanKattouw: Deploying category collation code changes (r83254, r83531) sitewide using scap
15:16 RoanKattouw: Updating MediaWiki on fenari from SVN to reintroduce category collation changes. For now, this is only for testing, so don't run sync-common-all or scap just yet
15:03 RoanKattouw: Looks fine
15:02 RoanKattouw: upgrade-1.17wmf1-final.php seems to have finished, running it again to verify it succeeded
13:28 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'NamespaceWithSubpages settings for Czech wikis, per Danny B'
12:44 mark: Changed 'wrr' scheduler to 'sh' on payments and owas LVS services
12:16 logmsgbot: midom synchronized php-1.17/extensions/FlaggedRevs/specialpages/PendingChanges_body.php 'syncing http://p.defau.lt/?9RQEZQiZ_WeMT7_f7TpX6A live too'
12:09 logmsgbot: midom synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'syncing livehack http://p.defau.lt/?BQwaKa6fH17X_CLt74khzA'
10:55 Tim: running upgrade-1.17wmf1-final.php to upgrade old masters
10:53 Tim: master positions: db36-bin.000001:106, db13-bin.000001:106, db31-bin.000001:106, db23-bin.001:79
10:53 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php
10:52 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php
10:52 Tim: switching master for s5: db14 -> db23
10:52 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php
10:52 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php
10:51 Tim: switching master for s4: db22 -> db31
10:51 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php
10:49 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php
10:48 Tim: switching master for s2: db30 -> db13
10:48 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php
10:48 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php
10:48 Tim: switching master for s1: db32 -> db36
10:08 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 'adding db33 back with snapshotting disabled - checking if that plus temptables were causing the melt'
08:53 logmsgbot: jeluf synchronized php-1.17/wmf-config/db.php 'removed db33 from s4'
08:51 JeLuF: db33 down, commons unavailable
07:31 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 're-enabled squid purging'
07:24 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh 'temp hack: no priority for htmlCacheUpdate'
07:22 Tim: it appears more likely that the overload was due to htmlCacheUpdate jobs updating page_touched, in a flood after jobs-loop.sh fixes
07:18 Tim: apaches overloaded after I fixed job queue squid purging. Disabled all squid purging and stopped job queue.
07:15 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php
06:47 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh
06:40 logmsgbot: tstarling synchronized php-1.17/includes/SquidUpdate.php
06:40 logmsgbot: tstarling synchronized php-1.17/includes/Title.php
06:39 logmsgbot: tstarling synchronized php-1.17/includes/DefaultSettings.php
06:06 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh
05:51 Tim: deployed new jobs-loop.sh script which prioritises fast important jobs like email notification. Thousands of emails are currently being sent.
05:23 Tim: apt-get upgrade on prototype.wikimedia.org
05:19 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'filemover $wgRemoveGroups'
05:05 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'added filemover group to enwiki, as on commons, per request on WP:VPR'
04:33 Tim: re-running upgrade-1.17wmf1-final.php to do db34, db21 and db18
04:31 Tim: new master positions: db39-bin.000001:106, db29-bin.001:79, db37-bin.000001:106
04:31 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php
04:30 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php
04:30 Tim: doing master switch s7: db18 -> db37
04:29 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php
04:29 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php
04:29 Tim: doing master switch s6: db21 -> db29
04:28 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php
04:25 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php
04:24 Tim: doing master switch: s3 db34 -> db39
02:05 Tim: apt-get upgrade on formey
01:47 Tim: apt-get upgrade on grosely and project2
01:35 Tim: apt-get upgrade on mchenry
01:22 Tim: apt-get upgrade on singer
00:53 Tim: manually upgraded python-memcache on several servers from 1.44-1wm1 to 1.44-1ubuntu1wm1~hardy1, which apt thought was a downgrade
00:22 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 'sending db33 back to duty'
00:20 RobH: filterxsshtml re-enabled on limesurvey, initial testing done

March 7

23:57 Tim: running apt-get upgrade on mediawiki-installation servers, for USN-1085-1
23:43 Tim: restarted upgrade-1.17wmf1-final.php, which had stopped due to user_last_timestamp existing on some wikis
23:42 Tim: updating schema change scripts to r83491 and restarting
23:30 RobH: disabled filterxsshtml in limesurvey install, per awjr
23:30 RobH: enabled filterxsshtml in limesurvey install
23:13 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryBase.php 'r83490'
21:45 logmsgbot: catrope ran sync-common-all
21:44 RoanKattouw: Running sync-common-all to deploy r83487
21:41 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27875 - Create a new usergroup Patroller with 'autopatrol' and 'patrol' right on simple.wikipedia'
21:24 logmsgbot: hashar: subversion working copy is now clean :)
21:22 logmsgbot: hashar: committing stuff in wmf-commit since the svn repository is available again
19:48 logmsgbot: midom synchronized php-1.17/wmf-config/db.php
12:34 hcatlin: Deploying mobile fix
11:02 Tim: removing some duplicate jobs from enwiki.job
10:18 Tim: running schema changes using upgrade-1.17wmf1-final.php
10:06 logmsgbot: root synchronizing Wikimedia installation... Revision: 83453:
10:06 Tim: running svn up/scap to get my latest schema change scripts
01:52 Tim: live-patched XSS vulnerability in mobile
00:20 logmsgbot: demon synchronized php-1.17/includes/media/Bitmap_ClientOnly.php 'r83417'
00:20 logmsgbot: demon synchronized php-1.17/includes/revisiondelete/RevisionDelete.php 'r83417'

March 6

18:54 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 27902 - Namespace fix on dawiki -- Help page talk too per IRC discussion'
18:46 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 27902 - Namespace fix on dawiki'
18:12 mark: Powered up sq35
17:43 richcole: swapping drive 7 on db15
15:02 hcatlin: Deployed to mobile again. Seems a little angry on ganglia.
14:28 hcatlin: Mobile deploy complete. Seems stable.

March 5

19:21 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 'Live hack to put mobile script back in <head>'
03:53 logmsgbot: andrew synchronized php-1.17/extensions/LiquidThreads/LiquidThreads.php 'Attempt to fix missing NSes on svwiktionary'
03:53 logmsgbot: andrew synchronized php-1.17/extensions/LiquidThreads/classes/Hooks.php 'Attempt to fix missing NSes on svwiktionary'

March 4

22:25 logmsgbot: hashar: Fixed outdated page by manually purging it (?purge + confirm). Issue resolved. Mark> you probably want to purge the squid/varnish caches for anything before 1.17 deployment :p
22:22 logmsgbot: hashar: page below is dated "Thu, 03 Feb 2011 12:18:28 GMT" HIT from amssq39.esams.
22:20 logmsgbot: hashar: got 404 errors on http://fr.wikipedia.org/wiki/H%C3%A9t%C3%A9rogramme : main-ltr.css points to 404 http://bits.wikimedia.org/skins-1.5/vector/main-ltr.css?283-18 , same with jquery.min.js http://bits.wikimedia.org/skins-1.5/common/jquery.min.js?283-18
00:12 logmsgbot: demon synchronized php-1.17/extensions/CodeReview/backend/CodeRevision.php 'r83186'

March 3

20:00 rainman-sr: modify lsearch-global-2.1.conf to point to an existing InitaliseSettings.php file (/home/wikipedia/common/php/wmf-config/InitialiseSettings.php). The file it was pointing before was removed a couple of days ago which made the incremental indexer broken.
19:23 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php
19:20 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js
19:20 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/redirector_test.js
19:19 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js
19:19 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/redirector_test.js
18:22 logmsgbot: catrope synchronized php-1.17/languages/classes/LanguageFi.php
18:22 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialProtectedpages.php
18:22 logmsgbot: catrope synchronized php-1.17/includes/DefaultSettings.php
18:21 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryAllpages.php
18:21 logmsgbot: catrope synchronized php-1.17/includes/filerepo/ForeignDBFile.php
18:21 logmsgbot: catrope synchronized php-1.17/includes/filerepo/LocalFile.php
18:21 logmsgbot: catrope synchronized php-1.17/includes/filerepo/File.php
18:21 logmsgbot: catrope synchronized php-1.17/includes/Title.php
18:19 RoanKattouw: Deploying r83154 with individual syncs
18:01 logmsgbot: catrope synchronized php-1.17/resources/jquery/jquery.placeholder.js 'r83152'
13:45 logmsgbot: midom synchronized php-1.17/wmf-config/db.php

March 2

22:50 Ryan_Lane: powercycling loudon, since we are locked out of the system via manual iptables rules, and the management console is non-responsive
21:01 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php
21:01 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js
20:59 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.i18n.php
20:58 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/redirector_test.js

March 1

23:49 Tim: removed "nomcomwiki" rows from the centralauth DB, they were causing fatal errors in Special:CentralAuth
22:07 logmsgbot: catrope synchronized php-1.17/includes/Wiki.php 'r83049'
22:07 logmsgbot: catrope synchronized php-1.17/resources/mediawiki/mediawiki.js 'r83051'
19:36 logmsgbot: catrope synchronized php-1.17/resources/mediawiki/mediawiki.js 'r83037'
13:45 logmsgbot: ariel synchronized php-1.17/languages/LanguageConverter.php 'make translate() public so zh abstract dumps run'
12:51 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php
12:29 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'stats for all wikis'
12:21 logmsgbot: tstarling synchronized php-1.17/includes/Setup.php 'removed more wfIncrStats calls'
12:14 logmsgbot: tstarling synchronized php-1.17/includes/filerepo/LocalFile.php 'disabling some unnecessary wfIncrStats() calls to reduce collector load'
06:41 logmsgbot: tstarling synchronized php-1.17/includes/GlobalFunctions.php 'wfIncrStats() aggregation hack'
06:35 logmsgbot: tstarling synchronized php-1.17/includes/Article.php 'profiling'
04:58 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'remove commented-out read only files'
04:22 Tim: moved liquidthreads.labs.wikimedia.org from all.dblist to deleted.dblist, so that its brokenness will offend people less
03:45 Ryan_Lane: changing the proxyfetch url to something non-version specific; should also repool all varnish servers
03:43 logmsgbot: tstarling synchronized docroot/bits/pybal-test-file
03:37 logmsgbot: tstarling ran sync-common-all
02:47 logmsgbot: root synchronized php-1.17/wmf-config/InitialiseSettings.php 'sync-file test'
02:41 logmsgbot: root synchronizing Wikimedia installation... Revision: 82948:
02:40 logmsgbot: root synchronizing Wikimedia installation... Revision: 82948:
02:39 Tim: running the new scap
02:24 Tim: updated the scap scripts in wikimedia-task-appserver, now racing puppet to update the package everywhere
02:02 Tim: updated scap and sync-file and related scripts in /home/wikipedia/bin to work with 1.17
01:22 ^demon: removed requirement for comment on bug resolution
00:56 Tim: fixed /home/wikipedia/bin/mysql-list, it makes a list of mysql servers
00:31 Tim: moving a lot of old scripts out of /home/wikipedia/bin to junk
00:19 Tim: fixed subversion versioning in php-1.17/wmf-config, checked in 1.17-related changes into the local subversion instance
00:10 logmsgbot: root ran sync-common-all
00:09 Tim: sync-common-all
00:07 Tim: except fonts
00:03 Tim: moved some old junk out of /home/wikipedia/common to /home/wikipedia/junk/common: nagios-fedora-plugins, fonts, lockfiles, proxies

February 28

23:53 Tim: tarred up the old MediaWiki directories in /home/wikipedia/common and moved them out to /home/wikipedia/lazy-backups/nostalgia
23:31 Tim: created job-runners group with a list of servers with job runners
23:26 Tim: cleaned up /etc/dsh/group somewhat, moved old files to the junk subdirectory
22:42 apergos: removed the wm-job-runner package on srv151 through srv190, so job runners don't accidentally get started back up on those hosts
21:59 apergos: stopped job runner on hosts that also serve as ext stores (srv151-srv190) to see if that deals with the "Too many connections" issue seen today for connections to srv160 and srv161
21:10 Ryan_Lane: repooling sq68
21:05 Ryan_Lane: pushing out load.php fix for domains like arbcom.nl.wikipedia.org
21:00 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryImageInfo.php 'r82960'
20:55 Ryan_Lane: adding another subdomain level for load.php in bits; depooling sq68 for testing
20:48 RobH: poking at blogs
19:05 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryAllUsers.php
19:05 logmsgbot: catrope synchronized php-1.17/includes/api/ApiBase.php
19:04 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryImageInfo.php
19:04 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryFilearchive.php
19:04 logmsgbot: catrope synchronized php-1.17/includes/Article.php
19:04 logmsgbot: catrope synchronized php-1.17/extensions/Nuke/Nuke_body.php
19:02 RoanKattouw: Deploying Reedy's recent merges (r82714, r82731, r82806, r82820, r82825) using individual file syncs
17:07 mark: Reinstalling lvs4
13:34 logmsgbot: tstarling synchronized php-1.17/includes/Setup.php 'profiling hack'
13:01 Tim: deployed wikimedia-job-runner 1.2
11:59 logmsgbot: ariel synchronized php-1.17/maintenance/dumpTextPass.php 'back off on XMLReader() due to performance issues'
11:42 mark: Power cycled sq70
11:38 mark: Disabled RP_filter on new LVS hosts with puppet
06:53 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php
06:51 logmsgbot: tstarling synchronized live-1.5/MWVersion.php 'disabling the multi-version hack, will caused test2wiki to go to 1.17'
06:39 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php
06:38 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php
06:33 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'more stats'
06:31 logmsgbot: tstarling synchronized php-1.17/includes/GlobalFunctions.php 'reverted 1.17 transition stats patch'
06:29 logmsgbot: tstarling synchronized php-1.17/StartProfiler.php 'profiling all wikis'
04:43 logmsgbot: tstarling synchronized php-1.17/wmf-config/wgConf.php 'fixing bug 27767 by sending commons requests to localhost'
01:45 Tim: added a "mediawiki" group to /svnroot/mediawiki/conf/authz, which allows for shell users to be omitted from MediaWiki r/w access
01:35 Tim: unsubscribed amcchord@dattobackup.com from wikitech-l

February 27

21:09 logmsgbot: demon synchronized php-1.17/includes/media/SVGMetadataExtractor.php 'r82912'
17:55 domas: added db25 to s3
17:54 logmsgbot: midom synchronized php-1.17/wmf-config/db.php
17:53 mark: NFS /home back up
17:48 mark: Removed (duplicate) service ip 10.0.5.8 from eth0 on nfs1; nfs2 is current master
17:44 richcole: moving lvs4
17:27 mark: Migrated text.pmtpa.wikimedia.org from lvs4 to lvs2
17:24 mark: Migrated bits.pmtpa.wikimedia.org from lvs4 to lvs2
17:19 mark: Migrated m.pmtpa.wikimedia.org LVS service from lvs4 to lvs2
16:51 mark: Reinstalled lvs6 with Lucid as well
16:39 richcole: moving nfs1
16:39 mark: Reinstalling lvs5 with Lucid
16:39 mark: Rich renamed transcode2 to lvs5 and transcode3 to lvs6
16:30 richcole: moving lvs2
15:57 richcole: moving transcode2
15:45 richcole: moving transcode3
11:38 logmsgbot: midom synchronized php-1.17/wmf-config/db.php

February 26

15:32 apergos: apt-get clean on srv219 and 222 to keep them from whining as much about disk space
09:09 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 27240 - Request for localized logo to be effected at tpi.wikipedia.org'

February 25

22:36 apergos: job queue reported slower than molasses, checked the job runners and there was one host running something. I did not restart the job runners though because the package has not been updated for 1.17 and was using the pre-1.17 deployment codebase. woops.
13:59 mark: Powered down transcode3 (preparing for move)
13:30 mark: Reinstalled lvs2 with Lucid, preparing as new LVS balancer
13:02 mark: Removed BGP session for lvs2 on csw5-pmtpa
12:36 mark: Setup csw1-sdtpa:11/20 for lvs2 (untagged vlan 100, tagged vlan 2)
12:12 mark: Powercycled amssq33, 35, 45
12:05 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add sidebar messages to $wgForceUIMsgAsContentMsg for mediawikiwiki'
11:50 logmsgbot: hashar: verified logos are live on kowikiquote and nnwikiquote. The later need a transparent background, user notified
11:47 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'logos for kowikiquote (bug 27548) and nnwikiquote (bug 27555)'
11:38 logmsgbot: hashar: running updateArticleCount on ttwiki for bug 27705
11:30 logmsgbot: hashar: srv169 & srv284 unreacheable. Maybe they should be removed. srv266 still in timeout :(
11:29 logmsgbot: hashar: checked that the three previous logos are correctly deployed
11:26 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'logos for knwiki (bug 27657), sawiki (bug 27661) and orwiki (bug 27704). Please note they are png thumbnails of the .svg images'

February 24

21:18 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27495 - adding user namespaces aliases in portuguese for ptwiki. Default is brazilian variant.'
19:34 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Another fix: apparently wikieditor-preview and wikieditor-previewDialog both exist, gotta hide both of them'
19:32 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Hide preview dialog preference, this feature is unstable. Was hidden before but got exposed because the preference key changed'
16:24 RoanKattouw: Restarted morebots as catrope. robh still has a zombie instance running that I can't kill

February 23

21:24 RoanKattouw: Adding skins-1.17 symlink to all docroots that also have skins-1.5
21:21 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads/classes/View.php 'Also stop LQT from including JUI CSS and add WikiEditor dialog CSS instead'
21:18 logmsgbot: catrope synchronized live-1.5/skins-1.17
21:17 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads/classes/View.php 'Fix LQT to not include its own JUI copy'
21:03 logmsgbot: catrope ran sync-common-all
21:02 RoanKattouw: Running sync-common-all to deploy r82700
17:08 mark: Migrated upload.pmtpa.wikimedia.org from lvs2 to lvs1
17:03 mark: Migrated owa.wikimedia.org from lvs4 to lvs1
16:57 mark: Migrated payments.wikimedia.org LVS to lvs1
16:49 mark: Reboot test of lvs1
16:49 logmsgbot: catrope synchronized php-1.17/includes/parser/Parser.php
16:48 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/ui/CodeCommentsListView.php
16:48 logmsgbot: catrope synchronized php-1.17/extensions/ProofreadPage/ProofreadPage_body.php
16:48 logmsgbot: catrope synchronized php-1.17/extensions/Collection/Collection.php
16:48 logmsgbot: catrope synchronized php-1.17/skins/common/wikibits.js
16:48 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesPt.php
16:47 RoanKattouw: Deploying r82678 with individual syncs
16:30 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Set account creation throttle to 300 for 194.64.239.* on elwiki. Requested by Ariel'
15:39 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable $wgHtml5 on all wikis'
15:26 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable $wgHtml5 on testwiki. If this goes well I will enable it on all wikis soon'
14:51 mark: Fixed puppet for lvs[234]
07:13 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'removed hacks to increase memory_limit to 100MB in certain cases'
07:12 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'increased memory_limit to 120MB'

February 22

02:33 logmsgbot: tstarling synchronized php-1.17/languages/LanguageConverter.php

February 21

21:43 logmsgbot: catrope ran sync-common-all
21:41 RoanKattouw: Running sync-common-all to deploy r82576.
14:21 logmsgbot: tstarling synchronized php-1.17/extensions/ProofreadPage/ProofreadPage_body.php

February 20

20:42 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesTt_cyrl.php 'r82535'
20:23 logmsgbot: catrope synchronized php-1.17/includes/OutputPage.php
20:18 logmsgbot: catrope ran sync-common-all
20:16 RoanKattouw: Deploying merges in r82533 with sync-common-all
18:31 RoanKattouw: Running clearMessageBlobs.php
17:34 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/modules/ext.wikiEditor.toolbar.js 'Touch this file to force module to regenerate'

February 19

17:23 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesMl.php 'Fix typo in category talk namespace name'
14:59 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 27495 - Change NS_USER and NS_USER_TALK namespaces at ptwiki'
14:38 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialListfiles.php
14:38 logmsgbot: catrope synchronized php-1.17/extensions/CentralAuth/SpecialCentralAuth.php
14:38 logmsgbot: catrope synchronized php-1.17/extensions/CentralAuth/ApiQueryGlobalUserInfo.php
14:38 logmsgbot: catrope synchronized php-1.17/extensions/skins/p2wiki/P2wiki.skin.php
14:38 logmsgbot: catrope synchronized php-1.17/extensions/ProofreadPage/ProofreadPage_body.php
14:38 logmsgbot: catrope synchronized php-1.17/extensions/FlaggedRevs/FlaggedArticleView.php
14:37 logmsgbot: catrope synchronized php-1.17/skins/vector/screen.css
14:37 RoanKattouw: Deploying updates in r82454 with individual syncs
13:59 RoanKattouw: Syncing php-1.17/cache/l10n/* to all Apaches manually using ddsh+rsync
12:47 logmsgbot: catrope synchronized php-1.17/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'r82450'
12:09 RoanKattouw: Running LocalisationUpdate with prospective fix patched in
11:47 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r82444'
11:46 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'r82444'
11:36 logmsgbot: hashar: running eval wfMsg( 'file-info-size') on all wiki for bug 27472
02:04 logmsgbot: LocalisationUpdate failed

February 18

21:08 RoanKattouw: Restarted import of Geograph images. Running in a screen on hume (as import.sh) with 30-min sleeps between batches
17:19 mark: Created mgmt.eqiad.wmnet and 65.10.in-addr.arpa DNS zones/ORIGINs
14:29 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'Bump filter cache version'
14:29 logmsgbot: catrope synchronized php-1.17/includes/libs/JavaScriptDistiller.php 'r82402'
05:55 logmsgbot: tstarling synchronized php-1.17/includes/resourceloader/ResourceLoader.php
03:48 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27367'
02:12 logmsgbot: LocalisationUpdate failed
00:35 Tim: fixed various cron jobs on hume
00:30 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh 'fix directory'
00:01 logmsgbot: tstarling synchronized php-1.17/wmf-config/secure.php 'wgStylePath fix'

February 17

23:57 logmsgbot: tstarling synchronized php-1.17/includes/DefaultSettings.php
23:54 logmsgbot: tstarling synchronized php-1.5/includes/DefaultSettings.php 'commenting out the $wgStylePath reference assignment, to try to fix the canary mismatch errors'
23:42 Tim: fixed logrotate on nfs2 and ran it with -f
23:16 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryPageProps.php 'r82363'
23:16 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialContributions.php 'r82363'
23:16 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialNewpages.php 'r82363'
23:08 logmsgbot: catrope synchronized php-1.17/includes/MessageCache.php 'r82362'
22:45 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/WikiEditor.hooks.php 'r82360'
22:38 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'Increment RL minifier cache version again'
22:37 logmsgbot: catrope synchronized php-1.17/includes/libs/JavaScriptDistiller.php 'r82358'
22:25 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'r82355. Will expire the entire ResourceLoader minifier cache'
22:08 logmsgbot: catrope synchronized php-1.17/includes/libs/JavaScriptDistiller.php 'Deploy minfier fixes'
21:37 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump style version appendix'
21:36 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads_alpha/jquery/plugins.js 'r82349'
21:35 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads/jquery/plugins.js 'r82349'
16:06 logmsgbot: mark synchronized php-1.5/../php-1.17/includes/media/SVGMetadataExtractor.php
15:55 mark: Disabled apparmor profile for apache2 on srv178, which was spamming the log
15:09 mark: Temporarily disabled wikidiff on srv151 (puppet will reenable soon)
14:53 mark: Cleaned up srv178
04:39 logmsgbot: demon synchronized php-1.17/includes/parser/Preprocessor_DOM.php 'Sync wfErrorLog removals, was filling up /home'
04:15 apergos1: killed udp2log on nfs2 and restarted it via init script after removing preprocessorfatal.log again, got back 160 gb, we had gone down to 6.5 gb free on /home on fenari
02:47 Tim: re-enabled snaprotate, bug 27471 does not involve data loss
02:22 logmsgbot: LocalisationUpdate failed
01:59 Tim: redirected /home/wikipedia/common/php symlink to php-1.17
01:36 Tim: disabled snaprotate.pl temporarily while I investigate bug 27471. This has to be reverted within a few days or else the snapshots will fill
00:04 Tim: rebooting spence, its load is too high to let me get a terminal

February 16

21:17 logmsgbot: catrope synchronized php-1.17/wmf-config/db.php 'Depool srv178 from ES'
21:09 Ryan_Lane|dc: on nfs2
21:09 Ryan_Lane|dc: killing udp2log and starting it via init script
21:08 Ryan_Lane|dc: restarting udp2log on nfs2
21:07 Ryan_Lane|dc: restarting rsyslog on nfs2
21:05 Ryan_Lane|dc: restarting rsyslog on fenari
21:00 RoanKattouw: Removing /h/w/logs/preprocessorfatal.log , was 146GB
20:59 logmsgbot: catrope synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Remove debug logging, 146 GB of the same error message over and over again is enough'
20:39 RobH: srv178 powered down, full disk, preventing syncs, needs to be powered backup and fixed later
20:12 hashar: Freenode seems down. I can connect using IPv6 though :)
- hashar: Most probably the IPv6 server had a netsplit.
20:10 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderWikiModule.php 'r82287'
20:07 RoanKattouw: Changed /home/wikipedia/bin/l10nupdate (L10Update cron job) to use php-1.17 instead of wmf-deployment
20:04 RoanKattouw: Fixed redirect loop on mlwiki a few minutes ago by changing the namespace names for NS_CATEGORY and NS_CATEGORY_TALK in MessagesMl.php
20:04 logmsgbot: catrope synchronized php-1.17/includes/Wiki.php 'Remove debugging code'
20:03 RoanKattouw: Restarted morebots as me on wikitech
17:35 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesKaa.php 'r82262'
17:35 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesCa.php 'r82262'
17:28 logmsgbot: catrope synchronized php-1.17/includes/Skin.php 'Fix stupid fatal'
17:21 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderStartUpModule.php 'r82260'
17:21 logmsgbot: catrope synchronized php-1.17/includes/Skin.php 'r82260'
16:36 logmsgbot: catrope synchronized php-1.17/languages/Language.php 'r82256'
16:36 logmsgbot: catrope synchronized php-1.17/includes/MessageCache.php 'r82255'
15:17 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Re-enable MWSuggest now that it can coexist with SimpleSearch'
15:07 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/modules/jquery.wikiEditor.iframe.js 'r82251'
15:06 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/modules/jquery.wikiEditor.js 'r82251'
14:49 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'Touch this file so changes to ResourceLoaderStartupModule.php take effect'
14:37 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderStartUpModule.php 'r82248'
14:34 logmsgbot: catrope synchronized php-1.17/includes/OutputPage.php 'r82248'
14:10 logmsgbot: catrope synchronized php-1.17/extensions/ProofreadPage/ProofreadPage_body.php 'Debugging'
14:01 logmsgbot: catrope synchronized php-1.17/extensions/CategoryTree/CategoryPageSubclass.php 'r82243'
13:58 mark: Added hyphen to character set in load.php matching regex on bits
13:52 logmsgbot: tstarling synchronized php-1.17/extensions/OpenSearchXml/ApiOpenSearchXml.php 'revert'
13:29 logmsgbot: tstarling synchronized php-1.17/extensions/OpenSearchXml/ApiOpenSearchXml.php 'debugging'
13:17 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Attempted fix for wikis with dashes in their name; they were still on 1.16'
13:15 logmsgbot: tstarling synchronized php-1.17/extensions/OpenSearchXml/ApiOpenSearchXml.php 'debugging'
13:12 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesKo.php 'r82237'
13:10 logmsgbot: tstarling synchronized php-1.17/extensions/OpenSearchXml/ApiOpenSearchXml.php 'debugging'
12:28 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'wgStyleSheetPath dereference'
12:25 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'wgStyleSheetPath dereference'
12:23 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'wgStyleSheetPath dereference'
12:21 RoanKattouw: Finished running fixUsabilityPrefs2.php a long time ago, forgot to log
12:04 logmsgbot: tstarling synchronized php-1.17/includes/DefaultSettings.php 'heap overflow part 2'
11:59 mark: Shutting down sq35 for bad drive
11:57 logmsgbot: demon synchronized php-1.17/extensions/CentralAuth/CentralAuthUser.php 'r82234'
11:55 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'possible fix for heap overflows'
11:54 RoanKattouw: Running fixUsabilityPrefs2.php again, now with wikieditor-% and usenavigabletoc fixes
11:52 mark: Restarted squid-frontend on sq35
11:48 RoanKattouw: Running maintenance/wmf/fixUsabilityPrefs2.php on all wikis
11:34 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'fix wfGetIP call'
11:30 Tim: all wikis -> 1.17
11:19 Tim: enwiki -> 1.17
10:59 mark: Restoring traffic to bits.esams
10:42 mark: Moving bits.esams traffic to bits.pmtpa
10:36 mark: Installing 2.6.37-12 kernel on knsq4, rebooting
10:14 mark: Deploying fix: User-agent sending in Varnish backend checks
10:11 Tim: enwiki -> 1.16
10:01 Tim: enwiki -> 1.17
09:18 Tim: whacking canary error moles
08:49 logmsgbot: catrope synchronized php-1.17/languages/classes/LanguageEo.php 'Fix cx->cxx problem on Esperanto wikis'
08:42 Tim: doubled APC cache size to 240MB
08:27 apergos: restarted apache on srv247, was seeing lots of segfaults
08:25 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 're-enable wmerrors'
08:23 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'disable wmerrors'
08:09 mark: Having puppet install screen and gdb on all machines (in base.pp)
07:56 logmsgbot: catrope synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.php 'Style version'
07:54 logmsgbot: catrope synchronized php-1.17/extensions/FlaggedRevs/client/flaggedrevs.css 'Fix for FR RTL issue'
07:43 logmsgbot: demon synchronized php-1.17/extensions/FlaggedRevs/specialpages/ProblemChanges_body.php 'Stop fatal'
07:40 logmsgbot: root ran sync-common-all
07:35 logmsgbot: demon synchronized php-1.17/includes/User.php 'r82222'
07:20 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'Turning NewUserMessage back on for 1.17 wikis'
07:01 Tim: frwiki -> 1.16
06:51 logmsgbot: catrope synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'Fix FlaggedRevs path bug'
06:48 Tim: zhwiki -> 1.17
06:46 Tim: enwiktionary -> 1.17
06:45 Tim: commonswiki -> 1.17
06:43 Tim: jawiki -> 1.17
06:38 Tim: dewiki, frwiki to 1.17
06:30 Tim: nlwiki to 1.17
06:23 Tim: ran sync-common on srv298
06:22 Tim: eowiki to 1.17
06:21 mark|coffee: Cleaned up space on srv219
06:12 Tim: running fixUsabilityPrefs.php
02:53 Tim: updated wmerrors in apt to version 1.1.2

February 15

22:03 RoanKattouw: Uncommented srv298 in mediawiki-installation node list. Comment says it's not installed, but it's pooled
20:59 logmsgbot: catrope synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.class.php 'Fix extension path in FlaggedRevs'
20:59 logmsgbot: catrope synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.php 'Fix extension path in FlaggedRevs'
18:58 mark: Singer has a bad HD /dev/sda, marked it as faulty in the RAID1 arrays
18:51 mark: powercycled singer
14:12 mark: powercycled amssq32
10:25 logmsgbot: tstarling ran sync-common-all
10:14 Tim: deploying latest version of 1.17wmf1 to the wikis that use it
08:53 Tim: added wmerrors requirement to puppet, with a configuration file that sends logs to UDP on fenari
08:38 Tim: set up temporary UDP log receiver for wmerrors on fenari, logging to /var/log/mw/fatal.log
07:01 Tim: added package php5-wmerrors to apt, and installed it on srv224
05:24 logmsgbot: aaron synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.php 'deployed r82159'

February 14

21:54 RoanKattouw: Running expensive queries on db39 to diagnose a database issue causing preferences bugs in 1.17
19:32 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'and codereview-associate too'
19:26 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Restrict CodeReview signoffs to coders'
17:55 ^demon: cleaned up bogus code_tags on mediawikiwiki containing commas from before we comma-delimited them
17:44 logmsgbot: demon synchronized php-1.17/extensions/CodeReview/ui/CodeTagListView.php 'Add some newlines between links'
16:57 logmsgbot: catrope synchronized php-1.5/wmf-config/ExtensionMessages.php 'This accidentally got 1.17-ified on Feb 2nd'
16:32 RoanKattouw: Running sync-common on srv298 because it was spewing errors about commented-out code
15:01 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/backend/CodeRevision.php 'Attempting to fix CR notif e-mail issue'
14:56 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/CodeReview.i18n.php
14:48 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Previous sync fixed mediawikiwiki for HTTP, was already working for HTTPS. Now fixed same bug for foundationwiki and oldwikisource so those will work once switched over'
14:46 logmsgbot: catrope synchronized live-1.5/MWVersion.php
14:45 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Attempted bugfix for mediawikiwiki'
14:41 RoanKattouw: Switching mediawikiwiki to 1.17
10:32 Tim: committed Mark's version of 1.16 CommonSettings.php to svn
10:14 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php
10:13 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php
10:11 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php
10:08 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Fix ptwikibooks (autothrottle broken)'

February 13

22:20 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'Turning NewUserMessage back off for 1.17 wikis'
22:04 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'Turning NewUserMessage back on for 1.17 wikis, with fixes'
21:59 logmsgbot: demon synchronized php-1.17/resources/Resources.php 'r82084'
00:06 logmsgbot: catrope synchronized php-1.17/resources/Resources.php 'r82044'

February 12

23:42 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'r82043'
23:38 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderFileModule.php 'r82041, the right file this time'
23:32 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderModule.php 'r82041'
22:02 logmsgbot: catrope synchronized live-1.5/resources 'Point this symlink to the 1.17 tree'
21:47 RoanKattouw: Fixed accidental svn1.6 contamination of php-1.17 by checking it out again with svn1.4, and copying modified, ignored and unversioned files. Kept the original tree around in /h/w/c/old-php-1.17-with-svn-1.6 in case something is missing, although diff -rq | grep -v .svn didn't think so
21:07 logmsgbot: demon synchronized php-1.17/extensions/NewUserMessage/NewUserMessage.class.php 'r82032'
21:07 logmsgbot: demon synchronized php-1.17/extensions/NewUserMessage/NewUserMessage.i18n.php 'r82032'
20:53 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Set $wgUseRCPatrol to true for test2wiki...one last time'
18:59 ^demon: scratch that, I should've waited for mark's fix to go live
18:58 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Set $wgUseRCPatrol to true for test2wiki...again'
17:38 Ryan_Lane: upgrading libvirt packages on nova-*.tesla for sheepdog support
08:47 domas: killed few more smssend processes...
00:43 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Set $wgUseRCPatrol to true for test2wiki'

February 11

22:49 Ryan_Lane: rebooting nova-controller.tesla
21:36 Ryan_Lane: changing nova-* packages to the bexar release version on nova-*.tesla
16:15 mark_: made Puppet catalog compiles fast
14:35 domas: disabled newusermessage
14:34 logmsgbot: midom synchronized php-1.17/wmf-config/CommonSettings.php
14:34 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php
12:20 logmsgbot: demon synchronized php-1.17/includes/ChangesList.php 'Fix warning, $classes should be an array for insertTags()'
12:18 logmsgbot: catrope synchronized php-1.17/extensions/ParserFunctions/ParserFunctions.i18n.magic.php 'Touching this file so the magic words fix takes effect'
12:12 logmsgbot: catrope synchronized php-1.17/includes/parser/Preprocessor_DOM.php
12:11 logmsgbot: tstarling synchronized php-1.5/includes/GlobalFunctions.php 'remove nlwiki and eowiki from 1.17 profiling'
12:10 logmsgbot: tstarling synchronized php-1.5/StartProfiler.php 'remove nlwiki and eowiki from 1.17 profiling'
12:09 logmsgbot: tstarling synchronized php-1.17/StartProfiler.php 'remove nlwiki and eowiki from 1.17 profiling'
12:00 logmsgbot: catrope synchronized php-1.17/extensions/CentralAuth/SpecialWikiSets.php 'r81956'
11:59 logmsgbot: catrope synchronized php-1.17/includes/LocalisationCache.php 'r81960'
11:06 logmsgbot: demon synchronized php-1.17/extensions/CentralAuth/SpecialWikiSets.php 'Fixing fatal without breaking SpecialWikiSets'
11:04 logmsgbot: demon synchronized php-1.17/extensions/CentralAuth/SpecialWikiSets.php 'Fixing fatal'
11:01 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Unbreak test on secure'
10:47 logmsgbot: catrope synchronized php-1.17/includes/parser/Preprocessor_DOM.php 'Trying to get a backtrace on preprocessor fatals'
10:28 logmsgbot: catrope synchronized php-1.17/languages/classes/LanguageEo.php 'r81954'
10:18 Tim: switched eowiki and nlwiki back to 1.16
10:13 Tim: switched the following wikis to 1.17: hewikisource eowiki enwikiquote enwikinews enwikibooks betawikiversity nlwiki hewikisource
10:11 logmsgbot: catrope synchronized php-1.17/resources/mediawiki.special/mediawiki.special.preferences.js 'r81953'
09:49 logmsgbot: catrope synchronized php-1.17/resources/jquery/jquery.client.js
09:48 logmsgbot: catrope synchronized php-1.17/resources/jquery/jquery.client.js 'Attempting to fix JS error'
09:45 mark: Deploying /etc/php5/conf.d/wikidiff2.ini using puppet
09:43 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 're-enable wikidiff2, use extension_loaded() to test for existence instead of file_get_contents()'
09:40 logmsgbot: catrope synchronized php-1.17/includes/EditPage.php 'r81947'
09:38 Tim: enabling wikidiff2 on all servers by manually creating wikidiff2.ini
09:29 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderUserModule.php 'r81944'
09:28 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderSiteModule.php 'r81944'
09:16 logmsgbot: catrope synchronized php-1.17/extensions/CentralAuth/SpecialGlobalGroupMembership.php 'Trying to fix GlobalUserRights->UserRights redirect'
09:01 logmsgbot: catrope synchronized php-1.17/extensions/PrefSwitch/PrefSwitch.hooks.php 'r81939'
09:01 logmsgbot: catrope synchronized php-1.17/extensions/PrefSwitch/PrefSwitch.php 'r81939'
08:55 logmsgbot: tstarling synchronized php-1.17/StartProfiler.php
08:54 logmsgbot: tstarling synchronized php-1.17/includes/GlobalFunctions.php
08:54 Tim: adding hewikisource to the relevant phase1 lists
08:54 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php
08:53 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Disable PrefSwitch links on 1.17 wikis'
08:50 Tim: upgraded metawiki and hewikisource to 1.17
08:48 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads/classes/View.php 'Attempting to fix JS error by loading in dialogs module'
08:41 logmsgbot: catrope synchronized php-1.17/wmf-config/liquidthreads.php 'Fix another broken LQT extensions path'
08:36 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads/LiquidThreads.php 'Fix LQT path to use $wgExtensionAssetsPath'
08:33 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'r81938'
08:22 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump style version appendix'
08:20 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads/lqt.js 'Testing fix for JS error'
07:58 RoanKattouw: Truncating module_deps on all wikis
07:48 Tim: switching strategy.wikimedia.org, simple.wikipedia.org and simple.wiktionary.org to 1.17
07:20 logmsgbot: catrope synchronized php-1.17/includes/MessageBlobStore.php 'r81933: Disable clear()'
07:19 logmsgbot: tstarling synchronized php-1.17/includes/MessageBlobStore.php
06:48 logmsgbot: catrope synchronized php-1.17/includes/MessageBlobStore.php 'Wrap clear() in a transaction to hopefully make stuff break less badly'
06:34 mark: Migrating srv248 and srv249 to bits appservers
06:24 logmsgbot: root ran sync-common-all
06:16 logmsgbot: tstarling synchronized php-1.17/extensions/CommunityVoice/Modules/Ratings.php
05:59 Tim: moving usabilitywiki to 1.17
05:19 logmsgbot: tstarling synchronized php-1.17/StartProfiler.php 'Synchronising StartProfiler.php with the 1.16wmf4 version'
05:14 logmsgbot: tstarling synchronized php-1.17/StartProfiler.php 'Synchronising StartProfiler.php with the 1.16wmf4 version'
05:10 logmsgbot: tstarling synchronized php-1.17/includes/GlobalFunctions.php 'pcache stats patch'
03:58 logmsgbot: ariel synchronized php-1.17/wmf-config/InitialiseSettings.php 'wgLogAutocreatedAccounts enabled for test2wiki'
02:22 logmsgbot: tstarling synchronized php-1.5/includes/GlobalFunctions.php '1.17 stats attempt 3'
02:18 logmsgbot: tstarling synchronized php-1.5/includes/GlobalFunctions.php '1.17 stats attempt 2'
02:14 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'stats for phase 1 deployment'
02:11 logmsgbot: tstarling synchronized php-1.5/includes/GlobalFunctions.php 'configurable profile ID for wfIncrStats()'
01:29 logmsgbot: tstarling synchronized php-1.5/StartProfiler.php 'set up profiling section for the first batch of wikis which will be moved to 1.17, for baseline'
00:48 logmsgbot: tstarling synchronized php-1.5/includes/parser/ParserCache.php 'remove age stats'
00:45 Tim: made http://noc.wikimedia.org/cgi-bin/report.py a redirect to http://noc.wikimedia.org/cgi-bin/ng/report.py

February 10

23:10 Ryan_Lane: rebooting nova-compute1 and nova-compute2
22:36 mark: bits.wikimeda.org on the backend is now only served by a special application server cluster (srv191, srv192), load balanced by Varnish
22:06 Ryan_Lane: upgrading qemu, kvm, qemu-kvm, qemu-common, purging kvm-pxe, installing libjpeg62 on nova-compute1
22:05 Ryan_Lane: installing libjpeg62 on nova-compute2, as it is a requirement for qemu-common
22:04 Ryan_Lane: upgrading those packages on nova-compute2
22:03 Ryan_Lane: upgrading qemu, kvm, qemu-kvm, qemu-common, purging kvm-pxe
19:58 mark: Depooled sq67 for testing
19:45 mark: Depooled srv192 as normal app server, converted to bits app server (in puppet)
19:41 Ryan_Lane: deploying a new squid config to all squids
19:35 mark: Depooled srv191 as normal app server, converted to bits app server (in puppet)
19:24 Ryan_Lane: restarting squid processes on sq34
19:09 Ryan_Lane: restarting gmetad on spence
19:07 Ryan_Lane: made a new ganglia group for api squids, made sq59 and sq60 aggregators for text squids, made sq31 and sq32 aggregators for api squid
19:07 Ryan_Lane: moved sq34 and sq35 to api squids
19:07 Ryan_Lane: depooling sq34, sq35
18:40 Ryan_Lane: adding sq34 and sq35 to api squids
18:35 Ryan_Lane: powering off sq32 - has bad disks
18:26 Ryan_Lane: powercycling sq32, it's dead, again
18:13 Ryan_Lane: powercycling sq32
16:41 Ryan_Lane: cleaning up logs on ekrem, and restarting apache
15:30 ^demon: made the 1.17.dblist available with the others on noc.wikimedia.org
12:22 logmsgbot: tstarling synchronized php-1.17/load.php
12:20 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'fix wgCacheDirectory'
11:57 logmsgbot: tstarling synchronized php-1.17/includes/resourceloader/ResourceLoaderFileModule.php 'profiling fix'
11:53 logmsgbot: tstarling synchronized php-1.17/includes/resourceloader/ResourceLoader.php
11:52 logmsgbot: tstarling synchronized php-1.17/extensions/Gadgets/Gadgets_body.php 'profiling fix'
11:46 logmsgbot: tstarling synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'profiling fix?'
11:41 logmsgbot: tstarling synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'profiling fix?'
07:09 logmsgbot: tstarling synchronized php-1.17/load.php
07:03 logmsgbot: root ran sync-common-all
06:56 logmsgbot: tstarling synchronized php-1.17/StartProfiler.php 'splitting out test2 with 1:1 sample rate for profiling testing'
04:34 Ryan_Lane: installing scsi-tools on nova-*.tesla
04:29 Ryan_Lane: adding 50GB of storage to each nova-*.tesla vm
04:05 Ryan_Lane: installing corosync on nova-*.tesla
04:04 Ryan_Lane: installing sheepdog on nova-*.tesla
03:35 RobH: mailing lists restarted
03:27 RobH: correction, halting mail LIST servers, not all mail servers.
03:27 RobH: halting mail servers to clean out an email from archives, will resume normal service shortly
02:53 Tim: restarting all apaches to get apc.ini change
02:31 Tim: deploying new version of php5-apc package to all apaches, to change a single line in /etc/php5/conf.d/apc.ini
00:12 logmsgbot: catrope synchronized php-1.17/StartProfiler.php 'Put 1.17 wikis in their own profiling category'
00:10 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Define ONE_SEVENTEEN for wikis running 1.17'

February 9

23:46 RoanKattouw: Het deploy is now working on secure
23:45 RoanKattouw: Syncing docroot
23:43 logmsgbot: catrope synchronized php-1.17/wmf-config/secure.php 'Fix $wgLoadScript on secure'
23:41 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Trying to make het deploy work on secure'
23:17 logmsgbot: tstarling synchronized live-1.5/apc-sma-info.php
23:16 logmsgbot: tstarling synchronized live-1.5/apc-sma-info.php
22:25 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Switch 1.17 stuff to a dblist'
22:18 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoader.php
22:18 logmsgbot: catrope synchronized php-1.17/includes/parser/ParserOptions.php
22:17 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads_alpha/classes/View.php
21:18 logmsgbot: catrope synchronized php-1.17/includes/db/Database.php
21:18 logmsgbot: catrope synchronized php-1.17/includes/MessageCache.php 'r81786'
21:14 logmsgbot: catrope synchronized php-1.17/extensions/Vector/modules/ext.vector.collapsibleNav.js
21:07 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Change paths to skins-1.17 and extensions-1.17'
21:00 logmsgbot: catrope synchronized live-1.5/extensions-1.17 'Sync extensions symlink too'
20:57 Ryan_Lane_: pushing out 1.17 aliases in wikimedia.conf file
20:19 logmsgbot: catrope synchronized live-1.5/skins-1.17 'Add skins-1.17 symlink'
20:07 logmsgbot: catrope synchronized php-1.17/LocalSettings.php 'Update path here too'
20:02 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Implement experimental hack to put 1.17 on test2wiki. Should not affect other wikis'
19:57 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Changes for 1.17 on test2wiki experiment'
19:53 mark: Reconfigured exim on mchenry and sanger to accept +whatever suffixes to e-mail address local parts
19:13 apergos: importing an old simplewiki dump to test2wiki, via importDump.php, running in screen on fenari. 35 000 pages, 200K revisions.
14:10 ^demon: granted myself +sysop +bcrat on test2wiki b/c Roan said I could
13:29 logmsgbot: catrope synchronized php-1.5/extensions/CentralAuth/CentralAuthUser.php 'Workaround for CentralAuth exceptions'
11:59 Tim: created wiki at test2.wikipedia.org
11:58 logmsgbot: tstarling ran sync-common-all
11:51 Tim: adding DNS entry for test2.wikipedia.org
11:31 mark: Dovecot on sanger restarted with doubled max memory size of 512 MB
02:46 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'test finished'
02:36 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'setting $wgCacheEpoch to test theory on 1.17wmf1 failure'
01:03 logmsgbot: tstarling synchronized php-1.5/includes/parser/ParserCache.php 'cache hit age instrumentation formatting'
01:01 logmsgbot: tstarling synchronized php-1.5/includes/parser/ParserCache.php 'cache hit age instrumentation'

February 8

22:26 logmsgbot: midom synchronized php-1.5/includes/GlobalFunctions.php 'msg profiling'
22:09 logmsgbot: midom synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling stats for enwiki'
22:00 logmsgbot: midom synchronized php-1.5/wmf-config/InitialiseSettings.php 'udpstats on testwiki'
19:16 RoanKattouw: Running maintenance/eval.php calling $wgMessageCache->clear(); on all wikis. This should fix the stuck messages issue
17:48 mark: Power cycled singer
17:23 logmsgbot: mark synchronized php-1.5/load.php
17:17 RoanKattouw: Servers going back to normal, second deployment attempt failed. Same symptoms as the first: pcache miss rate in the 40s
17:12 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 81334:
17:12 RoanKattouw: Rolling back 1.17 deployment again
17:04 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEo.php
16:56 logmsgbot: catrope synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Attempting to shut up fatals for good'
16:52 logmsgbot: catrope synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Another fatal workaround'
16:49 logmsgbot: catrope synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Workaround for preprocessor fatal'
16:45 logmsgbot: catrope synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Revert debug change'
16:44 logmsgbot: catrope synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Debugging'
16:41 logmsgbot: catrope synchronized php-1.5/wmf-config/secure.php 'Do not send load.php through bits on secure'
16:37 logmsgbot: catrope synchronized php-1.5/extensions/SiteMatrix/SiteMatrix.php 'Fix class SiteMatrix now found fatal'
16:32 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 81755:
16:32 RoanKattouw: Running scap
16:30 RoanKattouw: Switching cluster to 1.17
16:30 RoanKattouw: Switching /home to 1.17
16:29 logmsgbot: catrope ran sync-common-all
16:29 RoanKattouw: Running sync-common-all to push changes to php-1.17
16:28 RoanKattouw: Switching test back to 1.16 in preparation for sync-common-all
15:13 RoanKattouw: test is now running 1.17wmf1 again
14:42 RoanKattouw: Forgot to log that I put testwiki back on 1.16 about 30 mins ago
14:29 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesJa.php 'Testing to see if this will fix the message caching issue'
14:00 RoanKattouw: Moving directories on /home again, putting 1.17 on test so we can diagnose the cause of the overload
13:41 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 81334:
13:41 RoanKattouw: Running scap
13:40 RoanKattouw: 1.17-rollback done
13:39 RoanKattouw: Running 1.17-revert to move directories back cluster-wise
13:38 RoanKattouw: Moving directories back on /home
13:38 RoanKattouw: Rolling back 1.17 deployment
13:31 logmsgbot: catrope synchronized php-1.5/includes/parser/Parser.php 'r81738, fixes registerWatcher fatal'
13:30 logmsgbot: catrope synchronized php-1.5/extensions/NewUserMessage/NewUserMessage.class.php 'r81737, fixes fatal'
13:08 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 81732:
13:07 RoanKattouw: Running scap to make sure the move takes effect properly, caches are purged, etc.
13:07 RoanKattouw: Moving wmf-deployment to w-d.old and php-1.17 to wmf-deployment on all Apaches. Run 1.17-rollback to undo this
13:06 RoanKattouw: Moving wmf-deployment to w-d.old and php-1.17 to wmf-deployment on /home
13:05 logmsgbot: catrope ran sync-common-all
13:04 RoanKattouw: Rerunning sync-common-all
13:03 mark: Upgraded rsync to a version with higher tcp backlog on nfs2
13:01 RoanKattouw: sync-common-all failed for a few servers due to NFS problems. Mark is fixing it
12:58 logmsgbot: catrope ran sync-common-all
12:57 RoanKattouw: Moved old wmf-deployment/ and php-1.17/ back on /home , now syncing that out to the cluster
10:38 mark: Power cycled amssq37, 43, 49
10:33 mark: Power cycled amssq35
10:15 mark: Deploying special case for test.wikipedia.org (=> srv193) on bits.pmtpa (varnish)
10:12 logmsgbot: catrope synchronized live-1.5/load.php 'Revert failed hack'
10:01 logmsgbot: catrope synchronized live-1.5/load.php 'Temp hack to make load.php work on test'
09:28 logmsgbot: root ran sync-common-all
08:58 logmsgbot: root ran sync-common-all
08:41 mark: Unstuck SMS modem
07:50 Tim: restarted apache on srv174
07:22 mark: Started Nagios bot
06:15 ^demon: fixed svn permissions for wmf-deployment group

February 7

22:20 Ryan_Lane: upgrading nova-* on nova-*.tesla
20:02 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27195 - Grant anonymous users 'createpage' right and enable Extension'
19:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27209 - Request Extension of Quiz Addon to English Wikibooks'
17:18 logmsgbot: demon synchronized php-1.5/wmf-config/CommonSettings.php 'Removing proof-of-concept MWReleases extension'
17:18 logmsgbot: demon synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disabling proof-of-concept MWReleases extension'
13:18 Tim: running schema updates on all old masters. db29 already done.
13:17 Tim: new master positions: db22-bin.000001:106, db18-bin.000001:106, db34-bin.000001:106
13:16 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
13:16 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
13:15 Tim: doing master switch on s3 from db39 to db34
13:15 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
13:14 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
13:13 Tim: doing master switch on s7 from db37 to db18
13:13 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
13:13 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
13:12 Tim: doing master switch on s4 from db31 to db22
13:11 Tim: new positions: db21-bin.001:79, db32-bin.000001:106
13:07 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
13:06 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
13:06 Tim: doing master switch on s1 from db36 to db32
13:03 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
13:03 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
13:02 Tim: doing master switch of s6 from db29 to db21
08:02 Tim: running upgrade-1.17wmf1-initial.php again to upgrade db13 and db23
07:52 Tim: s5: New position: db14-bin.001:79
07:51 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
07:51 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
07:51 Tim: switching master on s5 from db23 to db14
07:41 Tim: s2: New position: db30-bin.000001:106
07:41 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
07:40 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
07:39 Tim: switching master on s2 from db13 to db30
04:35 apergos: running rebuild of the first of 4 history dump pieces on snapshot3, root, screen session

February 6

21:27 apergos: spliting up stubs into pieces in prep for resume of truncated history dumps fo en wiki; running on snapshot3 in screen session as root (see "part1,8,9,10.sh" in /backups-atg for what's running)
14:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'testwiki: Revert'
14:48 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'testwiki: Use unifont-5.1.20080907.ttf as default font for EasyTimeline'
14:43 logmsgbot: jeluf synchronized fonts/unifont-5.1.20080907.ttf 'GNU Unifont for timeline
13:57 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php '21497 CommonSettings.php'
13:38 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27191 - Re-point to local Wiki.png for kowiki'
13:35 logmsgbot: jeluf synchronized fonts/FreeSansWMF.ttf 'Merged FreeSerif into FreeSans (latest version). Provides Malayalam characters'
13:04 logmsgbot: jeluf synchronized fonts/FreeSansWMF.ttf 'Merged FreeSerif into FreeSans (latest version). Provides Malayalam characters'
13:02 logmsgbot: jeluf synchronized fonts/FreeSansWMF.ttf 'Merged FreeSerif into FreeSans (latest version). Provides Malayalam characters'
10:38 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'testwiki: Use FreeSansWMF as default font for EasyTimeline'
10:38 logmsgbot: jeluf synchronized fonts/FreeSansWMF.ttf 'Merged FreeSerif int FreeSans. *should* now provide Malayalam characters'
07:54 apergos: filled up root filesystem on snapshot3 by accident running an xml dump repair job, cleared out the file and restarted the job writing elsewhere, we whould be back to normal on that host now
06:49 apergos: four of the newly created en wikipedia history bz2 files are incomplete; they will be resumed after the 1.17 deployment has been pushed and is proven not to impact the dumps

February 5

22:35 logmsgbot: jeluf synchronized fonts/FreeSans.ttf 'New version from freefont-ttf-20100919'
22:33 logmsgbot: jeluf synchronized fonts/FreeSerif.ttf 'New version from freefont-ttf-20100919'
21:06 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'testwiki: Use FreeSerif as default font for EasyTimeline'
21:01 logmsgbot: jeluf synchronized fonts/FreeSerif.ttf '21497 - Change default font for EasyTimeline on ML projects to something that actually has glyphs for Malayalam characters'
20:04 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php '27172 CommonSettings.php'
20:04 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27172 - Increase $wgAccountCreationThrottle for Portuguese Wikibooks'
19:53 Ryan_Lane: adding the star certificate to lily, and changing lighttpd to use it
19:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26739 - wgLogo-variable points to wrong path'
12:44 logmsgbot: hashar: Marked bug 24078 as wontfix, wikis we have to request locallogo one by one
12:39 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'revering back to commons logo since some wiki have upload disabled (per IRC)'
12:20 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php '26739 - wgLogo-variable points to wrong path'

February 4

23:42 Ryan_Lane: disabling port forwarding on formey
22:38 Tim: resuming upgrade-1.17wmf1-initial.php ,stopped on usabilitywiki due to a schema idiosyncrasy there
22:22 Ryan_Lane: powercycling singer; it's non-responsive
17:04 Ryan_Lane: running apache-graceful-all
16:36 Ryan_Lane: pushing mod_expires fix to all apaches
16:33 Ryan_Lane: restarting apache on srv193 to force load of configuration
16:25 Ryan_Lane: testing mod_expires fix on test.wikipedia.org
04:49 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'adding a max lag so that servers executing schema changes will automatically be skipped'
04:43 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
04:41 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php
04:37 Tim: running schema upgrades in a screen on fenari using upgrade-1.17wmf1-initial.php

February 3

23:25 logmsgbot: tfinc synchronized php-1.5/extensions/WikimediaMobile/MobileRedirect.js
23:00 Tim: fixed refresh-dblist to use db.php instead of clusters.dblist for cluster assignments, moved clusters.dblist to junk
21:37 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25438 - Enable FlaggedRevs for Belarussian Wikipedia'
21:35 logmsgbot: jeluf synchronized flaggedrevs.dblist '25438 - Enable FlaggedRevs for Belarussian Wikipedia'
21:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '25438 - Enable FlaggedRevs for Belarussian Wikipedia'
20:48 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26985 - New namespace alias "WP'
20:41 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26817 - Create "Subject" namespace for Bengali Wikibooks'
12:08 RoanKattouw: Had to svn switch --relocate the tree from svn+ssh:// to http:// in order to be able to do so without knowing Priyanka's SSH key passphrase
12:07 RoanKattouw: Updated /srv/org/wikimedia/prototype/wikis/rc and its extensions on prototype

February 2

21:01 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'dpl nlwiki reverted to false'
20:55 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 8240 and 8563'
20:52 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 7952 dpl eswiktionary iswiktionary'
20:47 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 6163 DPL on nlwiki'
20:27 logmsgbot: pdhanda synchronized php-1.5/wmf-config/InitialiseSettings.php 'updated per bug 26062 - enabled abusefilter on ptwikisource'
19:31 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27069 - Define $wgUploadNavigationUrl for ptwikibooks'
18:55 Ryan_Lane: adding mlqt.tesla.usability.wikimedia.org to DNS
18:02 logmsgbot: hashar: srv169 srv266 and srv284 do not answer to ssh connection triggering error messages on sync :(
18:01 logmsgbot: hashar synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache for bug 26376'
18:01 logmsgbot: hashar synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache for bug 26376'
15:49 Ryan_Lane: running flush hosts on db9
14:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable subpages in API namespace on mediawikiwiki'
14:29 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set $wgSitename to Wikcionario on eswiktionary per IRC discussion and existing use'
13:54 mark: Unblocked some remoteLoaders in squid conf
00:48 logmsgbot: tstarling synchronized docroot/wikibooks.org/googled73abdc60a3b46a4.html
00:48 logmsgbot: tstarling synchronized docroot/wikipedia.org/googled73abdc60a3b46a4.html 'authorizing de.wikipedia.org for google webmaster tools'
00:46 logmsgbot: tstarling synchronized docroot/wikipedia.org/googled73abdc60a3b46a4.html 'authorizing de.wikipedia.org for google webmaster tools'
00:42 Tim: reset password for noc@wikimedia.org google web account

February 01

23:13 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27080 - Create an API'
23:10 Ryan_Lane: installing git-core on nova-controller.tesla
23:10 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26320 - Set project namespace of Portuguese Wikipedia to "Wikip??dia"'
22:57 logmsgbot: tstarling synchronized php-1.5/languages/Language.php
22:44 logmsgbot: tstarling synchronized php-1.5/languages/Language.php
22:41 logmsgbot: root synchronizing Wikimedia installation... Revision: 81334: r81334
21:05 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'Take live hack back out. Cause found, it's FlaggedRevs'
21:04 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'Live hack for debugging of noindex,nofollow issue'
02:26 river: subscribed all public mailing lists to the newsgate.tcx.org.uk NNTP gateway via Mailman add_member

January 31

21:37 logmsgbot: tstarling ran sync-common-all 'FlaggedRevs on trwiki, bug 26943'
19:40 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php '26366 - Thai Wikiquote project name & logo'
19:07 Ryan_Lane: purging bin logs to mysql-bin.001858 on db9
18:33 RobH: updated puppet to move user pdhanda into mortals group, able to push apache config changes
18:18 Ryan_Lane: uninstalling php-openid from nova-controller.tesla
17:14 Ryan_Lane: installing php-openid on nova-controller.tesla
09:06 tomaszf: installed php5-dev and PECL hash on grosley for david
09:06 tomaszf: installed php-mhash on grosley

January 30

17:32 Ryan_Lane: changing otrs to use the star cert
17:18 Ryan_Lane: adding star cert to williams
14:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26363 - Please Add Patroller User Group to fawikinews'

January 29

20:31 apergos: one more time for en wiki history dumps (looks like they might actually work this time). in screen on snapshot3, from /backups-atg.
08:46 apergos: starting enwiki history dumps (finally) with latest guesses at page ranges so we get more even no. of revisions per chunk... in screen on snapshot3, out of /backups-atg
08:11 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26988 - Changing the logo for Buginese Wikipedia'
08:00 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27003 - enable wgEnotifWatchlist for frrwiki'
07:47 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27015 - Change sitename for gdwiki'
07:40 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26451 - Install ReaderFeedback extension on ptwikibooks'
07:21 logmsgbot: ariel synchronized wmf-deployment/maintenance/dumpTextPass.php 'multiple prefetch files for xml dumps'
07:20 logmsgbot: ariel synchronized wmf-deployment/maintenance/backupPrefetch.inc 'multiple prefetch files for xml dumps'

January 28

21:52 Ryan_Lane: changing out bugzilla's certficiate and key
21:10 Ryan_Lane: adding mortals to emery, and adding sudoers.d entry for nimish
19:58 logmsgbot: catrope synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php 'r81158, r81159'
19:57 logmsgbot: catrope synchronized php-1.5/extensions/ProofreadPage/proofread.js 'r81156'
19:05 logmsgbot: catrope synchronized live-1.5/load.php 'Apparently this needs to be a wrapper'
18:57 logmsgbot: catrope synchronized live-1.5/load.php 'Add load.php entry point here as well, hopefully this will work'
18:50 logmsgbot: catrope synchronized php-1.5/load.php 'load.php rewrite rule testing'
18:23 Ryan_Lane: added url rewriting for load.php to bits
18:23 Ryan_Lane: repooling sq67
17:01 Ryan_Lane: depooling sq67
15:26 rainman-sr: searchidx1 is again out of disc space, freeing some space by deleting index snapshots etc, but we need more disc space soon!
13:05 ^demon: tweaked "no commit summary" hook on formey to be informative

January 27

22:46 Ryan_Lane: setting carp weight of sq63 to 30
22:43 Ryan_Lane: starting squid and puppet services on sq63
21:58 richcole: switching network cable out sq63
07:16 Ryan_Lane: stopping squid and disabling puppet on sq63
07:08 Ryan_Lane: changing carp weight of sq63 to 0
06:35 Ryan_Lane: restarting squid processes on sq63
02:55 Ryan_Lane: restarting pdns on ns2, it was hung
00:56 Ryan_Lane: adding mdale to engineering alias

January 26

21:38 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix typo'
18:44 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26915 - Modify $wgNamespaceRobotPolicies to noindex User and User talk namespaces on the French Wikipedia (fr.wikipedia.org)'
18:40 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26897 - Enable flag 'flood' in Spanish Wikinews (es.wikinews)'
18:37 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26946 - Create new group on Russian Wikipedia'
17:50 RobH: updated dns for harmon
17:23 mark: Moved ersch, alsted and harmon into the sandbox vlan
17:23 mark: Replaced ganglia aggregators for MySQL group
14:54 RobH: emery shutdown, needs to have drives swapped back to default
00:51 tomaszf: reset Renflauf pass on wikitech
00:13 richcole: thistle, ixia,lomaria going down for decommissioning a

January 25

22:55 richcole: db1-4 going down for decommission and wipe
22:20 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26889 - Set WB namespace alias to NS_PROJECT in Bengali wikibook'
22:14 RobH: project2 moved and back online
22:12 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26916 - Enable subpages in Template namespace on wikimania (and for future editions)'
22:08 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26913 - Please enable patrolled edits on the Hebrew Wikiquote (he.wikiquote.org)'
22:01 RobH: project2 (bugzilla testing) coming down for migration
21:39 logmsgbot: jeluf synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache for new wikitech entry'
21:22 Ryan_Lane: deleting review.tesla.usability.wikimedia.org, and removing from DNS
21:17 Ryan_Lane: removing public IP from analytics.tesla.usability.wikimedia.org and removing it from dns
20:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26932 - Add 'suppressredirect' right to 'eliminator' group on Portuguese Wikipedia'
18:02 RobH: srv281 repair working being done
16:38 Ryan_Lane: adding exim packages to nova-controller.tesla for mail sending from MediaWiki

January 24

23:53 Ryan_Lane: upgrading nova-* packages on nova-*.tesla
23:22 Ryan_Lane: rebooting nova-compute1/2
21:20 Ryan_Lane: rebooting nova-controller.tesla
18:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Kill another Nogomatch remnant, commented-out experiment from 2005'
18:49 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove Nogomatch, experimental since 2005 and doesn't seem to log anywhere'
18:04 RobH: sq86 still above normal ms4 connection thresholds, leaving it at increased rate for now
17:24 mark: Applied ipv4 route-map to ipv6 AS 6908 BGP session on br1-knams, so IPv6 routes via 6908 2828 are preferred, and IPv6 connectivity between pmtpa and esams is working again
17:11 RobH: sq86 is back online with lucid, set ms4 maxcon to 800 while it catches up
16:59 mark: Rebooting spence
16:27 mark: AMS-IX migration complete, all BGP sessions back up again
16:17 mark: Shutting down all AMS-IX sessions for SFP+ ER test migration
16:01 RobH: sq86 coming down for upgrade to lucid
15:27 mark: Shutdown project1 for decommissioning
13:33 logmsgbot: demon synchronized php-1.5/wmf-config/CommonSettings.php 'Reverting codereview-proxy removal, did not work'
13:31 logmsgbot: demon synchronized php-1.5/wmf-config/CommonSettings.php 'Removing codereview-proxy step from CR'
13:05 mark: Purged payments1/payments2 resources from the exported Puppet database on db9
12:57 mark: Restarted ntpd on srv236
12:55 mark: Added puppet classes ntp::client and ganglia to snapshot1-3
12:50 mark: Unstuck APT on knsq29, and fixed puppet
12:47 mark: Fixed puppet on Kaulen
12:46 mark: Removed APT pin -10 for Wikimedia repository on Kaulen - this has been fixed
12:44 mark: Unstuck APT on knsq30, and fixed puppet
12:38 mark: Finished upgrade of Kaulen
12:28 mark: Unstuck APT on knsq8, and fixed puppet
12:12 mark: Starting distribution upgrade on Kaulen
12:03 mark: Patched Bugzilla on Kaulen

January 23

15:19 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php '26886 - Move toggle for DoubleWiki to InitialiseSettings.php'
15:18 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26886 - Move toggle for DoubleWiki to InitialiseSettings.php'
14:41 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix logo of ruewiki'
14:34 JeLuF: created Rusyn Wikipedia, ruewiki
14:14 logmsgbot: jeluf ran sync-common-all '25924 - Create Wikipedia in Rusyn'
13:51 JeLuF: addwiki.php broken. SQL error: "1283: Column 'si_title' cannot be part of FULLTEXT index (10.0.6.49)"
11:58 logmsgbot: hashar: I did add eowikisource yesterday. Will write a report to wikitech-l asap
11:48 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php '18479 - Set $wgUseDynamicDates = false for the English Wikipedia'
01:28 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26323 - Requestin ReaderFeedback be enabled on ru.wikinews'

January 22

21:39 RobH: set sq83, sq84, and sq85 ms4 maxconn back to 100, all were well below and no longer needed increased max.
19:42 apergos: removing eowikisource, which was added to this file today but apparently doesn't exist; this breaks SUL-related things
19:42 logmsgbot: ariel synchronized all.dblist
18:29 logmsgbot: hashar synchronized php-1.5/wmf-config/wgConf.php 'indentation'
16:59 logmsgbot: 26853 - Local upload disabled at Swedish Wikisource
16:59 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php
14:19 hashar: ran rsync manually on srv287 after receiving this message from scap "rsync error: some files could not be transferred (code 23) at main.c(1385) [generator=2.6.9]"
14:10 logmsgbot: hashar ran sync-common-all 'Bug 16878 - Disable Special:CrossNamespaceLinks'
12:07 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 11776 - Showing also articles in the CategoryTree of categories in he-wiki'
11:49 logmsgbot: hashar synchronized langlist 'get ride of closed-zh-tw'
11:38 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26142 - really enable abusefilter'
11:20 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26584 - Add Portal namespace to Tarandine Wikipedia'
01:03 Ryan_Lane: powercycling storage3 - it's dead

January 21

23:11 logmsgbot: catrope synchronized php-1.5/extensions/CentralAuth/CentralAuthHooks.php 'Back out live hack, may be causing issues'
22:33 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26453 - Enable Collection extension on Romansh (rm) Wikipedia'
22:17 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26812 - Create namespace end namespaceAliases for Chechen (ce) Wikipedia'
22:11 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26855 - Create new Book namespace for gl.wikipedia'
22:02 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25740 - Enable Abuse Filter in gl.wp'
21:57 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26776 - Enable $wgUseRCPatrol for siwiki (si.wikipedia.org)'
21:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26777 - Enable AbuseFilter in si.wiki'
21:49 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 80711:
21:48 RoanKattouw: Running scap to deploy r80711 (i18n updates)
20:03 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26817 - Create "Subject" namespace for Bengali Wikibooks'
19:57 RobH: upload squids re-normalizing after sq85 resinstallation. sq83/84 set to 400 maxconns, sq85 set to 800, will change back when normalized
19:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26779 - Please create "Wikijunior" namespace for Bengali wikibook'
19:46 RobH: sq85 back online, frontend false, backend rebuilding, set to 800 maxconnection during rebuild
19:26 RobH: raised maxconn back to 400 on sq83, sq84, as they started to max out too often
19:18 RobH: upload services times back to normal, all upload squids online. taking down sq85 for upgrade to lucid.
19:06 RobH: reverting max_connections for sq83 & sq84 from 800 back to 100 as they have normalized well below the limit.
17:32 RobH: sq83/84 having empty caches on reinstall are maxing out ms4 connections for themselves, raising limits to 800 for them to catch up
16:53 RobH: sq85 coming down for lucid reinstallation
16:46 RobH: sq83 back in rotation
16:01 RobH: sq84 coming down for reinstallation to lucid
15:47 RobH: sq83 coming down for reinstallation to lucid
15:47 RobH: morebots died, restarted it on wikitech under my user
01:22 awjr: re-enabling automatic donation queue consumption in Hudson for queue2civicrm and recurring donation processing modules

January 20

23:42 logmsgbot: catrope synchronized php-1.5/wmf-config/ExtensionMessages.php 'Regenerated ExtensionMessages.php after removing unused extensions from extension-list and alphabetizing'
23:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove $wmgUseGeoLite'
23:36 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove GeoLite, enabled on test and meta but not actually used'
23:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove $wmgUseHoneypotIntegration'
23:18 logmsgbot: catrope synchronized php-1.5/wmf-config/secret-projects.php 'Remove HoneypotIntegration code'
23:18 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove HoneypotIntegration code, disabled everywhere'
23:16 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove commented-out code for ExternalPages, FixedImage and MiniDonation'
23:12 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove $wmgUseDrafts'
23:12 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove Drafts code; disabled on all wikis, breaks LiquidThreads'
23:10 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove $wmgUseConfigure'
23:10 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove Configure code, was disabled two years ago'
22:12 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Disable AjaxTest, it's ancient and useless'
21:46 logmsgbot: demon synchronized php-1.5/wmf-config/InitialiseSettings.php 'And let bcrats grant membership'
21:46 logmsgbot: demon synchronized php-1.5/wmf-config/CommonSettings.php 'Adding new svnadmins group for repository administration.'
21:42 awjr: deploying and enabling 'recurring' module on CiviCRM
21:41 awjr: temporarily setting the 'Days before considering transaction too old to automatically thank' setting to 50 (from 14) for thank_you module in CiviCRM in order to automatically send thank-you notes for old recurring payment messages that have not yet been consumed
21:40 awjr: temporarily suspending automatic donation queue consumption (done by queue2civicrm module) in Hudson to live-test the new recurring payment handling module
20:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/mc.php 'replace failed server'
19:14 Ryan_Lane: Patch python-nova on nova-controller.tesla for lp705195, lp705205, and lp705223
19:01 RobH: sq81 takes change, still running everything correctly, and updates disk permissions correctly now
19:00 RobH: committed puppet updates for udev rules on squids to templates, testing on sq81
18:38 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26314 - Creating new namespace in ca.wikinews'
17:02 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26252 - Enable autopatrolled and patroller groups on it.wikibooks'
16:16 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26615 - Set $wgBlockAllowsUTEdit to true for mediawikiwiki'
16:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26598 - Change wgNoticeProject on specieswiki'
16:04 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26545 - Enable NewUserMessage extension on Meta-Wiki'
16:02 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26532 - Enable Collection extension on Estonian (et) Wikipedia.'
15:29 JeLuF: Added vhost stats.wikipedia.org on spence. Provides a redirect to stats.wikimedia.org
15:03 JeLuF: added DNS alias stats.wikipedia.org -> stats.wikimedia.org
11:30 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '16410 - Enable e-mail notification for watchlist and minor edits on portuguese Wikibooks'
10:47 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26193 - Enable e-mail notification for watchlists on pt.wikiversity'
10:45 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26771 - Enable e-mail notification for movement roles wiki'
10:36 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26769 - Enable subpages on movement roles wiki'
10:28 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26783 - $wgEnotifWatchlist for WMPL wiki'
10:18 logmsgbot: jeluf synchronized docroot/commons/favicon.ico '#23917 - Commons favicon broken on Safari & Chrome (Mac) // Also distorted on other browsers'
04:05 Tim: set up IMAP account for Mark Hershberger
03:05 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable Drafts, causes issues in combination with LQT'
01:56 RoanKattouw: Restarted morebots
01:52 awjrichards: renamed fundcore_log table to queue2civicrm_log on drupal and dev_drupal databases on db9 in preparation for re-implementing fundcore module's logging feature in its replacement module, queue2civicrm
01:32 kaldari: deployed update to CentralAuth with new getCookieDomain method
01:22 logmsgbot: tfinc synchronized php-1.5/extensions/CentralAuth/CentralAuthUser.php
00:37 robla: added commit access for appy (Apekshit Sharma)
00:19 Tim: removed pilot.wikimedia.org, was an old insecure MediaWiki instance

January 19

23:26 Ryan_Lane: added agarrett to engineering alias
21:45 Ryan_Lane: upgrading nova-* packages on nova-*.tesla
21:41 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 26787 - English Wikisource: Have Template: namespace converted to be subpages'
20:47 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 26126 - New "Collection" namespace for en.wikiversity.org'
20:45 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'reorder wgExtraNamespace + folding'
20:25 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 26565 - Please enable upload on Persian Wikinews'
20:15 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 26572 - Enable subpages in main namespace on svwikiversity'
20:09 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'put db34 back'
19:44 hashar: frwiktionary WT was a namespace alias for NS_CATEGORY from 19:01 to 19:03 roughly. Now NS_PROJECT. Some page might show unwanted categories, issue can be solved by purging the page
19:03 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 26761 - was NS_PROJECT not 14!'
19:01 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 26761 - New namespace alias WT for project namespace on frwiktionary'
18:19 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'sync comments enabling vim folding (no code change)'
18:03 hcatlin: Mobile deploy complete and stable
18:02 RoanKattouw: Ganglia's fine
18:00 hcatlin: Ganglia seems to be down for me. "fsockopen error: Connection refused"
17:57 hcatlin: deploying mobile updates
17:44 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'remove db34 to dump s3'
17:38 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'put db16 back'
14:29 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'remove db16 to dump s7'
01:24 Ryan_Lane: fixing torrus

January 18

23:58 Ryan_Lane: upping CARP weight of sq79 to 30
23:50 Ryan_Lane: those last two sq49 logs were really sq79
23:50 Ryan_Lane: upping CARP weight for sq80-82 to 30, upping sq49 to 20
23:45 Ryan_Lane: upping CARP weight for sq80-82 to 20, upping sq49 to 15
23:28 Ryan_Lane: upping CARP weight for sq80-82 to 15
23:27 richcole: db 15 reboot
23:10 RobH: db25 & db27 shutdown
23:07 Ryan_Lane: upping CARP weight on sq79-82 to 10
22:40 mark: Increased max-conns to ms4 on sq80 from 100 to 400
21:55 Hashar: that one was adding code folding
21:55 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php
21:24 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php
20:36 RobH: sq79-sq82 reinstalled lucid, returned to service
19:52 RobH: sq79-sq82 down for reinstallation to lucid
19:52 RobH: updated puppet for udev rules on squid aufs exec.
19:52 RobH: had to shoot morebots and restart
11:23 mark: Changed subnet size, reenabled AMS-IX BGP sessions
11:21 mark: Shutdown all AMS-IX BGP sessions
11:19 mark: Starting AMS-IX interface subnet size change
06:29 apergos: restarted morebots, it was having issues

January 17

21:20 hashar: morebots is dead. Logged bug 26782
02:47 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
02:32 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
01:35 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'removing db16 to be my pillow'

January 16

20:31 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php
19:59 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php
19:57 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php '26633 - Give commonswiki group 'Image-reviewer' right to Add groups: Image-reviewer'
18:49 logmsgbot: hashar synchronized php-1.5/wmf-config/codereview.php 'codereview: fix repo_name case'
16:30 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'frwikibooks, allow bureaucrats to add/remove users from abusefilter'
16:30 logmsgbot: hashar synchronized php-1.5/wmf-config/abusefilter.php
16:16 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 25019 - trwikinews, create Yorum namespace'
15:34 logmsgbot: hashar synchronized php-1.5/wmf-config/CommonSettings.php 'Bug 23494 - Setup $wgCodeReviewDeferredPaths for Mediawiki.org'
15:32 logmsgbot: hashar synchronized php-1.5/wmf-config/codereview.php 'add file for mediawikiwiki $wgCodeReviewDeferredPaths'
12:54 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php
11:40 logmsgbot: hashar synchronized closed.dblist 'bug 26569 - Close usability.wikimedia.org to editing'
11:11 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php '26643 - Allow sysops to add/remove a bunch of groups on siwiki'
10:15 Tim-away: removed /tmp/gs_* on srv219-224, root partition was full

January 15

01:53 apergos: starting en wikipedia xml dump, one phase at a time. from screen session on snapshot3, this will be a parallelized run

January 14

23:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set $wgLogo to $stdlogo on cswiki'
23:15 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set enwiki logo back to $stdlogo so local sysops can set up a special logo for WP day'
20:05 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26634 - allow bureaucrats to add/remove for group abusefilter'
15:03 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php '26634 - Activation of AbuseFilter on Romanian language Wikipedia'
15:02 logmsgbot: hashar synchronized php-1.5/wmf-config/abusefilter.php '26634 - Activation of AbuseFilter on Romanian language Wikipedia'
14:55 RobH: rebooting dataset1, it was left in webbios and thus not controllable by serial
14:24 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php '26361 - wgLocaltimezone for koquote'
06:53 Tim: on all squids, turning off debug output on 44,1, which was filling up cache.log with "Failed to select source" warnings when the backend was down
06:42 Tim: backend squid on sq41-48 all crashed at once, because their root partitions filled up some time ago, and then something HUP'd them all at once. Deleted cache.log and restarted squid.
03:45 Tim: commented out the $wgCentralAuthCookieDomain = '' in CentralNotice.php
03:45 logmsgbot: tstarling synchronized php-1.5/extensions/CentralNotice/CentralNotice.php
03:33 mark_: db37 binlog: master_log_file='db37-bin.000028', master_log_pos=927356414;
03:29 mark_: db37 start pos was: master_log_pos=927356414;
03:29 mark_: domas: change master to master_host='db39', master_user='repl', master_log_file='db39-bin.000050', master_log_pos=720582706;
03:26 mark_: domas switched s3 master to db39 (position?), and s7 master to db37 (pos?)
03:24 mark_: domas depooled db25 from s3 and s7
03:18 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
03:18 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 's3 is now two-server 5.1 cluster'
03:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php
02:08 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'read-only on s3+s7'
01:55 Ryan_Lane: stopping mysql on db17

January 13

22:35 RobH: sq50 done with testing, bringing it back into service
22:26 richcole: shutting down dataset1 for recabling
20:31 RobH: pulling sq50 out of rotations, killing puppet and squid processes, and using as my testing for new udev rules on squids
20:05 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/backend/RepoStats.php 'r80187'
20:05 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/CodeReview.php 'r80187'
16:41 logmsgbot: catrope synchronized php-1.5/extensions/CentralAuth/CentralAuthHooks.php 'Live hack that tries to fix login issues'
12:57 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26697 - Set $wgForceUIMsgAsContentMsg for tenwiki'
05:36 Ryan_Lane: repooling sq49 and sq50
04:25 Ryan_Lane: depooling sq50 as well
04:09 Ryan_Lane: starting squid on sq49
04:08 Ryan_Lane: stopping squid on sq49
04:06 Ryan_Lane: stopping rsyslog, cleaning logs, and reenabling syslog in sq49
04:02 Ryan_Lane: depooling sq49
01:47 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialHideBanners.php
01:31 RobH: sq51-sq54 back online
01:29 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding AllowdHideBanenrs for test wiki'
01:24 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding AllowdHideBanenrs for foundation wiki'
01:23 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding AllowdHideBanenrs for foundation wiki'
01:20 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialHideBanners.php
01:20 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php
01:19 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'swapping out down servers'
01:13 Ryan_Lane: fixing torrus
01:10 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php
01:07 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.i18n.php
01:06 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/1x1.png
01:06 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialHideBanners.php
00:47 RobH: sq55-sq58 reinstalled lucid, back online, sq51-54 coming back down
00:09 RobH: sq55-sq58 coming down for reinstallation to lucid
00:05 RobH: sq51-sq54 reinstalled lucid, pushing back online

January 12

23:32 RobH: sq51-sq54 coming down for reinstallation to lucid
23:31 RobH: sq49, sq50 reinstalled lucid, online
23:27 RobH: sq46-sq48 reinstalled lucid and back online
22:53 RobH: sq46-sq50 coming down for reinstallation to lucid
22:51 RobH: sq42-sq45 reinstalled lucid, online
22:13 RobH: sq42-sq45 down for reinstallation to lucid
20:43 RobH: sq41 coming down for reinstallation to lucid
18:42 apergos: that file needs to be left there for awhile apparently (say the google instructions) so please don't just toss it yet
18:40 apergos: this is about domain verification for google storage.
18:39 logmsgbot: ariel synchronized docroot/www.wikimedia.org/google126853a33948578b.html
17:48 RobH: all text squids in pmtpa/sdtpa have been upgraded to lucid
17:48 RobH: sq75-sq78 back in service, lucid
17:45 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'Remove db15 from rotation'
17:22 RobH: sq75-sq78 coming down for reinstall to lucid
17:19 RobH: sq71-sq74 reinstalled to lucid and pushed back into service
16:39 mark_: Increased CARP weight of amssq31 to 30
16:33 mark_: Powercycled amssq38
16:30 RobH: sq71-sq74 coming down for reinstallation to lucid
16:27 RobH: sq65-sq66 reinstalled to lucid and online
16:20 mark_: Lowered CARP weight of amssq* non-SSD text squids from 10 to 8 to relieve disk and memory pressure
16:20 apergos: running cleanup on index.html and md5sums for the XML dump jobs that recorded bogus "recombine" jobs that weren't actually run
16:18 mark_: Stopped old puppet instance and restarted squid-frontend on amssq37
16:16 RobH: amsq31 upgraded to lucid and back in service
16:05 RobH: both srv182 and srv183 were not responsive to serial console, rebooted both.
15:42 RobH: sq65 & sq66 coming down for reinstallation to lucid
15:40 RobH: sq62-sq64 installed and online lucid
15:31 RobH: reinstalling amssq31
15:11 RobH: sq62-sq64 down for reinstall
15:06 RobH: sq59-sq61 online as lucid hosts
14:59 RobH: updated wmf repo, copying package for squid from karmic to lucid as well (seems to have been lost from other changes, as frontend was still there)
14:02 RobH: sq59-sq61 offline for reinstall
12:03 mark_: Enabled multicast snooping on csw1-sdtpa

January 11

23:39 Ryan_Lane: adding cron on nova-controller.tesla to svn up /wiki hourly
22:27 RobH: have to reinstall the squids, wrong version written into lease
22:24 awjr: archived hudson build files for 'donations queue consume' and emptied builds directory on grosley to allow the jobs to continue running (donations queue consume job was failing due to hitting max # of files in a dir for ext3 fs)
22:05 RobH: correction, sq65, sq66, & sq71
22:04 RobH: sq62-sq64 back online, sq65-sq67 coming down for reinstall
21:48 Ryan_Lane: adding exim::simple-mail-sender to owa1-3
21:33 RobH: sq59-sq61 in service, sq62-sq64 reinstalling
21:27 Ryan_Lane: powercycling amssq31
20:52 RobH: sq59-61 reinstalled and online, pooled, partially up
20:42 rainman-sr: enwiki.spell index somehow got corrupt, investigating and rebuilding it now on searchidx1
18:57 RobH: sq59-sq61 coming back offline, bad partitioning in automatied install, need to update squid configuration for these hosts
18:47 RobH: sq59 having reinstall issues, skipping it and moving on
18:36 RobH: sq61 reinstalled and back online
18:30 RobH: sq60 reinstalled, back in service
18:03 RobH: sq59-sq61 depooled for upgrade
16:15 RobH: sync-docroot run to push updated tenwiki favicon
14:27 mark_: Depooled amssq31 and amssq32 for SSD install

January 10

21:01 Ryan_Lane: patching python-nova on nova-compute*.tesla (see bug #lp700015)
20:59 Ryan_Lane: err make that #lp681164 for ldap driver
20:59 Ryan_Lane: patching ldap driver on nova-* (see bug #lp681030)
20:57 Ryan_Lane: patching ec2 api on nova-controller.tesla (see bug #lp701216)
19:56 RobH: updated dns with virt!-virt4 mgmt info
16:10 RobH: singer was crashed, investigating why it suddenly had issues. It was pulling down db9, but it halted before it could damage anything.
16:04 RobH: secure server, as well as blogs, offline, investigating server issue on singer
16:02 RobH: db9 having issues and singer is as well, taking singer down since its already crashed

January 9

22:04 Ryan_Lane: powercycling srv217
22:03 Ryan_Lane: powercycling srv271
22:01 Ryan_Lane: powercycling srv262
19:21 apergos: rebooting amssq61 hoping to clear up its problem. I guess puppet restarts squid instances every so often *sigh*
18:50 apergos: stopping squid front and back end instances on amssq61, has network issue
14:15 mark: Reduced Varnish max worker threads from 8000 to 2000 per threadpool
13:52 mark: Pooled knsq6 and knsq7 as bits.esams
13:43 mark: Converted knsq6 and knsq7 into bits.esams machines
13:16 mark: Stopping squid instances on knsq6 and knsq7
13:16 mark: Depooled knsq6 and knsq7 from the knams.text pool

January 8

16:14 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
16:14 logmsgbot: catrope synchronized php-1.5/skins/common/edit.js 'r79866'

January 7

23:42 davidstrauss: Cloning civicrm -> dev_civicrm on Fundraising infrastructure.
23:33 Ryan_Lane: installed dnsmasq on nova-controller.tesla
22:47 Ryan_Lane: rebooting nova-*
22:45 Ryan_Lane: reconfiguring network on nova-* to remove bridges (FlatDHCPManager handles that)
22:10 Ryan_Lane: rebooting nova-controller.tesla
22:01 Ryan_Lane: changing to FlatDHCPManager for networking driver on nova-controller.tesla
22:00 Ryan_Lane: installing bridge-utils on nova-controller.tesla
21:52 RobH: srv217 offline, but shows no errors in drac log, rebooted
19:03 Ryan_Lane: disabling apparmor on nova-compute1/2 to ensure it isn't causing instance creation to fail
16:40 Ryan_Lane|sleep: adding CNAME svn.mediawiki.org -> svn.wikimedia.org per bug 26474
14:44 RoanKattouw: Installed xdebug on prototype so I can turn it on when debugging fatals
00:51 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php
00:31 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php
00:30 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/centralnotice.js
00:30 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php
00:30 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialBannerController.php
00:29 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.i18n.php

January 6

22:21 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Set $wgCodeReviewRepoStatsCacheTime to 1 hour, requested by Sam'
22:18 Ryan_Lane: added awjrichards to storage3
16:29 Ryan_Lane: upgrade nova-* packages on nova-*.tesla, as the current branch packages are broken
12:37 mark: Powercycled amssq33
04:04 Ryan_Lane: powercycling mobile1, that is
04:04 Ryan_Lane: restarting mobile1
03:36 Ryan_Lane: updating nova* packages on nova-controller.tesla, this version of trunk packages is simply broken

January 5

22:22 Ryan_Lane: repooling payments1 and payments2
22:05 Ryan_Lane: depooling payments1 and payments2
21:35 Ryan_Lane: adding LVS realserver package and configuring it via puppet for payments1-2
20:33 Ryan_Lane: powercycling sq80
20:32 Ryan_Lane: powercycling knsq14
20:30 Ryan_Lane: powercycling amssq56
20:29 Ryan_Lane: powercycling amssq51
20:29 Ryan_Lane: powercycling amssq47
20:28 Ryan_Lane: powercycling amssq46
20:27 Ryan_Lane: powercycling amssq41
20:18 Ryan_Lane: fixing torrus
18:37 Ryan_Lane: enabling reporting for store and log on puppetmaster, and enabling reporting on spence for puppet
17:22 logmsgbot: demon synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26580 - try different interwiki prefix'
17:18 logmsgbot: demon synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26580'
16:18 Ryan_Lane: powercycling srv182
02:03 Ryan_Lane: uncommenting payment1 and payments1 in pybal config
02:02 Ryan_Lane: commenting out payments1 and payments2 to try to force a pybal check
01:23 Ryan_Lane: pooled payments1 and payments2

January 4

23:23 Ryan_Lane: moving payments2 from internal to squid vlan
23:23 Ryan_Lane: restarting network services on payments2
23:18 Ryan_Lane: restarting nginx on payments1
23:12 Ryan_Lane: switching IP address info for payments2
23:11 Ryan_Lane: moved payments1 from the internal to the squid vlan
22:12 Ryan_Lane: restarting network services on payments1
21:52 Ryan_Lane: changing IP address for payments1
20:07 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'changed tenwiki anon edit block to anon page creation block'
16:53 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove hack disallowing anon edits on tenwiki - moved to its proper place in InitialiseSettings.php'
16:52 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disallow anon edits on tenwiki'
16:50 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26554 - Set $wgAutoConfirmAge to 0 on tenwiki'
16:43 Ryan_Lane: removing login.tesla from DNS
15:42 apergos: snapshot1 removed from mediawiki-installation for now til set up again
15:35 Ryan_Lane: dropping nova database on nova-controller.tesla, letting nova recreate it
15:29 Ryan_Lane: changing ldap schema on nova-controller.tesla to newer (v2) openstack schema
14:24 logmsgbot: robh synchronized php-1.5/wmf-config/CommonSettings.php
14:12 Ryan_Lane: powercycling mobile3
12:17 RoanKattouw: srv182, srv183 seem to have gone into swap death
12:11 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'r79572'
09:54 apergos: restarted lsearcdh on search7 (in theory the one host in the search-pool3 ) to see if that helps
06:28 logmsgbot: root synchronizing Wikimedia installation... Revision: 79563:
06:27 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 79563:
06:26 Tim: svn up/scap to r79563 for clickjacking defences
02:47 Ryan_Lane: disabling the puppetmaster service, and enabling the opendj service on nova-controller.tesla
02:45 Ryan_Lane: clearing mediawiki database for nova-controller.tesla, and clearing out most of the LDAP data
02:31 Ryan_Lane: rebooting nova-*.tesla
02:21 Ryan_Lane: updating nova-* on nova-controller.tesla

January 3

23:19 Ryan_Lane: deleting login.tesla vm
23:05 Ryan_Lane: adding reedy account, putting samreed account back in, and ensuring the ssh key is absent for samreed
23:00 Ryan_Lane: adding rfaulk to storage3
22:58 Ryan_Lane: renaming samreed to reedy in admins.pp
22:51 Ryan_Lane: adding demon to kaulen
22:51 Ryan_Lane: giving robla access to modify-ldap-user, add-ldap-user, and delete-ldap-user on formey
18:19 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'db15 going live'
17:44 apergos: reboot snapshot1 for install of lucid
16:18 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'replacing srv266 since its down'
16:16 RobH: rebooting srv266
14:34 mark: Site should be stable again
14:08 mark: Depooled text squids again
13:59 mark: Adding text.esams squids into the bits.esams LVS pool
13:43 mark: Killed pybal on amslvs1
13:34 mark: Depooled knsq1 for installing debug symbols for oprofiling
10:55 logmsgbot: midom synchronized php-1.5/wmf-config/mc.php

January 2

23:12 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'db16 going live for s7 duty'

2000s

Archive 1: 2004 Jun - 2004 Sep
Archive 2: 2004 Oct - 2004 Nov
Archive 3: 2004 Dec - 2005 Mar
Archive 4: 2005 Apr - 2005 Jul
Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
Archive 6: 2005 Nov - 2006 Feb
Archive 7: 2006 Mar - 2006 Jun
Archive 8: 2006 Jul - 2006 Sep
Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
Archive 10: 2007 Feb - 2007 Jun
Archive 11: 2007 Jul - 2007 Dec
Archive 12: 2008 Jan - 2008 Jul
Archive 12a: 2008 Aug
Archive 12b: 2008 Sept
Archive 13: 2008 Oct - 2009 Jun
Archive 14: 2009 Jun - 2009 Dec

2010s

2020s