Jump to content

OCG/Deployments

From Wikitech
< OCG

Planned deployments, linked from Deployments. For a list of past deployments, look for 'ocg' in Server Admin Log.

See OCG#Deploying changes to learn how to deploy a new version of OCG.

2016-10-03

ocg-collection:

phab:T147211: Add en.wiktionary.org to blacklist temporarily to prevent DoS.

Deployed new service (d0f8e126f5d1ad53358c7c7380ca02f41c33ead6):

  • phab:T147211: Allow administrators to blacklist renders of certain metabooks.

Deployed new latexer (45c30840d22a7590cee904aff5f2e272c75d3bdb):

2016-08-17

ocg-collection:

Deployed new bundler (40ba27f770480cb1642ed58ef18b986ec67b0daf):

  • Remove rest.wikimedia.org, the server is being shut down Sept 1. (phab:T133001)

Deployed new service (51d2eb6d47dfc5225d8d5bc45444e81cc233b1c7):

  • Add metrics for number of items in collections.

Deployed new texter (63333e2c2d04f00c379ac35cac9d85af7096e767):

  • Add option to emit `blessed`-style markup.

2016-05-10

ocg-collection:

Deployed new service (ac2dff089f66fd5f6d50d7e7cd046cfd5558ac59):

  • Update npm dependencies; switch to `babybird` Promise implementation.

Deployed new bundler (e33963029ff99f63a54fae78515b742375d88a3c):

  • Update npm dependencies; switch to `babybird` Promise implementation.

Deployed new latexer (544a26ae19eb0e0c0ae9bea98e24630b8ca7982b):

  • Update npm dependencies; switch to `babybird` Promise implementation.

Deployed new texter (df26683c689eb282dba55b0f287fd71307c4a2e2):

  • Update npm dependencies; switch to `babybird` Promise implementation.

Dependencies were rebuilt.

The improved Promise implementation should fix the recursive nextTick issues encountered in phab:T120079.

2016-05-01

ocg-collection:

Deployed new bundler (d1b9c6b1e73766d056a3a80773da1e2cd38bd851):

  • Fix --domainname option when bundling a collection from the command line.

Deployed new latexer (77b0b9d45bf15021a7b2e79a8bee3f528e2e5943):

  • Add sawNonSpace internal flag to avoid problems when forcing a newline.
  • Skip images inside headings.

2016-04-27

ocg-collection:

Deployed new service (fc936c8ad7893d606ff67e5fa2fe4f62e755e5fe):

  • Allow decommissioning OCG hosts. phab:T120077
  • Periodically check for configuration file changes. (Not enabled in default configuration.)
  • Improve graceful shutdown.

2016-04-25

ocg-collection:

Deployed new service (4442920b712dad67e3ca7c7bf6af4a37a794405d):

  • Update README.

Deployed new bundler (c4f4c4b212aada9646dbc113b6db352f51408f75):

  • Don't fallback to rest.wikimedia.org.
  • Use domain for status messages if prefix is not available.
  • Resolve titles even when $wgArticlePath involves a query string. phab:T124762
  • Send Accept header to Parsoid/RESTBase to get data-mw attributes inline. phab:T130685

Deployed new latexer (3ff99a570acf7b8f3a8af3c47f02b02ac5aad01c):

  • Update README.

Deployed new texter (6b6ef440cb03c16d79c76e123044ee8e48b9c6b8):

  • Use released version 0.3.3. (No code changes.)

2016-03-17

ocg-collection:

Deployed new bundler (1fec608a4d55e7bddb115122dd0db0fd4bc88570):

  • Use underscores in titles sent to the RESTBase / Parsoid API.

2016-01-26 - 2016-01-27

Originally planned for 2016-01-20, deferred to 2016-01-25, then phab:T124720 prevented actual deploy to labs until ~9am PST 2016-01-26. However, the redis server was down in labs (phab:T124791,phab:T124677). That was fixed ~10am PST. Deployed to production 2016-01-27 in our usual deploy window.

  • Deployed to BETA: 2016-01-26
  • Deployed to PRODUCTION: 2016-01-27

ocg-collection:

Rebuilt dependencies.

Deployed new service (8a9626602a907ef6c2cb825119292933245efede):

  • Update dependencies; delint code.

Deployed new bundler (6491a016302de2db86a8ee14238ce5f25707df74):

  • Use body_only instead of bodyOnly in REST v1/Parsoid v3 API.
  • New --fetch-modules option (turned off by default). phab:T114788
  • Add username to the User-Agent header.
  • Remove most references to parsoid-lb.eqiad.wikimedia.org. phab:T110474
  • Handle protocol-relative API URLs.
  • Update dependencies; delint code.

Deployed new latexer (28793564325d1f5c941ce2d3be0b6cd288ae714f):

  • Update dependencies; delint code.

Deployed new texter (73f7b86871abff978c818ce3941c52072b0e53f5):

  • Update dependencies; delint code.

2015-09-16

ocg-collection:

Deployed new bundler (f673740b2a61b0bda64b0e58f7b7e596e3e7de4d):

  • Released version 1.3.0
  • Work around bug with --domain CLI option.

Deployed new latexer (2c2727487be7af4d55c78982a8891b6dd1ee01d2):

  • Released version 0.3.1.

Deployed new texter (3d4ab2dcdb3ab39cc781b757792a1ee8419fae6d):

  • Work around bug with --domain CLI option.
  • Improve layout of preformatted content.
  • Released version 0.3.2.

2015-09-14

ocg-collection:

Deployed new bundler (8d8cd3ec79367fa086a203768971e5aaa26811a8):

  • Add User-Agent header to API requests.
  • task T110782: Add Parsoid v3 API support.
  • Remove the (broken) --api option alias.

Deployed new texter (de5a441228dc6e6e8eafd68f198ea6fa33de6049):

  • Support "standalone mode" for single-article conversion.

2015-06-23 pm

ocg-collection:

Rebuilt dependencies on the wmf-deploy branch again, this time including the required request package.

2015-06-23 am (reverted)

Attempted to deploy d7c698d5bf730d34057945e912ac75dc542dd788 to production. Found that labs machines have native Ubuntu node-* packages installed which are not installed in production. This made production fail -- the most immediate cause was that the npm request package was missing from the wmf-deploy repo (due to an issue with the npm prune step of make production), but it wasn't noticed during the beta canary deploy because deployment-pdf01 had the Ubuntu node-request package installed.

Reverted production to ca4f64852de5b1de782b292b50038fbd2dd84266 (2015-05-20 deploy).

2015-06-22 (beta only)

ocg-collection:

Deployed new bundler (fe21e6f1577f571ea83226ba04dbe0285e50e710):

  • Update rimraf dependency.
  • Update sqlite3 dependency for compatibility with node > 0.10.
  • Convert ocg-bundler to make use of the https addresses.
  • task T75130: Add "AttributionRequired" metadata to OCG bundle.
  • Parsoid v2 API uses `body` as a query parameter, not `bodyOnly`.
  • Use SPDX license.

Deployed new latexer (dd0e83952ec12d0ca675b63396424b3a4330e1e6):

  • Update sqlite3 dependency for compatibility with node > 0.10.
  • task T99908: Use Devanagari script for Bhojpuri wikipedia.
  • Use SPDX license.

Deployed new service (2941c3dc0f25c654d7ddab1b82ecb726728551e1):

  • Update hiredis to latest version.
  • Update bunyan, bunyan-syslog, and gelf-stream.
  • Update node redis to support redis 2.8.x, which has been installed in beta/labs.
  • Update rimraf and mv dependency.
  • Use SPDX license.

Deployed new texter (bb897807e5ffa62788f8036bc58d31bf78b079f5):

  • Update sqlite3 dependency for compatibility with node > 0.10.
  • Use SPDX license.

Repeat of the earlier deploy attempt, after upgrading dependencies to fix redis issue. A patch to the Collection extension ( https://gerrit.wikimedia.org/r/220069 ) was also necessary to fix labs.

2015-06-22 (cancelled)

ocg-collection:

Deployed new bundler (0d70ff85138da5cdd5ec0147f6eaba5bb8f49a91):

  • Convert ocg-bundler to make use of the https addresses
  • task T75130: Add "AttributionRequired" metadata to OCG bundle.
  • Parsoid v2 API uses `body` as a query parameter, not `bodyOnly`.
  • Use SPDX license.

Deployed new latexer (9d81c73ae4bc1555b6f513639a025c5f5f09ffd6):

  • task T99908: Use Devanagari script for Bhojpuri wikipedia.
  • Use SPDX license.

Deployed new service (65ce1bb81bff63b810a9b135811edb9b5f147e3d):

  • Use SPDX license.

Deployed new texter (cbea03456f0c53212bb7d228bee1f09fcb88686e):

  • Use SPDX license.

This deploy was cancelled after being deployed to beta, since beta ran into redis problems. Redis apparently was upgraded to 2.8.x, which made node-redis unhappy.

2015-05-20

ocg-collection:

Deployed new latexer (e08a3d09a547d19b9c4b2a630851505f1ed5146d):

  • task T92817: improve text rendering for nepali languages.

2015-05-13

ocg-collection:

Deployed new bundler (24365a9154a86c6870e89b2256ee5bc1d88f5a49):

  • Add --php-api option for local debugging.
  • Fix file revision lookup failures.
  • task T93292: Allow bundler to fetch parsed articles from RESTBase.
  • Switch from es6-shim to core-js; update prfun and jshint.

Deployed new latexer (efae1a449bfaf2fa925f1968101fafedf87edb48):

  • Switch from es6-shim to core-js; update prfun and jshint.

Deployed new service (efa690d717ae257879a4dfbdd7ecc9227b586bed):

  • Remove deprecated download command.
  • Add script to allow date-based cache cleaning.
  • Update docs and config options to allow use of RESTBase.
  • Switch from es6-shim to core-js; update prfun and jshint.

Deployed new texter (e325556133e9ece9af20631d91635574a0b6f8d0):

  • Switch from es6-shim to core-js; update prfun and jshint.

2015-03-23

ocg-collection:

Deployed new latexer (39534c0e9f0a48b6c3565fe8b50f437103e0d218):

  • task T93008: use {{DISPLAYTITLE}} or subpage title where appropriate.

2014-12-11

ocg-collection:

Deployed new service (f6f3788da146e92dce8dcb11a14479c8b117f2be):

  • Ensure we exit when `clear-host-cache` is done.
  • Don't swallow output from helper scripts unless `--quiet` is given.
  • Separate out `download` and `download_file` commands.

2014-12-03

ocg-collection:

Deployed new service (4952462b0045d0f22ce31eaa80d671e9b3dc9350):

  • Don't halt gc due to readdir after clean; more verbose error reporting.
  • Ensure that directories are initialized in GC thread.
  • Fix GC of failed jobs.
  • Fix job progress status indication.
  • Record which host picks up a job.
  • Add script to allow host decommissioning.
  • Be sure that `console.log` appears on console output.

Deployed new bundler (8565c79549e107ffa10fa817d00348b85ba9dbb9):

  • Hook up npm-travis.

2014-12-01

ocg-collection:

Deployed new service (3bcfaa00595c8b331175577286733030328d146b):

  • Limit memory usage during directory clean up.
  • Don't halt gc if a directory is bad.

2014-11-26

ocg-collection:

Deployed new service (447940eba611ef65f9d251b86c63345c26b7adb5):

  • Don't use the writer as part of the renderTempDir.
  • Validate 'writer' parameter before creating a job object.
  • Remove stringified metabook from most log messages.

2014-11-10

ocg-collection:

Deployed new service (2b2816081120a5e65f5bf80c294b2a85a1de38f8):

  • Double cache lifetime of successful renders, to 4 days.
  • Send SIGKILL if SIGTERM doesn't do the job (after execution timeout).

Deployed new bundler (a93dd0c9935cb6db56da4f6289d66adbdc0710d3):

  • Better debugging if article name resolution fails.

Deployed new latexer (c4c7f7515ea337f05c2d4d851029ba90e469a8f8):

  • Add `--helper-exec-limit` option to protect against runaway image helpers.

2014-11-03

ocg-collection:

Deployed new service (379cea3f9b3678bb83963296fdb55e5b3e8887fa):

  • Allow prevention of retries of failed jobs.
  • Update dependencies.
  • Add _version API endpoint.
  • Switch from winston to bunyan for logging.

Deployed new bundler (c9bf3d430721dfd2553a49dbe31790113916e006):

  • Don't create pipes for P.spawn (avoid blocking unexpectedly).
  • Update dependencies.

Deployed new latexer (aad4dfbab9e6d6c8d05ec82679843541dfd504ca):

  • Prevent 100% CPU hang on certain input text.
  • Don't create pipes for P.spawn (avoid blocking unexpectedly).
  • Update dependencies.

Deployed new texter (8712aeb6e6490b558a22e964ec1bdf995cfd9ca0):

  • Don't create pipes for P.spawn (avoid blocking unexpectedly).
  • Update dependencies.

2014-10-27

ocg-collection:

Deployed new service (fe36c9cf355955870c8edb8eac216821899ce970):

  • Set TMPDIR for the renderer (don't clutter /tmp).

Deployed new bundler (7851aa2d51265eefdef0cf0edf012e7b6e418c3c):

  • Suppress console output during zip (avoid a possible hang).

Deployed new latexer (bd96f76038f9df8ece71e29ea2c1fbf005605077):

  • Suppress console output during unzip (avoid a possible hang).

2014-10-22

ocg-collection:

Deployed new service (f0f6951d09e570ef114567a0ca6e3180af0553d9):

  • Force images to be 600x600 pixels maximum.

Deployed new bundler (7296508d17f6634490cf079cb0fe03c175ff4ace):

  • Fix "image size too big" check.

This was User:Arlolra's first OCG deploy.

2014-10-21

ocg-collection:

Deployed new service (6a1b6a17a4d6e2f7d4b8c6bd0d6f033d425fd886):

  • Update bug URL for the package.
  • Factor out configuration file parsing and logging setup.

Deployed new bundler (f25c1956333a124e4cffec8a3ee52a2f4c5f76a3):

  • Add ability to limit the number of simultaneous requests for small wikis (bugzilla:71895).

Deployed new latexer (dda888e5f64008c71cc37221f942a404e3a32107):

Prepared 2014-10-9, Deployed 2014-10-20

ocg-collection:

Deployed new service (c4c7fbfb743306ad0dd7dfce1475ded536f68f97):

  • Log metabook on every request.

Deployed new latexer (820334a8452762f958e3ebf1a5342d9a667a7b0a):

  • Allow lists nested nine level deep (bugzilla:71896).
  • Fix non-curly quotes (`extquotedbl` in output).
  • Don't crash when the page title is RTL.

2014-10-8

ocg-collection:

Deployed new service (160404df14fb3e92b7b5fff3121a13ee55887bfd):

  • Use inline content disposition, instead of attachment.

Deployed new bundler (71100ec7c7b86fc7ded21aa9977dd303213f4281):

  • Command-line prefix guessing tweak.

Deployed new latexer (8097c38e172a0d358694d800feacc0af075d3cc2):

2014-10-7

ocg-collection:

Deployed new service (531bfb404ce39debfc028c3c2ae72e7f1c59d661):

  • Use article title as download filename (bugzilla:71720, bugzilla:71286).
  • The rl writer should be an alias for rdf2latex (fixes "I got plaintext when I tried to download my PDF using an old link").
  • Ignore backwards-compatibility settings field in metabook.

Deployed new bundler (f55bf164145bd1390a565480a87e04cc640deba4):

Deployed new latexer (fdde609f819679eca109d6a49851bf1e70006172):

  • Support Lao, Khmer, Tibetan, Macedonian, Aramaic, and Bosnian.
  • Improve CJK support.
  • Be careful about square brackets inside \item[...].
  • Support djvu images, but only extract first page by default.
  • Use narrower margins.
  • Equations should be flush left and not indented (bugzilla:68838).

2014-10-6 (afternoon)

ocg-collection:

Deployed new service (5fad457ac6536ccf2fb88df02b31215b2d14cb7b):

  • Avoid use of redis `hkeys`; fix OOM during gc.
  • Don't stop gc'ing if a file disappears.

Deployed new bundler (efbeee40948f80532a3e423e4d1813338c206f5f):

  • Add options to limit bundle and image size. (bug 71647)

Deployed new texter (b7a64572c93f11d75cb2c029c13b2a78f74ed480):

  • Fix line wrapping.

2014-10-6 (morning)

GC fixes.

ocg-collection:

Deployed new service (6579c1b1a01e09c6d3806a74b8e3e2152c293aae):

  • Use proper extension for downloaded file (partial fix for bug 71286)
  • Log the total user-experienced render time to statsd
  • Shrink expiry for temp files and postmortem files (to 6 hours and 1 day, respectively)
  • Reduce basic cache expiry time to 2 days (from 3)
  • Bug fix: utime was setting directory times 1000 years in the future.
  • Bug fix: we were only gc'ing the first subdirectory in the output directory.

2014-10-2

Remaining parts of mwlib deconfig: https://gerrit.wikimedia.org/r/164099

2014-10-1

Turned off mwlib PDF: https://gerrit.wikimedia.org/r/163609

ocg-collection:

Deployed new service (e603a4d79badfabdcd9c2cea6ec6606b18dfd116):

  • Reduce default job lifetime from 5 days to 3 days.
  • Disable folder size computation in health check unless explicitly requested. (bug 71260)
  • Improve statistics collection and measure cache effectiveness.
  • Various improvements to the load testing scripts.

Deployed new bundler (352f9e02316b698bce4fe4da42f0791ff5df591b):

  • Use released version 1.1.0.

Deployed new latexer (0e4ec243d6087a940bbb0954697ba79c1235f772):

  • Save and restore stack of inline decorations around environments, \item, etc.

Deployed new texter (07b0cc6f6ca1c13ec8ce3c1e06b665bb314f6653):

  • Minor changes to travis infrastructure.

2014-09-29

Made OCG default renderer in production: https://gerrit.wikimedia.org/r/163608

ocg-collection:

Deployed new service (037f01ca8c9c9beb66ddfca6d1f78c69b59df97f):

  • Fix redis garbage collection of status objects.
  • Fix redis key expiration time (should be seconds, not milliseconds).
  • Improve the gc cleanDir function to handle nested directories.
  • Write output files in subdirectories to avoid having an excessive number of files in one directory.
  • Remove some blocking operations in the backend.
  • Ensure postmortem moves complete before cleanup.

Deployed new latexer (de5b32c9579169067f3dd87ef04675184c278e77):

  • Treat zh language code as if it were zh-Hans (that is, Simplified Chinese)
  • Match language codes case-insensitively.

2014-09-25

ocg-collection:

Deployed new service (2946c8901cbd5fcb8abf7bb8cb63b579066bbe17):

  • Speed up/cache directory size computation in health check (bug 71260).

Deployed new latexer (144bdd2962702abaa468a18c343509a0945c11e4):

  • Support Simplified and Traditional Chinese scripts.
  • Don't crash if images are inside <dt> tags.
  • Fix TeX escaping of [].
  • Fix "There's no line here to end." crasher (<br> after <dd>).
  • Ensure that we switch to the latin font where necessary.
  • Indic language support (fixes for Assamese, Gujarati, Marathi, Oriya, and Punjabi).

2014-09-24

ocg-collection:

ocg-collection changes:

  • Added load testing tool.
  • Set parsoid endpoint for OCG on wikisource.

Deployed new service (34dce8513b9ce0f6c84b22f854f3d732c36c779e):

  • Clean up directory size computation, eliminate a spurious log message.
  • Add force_render option to help page.

Deployed new bundler (90d0700afb5937ec9ec5bf43f37b653dd1559828):

  • Improve Metabook.fromArticles so that it can be used from load testing tool.
  • Handle redirects from subpages (for example, en:Children's_Heritage/D_O_A).

Deployed new latexer (86c1e14305b5109acd68a6bbd3b129692563c195):

  • Don't crash if PNG has bogus resolution information (for example, [[File:Frederick-II-of-Prussia-Coloured-drawing.png]]).
  • Support inline images (like in en:⠦).

2014-09-23

ocg-collection:

Deployed new service (6d1c4a5e00d51ee55eb739ce5469cf7f47a4efa8):

  • Hash all metabook props for the collection id. (Don't cache too aggressively.)

Deployed new bundler (0d8e3664d9347eb6d5c02ec8b3e8db1bb546ca4c):

  • Reduce default image resolution to 150 dpi (bug 68576).
  • Don't crash on images in subpages, or images with invalid titles.
  • Don't crash on bad image metadata (work around bug 71135).

Deployed new latexer (d5ea61db485f6f01554c83abd47a88c2e709e8a0):

  • Fix CJK definition in polyglossia.
  • Add font and language information for additional indic languages.
  • Don't crash if section/chapter title contains a URL.
  • Don't crash if we encounter an uncontained <li>.
  • Don't crash if <div> begins a figure caption.
  • Don't crash on ;#.
  • Don't crash if chapter has no items.
  • Add support for metabook toc option == "auto".
  • Upgrade node-icu-bidi to v0.1.3.

2014-09-18

Beta is still unhappy: see https://bugzilla.wikimedia.org/show_bug.cgi?id=71031

ocg-collection:

Deployed new bundler (021a70cd7680ffe2c814df2c1dd8c94cb565e031)

  • Fix unreliable image streaming (bug 68929)
  • Tighter query for images
  • Use default papersize when bundling

Deployed new latexer (9686d0e937c7ed502353f346419abff84d420448)

  • Convert TIFF images to PNG (bug 70866)
  • Handle animated GIFs (bug 70865)
  • Accept papersize settings from metabook (bug 68836)

Rebuilt all dependencies:

  • easyimage removed
  • node-icu-bidi updated to 0.1.3.

2014-09-15

Deployed to prod, but skipped deploy to beta/labs due to deployment-salt issues.

ocg-collection:

Deployed new service (f00852507bed9f86fc24198db5a474dd1881f392):

  • Log metabook on error.
  • Allow logging extended information from the backends.

Deployed new bundler (55aa6bea33e29053b76b2043d2c96bcb2f4f1964):

  • Save title redirect information.
  • Increase retries of truncated downloads from 3 to 5.
  • Don't crash if a referenced image was deleted or otherwise missing.
  • Redirect more logging through new log framework.
  • Handle missing contributor information.

Deployed new latexer (03c8b04355e31d2b3d2aa80c505215bcd13a2676):

  • Properly escape HTML anchors and id attributes.
  • Pass additional details to logger on XeLaTeX failures.

2014-09-09

ocg-collection:

Rebuilt all dependencies on deployment-pdf01.eqiad.wmflabs using a documented process.

Deployed new service (526b7b1e90d3b32c65e1cc0cdc2cef2d71ffa2b0):

  • Unify logging through the service wrapper
  • Add a human-readable help page to quickly test/check the service
  • Make sure the collection id is specific to the selected backend
  • Ensure that default temp and output directories exist
  • Use /etc/mw-collection-ocg.js configuration file if present.

Deployed new bundler (05a34aef7e31d8073471327b2cbff989a962257d):

  • Use new log and status reporting framework

Deployed new latexer (c784b335525390b1da5d2f64a8503330175f08a3):

  • Use new log and status reporting framework

Deployed new texter (c0360962459362e2de8cd2bbc44e7516a6037f9d):

  • Use new log and status reporting framework
  • Add -T option to CLI to specify a temporary directory
  • Allow rendering from unzipped bundle

2014-08-13

ocg-collection:

  • master: d7d1571d2bbe7da7c85f618f4eec3b0d8749c49e
  • wmf-deploy: faee29a260a96cbc0cdc0d402658b754a7425af9

Deployed new bundler (e12b79121df18158403ca5f0b160fc95883d0ccc):

  • Don't use process.exit on a normal successful exit (avoid truncating files)
  • Sync output directories before bundling (avoid truncating files)
  • Check for truncated image files during download (avoid truncating files)

Deployed new latexer (f90a3141367eddbc72f5a657fe45c5ebe5ee1c98):

  • Allow page breaks between empty sections.
  • Ensure latin font is used for list item bullets.
  • Add Lohit fonts for Indic languages, and use them whenever possible (bug 28206, bug 68922)

Deployed new service (158d50bc9274ace22daaa76927e03c3ec6b1de71):

  • Remove useless configuration variable.
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=OCG/Deployments&oldid=877400"