Wikidata Query Service/Wikimedia Commons Query Service/Runbook

This page may be outdated or contain incorrect details. Please update it if you can.

Current setup

Wikimedia Commons Query Service is currently in beta and is served from an instance in Horizon. Instance name is wcqs-beta-01 (full hostname - wcqs-beta-01.wikidata-query.eqiad.wmflabs). Any puppet configuration is served from puppet master on wdqspuppet (wdqspuppet.wikidata-query.eqiad.wmflabs). Service uses OAuth 1.1a authentication, that is handled by nginx reverse proxy. Service is bound to https://wcqs-beta.wmflabs.org/ endpoint.

OAuth 1.1a setup

Our current setup for authentication is based on OAuth 1.1a, provided by Wikimedia (commons.wikimedia.org based). The authorization itself is handled by nginx reverse proxy (auth code) and simple java application (code in mw-oauth-proxy). Java application is piggybacked with WCQS deployments of Blazegraph.

OAuth requires credentials to be provided. Since currently service is in beta, we cannot use standard methods of having secret credentials, that is being used in production systems, is unavailable. To handle that, we use wdqspuppet instance to host our own puppetmaster, with the credentials saved as a local commit. Puppetmaster's puppet repo needs to be updated manually, but the credentials commit should be rebased onto changed production branch. Puppet repo clone is located at /var/lib/git/operations/puppet/.

Update

We don't have any deployment set up for, so it's being done manually. To do that, you need to log in to wcqs-beta-01 and execute following steps.

In the /srv/wdqs-package execute:

git pull
git fat pull
sudo systemctl restart wcqs-blazegraph

For potential updates to puppet, on wdqspuppet, in /var/lib/git/operations/puppet/:

sudo git fetch
sudo git rebase origin/production

On wcqs-beta-01, execute:

 sudo puppet agent -tv
 sudo systemctl restart wcqs-blazegraph

Data Reload

So far WCQS isn't updated real-time and it relies on a weekly process of reloading the data in zero-downtime manner - by indexing a standby namespace and switching it with active one after being done. Process is defined as a cronjob (see crontasks.pp), but of course can be also done manually, by executing script wcqs-data-reload.sh in /srv/wdqs-package/ directory.

Data Reload (Via Cookbook)

WCQS can also reload data via cookbook . To speed up the reload, you can get the latest timestamp from the dumpfile:

root@wcqs2001:/srv/query_service# bzcat /srv/query_service/latest-mediainfo.ttl.bz2 | head -100 | grep -A4 ^wikibase:Dump

Example output:

wikibase:Dump a schema:Dataset,
   owl:Ontology ;
   cc:license <http://creativecommons.org/publicdomain/zero/1.0/> ;
   schema:softwareVersion "1.0.0" ;
   schema:dateModified "2022-10-16T19:00:09Z"^^xsd:dateTime ;

The final line contains the timestamp you can use in your cookbook invocation:

cookbook sre.wdqs.data-reload wcqs2001.codfw.wmnet \
   --task-id T316236 \
   --reason 'reloading data' \
   --reuse-downloaded-dump \
   --depool \
   --reload-data=commons \
   --kafka-timestamp=2022-10-16T19:00:09

Query count summary

We have no easy way to push metrics out of the beta instance, but to provide at least simple quantity metrics we save the same metrics we normally send to event-gate, to storage. There is s script that summarizes them:

./summarizeEvents.sh -e <log event path, can be a prefix for rolling logs> -s <output csv>