Search Platform/Weekly Updates/2024-08-08

Summary

The WDQS graph split is loading onto its first production hosts. There were some hiccups during the first attempt, but the import is happening now and we'll be watching over the coming days to verify it worked.

Rollout of SUP for private wikis has some issues requiring debugging.

What we've accomplished

WDQS graph splitting

• Graph split rolling out to production hosts. https://phabricator.wikimedia.org/T370754
• Some Grafana graphs will need some retooling to reflect new arrangement of clusters.

Search Update Pipeline / Private Wikis

• Shipped NetworkSession to production, with troubleshooting on network connections underway. https://phabricator.wikimedia.org/T355267
• Debugging deployment of events for private wikis affecting jobs daily. https://phabricator.wikimedia.org/T371767
• Added support for authorizing API requests. https://phabricator.wikimedia.org/T345185
• Added a dashboard tracking SUP lag. https://grafana.wikimedia.org/d/8xDerelVz/search-update-lag-slo?orgId=1

Improve multilingual zero-results rate

• Review of old notes, Phabricator tickets, and patch notes to understand current configuration. There will be refactoring to disentangle ICU and ASCII folding. https://phabricator.wikimedia.org/T332342

Search backend replacement

• Decision record for OpenSearch shown to SRE-at-large, review requested for the week. https://www.mediawiki.org/wiki/Wikimedia_Search_Platform/Decision_Records/Search_backend_replacement_technology

Misc

• Repair ability to run airflow-dags test suite locally on a Linux host. https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/790
• Repair process_sparql_queries_hourly Airflow DAG. The task was running into out of memory errors, and it turns out it was also performing expensive work twice instead of reusing the result. https://phabricator.wikimedia.org/T371217
• Ship a fix for invalid events being generated from our frontend search logging events. Error rate has declined since train rolled forward, but not clear if this completely fixes the problem yet. https://phabricator.wikimedia.org/T286814
• Started a reindex of Wikidata on eqiad &codfw. https://phabricator.wikimedia.org/T371401