Jump to content

Search Platform/Weekly Updates/2023-10-20

From Wikitech

Summary

We've sent our first public communication about the WDQS graph split experiment (https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_backend_update/October_2023_scaling_update). The project is starting, with infrastructure already well on the way and software engineering making sure we have a working development environment. Things should pick up speed soon.

Search Update Pipeline is being deployed on our staging wikikube environment, we're ironing out the last configuration, network routes, and optimizing the code for production.

Search analysis work is being optimized. Since deployment and reindexing is an expensive operation, we want to bundle all changes together.

What we've accomplished

Search Update Pipeline

WDQS graph splitting

Operations

  • Failure of WDQS updater after switching to the newer kafka connector API. The cause is related to a bug in older versions of kafka that forget consumer offsets on idle streams - https://phabricator.wikimedia.org/T349147