In Q3, the Search Platform team will continue to primarily focus on fixing the Search Update Pipeline, to address architectural limitations that have been causing failures in edits being accurately reflected in search indexes for users.

We will be continuing the work to unpack language analyzers for emerging languages, to work towards making our content more discoverable for people who speak underserved languages.

We will address the Spark 3 migration as driven by the Data Engineering team.

While the technical work on the WDQS SLO should be complete in Q2, we will need to communicate it to our communities, with potentially some follow up to manage expectations and misunderstandings.

Q3 OKRs

Objective 1: Users can reliably query information on Wikidata, and search on wikis

KR: WDQS uptime and update lag >= 95%
KR: Search Update pipeline errors TBD
KR: Communicate WDQS strategy to community

Objective 2: Searchers of emerging languages can search in their own language

KR 2.1: Unpack 3 language analyzers from emerging languages

Objective 3: Paying down technical debt in infrastructure

KR 3.1: All of the Search Platform Spark jobs are running on Spark 3

Project plan (Q3)

Search Update Pipeline
Communication of WDQS uptime SLO
Spark 3 migration
Unpack language analyzers

Things we ship

Fix update pipeline (2+ quarters of work)

Higher level Objective: Users can reliably query and search for content, so that they can depend on Wikimedia Foundation projects as knowledge infrastructure (ERF: Technical Infrastructure, Product Platform)

KR1: End to end Search Update Pipeline for simple edits on one wiki

Description: The search update pipeline is currently broken, resulting in updates being sporadically lost often enough that users are reporting bugs. The Sanetizer is supposed to resolve this, but was shut off for a while, and restarted, but is running with ~4 weeks of lag. We want to resolve issues in the update pipeline so that our indexes are not out of sync by more than XX days

Phab: task T317045

Milestones (note that there are no dependencies between those milestones):

SLI and SLO are created for the Search update lag
Search Update Pipeline works with all edit types and optimizations (re-renders, deletes, redirects, event re-ordering)
Search Update Pipeline supports Weighted Tags
SLI and SLO created for Search update lag and error rates
Search Update pipeline is deployed on k8s with Flink operators

WDQS uptime SLO

Higher level Objective: Users can reliably query information on Wikidata and search on wikis

KR: Community is updated on expectations for WDQS maintenance-only mode

Description: Due to the WDQS backend scaling pause, the Search team will be moving to maintenance only mode for WDQS. In addition to the update lag SLO, we will create an uptime SLO for WDQS. This will allow us to set expectations for users for service availability, while also setting expectations for the Search team to know how much time to spend on working on WDQS. The community will need to be updated with the current state of WDQS scaling.

Phab: task T313751

Milestones:

Community comms for WDQS (and WCQS)
Communication and documentation for the larger SRE teams

Spark 3 migration

TBD: checking with Data Engineering team if they have an OKR to which we can attach.

Higher level Objective: Paying down technical debt in infrastructure

KR1: All of the Search Platform Spark jobs are running on Spark 3

Phab: task T322905

Milestones:

Convert python dependencies/environments to conda
Convert all spark jobs to Spark 3
Specific jobs are tracked in List of Spark2 jobs running in Hadoop
Affected teams are messaged
Outputs are validated to be same (within margin of error)

Unpack language analyzers

Higher level Objective: Searchers of languages can search for WMF in their preferred language

KR1: Unpack all language analyzers: Japanese (spillover), Armenian, Latvian, Hungarian

Description: Enable users of non mainstream languages to search for WMF content in their preferred languages. Reduce technical debt of having monolithic language analyzers, which make search efficacy opaque for those languages, and prevents general multilingual improvements to be deployed.

Phab: