Search Platform/Goals/OKR 2022-2023 Q4
In Q4, the Search Platform team will primarily focus on finishing the Search Update Pipeline work, to address architectural limitations that have been causing failures in edits being accurately reflected in search indexes for users.
Additionally, we will be taking advantage of the now unpacked language analyzers to harmonize language processing across wikis to improve search relevancy. At this stage, this will primarily focus on improving recall of search results.
There will be planning work to establish and instrument Search SLOs, in order to formalize the team’s commitments to operations work on the search stack, and creating a project plan/proposal for splitting the WDQS graph, to better outline the scope of work, milestones, and key decision points in that project.
Q4 OKRs
Objective 1: Users can reliably query information on Wikidata, and search on wikis
- KR: Start monitoring Search Update Pipeline KPIs (Key Performance Indicators)
- Lag
- Consistency
- Resource Usage
- KR: Reduce jobqueue usage by X%
- KR: Pipeline consistency quality improves by X%
Objective 2: Searchers of emerging languages can search in their own language
- KR: Reduce ZRR and/or increase number of results returned for 75% of relevant queries averaged across all wikis
Project plan (Q4):
- Search Update Pipeline
- Split the graph project plan
- Search SLOs
Things we ship
Fix update pipeline
Higher level Objective: Users can reliably query and search for content, so that they can depend on Wikimedia Foundation projects as knowledge infrastructure (ERF: Technical Infrastructure, Product Platform)
KR1: End to end Search Update Pipeline for simple edits on one wiki
Description: The search update pipeline is currently broken, resulting in updates being sporadically lost often enough that users are reporting bugs. The Saneitizer is supposed to resolve this, but was shut off for a while, and restarted, but is running with ~4 weeks of lag. We want to resolve issues in the update pipeline so that our indexes are not out of sync by more than XX days. Work needs to be done to understand the current rate of lost updates. We want to reduce the error rate to something close to 0.
The current update pipeline processes Mediawiki updates as a stream, with < 5 minutes of lag. Most additional data is processed as batch, with a lag > 1h. We want to harmonize the system so that all updates are processed as streams, reducing the lag of secondary data sources from > 1h to < 5 minutes.
Docs:
Phab:
Milestones (note that there are no dependencies between those milestones):
- SLI and SLO are created for the Search update lag and error rates
- Some progress on this in Q3 but still not done
- Search Update Pipeline works with all edit types and optimizations (re-renders, deletes, redirects, event re-ordering)
- Only page deletes were addressed in Q3
- Search Update Pipeline supports Weighted Tags
- Search Update pipeline is deployed on k8s with Flink operators
- Progress made on deploying the WDQS updater to the flink-operator in Q3
- /!\ We are waiting on Data Engineering for the Flink deployment
Improve multilingual zero-results rate
Higher level Objective: Searchers can search effectively in their preferred language
KR1: Reduce ZRR and/or increase number of results returned for 75% of relevant queries averaged across all wikis
Description: Following the work of unpacking all the language analyzers, we can now work on harmonizing language processing across wikis and deploy global improvements. In Q4 we will continue to focus on increasing recall (with decreasing zero-results rates and increasing number of results as proxy metrics), assuming that increased recall improves the odds of content discovery, especially on smaller language wikis. Note that this is an imperfect KPI for search relevancy overall.
Phab:
Milestones:
- Setup a framework to measure impact of changes
- Smarter handling of acronyms for word_break_helper in language analyzers
- Investigate applying aggressive_splitting everywhere, not just on English-language wikis
- Handle variation in apostrophe-like characters better
- Repair multi-script tokens split by the ICU tokenizer
- Standardize ASCII-folding/ICU-folding across analyzers
- Stretch: Look into enabling hiragana/katakana mapping everywhere
Things we plan
Create project plan for WDQS graph splitting
Higher level Objective: Users can reliably query information on Wikidata, and search on wikis
KR1: A project plan – complete with milestones, estimation, rough roadmap – is created for splitting the WDQS graph
Description: Splitting the WDQS graph has been identified as the highest priority work to scaling WDQS – the stability of the service is expected to be a high priority in the 2023-2024 annual plan. While we have some ideas about what needs to get done generally, a more concrete project plan/proposal should be documented and published in Q4. This will include details such as product requirements, key results, risks, milestones, etc – if there are still unknown variables that require research, we should be clear in estimating and scoping the amount of research that needs to be done, and how they will influence our decision to move forward with this project or not.
Docs:
Milestones:
- Establish milestones for project
- Establish tracking metrics
- High level estimate of work required
- Identify staffing/resourcing requirements
Finalize Search SLOs
Higher level Objective: Users can reliably query information on Wikidata, and search on wikis
KR1: Search SLOs are agreed upon and finalized
KR2: Instrumentation is in place to measure and monitor Search SLOs moving forward
Description: As part of establishing minimum standards around the quality of the search experience, especially as it pertains to performance of the search stack, the team will establish and maintain a list of Search SLOs that focus on the end user experience. This will both help prioritize operational work where necessary, as well as deprioritizing it where it is lower impact.
Docs:
Phab:
Milestones:
- SLOs are established
- SLO values are established
- Instrumentation for all SLOs is built
- SLO dashboard is created