Incidents/2025-02-27 wdqs 500 errors
document status: draft
Summary
| Incident ID | 2025-02-27 wdqs 500 errors | Start | 2025-02-27 14:10 UTC |
|---|---|---|---|
| Task | T387467 | End | 2025-02-27 14:50 UTC |
| People paged | 1 | Responder count | 3 |
| Coordinators | Brian King (inflatador, Search Platform SRE) | Affected metrics/SLOs | |
| Impact | For approximately 40 minutes, users of Wikidata Query service had a 2-12% chance of their queries failing. Normal failure rate is ~1% | ||
For approximately 40 minutes, the percentage of failed queries to the Wikidata Query Service increased by 2x, reaching 12x at its peak. More details in this Etherpad
Timeline
All times in UTC.
Note that this timeline assumes that today's Mediawiki extension deploy was the root cause. This has not been definitely proven, but I (Brian King) believe it's the best explanation based on the evidence.
- 14:04 This mediawiki extension CR was merged/deployed
- 14:10 WDQS query failure rate begins to rise
- 14:23 Alerts/pages on call FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx errors from wdqs.discovery.wmnet
- Problems with Mediawiki extension deploy noted in #wikimedia-operations:
- [14:28:12] <kart_> TheresNoTime: There are exceptions in debug servers. Anything going with it?
- [14:29:45] <TheresNoTime> kart_: ah, that looks related to your patch yes, shall we rollback?
- 14:41 CR reverted and revert deployed
- 14:50 WDQS query failure percentage begins to drop back to normal levels
Detection
Issue first detected by: l FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx errors from wdqs.discovery.wmnet
Did the appropriate alert(s) fire?
yes
Was the alert volume manageable?
yes
Did they point to the problem with as much accuracy as possible?
Not perfectly, but good enough
Conclusions
What went well?
- The issue was spotted quickly, and rolled back quickly.
What went poorly?
- The issue wasn't spotted until it was rolled out in production.
Where did we get lucky?
Links to relevant documentation
Actionables
- WikimediaCampaignEvents seems to have a dependency on WDQS. Review this dependency with developers and Service Ops to ensure stakeholder expectations are in line with the constraints mentioned in WDQS docs.
Scorecard
| Question | Answer
(yes/no) |
Notes | |
|---|---|---|---|
| People | Were the people responding to this incident sufficiently different than the previous five incidents? | y | |
| Were the people who responded prepared enough to respond effectively | n | ||
| Were fewer than five people paged? | y | ||
| Were pages routed to the correct sub-team(s)? | y | ||
| Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours. | y | ||
| Process | Was the "Incident status" section atop the Google Doc kept up-to-date during the incident? | n | |
| Was a public wikimediastatus.net entry created? | n | ||
| Is there a phabricator task for the incident? | T387467 | ||
| Are the documented action items assigned? | n | ||
| Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? | y | ||
| Tooling | To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are open tasks that would prevent this incident or make mitigation easier if implemented. | no | |
| Were the people responding able to communicate effectively during the incident with the existing tooling? | y | ||
| Did existing monitoring notify the initial responders? | y | ||
| Were the engineering tools that were to be used during the incident, available and in service? | y | ||
| Were the steps taken to mitigate guided by an existing runbook? | n | ||
| Total score (count of all “yes” answers above) | 10 | ||