Incidents/2017-06-01 Maps
Appearance
Summary
Migration of maps to upload caches resulted in query parameters being stripped, breaking some subservices (task T166735).
Timeline
This is a step by step outline of what happened to cause the incident and how it was remedied.
Conclusions
- There is never enough monitoring.
- An important change was considered ops-only, was never communicated to its product team, neither beforehand for coordination nor afterwards for a quick check that everything still works.
Actionables
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
- Done Add geoshapes and geolines to monitoring spec (task T166776)
- Done Services need external monitoring (task T167048)
- Not done Kartographer should handle external data errors gracefully (task T148883)