Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerDiskFull
Error / Incident
There's at least one toolforge worker node with less than 2.5 Gb of space left on their root partition, if they get under 2Gb they will start failing to allocate ephemeral storage for pods.
Debugging
If you can, ssh to the node and check disk usage with du -s /* or similar, common suspects are under /var/log.
Common issues
pacct log growing too much
One instance of logs getting out of control is the /var/log/account/pacct log, there's task T384250 to rotate it.
Related information
You can find a graph with the 20 workers with lower space left here:
https://grafana-rw.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview
Old incidents
Add here any new tasks for incidents you might encounter.
- task T382865 [jobs-api,jobs-cli] increased exit code 137 rate since 2024-12-14
Communication and support
Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:
- Chat in real time in the IRC channel #wikimedia-cloud connect or the bridged Telegram group
- Discuss via email after you have subscribed to the cloud@ mailing list
- Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
- Read the News wiki page
Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself
Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)