Performance/Graphite (synthetic instance)/Runbook

This is the run book for the Graphite instance used by QTE for performance and other related metrics.

Meta

• Issue tracker (Phabricator): synthetic-performance-testing
• Documentation: Graphite

Debug missing metrics

If metrics stops arriving in Grafana the reason can be two different reasons: Either something is wrong with Graphite or something is broken on the server that runs the performance tests. First check the #synthetic-tests-error-reporting Slack channel and see if you have any related errors in that channel. If there's problems with the Graphite server, the Slack channel should be flooded with messages that metrics cannot be sent to Graphite.

Log into the server and check the status Graphite. First switch to the correct user: sudo su - graphite

Run docker ps to check that the instance is up and running. If not you need to start it. Go to the settings directorycd /home/graphite/settings/ and start docker-compose up

If the Graphite is running but is broken somehow and the restart functionality hasn't fixed the problem you should check the logs:

docker logs --tail 100  settings-graphite-1

Update to new version

You very rarely need to update the Graphite version. However if you need, you update /home/graphite/settings/docker-compose.yml file by changing the version number for the Graphite container then stop and start the container.