GitLab/Monitoring
| GitLab Backup/Monitoring |
|---|
 |
|
GitLab is monitored by Prometheus and Icinga.
Metrics and dashboards
Metrics of GitLab instances are collected by Prometheus.[1] Multiple Grafana dashboards can be found in the folder GitLab.
A new GitLab Logstash dashboard with visualizations was added in phab:T375849.
Currently the following exporters are enabled:
| Exporter | Port | Path |
|---|---|---|
| nginx | 8060 | /metrics
|
| redis | 9121 | /metrics
|
| postgresql | 9187 | /metrics
|
| gitlab-workhorse | 9229 | /metrics
|
| rails | 8083 | /metrics
|
| sidekiq | 8082 | /metrics
|
| gitlab-server | 9168 | /metrics
|
| gitlay | 9236 | /metrics
|
| wmf_gitlab_exporter | 9196 | /
|
Related Task: T275170 and T354656
The wmf_gitlab_exporter is a custom exporter written in Python. The code can be found here: https://gitlab.wikimedia.org/repos/sre/gitlab-exporter. The exporter exposes metrics for special alerting and auditing use-cases mostly for the collaboration services team.
Alerting
Alerting happens with multiple Icinga and Prometheus blackbox checks. See the overview page. We mostly rely on the checks which come from the basic vm setup (like filesystem checks, puppet check). Additionally the public facing endpoints of production GitLab are checked as well[2]:
| Host | check |
|---|---|
| gitlab.wikimedia.org | https and cert check |
| gitlab.wikimedia.org | ssh check for git ssh daemon |
Individual GitLab components (like Postgresql or Redis) are not monitored currently.
There are additional alerts configured in Prometheus Alertmanager, see https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/refs/heads/master/team-sre/gitlab.yaml and all GitLab/Runbook.
In the future when individual GitLab components are separated adapting the existing upstream alerts is recommended.