User:Majavah/Loki notes
Appearance
This page is currently a draft. More information and discussion about changes to this draft on the talk page. |
This page collects some draft notes from investigating and testing Grafana Loki as a potential solution for Toolforge log aggregation
Collection
- promtail sidecar, with an emptyDir to log to
- TESTED (toolsbeta.test3): this works fine at least with uwsgi, interface will be "send your logs to this file"
- no formatting requirements! just ship your plain text
- custom docker image with promtail and a script to create the full config file with relevant labels (at least container name, possibly deployment name as well), credentials mounted from a k8s secret
- in a production deployment this would probably have a kubernetes mutating webhook to mount the sidecar based on a k8s annotation
Ingestion
- loki server
- ngx_http_auth_request_module and a custom microservice for auth + x-scope-orgid header
Authentication
- loki does not have native authentication built-in, relies on some proxy for that (but is multi-tenant, and expects the proxy to set a header)
- promtail (official ingestion agent) supports http basic auth / client tls
- we can probably write some code to provision some of those auth methods for each tool and write them to a kubernetes secret
- TESTED: proof of concept works!
Storage
- loki has code for swift, but does not list as supported in official docs?
- no support for scaling local file storage unlike OpenSearch
- need to estimate storage needs
Querying
- grafana is standard
- grafana is natively multi-tenant (organizations), and data sources are per org
- supports proxy auth but no account autocreation / support for reading groups from header
- Grafana + CAS + custom script to create orgs? can possibly re-use parts of the script used to create grafana.wikimedia.org users from ldap
Other
- Debian packages for Loki/Promtail? https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1001903