GitLab/Gitlab Runner/Security Evaluation
This page describes the access and permission model for different kind of Runners as well as additional security measures.
Access and permission model
Access to the different tiers of Runners depends on the kind of project, what source branch the CI job comes from and if the project has additional authorization for specific Runners. CI jobs won't be scheduled based on the author, but only on the project and branch. This model differs from the current Gerrit model but can be replicated quite similar using the feature-branch and protected-branch approach with limited maintainer permissions.
Personal/team projects
By default, projects in GitLab don't have CI capabilities. So projects in personal space or in arbitrary groups can not execute CI jobs. The only option to execute jobs is to configure your own GitLab Runners for this project (bring-your-own Runner). Please take a look at the official documentation on how to register a runner. In the future it may be possible to also use Cloud Runners.
Supported projects
Projects under the /repos group are considered officially supported and have CI capabilities. These projects can access Shared Group Runners configured for this group. All CI jobs for this projects will get executed on the Shared Runners. The Shared Runners are running inside WMCS and offer general purpose CI capabilities. Jobs from all branches (feature and main branch) will get executed on this workers.
Please note that this workers live outside of WMF production infrastructure and must be considered untrusted. So if CI jobs deal with sensitive credentials or produce production artifacts, make sure to execute this jobs with Trusted Runners instead of Shared Runners.
Supported projects with trusted builds
Projects with advanced requirements can request access to specific Trusted Runners. Projects with these requirements have to be explicitly allowed for these Runners. No project has access to these Runners by default. These Runners live inside the WMF infrastructure and builds on these Runners are considered trusted.
To make sure untrusted and unreviewed code changes are separated from reviewed code, commits on feature branches from contributors and developers will get executed on the group Shared Runners in WMCS. Only if these changes are reviewed and merged into the protected main branch, Jobs will get executed on the trusted Runners (see also /hieraddata/role/common/gitlab_runner.yaml for this configuration). Reversely, this also means maintainer permissions and permissions to the main branch should be treated with extra care! Make sure to protect the main branch.
Some jobs may want to be executed for protected branches only (like releasing a new image or package). This can be controlled by adding certain rules to the jobs in gitlab-ci.yml ( rules: [if: $CI_COMMIT_BRANCH == "main"]). Furthermore trusted runners need certain tags on every job they execute. So make sure to add tags: [protected] (tag name to be discussed) to all trusted jobs in gitlab-ci.yml.
Security measures
Shared and Trusted Runners follow GitLab security best practices. The are setup using the more secure Docker executor, which separates CI jobs from each other and from the host and makes it harder to persist any malicious changes on the Runner. privileged CI jobs, which could use privileged containers and change the GitLab Runner host are not allowed on all Runners.
Trusted Runners have additional security requirements. These Runners execute Jobs which might produce production artifacts or which might use production credentials. Furthermore Trusted Runners are able to reach other production workloads. So additional security measures are implemented for all Trusted Runners.
Jobs on protected branch only
Protected branches are a key element to manage who has access to Trusted Runners (see above). So to make sure no unauthorized jobs are executed, Trusted Runners will accept Jobs from protected branches only. This can be configured using the parameter --access-level="ref_protected" when registering a runner (see also profile::gitlab::runner::access_level for this configuration)
Tagged jobs only
To prevent Trusted Runners from executing arbitrary jobs, only jobs with a certain tag will get executed. This can be configured using the parameter --run-untagged="false" and --tag-list when registering a runner (see also profile::gitlab::runner::run_untagged for this configuration). This is more of an organizational than a security restriction, because any job can use any tag. However this configuration helps to explicitly control which job is executed on which Runners. See also using Trusted Runners.
Run gitlab-runner as non-root
To reduce the risk of privilege escalation on Trusted Runners the gitlab-runner process is executed as user gitlab-runner instead of root. This addresses mostly potential vulnerabilities in the gitlab-runner process itself. Access to the Docker daemon is similar to running the process with root (gitlab-runner user is member of group docker). See also profile::gitlab::runner::user for this configuration.
Firewall configuration
To reduce the attack surface for potential malicious jobs Trusted Runners have special firewall configuration. By default all Docker containers on Trusted Runners can not connect to the internal wmnet network (10.0.0.0/8.). A default REJECT ferm rule drops all TCP traffic. If certain services are needed in CI (like apt repository, chart museum or Docker registry), this services have to be explicitly allowed for Trusted Runners. See also profile::gitlab::runner::allowed_services and profile::gitlab::runner::restrict_firewall for this configuration.
Requests to the internet are allowed.
Restrict allowed images
All GitLab Runner have a allow-list of certain images which can be executed in CI jobs. This should make it harder to run typical malicious code like crypto miners. All Runners are initialized with a config-template which defines a list f allowed_images. This list contains the official WMF container registry and a handful of popular, trusted images.
Runner configuration changes
To ensure the secure default configuration remains unchanged, two measures have been implemented. First, the GitLab/Runbook#GitLabRunnerTrustedConfigMissing alert triggers when any critical configuration parameter is modified. Additionally, the edit page for Trusted Runners has been disabled for non-admin users, ensuring that only administrators can alter the configuration.
See puppet hiera config profile::gitlab::custom_nginx_config and profile::gitlab::trusted_runner_id_regex.
Rootless Docker and dropped Docker capabilities
It is possible to run Docker as non-root user in a dedicated user namespace for enhanced security and even smaller attack surface (see https://docs.docker.com/engine/security/rootless/ and cap_drop). For the current setup this is not implemented but may be added in the future.