Portal:Cloud VPS/Admin/Cloud roots and Cloud admins

This page is a reference describing Cloud VPS permissions and policy groups. To apply for any of these permissions, follow Help:Access policies.

The term Cloud root is used to describe someone with root permissions on WMCS servers. The term Cloud admin is used to describe someone with admin permissions on Cloud VPS.

This is more complex than it may sound, as it involves a set of different but interrelated permissions.

Cloud root permissions are more limited than Global root permissions, but are not a strict subset of them: people with Global root do not automatically have all the permissions listed in this page.

SREs in the Wikimedia Cloud Services team should be granted all the permissions listed in this page, but other people (including non-SREs in the Cloud Services team, other WMF staff and external technical volunteers) will only have a subset of them.

Ideally permissions should always be granted by adding people to a LDAP group, but at the moment some permissions are hardcoded in files or databases, more details below.

Root access to bare metal Cloud VPS hosts

Members of the wmcs-roots group (defined in modules/admin/data/data.yaml) have root privileges on most bare metal hosts named cloud* (a notable exception are clouddb* hosts, see #Wikireplicas root below).

Bare metal hosts named cloud* are the hosts running Cloud VPS and Toolforge and are split between the EQIAD and CODFW data centers. They include the cloudcumin hosts that can be used to run Cumin and Cookbooks against other cloud* hosts and against Cloud VPS vms.

Access is granted via /etc/sudoers.d/wmcs-roots, which is deployed by Puppet via the following hiera value:

profile::admin::groups:
  - wmcs-roots

Root access from cloudcumin hosts

Cloudcumin hosts can run commands as root on cloud* hosts through a separate mechanism: they contain two SSH private keys, cloud_cumin_master for root access to bare metal cloud* hosts and cumin_openstack_master for root access to Cloud VPS vms. The corresponding public keys are deployed by Puppet to hosts with profile::cumin::cloud_target (bare metal) and profile::openstack::{codfw1dev,eqiad1}::cumin::target (vms).

As they use two separate mechanisms, it's possible (but should be avoided) that members of wmcs-roots have access to hosts that are not accessible from cloudcumin, or vice versa.

Root access to Cloud VPS vms

If you add a public SSH key to root-authorized-keys.erb in the labs/private repo, that key will be deployed by Puppet to all Cloud VPS vms, in /etc/ssh/userkeys/root.

This file will be moved to the operations/puppet repo in phab:T317362.

The owners of those keys will be able to ssh as root to any puppet-managed Cloud VPS vm, even if they are not a member of the corresponding Cloud VPS project.

Please note that while members of Cloud VPS projects get access via sudo (i.e. they ssh as their user, then become root with sudo -i), this method requires you to ssh directly as root, i.e. ssh root@fqdn-of-vm.

Puppetless vms

In 2024 we introduced Unmanaged Cloud VPS instances where this mechanism does not work. Root access to those vms is managed by attaching a public SSH key to the vm via Horizon.

virsh console access

If you have root access to cloudvirt* hosts, you can also use virsh console to get a root shell in any vm.

OpenStack administrator privileges

OpenStack has a complex RBAC system that controls what you can and cannot do via the OpenStack APIs, CLIs and Web Interface (Horizon).

Some users have superadmin permissions on the whole cluster:

cloudcontrol1005:~$ sudo wmcs-openstack role assignment list --names --domain default

TODO: expand this section with more details.

Wikireplicas root

Wiki Replicas hosts (clouddb*), despite having a host name starting with "cloud" have a more restricted access. Members of the wmcs-roots group do not have root access. There is a separate group wikireplica-roots (defined in modules/admin/data/data.yaml) that is used to grant root access to those hosts.

Wikireplicas admins

For people who need to perform admin operations on Wiki Replicas (like running the maintain-views script or the sre.wikireplicas.update-views cookbook), but do not need root access on the Wiki Replicas hosts, there is an additional group named wmcs-admin (defined in modules/admin/data/data.yaml).

Cloud-admin mailing list

There is a mailing list cloud-admin@lists.wikimedia.org. List membership is moderated and restricted to relevant Wikimedia Foundation staff and volunteers, but the archives are public.

#wmcs-cloud-admin IRC channel

The #wikimedia-cloud-admin ^connect channel in IRC is open to everyone and it's publicly logged at https://wm-bot.wmcloud.org/logs/%23wikimedia-cloud-admin/