Portal:Cloud VPS/Admin/Users not in bastion project
In order to ssh into new cloud-vps instances, users must belong to the bastion project. By default, all new cloud-vps VMs provide access on port 22 to VMs in the 'bastion' project, and users are advised to set up proxyjump for access via a bastion host.
Toolforge users do NOT need to belong to the bastion project, as the toolforge project contains its own bastion host.
Error / Incident
This alert fires if one or more users have the member or reader role in a cloud-vps project but do not have the reader role in the bastion project.
Service users ("magnum", "wmflabsdotorgadmin", and similar) are excluded from this monitoring as they do not use ssh.
We run a custom hook in the Keystone service to automatically add a user to the bastion project any time they are granted the reader role.
Causes
If this alert fires, there was either a failure in our keystone hook, or a user was granted a non-reader role in a project without first receiving the reader role in any project.
First, determine which users are affected by running wmcs-bastionless on a cloudcontrol host.
andrew@cloudcontrol1006:~$ sudo wmcs-bastionless
Project mediawiki-quickstart has unbastioned member: ['zfilipin']
Then check what roles the affected user does have:
andrew@cloudcontrol1006:~$ sudo wmcs-openstack role assignment list --user zfilipin
+----------------------------------+----------+-------+----------------------------------+--------+--------+-----------+
| Role | User | Group | Project | Domain | System | Inherited |
+----------------------------------+----------+-------+----------------------------------+--------+--------+-----------+
| 38676f30eaeb44518bf7e144a73c8da6 | zfilipin | | 556a765653d3442aac428801f89cc5da | | | False |
+----------------------------------+----------+-------+----------------------------------+--------+--------+-----------+
andrew@cloudcontrol1006:~$ sudo wmcs-openstack role show 38676f30eaeb44518bf7e144a73c8da6
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | None |
| domain_id | None |
| id | 38676f30eaeb44518bf7e144a73c8da6 |
| name | member |
| options | {} |
+-------------+----------------------------------+
In the above example you can see that the user has the 'member' role but not the 'reader' role. This is the cause of the problem.
Fixing
The quick fix is to adjust roles as necessary. Either grant the affected user the reader role (presuming the keystone hook is working)
andrew@cloudcontrol1006:~$ sudo wmcs-openstack role add --project 556a765653d3442aac428801f89cc5da --user zfilipin reader
or add them directly to the bastion project
andrew@cloudcontrol1006:~$ sudo wmcs-openstack role add --project bastion --user zfilipin reader
Debugging
Of course, resolving the issue is not the same as understanding what caused it.
Typically a user is granted a role via Horizon or via the wmcs.vps.add_user_to_project cookbook. In some cases a cloud admin may add the user directly using 'openstack role add'. If a user is missing the 'reader' role, then it is likely that one of these processes is broken.
If a user has the reader role but is nonetheless not in the bastion project, it is likely that the keystone hooks have failed. Look in logstash for keystone errors, especially including the string 'wmfkeystonehooks'. The code for the keystone hook itself is in puppet, under modules/openstack/files/<current openstack version>/keystone/wmfkeystonehooks/
Old incidents
Add your incident here: