Portal:Cloud VPS/Admin/OpenTofu

Tracked in Phabricator
Task T370037

We use OpenTofu to manage some cluster-wide OpenStack resources using the code in gitlab:repos/cloud/cloud-vps/tofu-infra.

Source of truth

As of this writing, the tofu-infra repo is the source of truth for a number of admin-controlled openstack elements, for example (this list is likely outdated now):

nova flavors
neutron networks, subnets, routers and routers ports
projects
DNS zones, and some DNS records

Any changes to these items should be made via tofu-infra.

Usage

We are improving tofu workflows as we increase usage, and identify patterns, etc.

Automated workflow via cookbook

This cookbook-based procedure will do everything you need. Please note that by default the cookbook operates on all deployments/regions (i.e, eqiad1-r and codfw1dev-r).

To apply the latest configuration in the repository (main branch):

user@cloudcumin1001:~ $ sudo cookbook wmcs.openstack.tofu --apply

To plan the changes associated with a given gitlab MR:

user@cloudcumin1001:~ $ sudo cookbook wmcs.openstack.tofu --gitlab-mr 27 --plan

Only plan a MR on a given region:

user@cloudcumin1001:~ $ sudo cookbook wmcs.openstack.tofu --gitlab-mr 27 --plan --cluster-name codfw1dev

Manual workflow

This procedure describes the steps required to run opentofu manually for a given openstack deployment.

Log in to a cloudcontrol on the deployment you want to run tofu on
Run Puppet agent (to pull latest changes from the Git repo)
$ cd /srv/tofu-infra
$ sudo tofu plan
$ sudo tofu apply

Automated workflow via gitlab CI

There is some gitlab CI integration. But at the time of this writing, it just does basic validation and linting. Therefore gitlab CI is not very involved in the workflow at this very moment.

However, having a workflow automated via gitlab CI is probably the desired end state.

Setup

There's a dedicated service account that OpenTofu authenticates with. The password for this account is in cloudvps-tofu-admin-account pwstore file.

That account has full OpenStack access to the default domain:

$ sudo wmcs-openstack role add --domain default --inherited --user tofuadmin admin

Then, the tofu binary is actually a thin wrapper to load credentials from /etc/tofu.env, something like this:

export AWS_ENDPOINT_URL_S3="https://object.codfw1dev.wikimediacloud.org"
export AWS_REGION="codfw1dev-r"
export AWS_ACCESS_KEY_ID="SOMETHING"
export AWS_SECRET_ACCESS_KEY="SOME-SECRET-KEY"
export OS_CLOUD="tofu"
export OS_REGION_NAME="codfw1dev-r"
export TF_VAR_cloudvps_region="codfw1dev-r"

cookbook setup notes

The wmcs.openstack.tofu cookbook uses a gitlab token to be able to write comments to a merge request.

The token is valid for 1 year.

To regenerate:

go to https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/settings/access_tokens
generate a new token with api scope and reporter role, valid for 1 year
put the token in the the puppetserver secret git repo, in file hieradata/role/common/cluster/cloud_management.yaml hiera key profile::wmcs::gitlab_tokens::tofu_infra

Known problems

We are in the initial stages of adopting this IaC solution for Cloud VPS. There are some rough edges that are documented here.

radosgw endpoint 500 error

Happens from time to time for unkonwn reasons. Just try the tofu operation again.

DNS records of type NS in root zone cannot be updated

When updating the NS recordset of a root zone, the openstack designate API won't let a POST action update the data. You will need to delete the record before running tofu, which will then create it from scratch.

DNS records cannot be imported

This is a bug in the upstream provider, see https://github.com/terraform-provider-openstack/terraform-provider-openstack/pull/1778

The patch is already in the main branch, but we will need to wait for the next release before using the import feature.

As a workaround, delete the records by hand before running tofu, which will then create them from scratch and track them in the state.