Nova Resource:Integration/Setup

Roles

integration-agent-{type}-XXXX

Updated September 2019 based on T226233 Updated January 2021

The instances are created via https://horizon.wikimedia.org/project/instances/ , you will need a source image to pick and an instance flavor.

• Source: pick the debian-11.0-bullseye image now that T252071 is complete

For the flavor the important parts are:

• have enough disk space (docker role notably requests 24G for /var/lib/docker and you would need enough disk remaining for /srv).
• have a 4xiops flavor which dramatically boost the underlying Disk IO rate limiting applied to all WMCS instances.

• Flavor: pick g3.cores8.ram24.disk20.ephemeral90.4xiops

• Create a new instance named integration-agent-{type}-XXXX where {type} is a role (example: docker) and XXXX increments starting from 1001.

Wait a few minutes (during which the instance is created, initial setup happens). Then connect to the instance over SSH and fix puppet:

• sudo rm -fR /var/lib/puppet/ssl && sudo puppet agent -tv
• If that complains:
  • get the instance fully qualified domain name (FQDN): hostname --fqdn
  • On integration-puppetmaster-02.integration.eqiad.wmflabs, clean the old and invalid certificate(s): sudo puppet cert clean <FQDN OF INSTANCE HERE

Apply the Puppet role:

• https://horizon.wikimedia.org/project/instances/
• Click the instance then head to the tab Puppet
• Pick role::ci::slave::labs::docker
• For instances with a 90G ephemeral disk, set the Hiera value profile::ci::dockervolume::size: '45G'

The Docker agent will have a 24G (or 45G) /var/lib/docker partition, the remaining disk space is allocated to /srv (LVM 100%FREE allocation).

Run Puppet on the instance (puppet agent -tv) and verify:

• If a Docker agent, make sure there is a /var/lib/docker partition for Docker
• Clean unused packages: apt-get autoremove --purge
• Upgrade packages: apt-get -y dist-upgrade

Reboot the instance (Before adding to Jenkins). This cleans state, take in account the new Linux kernel if any, launches daemons. Once it is back, you can then add it to Jenkins

Add the instance to Jenkins

1. Create "New Node" in Jenkins management
   • Name: (short hostname of instance)
   • Type: Permanent Agent
   • Executors: 1 (for Docker agents: 4, for Qemu agents: 1)
   • Remote root directory: /srv/jenkins
   • Labels:
     • For Docker agents: Docker
     • For Qemu agents: Qemu
   • Usage: EXCLUSIVE (Only build jobs with label restrictions matching this node)
   • Launch method: SSH
     • Host: (internal IP of instance)
     • Credentials: jenkins-deploy (key to connect to labs instances set up with role::ci::slave::labs::common)
     • Host Key Verification Strategy: Manually trusted key Verification Strategy
   • Availability: Always (Keep this slave on-line as much as possible)

The Jenkins master will automatically trust the ssh key upon the first connection.

integration-dev

1. Create instance:
   • m1.medium
   • Security group: Default
2. Wait 10 minutes
3. Reconfigure instance from wikitech: Enable role::ci::slave::labs.
4. Via SSH, force a puppet run (applies role).

Utilities

puppet

This page may be outdated or contain incorrect details. Please update it if you can.

Use sudo /usr/local/sbin/puppet-run &. Don't use sudo puppet agent -t, because that is not what cron uses and leads to inconsistencies with e.g. umask and other factors affecting default values used at runtime.