Jump to content

User:Thcipriani/Scap3 Stagingdev

From Wikitech

Need a way to dev scap3 mediawiki deploys. We can't do it on beta without being disruptive. *Maybe* we can do it on staging. It has to happen somewhere.

Staging Setup

  1. Create Hiera pages for 3 hosts. The deployment server (Hiera:Staging/host/thcipriani-tin) should have the content:
    ---
    "classes":
        - role::deployment::server
    deployment_server: thcipriani-tin.staging.eqiad.wmflabs
    "scap::deployment_server": thcipriani-tin.staging.eqiad.wmflabs
    "scap::wmflabs_master": thcipriani-tin.staging.eqiad.wmflabs
    scap::dsh::scap_proxies:
        - thcipriani-proxy.staging.eqiad.wmflabs
    
    The proxy and appserver (Hiera:Staging/host/thcipriani-proxy and Hiera:Staging/host/thcipriani-mediawiki) should have this content:
    ---
    classes:
        - role::beta::mediawiki
        - role::mediawiki::appserver
    deployment_server: thcipriani-tin.staging.eqiad.wmflabs
    "scap::deployment_server": thcipriani-tin.staging.eqiad.wmflabs
    "scap::wmflabs_master": thcipriani-tin.staging.eqiad.wmflabs
    "scap::dsh::scap_proxies":
        - thcipriani-proxy.staging.eqiad.wmflabs
    
    Example hiera pages:
    1. Hiera:Staging/host/thcipriani-tin
    2. Hiera:Staging/host/thcipriani-proxy
    3. Hiera:Staging/host/thcipriani-mediawiki
  2. Create the 3 hosts:
    1. thcipriani-tin -- Will act as your deployment-host (tin)
    2. thcipriani-proxy -- Will act as your deployment-proxy (mwXXXX)
    3. thcipriani-mediawiki -- Will act as your deployment-target (mwXXXX)
  3. Finally, to use scap to deploy MediaWiki, we need to ensure that the dsh files exist, and keyholder is armed on thcipriani-tin.staging.eqiad.wmflabs
    $ sudo touch /etc/dsh/group/mediawiki-{installation,api-canaries,appserver-canaries}
    $ sudo sh -c 'printf "%s\n%s\n" "thcipriani-mediawiki.staging.eqiad.wmflabs" "thcipriani-proxy.staging.eqiad.wmflabs" > /etc/dsh/group/mediawiki-installation'
    $ sudo /usr/local/sbin/keyholder arm && sudo service keyholder-proxy restart
    
  4. You should be able to run a mediawiki deployment with just that

Developing Scap on Staging

Now that we have a few working MediaWiki appservers, we can test deployments to them using scap; however, to test a development version of scap, we must add a version that shadows the version installed on these dev boxes via apt. These instructions create a working scap clone on thcipriani-tin that can be deployed to thcipriani-proxy and thcipriani-mediawiki via scap.

deploy scap via scap

a.k.a scapception

Clone scap on thcipriani-tin and add to your $PATH:

$ sudo mkdir -p /srv/deployment/scap && sudo chown -R $(whoami):wikidev /srv/deployment/scap
$ git clone https://phabricator.wikimedia.org/diffusion/MSCA/scap /srv/deployment/scap/scap

Add a scap.cfg to your scap repo to deploy scap via scap:

/srv/deployment/scap/scap/scap/scap.cfg (yeah, it's ridiculous):

[global]
git_repo: scap/scap
dsh_targets: mediawiki-installation

Create the scap directory on the scap targets: thcipriani-mediawiki and thcipriani-proxy:

$ sudo mkdir -p /srv/deployment/scap && sudo chown -R mwdeploy:wikidev /srv/deployment/scap

Create a virtualenv and install scap and dependencies:

$ sudo apt-get install python-pip python-dev
$ sudo pip install virtualenv
$ cd /srv/deployment/scap/scap
$ virtualenv .venv
$ . .venv/bin/activate
$ pip install -r requirements.txt
$ python setup.py install
$ echo '. /srv/deployment/scap/scap/.venv/bin/activate' >> "$HOME/.bashrc"

Now you can deploy scap to thcipriani-mediawiki and thcipriani-proxy by logging into thcipriani-tin and running scap deploy from /srv/deployment/scap/scap. You can then use scap in /srv/mediawiki-staging to test out changes to MediaWiki deploys.

Meta Notes

Had to add this this to Hiera:Staging to ensure that we could crate deployment servers.

mediawiki_memcached_servers:
    - 127.0.0.1:11211:1
"mediawiki::redis_servers::eqiad":
    shard01:
        host: 127.0.0.1
        port: 6379
"scap::deployment_server": scaptest.staging.eqiad.wmflabs
"scap::wmflabs_master": scaptest.staging.eqiad.wmflabs
"role::mediawiki::webserver::pool": one-pool-to-rule-them-all