Spicerack

Wikimedia infrastructure

Data centers

Networking

Global traffic routing

MediaWiki SRE

MediaWiki Engineering

Multimedia

Data Engineering

Search

SRE Data Persistence

SRE Infra Foundations

• CAS-SSO
• Conftool
  • Dbctl
• Cumin
• Debian packaging
  • APT repository
  • DebMonitor
  • Reprepro
• Ganeti
• Puppet-compiler
• Homer
• irc.wikimedia.org
• Netbox
• Network Error Logging
• Spicerack and cookbooks

SRE Observability

Wikimedia Performance

Event Platform

Release Engineering

[]

Spicerack is a Python library to orchestrate tasks in the Wikimedia Foundation production environment. It comes with an easy API and a cookbook entry point script that allows to write simple Cookbooks to automate and orchestrate tasks.

Documentation

See the automatically generated Spicerack documentation.

Explore Spicerack

To explore spicerack features when writing cookbooks, perform some read-only investigation and occasionally run that one-off command that might be needed in some occasions, there is a spicerack-shell binary available on all cumin (cumin1002.eqiad.wmnet, cumin2002.codfw.wmnet ) and cloudcumin hosts.

Do not abuse the usage of spicerack-shell. Is not meant to be used to perform regular read-write actions on the infrastructure, use cookbooks for that.

$ sudo spicerack-shell -h
usage: spicerack-shell [-h] [--live]

Interactive Spicerack Shell

Opens a Python REPL shell with the same spicerack instance available in the cookbooks.

optional arguments:
  -h, --help  show this help message and exit
  --live      Sets Spicerack with DRY-RUN=False and allow to make live changes to the production systems. (default: False)

When executed, it will drop the user in a Python REPL shell with a spicerack object like the one available in the cookbooks initialized and with the DRY-RUN mode set or not based on the command line arguments:

$ sudo spicerack-shell
##### SPICERACK INTERACTIVE SHELL #####
#
# This interactive shell allows to test and use specific Spicerack bits in the same way they are
# available inside the Cookbooks.
#
# There is one available object, spicerack, that is an instance of spicerack.Spicerack like the
# one available inside the cookbooks.
# By default the shell will setup Spicerack with DRY-RUN set to True. If --live is passed the
# DRY-RUN flag will be set to False and the instance will actually modify production systems hence
# BE CAREFUL when using it.
# The shell has the default Python's autocompletion and is possible to get help on objects, e.g.:
#
#    help(spicerack)
#
# The Spicerack documentation is available at:
#
#   https://doc.wikimedia.org/spicerack/master/api/index.html
#
# The logging for the Spicerack modules is already set up and will log into:
#
#   /home/YOUR_USERNAME/cookbooks_testing/logs/interactive
#
# Like the cookbooks there are two log files, one at INFO level and the other at DEBUG level.
#
# Example usage:
#
#    >>> hosts = spicerack.remote().query("A:sretest")
#
# DRY-RUN = True
#
>>>

How to contribute

Bug fixes and small feature updates

For any trivial changes simply send a Gerrit request for review. Context is always in demand, so we'll greatly appreciate a Phabricator task with the SRE-Tools tag and include some background that would help us understand your request better.

In case of any uncertainty or questions, feel free to contact the SRE Infrastructure Foundations team.

Adding new module or change in core behaviour

For any bigger changes like adding a new module or changing core behaviour, please create a Phabricator task with Spicerack tag documenting your proposal.

The ideal proposal would include

• A problem statement

  Here please detail the benefits that the proposed changes will bring or the issues they will assist with.

  In case of a new or refactored module, describe the high level API definition including:

  • Any new data structures
  • All affected public methods signatures with type hints to clearly describe parameters and return types
  • The spicerack accessor(s) to be added/modified to spicerack.Spicerack (how the cookbook will access the module features)

• Third party dependencies

  Any additional third party dependency that would be required with version constraints and if they are available as Debian packages.

• Additional configuration

  Any additional configuration file and its structure.

Alternatively

At an early stage of a project, you might not be sure about API structure, dependencies and configuration needed - that’s fine, in such a case just create a task with a problem statement and we can backfill the proposal template as we work through the solution together.

This will allow the SRE Infrastructure Foundations team to provide early feedback and guidance on where best in the Spicerack source tree the code should live, or if the code fits more logically in a different library such as wmflib.

We can also provide feedback on any current Spicerack interfaces which may assist with the implementation, either in their current form or with minor refactoring.  Finally it allows the SRE Infrastructure Foundations team to provide guidance on how best to organise the module to ensure its interface is consistent with the current Spicerack modules.

Automation and Tooling office hours

Further to this all are welcome to attend the Automation and Tooling office hour to discuss new ideas or specific proposals. Please ask us on IRC for more details and an invite to the office hours meeting.

Test newly released Spicerack features

In order to test changes included in a new Spicerack release, upgrade the spicerack package on a single host (usually cumin2002) and then use spicerack-shell to test in isolation the newly released features.