Swift/Deploy Schedule - Originals

This page details the steps necessary to get originals running in swift, with more detail around the actual cutover process.

Overall Plan

These steps are in the rough order of execution but is not strictly ordered.

increase the size of the swift cluster with new hardware (waiting on new racks to get set up)
deploy mediawiki code to write a log of all media changes ('changelog') to use to synchronize backend stores DONE
deploy configuration to enable the media changelog DONE
rewrite cloudfiles to allow parallel purging DONE
FileBackend based thumbnailer (more detail) DONE
- aka make mediawiki write thumbnails to swift instead of the 404-handler rewrite module DONE
start process to migrate all data from ms7 to swift (initial snapshot) (eta start time week of 5/1)
write apache rewrite rule to allow image scaler apaches to handle upload.wikimedia style URLs (to let us take ms5 out of the loop)
wait for data migration/population into swift to complete
cut over to swift as the primary backend store

Things not in the above list that we should do

test originals in swift in labs
create and test (in labs) rollback plan

Things not in the above list that don't actually block originals but it would be nice if they happened first

mirror swift cluster to eqiad
upgrade swift version

Mediawiki Writes Thumbnails

At the moment, the image scalers write thumbnails to NFS and swift's rewrite module writes thumbnails into swift. The goal for this change is to have Mediawiki write the thumbnails into swift directly in addition to NFS. ms5 will stay in the image request path. A new config parameter is added to rewrite to allow us to choose to write thumbnails based on the container.

Plan is to make the following changes for a few wikis Thursday 5/18 and the rest of them Tuesday 5/20:

adjust rewrite.py config (in puppet); set write_thumbs to 'most' and the list to the containers for which MW will write the thumbs
adjust MW config to list the containers (wikis) for which it will write thumbs
test!