Nova Resource:Dumps

Project Name	dumps
Details, admins/members	openstack-browser
Monitoring	grafana SAL

Dumps

Description

This is a project that archives the public datasets generated by Wikimedia.

Purpose

Archive the public Wikimedia datasets.

Anticipated time span

indefinite

Project status

currently running

Contact address

https://groups.google.com/forum/#!forum/wikiteam-discuss

Willing to take contributors or not

not willing

Subject area narrow or broad

broad

Project information

Introduction

This project was created to provide a dedicated space just for transferring Wikimedia dump files to the Internet Archive. These dumps were created as a possible backup in the case of cluster-wide hardware failure, and its also often used by researchers/bots. Sometimes, these files are generated for forking of any Wikimedia project, when lots of people of a project has different aims from the original Wikimedia goal.

More information about the archiving process is available at Nova Resource:Dumps/Archive.org

Data currently being archived

Here are some information and links regarding the data that this project is archiving:

• Wikimedia main database dumps
• Wikimedia incremental dumps
• Wikidata JSON dumps
• Wikimania videos
• OpenStreetMap datasets

Servers

• dumps-N (where N is an integer): Main archiving servers
• dumps-stats: Wikimedia data manipulation, including dumps above and other stuff of relevance for Wikimedia research.

Storage:

• Before the eqiad migration we used to have a 900 GB quota (hardly sufficient for comfortable work).
• Currently all heavy operations are conducted on /data/scratch/. We currently keep to a soft limit of using only 3 TB of space, but such disk usage is always temporary and will be deleted once the data is pushed to the Archive.
• Everything is retained locally only for very short periods, just the time needed for packing on archive.org.

Links

Our resources:

• Main WikiTeam repository
• Balchivist repository (main archiving infrastructure)

Wikimedia data:

• Wikimedia dumps server
• Wikimedia Downloads collection on the Internet Archive

OSM data:

• OpenStreetMap datasets
• OpenStreetMap data collection

Edit documentation

Server admin log

2024-08-05

• 08:52 dcaro: restarted kiwix-mirror-update, was stuck on name resolution (works now)

2024-06-25

• 13:35 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1)
• 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs

2024-06-18

• 10:48 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1)
• 10:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs

2023-09-29

• 09:19 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs... (more)