Jump to content

Nova Resource:Dumps

From Wikitech
Project Name dumps
Details,
admins/members
openstack-browser
Monitoring

Dumps

Description

This is a project that archives the public datasets generated by Wikimedia.

Purpose

Archive the public Wikimedia datasets.

Anticipated time span

indefinite

Project status

currently running

Contact address

https://groups.google.com/forum/#!forum/wikiteam-discuss

Willing to take contributors or not

not willing

Subject area narrow or broad

broad

Project information

Introduction

This project was created to provide a dedicated space just for transferring Wikimedia dump files to the Internet Archive. These dumps were created as a possible backup in the case of cluster-wide hardware failure, and its also often used by researchers/bots. Sometimes, these files are generated for forking of any Wikimedia project, when lots of people of a project has different aims from the original Wikimedia goal.

More information about the archiving process is available at Nova Resource:Dumps/Archive.org

Data currently being archived

Here are some information and links regarding the data that this project is archiving:

  • Wikimedia main database dumps
  • Wikimedia incremental dumps
  • Wikidata JSON dumps
  • Wikimania videos
  • OpenStreetMap datasets

Servers

  • dumps-N (where N is an integer): Main archiving servers
  • dumps-stats: Wikimedia data manipulation, including dumps above and other stuff of relevance for Wikimedia research.

Storage:

  • Before the eqiad migration we used to have a 900 GB quota (hardly sufficient for comfortable work).
  • Currently all heavy operations are conducted on /data/scratch/. We currently keep to a soft limit of using only 3 TB of space, but such disk usage is always temporary and will be deleted once the data is pushed to the Archive.
  • Everything is retained locally only for very short periods, just the time needed for packing on archive.org.

Our resources:

Wikimedia data:

OSM data:

Server admin log

2024-08-05

  • 08:52 dcaro: restarted kiwix-mirror-update, was stuck on name resolution (works now)

2024-06-25

  • 13:35 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1)
  • 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs

2024-06-18

  • 10:48 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1)
  • 10:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs

2023-09-29

  • 09:19 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs... (more)