Parser cache

Wikimedia infrastructure

Data centers

Networking

Global traffic routing

MediaWiki SRE

MediaWiki Engineering

• Parser cache
• MediaWiki JobQueue
• Performance review
• performance.wikimedia.org
• Web Perf Hero award
• Guides:
  • Frontend best practices
  • Backend best practices
  • more…

Multimedia

Data Engineering

Search

SRE Data Persistence

SRE Infra Foundations

SRE Observability

Wikimedia Performance

Event Platform

Release Engineering

[]

The ParserCache at WMF is an instance of the MediaWiki parser cache that primarily uses a dedicated MariaDB cluster, with hot keys also residing in Memcached.

• 
  ParserCache in Memcached (tier 1 and tier 2).
• 
  ParserCache DB (tier 3). MediaWiki uses consistent hashing to distribute entries over the pc1, pc2, and pc3 sections.

Current configuration

• Retention: Defaults to 30 days (as of Dec 2023).
• MediaWiki config: It is configured in MediaWiki via the wgParserCacheType and wgObjectCaches configuration variables, which we set in wmf-config to a MultiWriteBagOStuff that reads and writes using both Memcached and a list of sharded MySQL hosts.

• Database pool: The list of DB hosts is managed via dbctl.
• Statistics: Parser Cache dashboard in Grafana

The Eqiad and Codfw data centers each have their own ParserCache databases. MediaWiki's ParserCache is designed to store values by page rather than revision. MediaWiki validates the value at runtime against the expected revision and touched timestamp. The page-based key naturally limits/updates storage, manages regeneration load, and allows serving of stale values during times of high load (PoolCounter). While the data centgers use best-effort MySQL replication between them, the values may (consistently) differ between data centers if both are valid and fresh. The purge action in MediaWiki bumps the page_touched timestamp in the core database, thus invalidating the ParserCache.

You can read more about how ParserCache and its garbage-collection script works in T282761 (2021).

History

Prior to 2011, the parser cache was solely memcached-based.

In 2011, we introduced a MySQL-based parser cache as second tier in addition to Memcached. See also the Disk-backed object cache initiative (Project pitch, and Project status) for details of the 2011 project, and e.g. the evaluation of Ehcache before choosing MySQL.

From 2020 to 2023, there was briefly a third tier in the form of a local on-host Memcached on each webserver (T264604).

In May 2021, ParserCache retention for talk pages was temporary reduced from 30 days to 10 days, and other pages were similarly reduced from 30 to 21 days due to reaching capacity limits (T280605, change 685181).

In Dec 2023, after adding additional database hosts to the ParserCache cluster, ParserCache retention defaults were raised back to 30 days for all pages (T280604).