OpenStreetMaps Extract Transform Load (ETL) procedure

The WMF maps stack use OSM to render static and dynamic maps. Imposm3 is the tool behind the ETL procedure that loads data into a PostgreSQL DB schema that conforms with the maps applications' needs.

Database requirements

Postgres 11
PostGIS 3.1+ (this version allows the use of the function ST_GeomAsMVT to generate vector tiles with Tegola)

Imposm3 mapping and DB schema

The database schema is defined in a configuration file that is available in the operations/puppet repository.

Import OSM data

In order to import OSM data, we have a production-ready script that needs to be executed with root access and the latest planet dump from OSM, with a caveat, always choose a specific revision to execute the script instead of the one tagged with latest, here is an example of the command to be executed:

 sudo -s
 screen -DR # always execute with screen because the script takes several hours to finish
 imposm-initial-import \
   -d 210906 \
   -x webproxy.eqiad.wmnet:8080

The script will take care of the whole procedure, but if you want to know how the process works and execute it locally, please refer to the guide present in maps-config

Rendering Maps

Layers Functions

As part of the Maps 2.0 work, it was introduced the concept of SQL functions to query the layers of the map. See the SQL functions that can be accessed.

Updating Layers Functions

When changing a SQL function behavior that introduces significant changes or needs to be easily rolled-back, we decided to add version suffix in the function name and updating the applications usage of the functions accordingly. This means that, the next changes in public.layer_admin will be applied in public.layer_admin_v2.