Search Platform/Weekly Updates/2024-09-13
Appearance
Summary
Now that WDQS graph split endpoints are public, we are seeing some discussions and user experimentation with those endpoints. Big thanks to David for following up on questions from our communities!
Elasticsearch is moving back to an open source license. We are reviewing our plan to migrate to OpenSearch in light of this licensing change. Both OpenSearch and a recent version of Elasticsearch would provide support for vector search, and a modern and supported software stack. We'll need time to let things settle, understand the implications of the new licensing of Elasticsearch, and reach a decision on which stack is the right one in our context.
What we've accomplished
WDQS graph splitting
- Answering questions on wiki (https://www.wikidata.org/w/index.php?title=Special%3AContributions&target=DCausse+%28WMF%29&namespace=all&tagfilter=&start=&end=)
- New property proposal created by Charles https://www.wikidata.org/wiki/Wikidata:Property_proposal/publication_type_of_scholarly_article
Search backend replacement
- Awaiting licensing feedback. Wrote high level checklist, tasks will need to be created. https://phabricator.wikimedia.org/T370148
Misc
- Add flag revision_based to page_weighted_tags_changed schema - https://phabricator.wikimedia.org/T372914
- Started to investigate why the wdqs-updater misbehaved after kafka-main@codfw node replacements, suspecting some weird behaviors of the transactional kafka producer and the way we re-used the same broker.id on two different nodes - https://phabricator.wikimedia.org/T374628
- Some work to cleanup wdqs in puppet (better separation of the categories endpoint)
- Small patch to add support for SSL in the wdqs updater (long overdue)
- Data platform industry landscape slide preparation