Jump to content

Search Platform/Weekly Updates/2024-09-13

From Wikitech

Summary

Now that WDQS graph split endpoints are public, we are seeing some discussions and user experimentation with those endpoints. Big thanks to David for following up on questions from our communities!

Elasticsearch is moving back to an open source license. We are reviewing our plan to migrate to OpenSearch in light of this licensing change. Both OpenSearch and a recent version of Elasticsearch would provide support for vector search, and a modern and supported software stack. We'll need time to let things settle, understand the implications of the new licensing of Elasticsearch, and reach a decision on which stack is the right one in our context.

What we've accomplished

WDQS graph splitting

Search backend replacement

Misc

  • Add flag revision_based to page_weighted_tags_changed schema - https://phabricator.wikimedia.org/T372914
  • Started to investigate why the wdqs-updater misbehaved after kafka-main@codfw node replacements, suspecting some weird behaviors of the transactional kafka producer and the way we re-used the same broker.id on two different nodes - https://phabricator.wikimedia.org/T374628
  • Some work to cleanup wdqs in puppet (better separation of the categories endpoint)
  • Small patch to add support for SSL in the wdqs updater (long overdue)
  • Data platform industry landscape slide preparation