Jump to content

Metrics Platform/How to/Validate Events

From Wikitech

During active development, it is important to be able to test that events can be validated by EventGate against a specified JSONSchema. Each of the client libraries contain unit and integration tests for ensuring that events are properly formatted and serialized for submission. However testing end-to-end is a valuable exercise in instilling confidence that events will be produced to Kafka and ensuing data is able to be consumed.

The basic process for testing end-to-end requires having a local EventGate instance. See Metrics_Platform#Quick_Start for development environment recommendations.

Javascript

Because the Javascript client library is provided through the MediaWiki EventLogging extension, having an instance of MediaWiki running in a local development environment is required. Once a change set is submitted as a patch to the EventLogging extension (presumably after corresponding changes have been merged in the Javascript client library), the following steps should be followed: 

$wgEventStreams = [
  'test.metrics_platform.interactions' => [
    'schema_title' => '/analytics/product_metrics/web/base',
    'destination_event_service' => 'eventgate-analytics-external',
    'producers' => [
      'metrics_platform_client' => [
        'provide_values' => [
          'performer_is_logged_in',
          'mediawiki_skin',
        ],
      ],
    ],
  ],
];

// When $wgEventLoggingStreamNames is false (not falsy), the EventLogging
// JavaScript client will treat all streams as if they are configured and
// registered.
$wgEventLoggingStreamNames = [ 'test.metrics_platform.interactions' ];

mw.eventLog.submitInteraction('test.metrics_platform.interactions', '/analytics/product_metrics/web/base/1.0.0', 'init');
  • Observe that an event has been sent to, validated and accepted by the local EventGate instance.

Source: https://phabricator.wikimedia.org/T351293#9337205

PHP

The exact same process as for Javascript can be used.

Java

While the Java client library includes end-to-end integration testing from reading a local stream config data fixture to building a metrics client to submitting events and then stubbing expected responses, it does not ultimately provide a true end-to-end testing experience wherein production config is read and the metrics client submits an event to EventGate whereupon it is validated and accepted.

There are a few ways to confirm the validation of events. One method can be done with a local EventGate instance. Another can be accomplished by utilizing the EventStreams Beta API.

Validate using local EventGate

To emulate a true end-to-end testing scenario locally, we can create a test that instantiates a metric client which fetches production stream configs (from the meta api) and submits an event using a production stream associated with a production schema and have the event validated with a local EventGate instance.

By using production config, we can submit an event using an example stream:

@Test void submitEventTimerStreamConfig() throws IOException, InterruptedException {
        // Create the metrics client.
        MetricsClient testMetricsClient = MetricsClient.builder(testClientData).build();
        await().atMost(10, SECONDS).until(testMetricsClient::isFullyInitialized);

        testMetricsClient.submitInteraction(
                "android.metrics_platform.find_in_page_interaction",
                DataFixtures.getTestClientData(getExpectedEventClick()),
                DataFixtures.getTestInteractionData("TestClick")
        );
        Thread.sleep(10_000);
        await().atMost(30, SECONDS).until(() -> testMetricsClient.getEventQueue().size() == 0);
    }

In the example above, the MetricsClient::submitInteraction() method takes the sample production stream name as a parameter along with some test ClientData and test InteractionData. The default schema id is set as the app base schema inside the library.

Once the metrics client has fetched the production stream configs, it batch processes the event queue to send serialized events to the DestinationEventService which in this case should be set as a local EventGate instance.

Because the sample production stream android.metrics_platform.find_in_page_interaction adds only a few requested values in its producer config, we can prevent ContextController:enrichEvent() (which enriches the event with configuration-specified contextual values) from filtering the ClientData and simply have all available contextual values sent with the event by commenting out the lines the method that resets the ClientData.

The test above waits until the event queue is empty so the following method can be added to the MetricsClient class though it is not currently in the latest release:

public BlockingQueue<EventProcessed> getEventQueue() {
    return this.eventQueue;
}

Make sure local eventlogging is running and then run the test. Observe that the test event is sent to and validated/accepted by the local EventGate instance:

Source: https://phabricator.wikimedia.org/T351292#9347381

Troubleshooting

If one is using Docker for the local EventGate instance and has it running on port 8192, note that WireMock also runs on this port so lines related to WireMock need to be commented out as well. Otherwise try running a local EventGate on a different port.

Validate using EventStreams Beta API

To validate events using the EventStream API (Beta Cluster), client code needs to include the Metrics Platform Java library which should be installed locally as well. For example, using the Wikipedia Android app repository as the client code base, one can trigger the sending of events using their dev debug build (see Android's getting started documentation for local development).

The basic steps include:

  • Installing a local version of the Metrics Platform Java library with the EventStreams API Beta url.
  • Rebuilding the Android installation using the local MP Java installed version.
  • Setting up the Android emulator or hardware device for debugging.
  • Triggering events with known streams in production.
  • Observing the triggered event in the EventStreams UI.
Install a local MP Java library version

In order to see events streaming in real time, the MP Java client library needs to send events to the EventStreams API Beta intake service which is: 

https://intake-analytics-beta.wmflabs.org

Because DestinationEventService::ANALYTICS in the Java library is set with the production url, this value needs to be replaced in the library (and associated tests in order to compile/build) by the beta url. Replace https://intake-analytics.wikimedia.org in DestinationEventService::ANALYTICS with https://intake-analytics-beta.wmflabs.org and update the assertions in DestinationEventServiceTest::testDestinationEventServiceAnalytics() and DestinationEventServiceTest::testDestinationEventServiceAnalyticsDev() with the base beta url.
Note that these changes should not be committed.

Once the updates are made in your local MP repo, build a local installation of the library by running the following command from the root MP directory: 

./mvnw -f java install

This will build the library locally and bump the next major version number with -SNAPSHOT appended. So if the current version of the Java library is 2.3, the local installed version will be 2.4-SNAPSHOT.

Rebuild the client code app with the local MP version

Once you have built a local version of your library with the updated DestinationEventService::ANALYTICS enum, go to your local version of the Android app and replace the value of metricsVersion in the build.gradle (Module app) file with the local installed version (i.e. 2.4-SNAPSHOT). Make sure that the active build variant of the Android app is devDebug.

Clean the project (from the Android Studio toolbar - Build > Clean Project), sync the Gradle files (icon: Sync Project with Gradle Files), then rebuild the project (from the Android Studio toolbar - Build > Rebuild Project).

Set up the Android emulator or hardware device for debugging

Assuming Android Studio is the IDE, one can run installations of the Wikipedia Android app using an emulator or a hardware device. You can verify that your local Gradle build is running the local snapshot version of the MP library dependency by navigating to File > Project Structure, and confirming that the Metrics Platform version matches the local installed version: 

org.wikimedia.metrics:metrics-platform:2.4-SNAPSHOT

Reinstall the app on your virtual or hardware device.
For virtual devices, you can run the app on the emulator.
For hardware devices, you can use either USB or Wi-Fi to run the app.

Trigger events

If you know which instruments are currently in production, you can perform the triggering action in the dev build of the Android app. As of this writing, there are 4 article-related instruments in production that collect event data related to clicking on preview links, interacting with the toolbar, interacting with the table of contents, and utilizing the Find in page feature. Performing any of the above actions will trigger the sending of events to the EventStreams API Beta UI.

Observe events

Prior to triggering the event, navigate to https://stream-beta.wmflabs.org/v2/ui/#/ and select the streams associated with the instruments you would like to test. Once you've selected your streams, click the Stream button and trigger the events in the dev build of the app. In the example of the article instruments above, by triggering an event, you should see the corresponding event data in the stream if they pass validation against their respective schema ids.

Troubleshooting

The Android documentation has excellent instructions on debugging.

For hardware devices, it is as simple as unplugging the USB cable of your device to eject it and re-plugging it in to see the device in Android Studio. It is important to reinstall the app on the device by clicking the Run 'app' icon at the top of the Android Studio interface every time you make changes to the library dependency or to the Android app itself.

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Metrics_Platform/How_to/Validate_Events&oldid=2214925"