Jump to content

Data Platform/Systems/Presto/Query Logger

From Wikitech

Currently work in progress. Will be as specified in https://phabricator.wikimedia.org/T269832.

Repository is at https://github.com/razzius/wmf-presto-query-logger

We want to get the information out of the `tasks` table:

presto:runtime> describe tasks;
         Column          |   Type    | Extra | Comment
-------------------------+-----------+-------+---------
 node_id                 | varchar   |       |
 task_id                 | varchar   |       |
 stage_execution_id      | varchar   |       |
 stage_id                | varchar   |       |
 query_id                | varchar   |       |
 state                   | varchar   |       |
 splits                  | bigint    |       |
 queued_splits           | bigint    |       |
 running_splits          | bigint    |       |
 completed_splits        | bigint    |       |
 split_scheduled_time_ms | bigint    |       |
 split_cpu_time_ms       | bigint    |       |
 split_blocked_time_ms   | bigint    |       |
 raw_input_bytes         | bigint    |       |
 raw_input_rows          | bigint    |       |
 processed_input_bytes   | bigint    |       |
 processed_input_rows    | bigint    |       |
 output_bytes            | bigint    |       |
 output_rows             | bigint    |       |
 physical_written_bytes  | bigint    |       |
 created                 | timestamp |       |
 start                   | timestamp |       |
 last_heartbeat          | timestamp |       |
 end                     | timestamp |       |
(24 rows)