Analytics/Archive/Data/Mobile requests stream
The Mobile requests stream is holding request logs from the mobile caches.
This stream is owned by the Analytics Team.
Contained data
The contained data only covers mobile caches, but that does not mean, that they cover all mobile traffic. For example mobile traffic to upload, bits, or api caches are not covered by this stream.
Availability
stat1002.eqiad.wmnet /a/squid/archive/mobile
The stream is available in Cache log format with a sampling rate of 1:100 as gzipped files at /a/squid/archive/mobile/mobile-sampled-100.tsv.log-*.gz on stat1002.eqiad.wmnet.
The date in the file name does not mean that all logs of that day are in that file. Instead, the files contain logs from ~06:30 of the previous day until to ~06:30 of the day in the file name. So for example /a/squid/archive/mobile/mobile-sampled-100.tsv.log-20130930.gz contains data from ~2013-09-29T06:30.00 until ~2013-09-30T06:30:00.
| Avg. size / gzipped file | 258 MiB |
|---|---|
| Avg. size / uncompressed file | 1537 MiB |
| Avg. lines / uncompressed file | 3705 K |
| Avg. lines / second | 43 |
| Avg. requests / second | 4288 |
This stream gets used for:
- adhoc research
Events and known problems since 2013-09-01
| Date from | Date until | Bug | Details |
|---|---|---|---|
| * | Only covers data from mobile caches, not all mobile traffic. | ||
| * | The stream may suffer from packet drop on udp2log. This should be <5%. | ||
| * | âzeroâ markers got set not only for wikipedia, but also for sister projects (wiktionary, ...) | ||
| * | Lines that would be longer than ~8K get chopped off at that border (no newline gets added). (Affects <1 line/day on average) | ||
| * | bug 60315 | The stream does not contain the SSL requests that come to the SSL terminators, but only forwarded ones from the terminators. | |
| * | 2013-09-26 | bug 53806 | Until around 2013-09-26 ~22:57, the client ip might have been garbled. |
| 2013-09-26 | 2013-10-01 | bug 54779 | No âmf-mâ markers in stream between 2013-09-26 ~22:44 and 2013-10-01 ~13:33. |
| 2013-11-13 | 2013-12-16 | bug 58764 | <40 lines/day have been concatenated due to puppet runs unnecessarily restarting the udp2log filtering. First occurrence on 2013-11-13T17:29:22. Last occurrence on 2013-12-16T17:36:17. |
| 2013-12-18 | n/a | bug 58889 | Increase in zero=470-01 (Grameenphone Bangladesh) tagged traffic, due to the advertisement by the carrier |
| 2014-01-05 | 2014-01-06 | bug 59722 | Udp2log relay went down. There are no log lines between 2014-01-05T03:39:25 and 2014-01-06T17:45:10. |
| 2014-03-21 | 2014-03-21 | bug 62922 | Sometimes zero tags are doubled like âzero=250-99;zero=250-99â. The first occurrence is on 2014-03-21T00:16:52. Last occurrence is on 2014-03-21T17:13:22. |
| 2014-04-18 | 2014-04-22 | bug 64145 | Requests from cp301[34] are missing. That is ~50% of esams mobile traffic, and ~20% of overall mobile traffic. Missed requests start ~2014-04-18T19:02:16 and stop ~2014-04-22T17:53:09. |
| 2014-05-22 | 2014-06-24 | bug 66833 | Zero tags need not have a trailing characters stripped (like âzero=404-01bâ instead of âzero=404-01â). Last occurrence is on 2014-06-24T15:24:16. |
| 2014-07-08 19:00 | 2014-07-08 22:00 | bug 67694 | A 2014 FIFA World Cup (soccer) related traffic spike caused udp2log overload and lead to up to ~10% packetloss during this period of time. |
| 2014-07-09 09:00 | 2014-07-10 09:00 | bug 68199 | Traffic has been rerouted from ulsfo to eqiad for ULSFO floor move. No data has been lost, but host column may show eqiad caches for traffic that could be expected to go to ulsfo. |
| 2014-07-13 19:00 | 2014-07-13 23:00 | bug 67694 | A 2014 FIFA World Cup (soccer) related traffic spike caused udp2log overload and lead to up to ~25% packetloss during this period of time. |
| 2014-07-25 ~14:00 | 2014-07-25 ~17:00 | bug 69112 | Carrier 250-99 was not properly zero tagged. |
| 2014-07-29 01:35 | 2014-07-29 01:42 | bug 68796 | cp3013, cp3014 (half of esams) missing between 2014-07-29T01:35:45 and 2014-07-29T01:42:00 due to flapping network link (~5% of total mobile traffic around that time) |
| 2014-07-30 ~00:54 | 2014-08-04 ~21:00 | bug 69112 | Carrier 250-99 was not properly zero tagged, and some of the carrier's requests came with zero=ON instead. |
| 2014-08-16 ~22:43 | 2014-08-16 ~22:49 | bug 69663 | Root mount on oxygen went full, which caused services to panic and udp2log dropped requests during that time |
| 2014-08-17 ~06:26 | 2014-08-17 ~06:30 | bug 69663 | Root mount on oxygen went full again, which caused services to panic and udp2log dropped requests during that time |
| 2014-10-08 22:00 | 2014-10-08 24:00 | bug 71879 | ULSFO having connectivity issues leading to partial message loss |
| 2014-10-20 13:06 | 2014-10-20 13:27 | bug 72306 | ULSFO connectivity issues causing packet loss between 6% and 47% for ulsfo caches. |
| 2014-10-21 ~10:30 | 2014-10-21 ~11:43 | bug 72355 | Ulsfo connectivity issues causing packet loss for ulsfo caches. |
| 2014-11-30 ~03:50 | 2014-11-30 ~10:13 | task T76334 | No data while analytics infrastructure suffered eqiad network issues. |
| 2015-01-13 ~22:20 | 2015-01-13 ~23:18 | task T86973 | No data due to firewall problems |
stat1002.eqiad.wmnet /a/log/webrequest/archive/mobile
The stream is available in Cache log format with a sampling rate of 1:100 as gzipped files at /a/log/webrequest/archive/mobile/mobile-sampled-100.tsv.log-*.gz on stat1002.eqiad.wmnet (using kafka as backend).
Each file covers the full day of the date in the file name.
Events and known problems since 2015-01-01
| Date from | Date until | Bug | Details |
|---|