User:triciaburmeister/Sandbox/requestctl

This page is currently a draft.
More information and discussion about changes to this draft on the talk page.

requestctl is a command-line tool to control the configuration that manages access and routing of web requests. Wikimedia SREs use this tool to throttle and block certain requests patterns in Varnish frontend, in our edge caching.

Get started

User guide

Identify problematic traffic patterns
Look for existing request pattern
Add a new pattern
Sync pattern objects to etcd
Define an action
List existing actions
Sync action objects to etcd
Enable / disable and commit action
Remove an object
Modify and sync any object

Quick links

Command-line reference
Relevant code repositories
Admin docs

TODO: styling of "requestctl" should be <code>requestctl</code> when it's a command, but just "requestctl" when referring to the tool itself.

Essential terminology

To configure IP ranges and rate-limiting/ban rules, requestctl uses a custom schema that defines three types of objects:

pattern objects describe specific patterns of an HTTP request.
ipblock objects group specific IP ranges into logical groups.
action objects describe an action to be performed on a request that matches specific combinations of patterns and ipblocks. Actions are what is enabled on varnish.
- haproxy_action objects are similar to action objects, but allow a different set of actions because of the capabilities of haproxy.

For more details and explanations, read the requestctl Overview.

User guide

For a full list of commands, see the Command line reference.

TODO: add links from each of the commands in the user guide section to the command line reference for that command

TODO: need to add sections relevant for IPblock objects

Identify problematic traffic patterns

Filter requests in turnilo and superset live data based on which requestctl actions match them
Inspect the current or potential impact of requestctl actions in turnilo/superset
Check varnish logs for matching requests

Look for existing request pattern

Files are in /srv/git/private/requestctl/request-patterns/req/.

# All patterns
requestctl get pattern
# Request a specific pattern
requestctl get pattern ua/requests

To list all actions to which a pattern is applied to:

requestctl find pattern

Add a new pattern

If no existing request pattern matches what you need, add a new one. A pattern object should describe, with good flexibility, the large majority of the characteristics you want to match in a request.

Each pattern has an associated “scope” tag. (TODO: "scope" is unclear, even when also referenced in the IPBlock overview). The fields of each record are:

method, the http method
request_body a regex to match in the http body. CURRENTLY UNSUPPORTED IN VARNISH.
url_path the path part of the url, will be used as a regexp
header an header name to match, using the regexp at header_value;
header_value the regexp to match the value of header to. If left blank when a header is defined, the pattern means “the header is not present”
query_parameter and query_parameter_value are a parameter and a regexp for the value of a query parameter to match. An empty value will be interpreted as “for any value”.

Sync pattern objects to etcd

After you create or modify a pattern object, sync to etcd so you can use the pattern in an action.

SSH to a puppetserver frontend, then run:

puppetserver1001:~$ sudo requestctl sync -g /srv/git/private/requestctl pattern

Define an action

Action objects define what happens to the traffic matching the pattern you defined. Should matching traffic be blocked, or rate limited? What HTTP status should be served? What message?

Actions are defined in YAML file(s) in the puppet private repo under /srv/git/private/requestctl/request-actions.

List all actions
Enable or disable actions
Modify an existing action

Action objects are associated to a specific cluster (cache-text or cache-upload at the time of writing) and have a name. Their fields are:

enabled boolean. If false, the pattern will not be included in VCL
sites a list of datacenters where to apply the rule. If empty, the rule will be applied to all datacenters.
cache_miss_only boolean. If false, the pattern will be applied also to cache hits.
comment a comment to describe what this action does.
expression a string describing the combination of patterns and ipblocks that should be matched. The BNF of the grammar is described in cli.Requestctl.grammar, but in short:
- A pattern is referenced with the keyword pattern@<scope>/<name>
- An ipblock is referenced with the keyword ipblock@<scope>/<name>
- Patterns and ipblocks can be combined with AND/AND NOT and OR/OR NOT logic and groups can be organized using parentheses.

Example valid expressions: ( pattern@ua/requests OR pattern@ua/curl ) AND ipblock@cloud/aws AND NOT pattern@site/commons

resp_status the http status code to send as a response
resp_reason the text to send as a reason with the response
do_throttle boolean to say if we should throttle requests matching the expression (true) on just respond with resp_status unconditionally (false)
throttle_requests, throttle_interval, throttle_duration are the three arguments of vsthrottle in VCL to control the rate-limiting behaviour.
throttle_per_ip boolean makes the rate-limiting per-ip rather than per-cache-server
log_matching if true, it will record in X-Requestctl if a request matches the rule. It will thus be included into the vcl objects even if disabled; it will just not perform any banning / ratelimiting action.

List existing actions

# All actions.
requestctl get action -o yaml
# A specific action
requestctl get action cache-text/generic_ua_clouds
# All enabled actions
requestctl get action -o json | jq 'to_entries[] | select(.value.enabled == true)'

Sync action objects to etcd

TODO: different docs have sync and commit commands in alternating orders; what is the correct order?

After you create or modify an action object, to sync it in the datastore, run sudo requestctl sync -g /srv/git/private/requestctl action.

Enable / disable and commit action

To actually get changes to an action injected into the varnish configuration, run enable or disable and then commit to the puppet private git repo (/srv/git/private):

# Writes to the datastore, needs sudo
sudo requestctl enable cache-text/generic_ua_clouds && sudo requestctl commit
sudo requestctl disable cache-text/generic_ua_clouds && sudo requestctl commit

Remove an object

If you're removing a pattern / ipblock, ensure it's not referenced by any action object (TODO: how, which commands).

To list all actions to which a pattern is applied to:

requestctl find pattern requestctl doesn't allow you to remove a pattern/ipblock if they’re still referenced in an action. (TODO:what error will i see?)

Remove the object file from the git repository
Commit the change to the puppet private git repo (/srv/git/private)
Run sudo requestctl sync --purge -g /srv/git/private/requestctl {pattern,ipblock,action}

Modify and sync any object

TODO: different docs have sync and commit commands in alternating orders; what is the correct order?

SSH to a puppetserver frontend
Modify the YAML file under /srv/git/private/requestctl
Commit the change to the puppet private git repo (/srv/git/private)
Run sudo requestctl sync -g /srv/git/private/requestctl {pattern,ipblock,action}

TODO: different docs have sync and commit commands in alternating orders; what is the correct order?

Commit the change to the puppet private git repo (/srv/git/private)
Run requestctl sync in the following order:

sudo requestctl sync -g /srv/git/private/requestctl pattern
sudo requestctl sync -g /srv/git/private/requestctl ipblock
sudo requestctl sync -g /srv/git/private/requestctl action

OR just run:

sudo requestctl sync -g /srv/git/private/requestctl {pattern,ipblock,action}
sudo requestctl commit

Tutorial: Add a new action in varnish

Let's say we want to throttle per ip requests that don't have an accept-encoding header, have Connect: keep-alive as a header, and go to a special page, coming from azure.

Get ipblocks

We already have the ipblocks from azure, originating from a cronjob running on the puppetservers, in the file requestctl/request-ipblocks/cloud/azure.yaml:

:~$ requestctl get ipblock -o json | jq -r 'keys[]'
abuse/blocked_nets
abuse/bot_blocked_nets
abuse/bot_posts_blocked_nets
abuse/phabricator_abusers
abuse/text_abuse_nets
cloud/akamai
cloud/aws
cloud/azure
cloud/digitalocean
cloud/gcp
cloud/linode
cloud/oci
cloud/public_cloud_nets
known-clients/googlebot

Look for request pattern

Now let's check if we have a request pattern that corresponds to not having an accept-encoding header:

:~$ requestctl get pattern
name                pattern
------------------  --------------------------------
req/cache_buster
req/cache_buster_q  ?q=\w{12}
req/specific_page
ua/urllib3          User-Agent: ^python-urllib3/.*$
ua/requests         User-Agent: ^python-requests/.*$
ua/curl             User-Agent: ^curl/.*$
ua/MediaWiki        User-Agent: ^MediaWiki/.*$
sites/commonswiki   Host: commons.wikimedia.org
sites/wikidata      Host: www.wikidata.org
sites/enwiki        Host: en.wikipedia.org
url/api             url:^/w/(api|rest).php
url/docroot         url:^/[?$]
url/page            url:^/wiki/
url/semicolon_page  url:^/wiki/.+:+

It doesn't look like it's the case!

Add a pattern

So let's add a file named /srv/git/private/requestctl/request-patterns/req/no_accept_encoding.yaml, with the following content:

header: 'Accept-Encoding'

Omitting any header_value this will translate to "no header present" (see the README, again (TODO)).

Sync to etcd

Now let's sync our objects to etcd:

puppetserver1001:~$ sudo requestctl sync -g /srv/git/private/requestctl pattern
2022-03-28 14:56:23,995 - reqctl (cli:_write:359) - INFO - Updating pattern ua/MediaWiki
2022-03-28 14:56:24,005 - reqctl (cli:_write:359) - INFO - Updating pattern ua/curl
2022-03-28 14:56:24,014 - reqctl (cli:_write:359) - INFO - Updating pattern ua/urllib3
2022-03-28 14:56:24,024 - reqctl (cli:_write:359) - INFO - Updating pattern ua/requests
2022-03-28 14:56:24,034 - reqctl (cli:_write:359) - INFO - Updating pattern sites/wikidata
2022-03-28 14:56:24,044 - reqctl (cli:_write:359) - INFO - Updating pattern sites/commonswiki
2022-03-28 14:56:24,054 - reqctl (cli:_write:359) - INFO - Updating pattern sites/enwiki
2022-03-28 14:56:24,064 - reqctl (cli:_write:359) - INFO - Updating pattern req/cache_buster
2022-03-28 14:56:24,073 - reqctl (cli:_write:359) - INFO - Updating pattern req/specific_page
2022-03-28 14:56:24,085 - reqctl (cli:_write:359) - INFO - Updating pattern req/cache_buster_q
2022-03-28 14:56:24,094 - reqctl (cli:_write:362) - INFO - Creating pattern req/no_accept_encoding
2022-03-28 14:56:24,103 - reqctl (cli:_write:359) - INFO - Updating pattern url/semicolon_page
2022-03-28 14:56:24,113 - reqctl (cli:_write:359) - INFO - Updating pattern url/api
2022-03-28 14:56:24,122 - reqctl (cli:_write:359) - INFO - Updating pattern url/docroot
2022-03-28 14:56:24,133 - reqctl (cli:_write:359) - INFO - Updating pattern url/page

(note that our object has been created).

Add another pattern

Now we can do the same with Connect: keep-alive, we'll create the file /srv/git/private/requestctl/request-patterns/req/keepalive.yaml containing:

header: Connect
header_value: keep-alive

...and sync again with the same command:

puppetserver1001:~$ sudo requestctl sync -g /srv/git/private/requestctl pattern

Write the action

Now that we have all the ingredients, we can write the action at /srv/git/private/requestctl/request-actions/cache-text/bot_from_azure.yaml

# This should tell anyone what this rule does
comment: "Throttle requests with keepalive but no accept-encoding, coming from azure."
# This is the default. For now add it.
enabled: false
# each pattern and ipblock is referred to using {pattern,ipblock}@<scope>/<name>
expression: pattern@req/keepalive AND pattern@req/no_accept_encoding AND ipblock@cloud/azure
# Only bother with cache misses
cache_miss_only: true
# We want to throttle individual ips
do_throttle: true
throttle_per_ip: true
# Allow 10 rqp per 10 seconds, and if exceeeded, ban for 1 minute
throttle_requests: 100
throttle_interval: 10
throttle_duration: 60

To sync your action object in the datastore, run requestctl sync -g /srv/git/private/requestctl action:

:~$ sudo requestctl get action cache-text/bot_from_azure -o yaml
cache-text/bot_from_azure:
  cache_miss_only: true
  comment: Throttle requests with keepalive but no accept-encoding, coming from azure
  do_throttle: true
  enabled: false
  expression: pattern@req/keepalive AND pattern@req/no_accept_encoding AND ipblock@cloud/azure
  resp_reason: ''
  resp_status: 429
  sites: []
  throttle_duration: 60
  throttle_interval: 10
  throttle_per_ip: true
  throttle_requests: 100

Check your new action

This won't show up now in varnish, but you can still check the action you created. Use the requestctl vcl command:

puppetserver2001:~$ requestctl vcl 'cache-text/bot_from_azure'

// FILTER bot_from_azure
// Throttle requests with keepalive but no accept-encoding, coming from azure
// This filter is generated from data in etcd. To disable it, run the following command:
// sudo requestctl disable 'cache-text/bot_from_azure'
if (req.http.Connect ~ "keep-alive" && !req.http.Accept-Encoding && req.http.X-Public-Cloud ~ "azure" && vsthrottle.is_denied("requestctl:bot_from_azure:" + req.http.X-Client-IP, 100, 10s, 60s)) {
    return (synth(429, ""));
}

This allows you to do a first check of the action you'll be creating. To add an additional layer of security to your rollout, you can also obtain a VSL expression to match the same condition in logs of a cache server using varnishlog:

puppetserver2001:~$ requestctl log 'cache-text/bot_from_azure'

Monitor requests matching this action using the following command:
sudo varnishncsa -n frontend -g request \
  -F '"%{X-Client-IP}i" %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" "%{X-Public-Cloud}i"' \
  -q 'ReqHeader:Connect ~ "keep-alive" and not ReqHeader:Accept-Encoding and ReqHeader:X-Public-Cloud ~ "azure" and VCL_ACL eq "NO_MATCH wikimedia_nets"'

Inject to varnish and commit

To actually get the action injected into the varnish configuration, run:

puppetserver1001:~$ sudo requestctl enable cache-text/bot_from_azure

And finally commit all of your changes to the injected vcl:

puppetserver1001:~$ sudo requestctl commit
--- cache-text/global.old

+++ cache-text/global.new

@@ -1,3 +1,12 @@

+
+// FILTER bot_from_azure
+// Throttle requests with keepalive but no accept-encoding, coming from azure
+// This filter is generated from data in etcd. To disable it, run the following command:
+// sudo requestctl disable 'cache-text/bot_from_azure'
+if (req.http.Connect ~ "keep-alive" && !req.http.Accept-Encoding && req.http.X-Public-Cloud ~ "azure" && vsthrottle.is_denied("requestctl:bot_from_azure:" + req.http.X-Client-IP, 100, 10s, 60s)) {
+    return (synth(429, ""));
+}
+
 
 // FILTER parameter_1
 // Common cache-busting attack that is recurring

==> Ok to commit these changes?
Type "go" to proceed or "abort" to interrupt the execution
> abort

At this point, if you type "go" instead of "abort" at the input, the action will appear on all cache-text nodes. That is because you didn't define the sites property for your new action object.

BEFORE YOU GO: Commit any changes to files under /srv/git/private/requestctl/ to the private puppet git repo.

Tutorial: Add a new action in haproxy

inject requestctl rules in the TLS termination layer at the edge (haproxy) to limit bandwidth usage and/or concurrency for specific users or request patterns

TODO: all the steps

Command line reference

commit

Commits changes to actions to the compiled datastores. Interactive by default; pass -b if you want to run in batch mode.

$ requestctl enable cache-text/requests_ua_api
$ requestctl commit
--- cache-text/global.old

+++ cache-text/global.new

@@ -1,3 +1,12 @@

+
+// FILTER requests_ua_api
+// Disallow python-requests to access restbase or the action api
+// This filter is generated from data in etcd. To disable it, run the following command:
+// sudo requestctl disable 'cache-text/requests_ua_api'
+if (req.http.User-Agent ~ "^python-requests" && (req.url ~ "^/api/rest_v1/" || req.url ~ "/w/api.php") && vsthrottle.is_denied("requestctl:requests_ua_api", 500, 30s, 1000s)) {
+    set req.http.Requestctl = req.http.Requestctl + ",requests_ua_api";
+    return (synth(429, "Please see our UA policy"));
+}
+

 // FILTER enwiki_api_cloud
 // Limit access to the enwiki api from the clouds

==> Ok to commit these changes?
Type "go" to proceed or "abort" to interrupt the execution
>

\

Once all varnish changes are merged, the haproxy actions will be committed as well.

enable / disable

Enables / disables actions. Note: the enabled field in actions is explicitly excluded from syncing.

$ requestctl enable cache-text/foobar  # enables cache-text/foobar in varnish
$ requestctl disable -s varnish cache-text/foobar # disables the same action in varnish.
$ requestctl enable -s haproxy cache-text/foobar # enables a similarly named haproxy_action

find

Finds which actions include a specific pattern or ipblock:

$ requestctl find ua/requests
action: generic_ua_aws, expression: (pattern@ua/requests OR pattern@ua/curl) AND ipblock@cloud/aws
haproxy_action: generic_ua_aws, expression: (pattern@ua/requests OR pattern@ua/curl) AND ipblock@cloud/aws

get

Gets the data from the datastore and displays them in the desired format. Can be used to fetch all objects or just one.

Examples:

  $ requestctl get pattern
  +------------------------+-------------------------------+
  |          name          |            pattern            |
  +------------------------+-------------------------------+
  |   cache-text/docroot   |          url:^/[\?$]          |
  | cache-text/bad_param_q |           ?q=\w{12}           |
  |   cache-text/enwiki    |    Host: en.wikipedia.org     |
  |  cache-text/restbase   |      url:^/api/rest_v1/       |
  | cache-text/action_api  |        url:/w/api.php         |
  | cache-text/requests_ua | User-Agent: python-requests.* |
  |  cache-text/wiki_page  |     url:/wiki/[^:]+(\?$)      |
  |      ua/requests       | User-Agent: ^python-requests  |
  +------------------------+-------------------------------+



  $ requestctl get pattern ua/requests -o json | jq .
  {
    "ua/requests": {
      "method": "",
      "request_body": "",
      "url_path": "",
      "header": "User-Agent",
      "header_value": "^python-requests",
      "query_parameter": "",
      "query_parameter_value": ""
    }
  }

  $ requestctl get pattern ua/requests -o yaml
  ua/requests:
    header: User-Agent
    header_value: ^python-requests
    method: ''
    query_parameter: ''
    query_parameter_value: ''
    request_body: ''
    url_path: ''

haproxycfg

Outputs the haproxy configuration fragment generated from the haproxy_action

$ requestctl haproxycfg cache-text/requests_ua_api

# ACLs generated for requestctl actions
acl ua_python_requests hdr_reg(User-Agent) -i "^python\-requests"
acl url_rest_api path_reg -i "^/api/rest_v1/"
acl url_action_api path_reg -i "/w/api.php"

# requestctl haproxy action cache-text/requests_ua_api
# Disallow python-requests to access restbase or the action api
# This action is generated from data in etcd. To disable it, run the following command:
# sudo requestctl disable -s haproxy 'cache-text/requests_ua_api'

http-request deny status 429 reason "Please see our UA policy" if ua_python_requests url_rest_api || url_rest_api url_action_api

log

Outputs the varnishncsa command to run on a cache host to see requests matching our action.

$ requestctl log cache-text/requests_ua_api

You can monitor requests matching this action using the following command:
sudo varnishncsa -n frontend -g request \
  -F '"%{X-Client-IP}i" %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" "%{X-Public-Cloud}i"' \
  -q 'ReqHeader:User-Agent ~ "^python-requests" and ( ReqURL ~ "^/api/rest_v1/" or ReqURL ~ "/w/api.php" ) and  not VCL_ACL eq "MATCH wikimedia_nets"'

There is no corresponding functionality for haproxy actions

sync / dump

For each category of objects, copies the directory from a specific repository:

TODO: would these commands always be used together? if so, that should be made clearer. if not, they should have separate sections.

# This is actually ~ copying the directory tree, via etcd
$ requestctl sync -g base_dir action [-i] [--purge]
$ requestctl dump -g dump_dir action

Parameters:

-g, --git-repo identifies the base directory
-i, --interactive (sync only) indicates if we want to be prompted before any write/delete operation happens
-p, --purge if we want to delete stale entries from the datastore. The purge operation should be considered generally omitted for anything but actions (TODO does this mean the user should omit it, or that the tool omits it?). Only use --purge to explicitly remove an object. requestctl doesn't allow you to remove a pattern/ipblock if they’re still referenced in an action.

The structure of the directory tree should contain one file per object we want to upload to the datastore, with path as follows:

<root>/request-{ipblocks,actions/patterns}/<tag>/<name>.yaml

validate

Validates objects written in a repository. Useful for CI.

$ requestctl validate base_dir
$

It will exit with non-zero exit status if any error is present.

vcl

Outputs the vcl fragment generated from the action.

$ requestctl vcl cache-text/requests_ua_api

// FILTER requests_ua_api
// Disallow python-requests to access restbase or the action api
// This filter is generated from data in etcd. To disable it, run the following command:
// sudo requestctl disable 'cache-text/requests_ua_api'
if (req.http.User-Agent ~ "^python-requests" && (req.url ~ "^/api/rest_v1/" || req.url ~ "/w/api.php") && vsthrottle.is_denied("requestctl:requests_ua_api", 500, 30s, 1000s)) {
    return (synth(429, "Please see our UA policy"));
}

Code reference

Private Puppet repo

In production, requestctl objects are under the private Puppet git repo /srv/git/private/requestctl/. Updates to object definitions happen on the primary puppetserver frontend (puppetserver1001 as of early 2024). Your changes modify data that resides in our main Etcd cluster under /conftool/v1/request-{ipblock,action,pattern}s/.

TODO: is there a subdir structure i.e requestctl/request-ipblocks/ that should be explained to make it easier to pick which directory to use / look in?

Schema

requestctl schema
TODO: other useful refs?
- https://wikitech.wikimedia.org/wiki/Conftool

Related playbooks

(restricted access) (D)Dos Playbook