Service/IPoid
More information and discussion about changes to this draft on the talk page.
The ipoid service provides an API for retrieving reputation data for individual IP addresses, as well as retrieving lists of IP addresses matching a particular set of labels (e.g. list of IP addresses that are associated with VPNs).
As of October 2023, the primary consumer of the service is the IPInfo extension, which allows a small number of users with privileged access to view IP reputation data.
Querying
From mwmaint:
Terminalcurl -s http://localhost:6035/feed/v1/ip/[ip]
To find IPs matching a prefix:
Terminalcurl -s http://localhost:6035/feed/v1/ip/[IP]?usePrefixSearch=1
from deploy servers:
Terminalcurl -s https://ipoid.svc.eqiad.wmnet:30443/feed/v1/ip/2.2.2.2
Terminalcurl -s https://ipoid.svc.codfw.wmnet:30443/feed/v1/ip/2.2.2.2
Data pipeline
The extract, transform, load pipeline is described in more detail in IPoid's README. At a high level:
- Once a day, download IP reputation data from Spur.us
- Compare existing data with newly downloaded data and update IPoid's database with the latest information
The import process, as of October 2023, peaks at about 600mb of RAM in local development environment tests.
Service Dependencies
Database
MariaDB instance on m5. m5-master.eqiad.wmnet is anchored to eqiad for writes, while it can read from m5-master.codfw.wmnet, that means that, the service will be active/active from the user perspective, but its cronjob, which is the only path where there are writes in the database, will run always from eqiad.
Service Monitoring
Service Kubernetes dashboard: https://grafana-rw.wikimedia.org/d/6C9Bm6uVz/ipoid?orgId=1
Ownership
Trust and Safety Product team.
Cronjobs
There is a cron job for processing daily updates, ipoid-production-daily-updates.
Note that if you update the schedule for the Kubernetes cronJob, helmfile apply will not show any update in its diff about the newly applied schedule.
Deployment
 | The canonical documentation is at Deployments on kubernetes. This is a helper for those deploying updates to ipoid. |
Prepare the deployment patch
Make a patch in operations/deployment-charts that updates the value of the main_app.version field in helmfile.d/services/ipoid/values.yaml, to the new image tag was built automatically with the merging of a GitLab merge request in the iPoid repo. The image tag can be found at IPoid's Docker registry but may take some time to show up. If the registry hasn't updated yet, the tag can also be found in the logs of the merge's `publish-production-build-and-publish-image` job.
ipoid: Bump version
* [misc] Use date/time image version name
T338224 / MR !5
Bug: T338224
Commit message guidelines
- Subject line can always be: "ipoid: Bump version"
- Add a bullet point for patch in
repos/mediawiki/services/ipoidthat is part of this release. The first line should specify what relevant code is affected (api, app, etc) followed by the subject line of the commit. On the second line, include a reference to the task from the patch and a reference to the relevant merge request number(s) - Finally, the last line should include "Bug: " and reference the relevant phabricator task for this deployment.
All of the above guidelines in the commit message are helpful for paper trail and for documenting what was deployed, and when.
diff --git a/helmfile.d/services/ipoid/values.yaml b/helmfile.d/services/ipoid/values.yaml
index b843d7f..025e203 100644
--- a/helmfile.d/services/ipoid/values.yaml
+++ b/helmfile.d/services/ipoid/values.yaml
@@ -18,7 +18,7 @@
limits:
cpu: 2
memory: 4Gi
- version: 2023-06-25-123417-production
+ version: 2023-06-26-123417-production
monitoring:
enabled: false
See also
See Deployments on kubernetes for tips, and note that 1) self merges are OK in this repository, and 2) a cron script on the deployment server will fetch the latest contents of the repository every minute.
Deploy the patch
Now, SSH to a Deployment server. Using tmux is recommended.
staging
Staging$ cd /srv/deployment-charts/helmfile.d/services/ipoid/ $ git log # Make sure your deployment patch is there $ helmfile -e staging -i apply # scan output to see if the changes are expected, press "enter" $ service-checker-swagger ipoid.k8s-staging.discovery.wmnet https://ipoid.k8s-staging.discovery.wmnet:30443?spec # Manually verifying requests $ curl "https://ipoid.k8s-staging.discovery.wmnet:30443/_info" $ curl "https://ipoid.k8s-staging.discovery.wmnet:30443/feed/v1/ip/{ip}"
eqiad
eqiad$ cd /srv/deployment-charts/helmfile.d/services/ipoid/ $ git log # Make sure your deployment patch is there $ helmfile -e eqiad -i apply # scan output to see if the changes are expected, press "enter" # Manually verifying requests $ curl "https://ipoid.svc.eqiad.wmnet:30443/_info" $ curl "https://ipoid.svc.eqiad.wmnet:30443/feed/v1/ip/{ip}"
codfw
codfw$ cd /srv/deployment-charts/helmfile.d/services/ipoid/ $ git log # Make sure your deployment patch is there $ helmfile -e codfw -i apply # scan output to see if the changes are expected, press "enter" # NB the following requests will go to the active datacenter, so if eqiad is active and you're deploying to codfw, these requests will go to eqiad. # Manually verifying requests $ curl "https://ipoid.svc.eqiad.wmnet:30443/_info" $ curl "https://ipoid.svc.eqiad.wmnet:30443/feed/v1/ip/{ip}"
Checking output from a container
Terminal$ kube_env ipoid staging $ kubectl get pods NAME READY STATUS RESTARTS AGE ipoid-staging-7476db744d-w8bms 3/3 Running 0 7h47m tiller-974b97fc7-rq4dn 1/1 Running 0 30h $ kubectl logs -f ipoid-staging-7476db744d-w8bms Error from server (BadRequest): a container name must be specified for pod ipoid-staging-7476db744d-w8bms, choose one of: [ipoid-staging ipoid-staging-tls-proxy] $ kubectl logs -f ipoid-staging-5465b6997d-z5hhd -c ipoid-staging
Monitor progress after deployment
To use the kubectl command:
$ export KUBECONFIG=/etc/kubernetes/ipoid-deploy-eqiad.config
- To see the running (and stopped but not deleted) pods:
kubectl get pods
- To see logs for a running pod:
kubectl logs -f <pod NAME>
- To explore file system for a running pod:
kubectl exec --stdin --tty <pod NAME> -- /bin/bash
To explore the database (will prompt for password):
$ mysql -uipoid_ro -p -h m5-master.eqiad.wmnet ipoid