Jump to content

PyBal

From Wikitech

PyBal is an automated manager for LVS. We use it to continously monitor Varnish or Apache servers and change the LVS load balancer pooling and weights accordingly. The service is written in Python using the Twisted framework.

For more information about Wikimedia's LVS setup in general, see LVS.

Features

PyBal distinguishes itself from lvsmon in a few aspects:

  • It's using asynchronous communication, and thus runs all checks in parallel instead of sequentially
  • It has an extra monitoring method called IdleConnection, which keeps an idle connection open to all squids, and therefore notices immediately when the Squid processes are shut down / crashing
  • It can fetch server lists over HTTP as well as from the local filesystems

Setup

PyBal is currently installed on our LVS hosts, in directory /usr/sbin. Start or stop it with systemctl start/stop pybal.service

Configuration is in /etc/pybal/. pybal.conf defines the LVS service parameters

The list of pooled hosts resides wherever the pybal::web class is installed via puppet, under the directory /srv/pybal-config and it will be reachable via the internal address http://configuration-master.$site.wmnet/pybal, with one file per LVS service. Attributes:

  • weight: a larger number means that more requests get sent to this server in comparison with others
  • enabled: either True or False, depending on whether you want requests to be sent to this server

The format should be fairly self explanatory; the files more or less use Python assignment / dictionary syntax.

PyBal supports multiple LVS services through a single instance and configuration file pybal.conf, e.g.:

[text]
protocol = tcp
ip = 145.97.39.155
port = 80
scheduler = wlc
config = file:///etc/pybal/text-squids

[images]
protocol = tcp
ip = 145.97.39.156
port = 80
scheduler = wlc
config = file:///etc/pybal/upload-squids

Beware, the code as checked out from git has DryRun = True set in ipvs.py, meaning that it will not modify any actual IPVS state but only show the commands for debugging. This should be changed to a command line option, but for now edit that file to DryRun = False.

The configuration files are generated via puppet.

How to

See LVS.

Updating PyBal on LVS instances

After testing new releases on pybal-test2003.codfw.wmnet, PyBal should be updated in the following order:

  • ulsfo
  • eqsin
  • drmrs
  • esams
  • codfw
  • eqiad

Within each datacenter, first update and check that everything is fine on the passive instances and then go for the active instances. As long as BGP is enabled, redirecting traffic from the active instances to the passive one it should as easy as stopping PyBal in the active instance. After stopping it, you should see an increase of the active connections on the passive instance running the ipvsadm command described below.

After updating a BGP enabled PyBal instance you can check that everything is good on the router side with the following commands:

show bgp summary | match <pybal_ip>
show bgp neighbor <pybal_ip>
show route receive-protocol bgp <pybal_ip>

On the PyBal instance the following commands are useful:

ipvsadm -Ln
/usr/local/lib/nagios/plugins/check_pybal_ipvs_diff --prometheus-url http://<pybal_ip>:9100/metrics

Testing

New PyBal releases can be tested on pybal-test2003.codfw.wmnet. The systems are deployed with role(pybaltest). Configuration example:

# /etc/pybal/pybal.conf on pybal-test2001
[global]
bgp = yes
bgp-local-asn = 64496
bgp-peer-address = 10.192.16.140
#bgp-as-path = 64460
bgp-nexthop-ipv4 = 10.192.16.139
bgp-nexthop-ipv6 = 2620:0:860:101:10:192:1:3
instrumentation = yes
instrumentation_ips = [ '127.0.0.1', '::1', '10.192.16.139' ]
# Lower is prefered
bgp-med = 50

# Service definition
[textlb6_80]
protocol = tcp
ip = 2620:0:860:ed1a::1
port = 80
scheduler = sh

config = etcd://conf2001.codfw.wmnet/conftool/v1/pools/codfw/cache_text/varnish-fe/

depool-threshold = .5
monitors = ["IdleConnection"]

# IdleConnection monitor configuration
idleconnection.max-delay = 300
idleconnection.timeout-clean-reconnect = 3
# /etc/pybal/pybal.conf on pybal-test2003
[global]
bgp = yes
bgp-local-asn = 64496
bgp-peer-address = 10.192.16.140
#bgp-as-path = 64460
bgp-nexthop-ipv4 = 10.192.16.141
bgp-nexthop-ipv6 = 2620:0:860:101:10:192:1:3
instrumentation = yes
instrumentation_ips = [ '127.0.0.1', '::1', '10.192.16.141' ]
#Lower is prefered
bgp-med = 100

# Service definition
[...]

A Quagga instance is installed on pybal-test2002 and can be used to test the BGP component of PyBal:

log file /var/log/quagga/quagga.log
!
debug zebra rib
debug bgp events
debug bgp updates
debug bgp zebra
!
password SECRET
!
interface eth0
 ipv6 nd suppress-ra
!
interface lo
!
router bgp 64460
 bgp router-id 10.192.16.140
 no bgp default ipv4-unicast
 network 127.0.0.2/32
 neighbor 10.192.16.139 remote-as 64496
 neighbor 10.192.16.139 description PyBal on pybal-test2001
 neighbor 10.192.16.139 activate
 neighbor 10.192.16.139 prefix-list NONE out

 neighbor 10.192.16.141 remote-as 64496
 neighbor 10.192.16.141 description PyBal on pybal-test2003
 neighbor 10.192.16.141 activate
 neighbor 10.192.16.141 prefix-list NONE out
!
 address-family ipv6
 network 2620:0:860:102::/64
 neighbor 10.192.16.139 activate
 neighbor 10.192.16.141 activate
 exit-address-family
!
ip prefix-list NONE seq 5 deny any
!
ip forwarding
ipv6 forwarding
!
line vty
!

The IPv4 routing table can be inspected with:

vtysh -c 'show ip route'

Similarly, to inspect the IPv6 routing table:

vtysh -c 'show ipv6 route'

Alerts

PyBal IPVS diff check

The alert fires whenever pybal and ipvs disagree on the current configuration.

Services in IPVS but unknown to PyBal

For example upon removing services from pybal (or changing ports) the stale ipvs virtual services might not get removed. (In other words, this is shown: CRITICAL: Services in IPVS but unknown to PyBal: set(['addr:port'])). For such cases it is sufficient to delete the stale TCP service from the lvs pair:

 ipvsadm --delete-service --tcp-service addr:port

Services known to PyBal but not to IPVS

This alert is usually temporary and is caused by new services being setup (i.e. in etcd, pybal knows about them) but pybal hasn't been restarted yet, and thus hasn't had a chance to program ipvs correctly. The fix is to restart pybal.

For example:

 PyBal IPVS diff check on lvs2009 is CRITICAL: CRITICAL: Services known to PyBal but not to IPVS: set([10.2.1.54:443])
 

Pybal service has not been restarted

This alert means that the Pybal configuration file was changed but the service has not been restarted. To fix this, simply restart the Pybal service on the respective host. Note that this will pick up all changes since the configuration was changed, as expected.

See also

  • lvsmon: The predecessor to PyBal