Fundraising/techops/procedures/services-redis-frqueue

This page documents to the 'frqueue' redis service which handles contribution tracking messages and donor preference change messages. This service runs on the 'frqueue' server role in the Fundraising server cluster.

Overview

• We run redis basic replication
• There are two redis instances running as separate services
  • redis-frqueue (tcp/6379)
  • redis-donor-prefs (tcp/6380)
• The redis-frqueue instance uses key "sequence_contribution-tracking" as a sequence generator for contribution tracking.
• When switching origin servers, shut down the old origin server before making replication changes to be sure it doesn't receive any new messages after replication is changed.
• You can invoke redis commands from a shell using redis-cli, or you can log in via redis-cli and run commands within in a client.

Origin Swap

If the old redis origin server is dead, verify the new origin server has the latest Contribution ID.

• The last ID imported by queue consumers can be obtained from the drupal database.

SELECT id,contribution_id FROM drupal.contribution_tracking WHERE contribution_id IS NOT NULL ORDER BY id DESC LIMIT 10;

• Use redis-client to compare the autoincrement counter to the latest contribution ID.

# redis-cli
127.0.0.1:6379> AUTH <password>
127.0.0.1:6379> GET sequence_contribution-tracking
127.0.0.1:6379> exit

Problems

• If replication died in some wierd way, it's possible different replicas have different autoincrement values. In this case choose the one with the greatest value as the new origin.
• If contribution ID from the database is higher than the redis autoincrement, there is a risk of corrupting the database if you don't adjust the redis autoincrement. FR Tech should be consulted for how to proceed.

Promote new origin server for redis-frqueue

• On the new queue server stop replication, retaining dataset, become origin. Check if things look ok.

# redis-cli
127.0.0.1:6379> AUTH <password>
127.0.0.1:6379> REPLICAOF NO ONE
127.0.0.1:6379> INFO
127.0.0.1:6379> GET sequence_contribution-tracking
127.0.0.1:6379> exit

# redis-cli -p 6380
127.0.0.1:6380> AUTH <password>
127.0.0.1:6380> REPLICAOF NO ONE
127.0.0.1:6380> INFO
127.0.0.1:6380> exit

On each replica, change origins (discarding dataset) and resynchronize, check status.

# redis-cli
127.0.0.1:6380> AUTH <password>
127.0.0.1:6379> REPLICAOF <new redis origin FQDN> 6379
127.0.0.1:6379> INFO
127.0.0.1:6379> exit

# redis-cli -p 6380
127.0.0.1:6380> AUTH <password>
127.0.0.1:6380> REPLICAOF <new redis origin FQDN> 6380
127.0.0.1:6380> INFO
127.0.0.1:6380> exit

Adjust redis-related configuration in puppet.

This adjusts puppetized clients including civicrm, payments-listener, and sets redis replication configuration to survive a service restart.

• edit (frack) puppet/hieradata/site/common.yaml

queue_redis_primary: '<new redis origin FQDN>'

• git commit, git push, puppet-merge