Fundraising/Normalized donation messages
 | This documentation is outdated. For example, headers are no longer separate from message content since migrating off activemq. |
This page documents the in-flight format for donation messages.
Overview
Incoming donations are sent between subsystems as a single packet of data, in a flat dictionary. We use JSON over the wire, and PHP arrays internally. Subscription creations and modifications, refunds, and unsubscribe requests are also sent using similar techniques, and are documented here as well.
We use donation queues to decouple the CRM database from public payments servers, and to not block performance-sensitive tasks.
Legacy format notice: This is the current schema for our messages. Please do not make edits here unless they reflect the existing queue consumer code. Message formats have developed over time and suffer from inconsistency and accumulated cruft, there is a message roadmap with plans to fix this.
Headers
Some data is sent in the message headers in wire format, to make it available for querying.
- correlation-id
- Certifiably unique identifier sometimes used as the primary key for queuing and dequeuing operations. The default way to construct this ID is GATEWAY-GATEWAY_TXN_ID. If the message does not have a correlation-id, then the native queue server message ID is used instead.
- source_name
- freeform name of the application which generated this message
- source_type
- category of application.
- payments
- came in direct from donatewiki or a banner (payments.wikimedia.org)
- listener
- real time messages coming in via a listener service created by the payment provider
- audit
- created by the nightly reconciliation job
- direct
- message was generated within our CRM system itself, for example a hand-entered donation or refund.
- source_host
- machine name where initial intake interfacing occurred.
- source_run_id
- process identifier of the generating code
- source_version
- revision level of the originating code
- source_enqueued_time
- unix timestamp encoding when the message was first added to a queue. This header is not updated during requeueing or other operations.
One-time donations
This is the most common type of donation message. They land in the "donations" queue, and are processed by the queue2civicrm consumer.
- gateway
- Gateway identifier string, e.g. "paypal" or "globalcollect".
- gateway_txn_id
- Transaction ID string used by the gateway. Sadly, several of our processors cannot guarantee that this is a unique identifier.
- date
- Time donation was received, in unix timestamp seconds since epoch.
- currency
- (required) Original currency of transaction. Do not use the deprecated original_currency or original_gross fields, these are too confusing. We'll introduce "settled_currency", etc., when it becomes necessary to track FOREX across processor accounts.
- gross
- (required) Total amount of transaction, in original currency.
- fee
- (optional) Fees charged by the payment processor, in original currency. If unspecified, this will be assumed zero, or calculated from gross - net if available.
- net
- (optional) Amount after subtracting fees.
- settled_date
- (NOT IMPLEMENTED YET) Unix timestamp at which the transaction was settled.
- settled_currency
- (NOT IMPLEMENTED YET) Currency code in which settlement was made.
- settled_gross
- (NOT IMPLEMENTED YET) Total amount settled. (Should this be settled_amount? - it's usually reported after fees)
- settled_fee
- (NOT IMPLEMENTED YET) Processor fee, in settlement currency.
- Donor's email. If unspecified, we may substitute with "nobody@wikimedia.org" for validation purposes. This default is stripped out again before storing to the database.
- first_name
- middle_name
- last_name
- organization_name
- (optional) If given in place of first/last_name, an Organization contact will be created rather than an Individual.
- street_address
- supplemental_address_1
- supplemental_address_2
- city
- state_province
- postal_code
- country
- Billing or mailing address country—not necessarily the same as the contribution_tracking country of web origin.
- payment_method
- Primary payment method, e.g. "cc". TODO: enumerate.
- payment_submethod
- Payment method details, e.g. "visa". TODO
- gateway_status
- Raw gateway status code, if available.
- notes
- Text blob will be stored as a CiviCRM note, associated with the contact.
- contact_source
- TODO WAT?
- gateway_account
- Our name for the gateway account, e.g. "default", "Euro". We support multiple accounts per processor.
- letter_code
- TODO
- gift_source
- TODO
- restrictions
- TODO
- import_batch_number
- Conceptually broken, this should be renamed. It's a number internal to AZ Lockbox.
- check_number
- contact_type
- (optional) "Organization" if you want to create an Org record in Civi. This is normally implicit, according to which name fields were filled out.
- anonymous
- (unused)
- optout
- (unused)
- language
- Our best guess at the donor's preferred contact language.
- recurring
- Always "0" TODO: EH?
- utm_campaign
- Mapped into the direct_mail_appeal custom field
- direct_mail_appeal
- contribution_tracking_id
- completion_message_id
- If a message does not contain enough information to import a donation into Civi, this property gives the ID of a message in the pending queue with the rest of the data. Currently used for AstroPay and Amazon IPN messages.
- log_id
- Like completion_message_id, but for looking up missing information in payment cluster logs. Used for AstroPay and Amazon audit file messages.
Recurring PayPal donations
- gateway
- txn_type
- PayPal-style recurring subtype. One of (subscr_signup, subscr_modify, subscr_payment, subscr_cancel, subscr_eot, subscr_failed)
- subscr_id
- Gateway subscription ID
- subscr_date
- payment_date
- mc_currency
- mc_amount3
- "gross" for all subtypes except subscr_payment
- amount3
- another name for mc_amount3
- mc_gross
- "gross" field for subscr_payment
- mc_fee
- period3
- Must be "1 M", meaning we charge every month.
- recur_times
- Number of installments. "0" means forever (default)
- payer_email
- first_name
- last_name
- address_street
- address_city
- address_zip
- address_state
- address_country_code
- address_name
- recurring
- Always "1"
- cancel
- Cancellation timestamp
- option_selection1
- Premium size
- option_selection2
- Premium language
Recurring other gateway donations
- txn_type
- subscr_id
- subscr_date
- frequency_unit
- one of (day, week, month, year), or empty
- frequency_interval
- installments
- recurring
- must be 1-ish
- cancel
Other fields are the same as for one-time donation messages
Refunds
- gateway
- type
- Refund subtype, one of (chargeback, refund)
- date
- gross_currency
- gross
- total charged to us
- net
- total refunded to donor (not yet implemented)
- fee
- fee associated with refund or chargeback (may be high for chargebacks, not yet implemented)
- gateway_parent_id
- Gateway transaction ID of the original transaction
- gateway_refund_id
- Gateway transaction ID of the refund
Unsubscribe
- contribution-id
Damaged
When invalid messages are kicked out of a queue, they are copied into the "ORIG_QUEUE-damaged" queue using the following format:
- (header) error
- WmfException error code, like "INVALID_MESSAGE"
- (header) correlation-id
- Same as the original message's correlation ID
- error
- error message
- original
- json-encoded original message