VRT System/Troubleshooting

Troubleshooting

Mail delivery

When user/group for the exim pipe were incorrect, otrs.Postmaster.pl logged permissions to /var/log/mail.log about errors on attempts to write in /opt/otrs/var/tmp/CacheFileStorable. We fixed this by configuring the exim pipe to use group=www-data.
Mail hosts need mysql access to the otrs database. If mx IP addresses change or the database is inaccessible, mail defers on whichever mx is trying to do an address lookup. When we saw this happen, exim wasn't very informative about why. When in doubt double-check mysql access from the mx's command line.
Spamassassin runs locally, and logs in /var/log/mail.log.

Apache permissions errors

as user otrs run:

/opt/otrs/bin/otrs.SetPermissions.pl --otrs-user=otrs --otrs-group=otrs --web-user=www-data --web-group=www-data /opt/otrs

SpamAssassin stops reporting Bayes results

This happened 2014-04-24, 2016-08-06, 2016-12-21 and we discovered it was unhappy about the bayes database.
/var/log/syslog was full of this:

bayes db version 0 is not able to be used, aborting! at /usr/share/perl5/Mail/SpamAssassin/BayesStore/DBM.pm line 203, <GEN88>

We tried backup/restore the database (the verify failed), and the database shrank from ~24M to ~14M and SA stopped complaining. But SA continued to pass through mail with no Bayes results (the BAYES_XX where XX in [00,90] Header added to the message was missing)
So we moved aside the old database, modifed otrs.TicketExport2Mbox.pl not to skip previously-seen messages, and created one-time GenericAgent [within OTRS] jobs to re-export a couple of days worth of ham/spam. Then we ran train_spamassaassin manually to train on all this data. Note otrs.TicketExport2Mbox now has --rebuild mode to support this process.
On the 2016-08-06 incident the log statement below was found in the logs:

  Aug  6 09:56:59.752 [1619] dbg: bayes: not available for scanning, only 126 ham(s) in bayes DB < 200

after running a:

 sudo -u debian-spamd spamassassin -D bayes < /tmp/sample_email.eml

and a

 sudo -u debian-spamd sa-learn --dump magic

confirmed it.

The fix was the re-exporting and training of spamassassin as mentioned above. Extra care should be take to make sure the spam/ham messages exported are above 200 in every case.

On the 2016-12-21 incident both the hams and the spams in the database were below 200. That was not logged however as the message about the hams above, leading the investigation off track for a while. Exporting quite a few messages and training spamassasin on them as above fixed the issue. The database in this case was NOT being marked as corrupted by db_verify but it was truncated in the end just for good measure manually

Add an email to the exim4 deny list

Access the private repo on puppetserver1001

Log in to the puppetserver1001 server.

Go to the private repo directory:

cd /srv/git/private/hieradata/common/profile

Open the file vrts.yaml.

Update the `profile::vrts::exim_deny_senders` list

In vrts.yaml, locate the key profile::vrts::exim_deny_senders.

Add the new sender(s) that need to be blocked. For example:

profile::vrts::exim_deny_senders: "address@domain.com"

Save the file.
Commit the change in the private repo.
In the same directory, check your changes:

git status
git diff

Commit and push the changes:

git add vrts.yaml 
git commit -m "Add new addresses to the Exim4 deny list" 
git push

Verify and propagate the Puppet configuration

In the public repo, the necessary changes to support this mechanism have already been implemented. Ensure that your commit in the private repo does not break Puppet compilation by checking:

The puppet compiler output.
CI and automated validation feedback.

Once the compilation is confirmed, deploy (merge) the changes.

Confirm the noop on the target host

On vrts1003 (and any other host using this profile), run or wait for the Puppet run.
Verify that the run is a noop.
The new addresses will then be included in the Exim4 deny list and automatically enforced.