grosley
grosley | |
Location: | pmtpa |
Status | |
---|---|
Overall: | This device has been decommissioned.
|
Icinga?: | host status services status |
Hardware | |
Software |
gosley was the failover host for aluminium. See Grosley-Aluminium-Migration for notes from switchover in November 2011.
Services
Various fundraising services, including:
- civicrm.wikimedia.org (Apache, PHP & CiviCRM)
- fundraising.wikimedia.org
- Non-web accessible fundraising scripts can be found in
/opt/fundraising_misc
.
Hudson
Grosley has Hudson CI installed to facilitate the periodic running of maintenance scripts. It's flexibility, particularly for logging/capturing output, is why we are preferring it over cron. Hudson CI ver. 1.368 running. It is accessible at: http://grosley.wikimedia.org:8080/
Potential issues
Hitting ext3 subdir limit in builds directories
Every time a Hudson job runs, a new directory gets created on Hudson where information about that particular build gets stored. Build dirs get created in:
/var/lib/hudson/jobs/<job name>/builds
Particularly for frequently running jobs (eg queue consumption which runs every 5 minutes), we run the risk of hitting the ext3 limit on subirectories (32000). When this limit gets reached, build jobs will fail silently until the issue gets resolved
Prevention
We have prepared a Nagios script which we are hoping to have monitor the job directories. Warning thresholds should be set at 75% of subdirectory limit and critical at 85%.
Resolution
As the limit gets approached, build directories should be archived to storage3 and then cleaned out:
# tar czf hudson_builds_<build name>_<starting date>-<ending date>.tar.gz /var/lib/hudson/jobs/<build name>/builds # scp hudson_builds_<build name>_<starting date>-<ending date>.tar.gz user@storage3.pmtpa.wmnet:/archive/fundraising/hudson_builds/ # rm -rf /var/lib/hudson/jobs/<build name>/builds/*
Mail setup
grosley runs a simple Exim setup, which can send out e-mail directly to the outside world using its dnslookup router and remote_smtp transport. It's also configured to accept mail for *@donate.wikimedia.org, and deliver this mail into a Maildir box in /var/mail/civimail, using the civimail router and civimail_delivery transport. This is basically a simpler version of the configuration described in the Mail article.
Dovecot is installed for IMAP access by CiviMail. Its configuration is almost default, but it's configured to listen using non-SSL IMAP on the loopback interface only, and to retrieve mail using Maildir from /var/mail/user. Authentication is done using PAM and system users, so a system user civimail has been added.
Configuration changes
- localhost IPs (v4 and v6) added to relay_from_hosts in exim4.conf --David Strauss 02:37, 7 May 2009 (UTC)
- Truncated /var/log/exim4/paniclog, which had an old configuration syntax error notice in it --David Strauss 02:38, 7 May 2009 (UTC)