Conftool2git

Conftool2git is a service that listens for HTTP POST requests of conftool audit log events, and combines them with querying the etcd backend to create changes to a git repository that is, to all effects, recording all history of what was changed by whom in conftool. Conftool will only send such audit log events if the conftool2git_address configuration key is set. The service runs on one of the puppetservers, indicated by profile::conftool2git::active_host, and saves the data it collects to /srv/git/conftool/auditlog, and that repository is then replicated to all puppetservers.

Some details

The service is written in async python, and spawns an aiohttp web server to receive POST requests at the root url.

It listens on the address indicated by profile::conftool2git::address, and once it receives a message it json-decodes it, and sends it to a queue. Another coroutine will consume from this queue, fetch the relevant data from etcd, dump it to the appropriate file on disk, then commit the changes. Once the commit is done, the post-commit hooks are run, which are installed by the git::replicated_local_repo puppet class, pushing the new commit to all puppetservers.

On startup, the service will try to perform a full sync between what is on the backend and what's currently on disk, so that any lost messages or other issues (like a server/service downtime) are amended.

What to do if it breaks

Just restart the service, that should solve most issues that are inherent to conftool2git. If the problem comes from the git commit process, that's another kind of issue and you should go look at the postcommit hook in the repository and try to debug that.

If any attempt at debugging the problem fails, you should just force push from the active host to the others.