We use Redmine here at CMD for all our project management needs. In fact, this Open Source solution is at the heart of our day-to-day operations. Over the time, we have made a number of contiributions to the project and extended, or modified, Redmine's behavior to suit our needs.

For example, one of the things we did was integrate Zabbix-based PostgreSQL monitoring system with Redmine. This allows us to keep track of all monitoring alerts in a centrally managed place, turn them into actionable tickets when necessary, discuss and work out specific problems either internally or with a customer.

Per our SLA, we're often expected to respond in a swift fashion to emergencies that do occur from time to time. In most cases our systems work very reliably and an e-mail notification about a Disaster-level monitoring event reaches one of our employees quickly enough, which allows us to provide service that meets the requirements outlined in the SLA.

However, we felt that we also needed to make sure that each Disaster-level monitoring event results in a phone call. Just in case e-mail fails.

As a matter of fact, same applies to emergency Redmine tickets. These tickets are basically normal tickets that were assigned a custom "Emergency (Critical)" priority. Either when a new ticket was created or when an existing ticket was promoted to the "Emergency (Critical)" priority, a way for our customers to let us know that they're in an urgent need of our help.

We tried a couple of approaches and eventually ended up using Twilio, which has worked for us really well so far.

Twilio is a cloud communications platform. A programmable phone, if you will. Something that we use to call our on-call employees and inform them about an emergency.

In this post we wanted to show you how you could use Twilio to generate phone calls for Redmine emergency tickets.

procmail-twilio Overview

As an example, we'll take a look at a solution we created and use ourselves here at CMD.

It is called procmail-twilio and is basically a combination of Redmine, procmail, a simple BASH script that calls another Python script that leverages Twilio Python REST API Client and a HTTP server.

It all works together rather simply.

• All e-mail generated by Redmine goes through a procmail filter
• which looks for an Emergency priority ticket
• and, when found, parses it
• and extracts key information about the ticket: ticket number, project name, who created or updated the ticket,
• generates a TwiML file based on the extracted information,
• executes a BASH script that initializes a Python virtual environment
• and then runs a Python script that makes an actual call.
• The HTTP server is required to serve the TwiML file to Twilio via a REST call.

That's it in a nutshell.

Now, let's take a closer look at each stage of this process to get a better idea of how it really works.

Redmine

Assuming Redmine is already configured to deliver e-mail notifications for all ticket updates, all you need is a dedicated Redmine user, e.g. procmail-twilio@yourdomain.com, with a corresponding e-mail account (in this example procmail-twilio) on a server where all Redmine e-mail notifications are delivered.

This user must be a member of a group that gives it access to all Redmine projects. This ensures that the user receives all e-mail notifications generated by Redmine ticket updates in any of the existing projects.

procmail

In procmail-twilio user's home directory on your e-mail server, there will be a ~/procmailrc file with three recipes that look for two distinct types of Emergency priority tickets.

First, existing tickets that were escalated to Emergency priority:

:0B
* ^Priority changed from [A-Za-z]* to Emergency \(Critical\)$
{
  :0B
  {
   WHO=`sed -e '1,/^$/ d' | grep -m 1 "Issue \#[0-9]* has been \(updated\|reported\) by" | cut -d " " -f 7,8 | sed -e "s:\.$::g"`
   PROJECT=`formail -xX-Redmine-Project: | sed -e 's/^\s*//g' -e 's/\s*$//g'`
   ISSUE=`sed -e '1,/^$/ d' | grep -m 1 "Issue \#[0-9]* has been \(updated\|reported\) by" | cut -d " " -f 2 | sed -e "s:#::g" -e "s/./& /g;s/ $//"`
   GOODLUCK=`shuf -n 1 ${GOODLUCKFILE}`
   TWLMSG="<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n<Response>\n\t<Say voice=\"woman\" language=\"en\" loop=\"3\">Hello! Issue number, ${ISSUE}, has just been escalated to Emergency status, by user, ${WHO}. Please, respond immediately. This is issue, number, ${ISSUE}. Project is, ${PROJECT}. The contact is, ${WHO}. Oh, and one more thing... ${GOODLUCK}!</Say>\n</Response>\n"
   :0w:procmail-twilio-existing.lock
   | echo "${TWLMSG}" > "${TWLMSGFILE}" && bash ${CALLSCRIPT}
  }
}

Then, new Emergency tickets:

:0H
* ^Subject:.*\(New\)
{
  :0BA
  * ^\* Priority: Emergency \(Critical\)$
  {
    WHO=`sed -e '1,/^$/ d' | grep -m 1 "Issue \#[0-9]* has been \(updated\|reported\) by" | cut -d " " -f 7,8 | sed -e "s:\.$::g"`
    PROJECT=`formail -xX-Redmine-Project: | sed -e 's/^\s*//g' -e 's/\s*$//g'`
    ISSUE=`sed -e '1,/^$/ d' | grep -m 1 "Issue \#[0-9]* has been \(updated\|reported\) by" | cut -d " " -f 2 | sed -e "s:#::g" -e "s/./& /g;s/ $//"`
    GOODLUCK=`shuf -n 1 ${GOODLUCKFILE}`
    TWLMSG="<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n<Response>\n\t<Say voice=\"woman\" language=\"en\" loop=\"3\">Hello! New Emergency issue, number, ${ISSUE}, has just been reported, by user, ${WHO}. Please, respond immediately. This is issue, number, ${ISSUE}. Project, ${PROJECT}. The contact is, ${WHO}. Oh, and one more thing...  ${GOODLUCK}!</Say>\n</Response>\n"
    :0w:procmail-twilio-new.lock
    | echo "${TWLMSG}" > "${TWLMSGFILE}" && bash ${CALLSCRIPT}
  }
}

This particular order is important.

When an Emergency ticket e-mail notification is encountered, it is parsed and the most relevant information is extracted, which then is used to generate a TwiML file – essentially a voice message that is played during a call.

Otherwise, e-mail is discarded by redirecting it to /dev/null.

:0
/dev/null

This procmail recipe file won't keep any copies of e-mail on disk. However, by default it will log its actions to a logs/procmail.log file. It is optional and can be disabled if desired. Just comment out LOGFILE= variable in ~/.procmailrc file.

Twilio Call Scripts

procmail will eventually execute a simple BASH script procmail-twilio.bash that initiates a Python virtual environment and runs procmail-twilio.py:

#!/bin/bash

set -e

twldir="{{virtualenv_path}}"
phones="{{phonesdir}}/phones.sh"

source "$phones"

# Change directory to our python virtualenv
cd $twldir

# Activate the virtualenv
source bin/activate

# Execute the python script that makes the call
python procmail-twilio.py $oncall $backup $tertiary

In our setup, the BASH script relies on phones.sh file to look up on-call, backup and tertiary numbers. It is part of a larger Twilio-based solution that we also developed, and you don't really have to use it. You could use a similar file, if you prefer to keep phone numbers separate from the script, or you could define those numbers directly in procmail-twilio.bash:

#!/bin/bash

set -e

twldir="{{virtualenv_path}}"

oncall="+1xxxxxxxxxx"
backup="+1xxxxxxxxxx"
tertiary="+1xxxxxxxxxx"

# Change directory to our python virtualenv
cd $twldir

# Activate the virtualenv
source bin/activate

# Execute the python script that makes the call
python procmail-twilio.py $oncall $backup $tertiary

The procmail-twilio.py script makes an actual call, handles call tree logic, call status and provides simple logging of call events. We make several attempts to reach primary on-call and if that fails, we call backup on-call and if that somehow falls too, the call goes out to a third person, usually our boss. We wait 5 minutes between initiating a call and polling for a call status.

Consider this abridged version of the script:

#!/usr/bin/python
        
# Tested with Python 2.7
# Requires TwilioRestClient

# Download the library from twilio.com/docs/libraries
from twilio.rest import TwilioRestClient
import sys
import time     
import datetime 

# Get these credentials from http://twilio.com/user/account
account_sid = "BZ62270014z14e1354aej1d66d5ea932v6"
auth_token = "57f3e13321841e31cefd4c137f5baeff1"
# Twilio phone number
from_="+1xxxxxxxxxx"
# oncall, backup and tertiary call recipient numbers
# are passed down as arguments from the calling BASH 
# script.       
oncall=sys.argv[1]
backup=sys.argv[2]
tertiary=sys.argv[3]

...

# Makes the call and returns session id
def makecall(to_num, from_num) :
    try:        
        call = client.calls.create(to=to_num, from_=from_num,
                                   url="https://yourdomain.com/procmail-twilio/twiml.html")
        return call.sid
    
    except TwilioRestException as e:
        logerror(e)

# Uses session id returned by makecall() to obtain call status
def callstatus(sid) :
    try:
        callinfo = client.calls.get(sid)
        return callinfo.status
        
    except TwilioRestException as e:
        logerror(e)

...

#
# Actual program starts here
#

client = TwilioRestClient(account_sid, auth_token)

count = 0
status = "nothing"

while count < 7 :

        count = count + 1

        if status in ['completed', 'in-progress'] :

                break

        if count == 1 :            
                to=oncall
                sid = makecall(to, from_)
                time.sleep(300)
                status = callstatus(sid)
                logcall(sid, count, status, to)

...

        elif count == 4 :
                to=backup
                sid = makecall(to, from_)
                time.sleep(300)
                status = callstatus(sid)
                logcall(sid, count, status, to)

...

        elif count == 7 :
                to=tertiary
                sid = makecall(to, from_)
                time.sleep(300)
                status = callstatus(sid)
                logcall(sid, count, status, to)

        else :
                logerror("Internal error. Check " + sys.argv[0])

Of a particular interest is the makecall() function:

call = client.calls.create(to=to_num, from_=from_num,
                                   url="https://yourdomain.com/procmail-twilio/twiml.html")

The url= parameter is what tells Twilio where to look for the TwiML file. This is where you need a HTTP server.

HTTP Server

We use Apache, but it could be nginx or some other HTTP server. Requirements are basic. The purpose of having a HTTP server is as simple as to serve the TwiML file to Twilio via a HTTP(S) URL. It essentially comes down to exposing the TwiML file via HTTP. No different than any other normal file.

Apache runs on the same server where Redmine e-mail notificaitons are delivered and processed by procmail.

procmail-twilio is meant to be installed and run as an unprivileged system user. The TwiML file is generated in a directory outside of Apache's DocumentRoot to avoid giving procmail-twilio system user extra permissions to manipulate files in Apache's DocumentRoot. Instead, we use a symlink that can be located in an arbitrary path of the DocumentRoot.

For example, if your TwiML file is /home/procmail-twilio/prctwl-data/twiml.html, Apache DocumentRoot is /srv/vhosts/main/ and you want the TwiML file to be accessible at https://yourdomain.com/procmail-twilio/twiml.html you need to create a symlink /srv/vhosts/main/procmail-twilio/twiml.html that points to /home/procmail-twilio/prctwl-data/twiml.html:

admin@http:~$ sudo mkdir /srv/vhosts/main/procmail-twilio
admin@http:~$ sudo ln -s /home/procmail-twilio/prctwl-data/twiml.html /srv/vhosts/main/procmail-twilio/twiml.html

So, when the makecall() initiates a call basically what happens, is that it tells Twilio to open url=, which is expected to be served by your HTTP server.

Log Files

By default, ~/.procmailrc file is configured to log all its actions in logs/procmail.log file.

Typically, you will see two types of log entries. This one tells us that an e-mail was discarded, because it doesn't appear to be a an Emergency priority ticket:

From projectid@yourdomain.com Sat Apr 1 13:36:48 2017
 Subject: [Project Name - Support #82795] Restore/Generate Monthly Reports
 Folder: /dev/null

In this example we can see that a new Emergency ticket was found, a TwiML file was generated and ${CALLSCRIPT} was run to make a call:

From projectid@yourdomain.com Mon Mar 27 09:36:16 2017
 Subject: [Project Name - Support #86855] (New) Our database is down
 Folder: echo "${TWLMSG}" > "${TWLMSGFILE}" && bash ${CALLSCRIPT} 3558

Each call attempt and its status is logged in a log file logs/procmail-twilio-calls.log:

2017-03-27 09:41:19.145112 Session ID: ZBb7xx16d4E1295ctf84e5b25121ea46kk, Call attempt number: 1, Call status: completed, Call recipient: +1**********.

This helps keep a clear history of all calls and whether they were successfull or not.

All log files are managed by logrotate, so they shouldn't take up too much space on disk.

Download and Install procmail-twilio

If you want to give it a try, or build on top of it something else, procmail-twilio can be downloaded from CMD's github repository as an Ansible playbook.

To install just run this command:

admin@workstation:~/playbooks$ git clone https://github.com/commandprompt/procmail-twilio.git
admin@workstation:~/playbooks$ cd procmail-twilio
admin@workstation:~/playbooks/procmail-twilio$ ansible-playbook run.yml --ask-vault-pass -K

Before you do so, however, you need to configure your vars file and decide if you need to make changes to any of the template files, etc. This example also assumes the vars file was first encrypted with ansible-vault command (it contains your Twilio account username and password):

admin@workstation:~/playbooks/procmail-twilio$ ansible-vault encrypt roles/procmail-twilio/vars/main.yml

You may also want to modify some template files but you don't really have to.

In Place of Conclusion

An important point to keep in mind is that Redmine here is just an example. It could be JIRA, BugZilla or something else. Even your personal GMail account. Just replace procmail recipes with your own and you're good to go.

And hey, if you use procmail-twilio or used it to build something else, let us know.

We'd love to hear your story.