In my previous few posts on Open BlueDragon for Google App Engine (GAE) I’ve done a basic introduction to GAE, discussed the Google Datastore, and dug into how to use the virtual file system and the GAE Image Service to handle file uploads and simple image processing.
In this post we’ll pick back up with my sample app (.tar.gz, 15.3 MB) from OpenCF Summit 2011 and look into two more of the excellent services available to you on GAE, specifically the mail and XMPP services.
Mail? So What?
Sending email is of course a big yawn–anyone can do that from any platform quite easily. But what if you want your application to be able to receive mail and take actions based on the mail it receives?
The standard solution is typically to set up a dummy email address (myapp@foo.com) and then have your application poll the mailbox for new mail, which in the case of a CFML app means setting up a scheduled task to do a CFPOP call to the mailbox in question. You then get back a list of messages and have to handle them as needed for your application.
This works, but it’s a pretty passive, braindead approach to solving the problem, and you wind up having to write a bunch of code to manage a mailbox designed for humans that’s being used by your application.
With GAE you can have your application respond to incoming email directly without having to set up any mailboxes, because the ability for your app to receive mail is built right into the GAE platform. You simply enable mail services in your appengine-web.xml file, configure specific email addresses or wildcard email addresses in web.xml, and your application can receive mail.
Configuring Inbound Services
Let’s take a look at appengine-web.xml, which if you’re following along with the sample app is in the war/WEB-INF directory. Towards the bottom of the file you’ll see a block of inbound services defined:
<inbound-services>
<service>xmpp_message</service>
<service>xmpp_presence</service>
<service>xmpp_subscribe</service>
<service>mail</service>
</inbound-services>
Note that by default there are no inbound services enabled on your GAE application, so you do have to specify these. We’ll be covering the XMPP services in this post as well, so you can see in the block above this is where specific granular XMPP services such as messaging, presence, and subscription are enabled.
That’s it! With that simple block in appengine-web.xml when you deploy your application to GAE, it will be ready to receive email and interact with XMPP messages.
At this point you’re probably wondering basic things like the address to which you can send email, how to handle the email once it’s received, how to configure specific email addresses, and a host of other things, so let’s dig in a bit more.
How the Magic Works
Once you have mail defined as an inbound service, your application essentially gets a mail server on the internet at the address YOUR_APP_ID.appspotmail.com. So with mail in the inbound services block in your config file, once you fire up your app on GAE mail can be sent to ANY_ADDRESS@YOUR_APP_ID.appspotmail.com. We’ll talk more about how the ANY_ADDRESS bit works in a minute when we look at writing code to handle incoming mail.
Google very cleverly designed many of their services to send messages to your application via an HTTP POST to a specific URL. In the case of inbound mail, any mail sent to ANY_ADDRESS@YOUR_APP_ID.appspotmail.com generates an HTTP POST to the URL /_ah/mail/{EMAIL_ADDRESS} with email address representing the full email address including the YOUR_APP_ID.appspotmail.com domain.
Since there’s a lot of magic behind the scenes that you don’t have to configure, let’s review how this works before moving on to the code for handling incoming mail.
- You enable mail as an inbound service in appengine-web.xml
- When deployed to GAE, your app can then receive email at ANY_ADDRESS@YOUR_APP_ID.appspotmail.com
- When mail is received at YOUR_APP_ID.appspotmail.com, this generates an HTTP POST to the url /_ah/mail/{EMAIL_ADDRESS} within your application
Now let’s look at how to write code to process incoming mail.
Processing Inbound Mail
With very little effort on our part our application can receive mail. That’s cool in theory, but how do we write code to react to this inbound mail?
Since all mail comes in as an HTTP POST, the way you handle inbound mail is more or less the same way you’d handle any POST request to your application. This makes handling all of this very simple since handling POST requests is something with which we’re all too familiar.
From a CFML standpoint there’s a bit of a disconnect here since, at least until I figure out a clever hack, the thing that receives the mail has to be a Java servlet. Not to worry, the code’s quite straight-forward and we can hand off any real work we want to do to CFML once we get the mail in the servlet.
Bit of a detour to explain why (at this point) CFML can’t handle the inbound mail directly. In order for your application to take action when mail is received, you have to define a servlet mapping for the special incoming mail URL of /_ah/mail/{EMAIL_ADDRESS}. Servlet mappings do support wildcards, so for the sample application I’ve defined a servlet mapping of /_ah/mail/* that will route all inbound mail, regardless of the specific email address, to the same servlet.
You could of course define specific servlet mappings for specific addresses, e.g. one servlet mapping for /_ah/mail/user1@yourappid.appspotmail.com and another for /_ah/mail/user2@yourappid.appspotmail.com if you needed two completely separate servlets to handle the different users. I hinted at this above, but I’ll point out here again that you do not need to configure mailboxes or anything along those lines for specific users. Your application will receive absolutely any mail sent to ANYONE@yourappid.appspotmail.com and how you handle the ANYONE part is up to you to define in your code.
Back to the CFML discussion. When I was first messing with the in the local GAE environment, I first thought I’d be clever and just define a real directory path in my application of /_ah/mail, throw an index.cfm file in there to act as a controller, and I’d be off to the races. That actually kind of worked locally, but the issue is when you deploy to GAE it won’t let you deploy your application if you have /_ah as a real directory since that’s a reserved directory for things like the admin console.
With that door shut, I next figured I could define the servlet mapping for /_ah/mail/* to point to the CFML servlet like so:
<servlet-mapping>
<servlet-name>cfmServlet</servlet-name>
<url-pattern>/_ah/mail/*</url-pattern>
</servlet-mapping>
This kind of worked as well, meaning I could at least see it was hitting the CFML servlet since I could dump things in Application.cfc and see what was going on. The issue here turned out to be that I coudln’t really leverage anything like onRequestStart in Application.cfc and take any action there, because when a POST comes in to /_ah/mail/* there’s no page argument, which is what onRequestStart expects.
I’m convinced there’s a way to get this to work (or I’m at least convinced I haven’t yet exhausted all avenues of failure here), but since I was under a time crunch to get this rolling for OpenCF Summit, I decided to write a simple servlet to handle inbound mail and then hand things off to CFML for some additional work. I haven’t had time yet to revisit this so if anyone has any ideas based on what I’m describing of how we could get mail to hit CFML directly I’d be very interested to hear them.
The MailReceiverServlet
In the sample app project the code for the MailReceiverServlet is in the src directory, and then in the package org.opencfsummit. As I mentioned above it’s pretty straight-forward, but let’s go over the mail specific parts since this will help shed some light on how mail comes in via the POST to /_ah/mail.
Inside the servlet’s doPost method we first declare some variables and set some defaults starting on line 23:
Properties props = new Properties();
Session session = Session.getDefaultInstance(props, null);
MimeMessage message;
Address[] toAddresses = null;
Address[] fromAddresses = null;
ArrayList<String> recipients = null;
ArrayList<String> senders = null;
Couple of things to note here. First, just so there’s no confusion the Session in this case is a javax.mail.Session, not an HttpSession. So you can see here we’re dealing with actual Java mail stuff when the POST hits our servlet, which is also evidenced by the MimeMessage and javax.mail.Address arrays. The two string ArrayLists at the end we’ll use in a minute as an easy way to pass off the data we want to be available in CFML.
Next up in MailReceiverServlet we get the data from the inbound mail and parse out the to and from addresses:
try {
message = new MimeMessage(session, request.getInputStream());
toAddresses = message.getAllRecipients();
recipients = new ArrayList<String>();
fromAddresses = message.getFrom();
senders = new ArrayList<String>();
for (Address address : toAddresses) {
recipients.add(address.toString());
}
for (Address address : fromAddresses) {
senders.add(address.toString());
}
} catch (MessagingException me) {
// gulp
}
Let’s step through that code briefly.
First, we instantiate a new MimeMessage using a mail session and the data coming from the request’s input stream. Once we have a handle on the message, we can get all of the to addresses contained in the message (message.getAllRecipients()) and all the from addresses as well (message.getFrom()).
We then loop over each address array (toAddresses and fromAddresses) and add each address as a String to the ArrayLists (recipients and senders).
With the simple string versions of the addresses in ArrayLists, we’ll then set these as attributes in the request object and forward the request to a CFML page:
request.setAttribute("senders", senders);
request.setAttribute("recipients", receipients);
getServletContext().getRequestDispatcher("/mailhandler.cfm").forward(request, response);
What this last chunk of code does it puts the senders and recipients ArrayLists in the request object, which translates into the request scope on the CFML side, and then forwards the request to mailhandler.cfm for some additional work.
And of course we have to define this servlet and a servlet mapping in our web.xml:
<servlet>
<servlet-name>mailReceiver</servlet-name>
<servlet-class>org.opencfsummit.MailReceiverServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>mailReceiver</servlet-name>
<url-pattern>/_ah/mail/*
</servlet-mapping>
mailhandler.cfm
With the initial incoming mail request handled by our MailReceiverServlet and the request forwarded to mailhandler.cfm (which is in the war directory if you’re following along in the sample app), we can make quick work of the rest of our task.
This might not be the most practical example, but for the purposes of demonstration what we’re doing in mailhandler.cfm is if anyone sends an email to stallmanbook@opencfsummit2011.appspotmail.com the application will respond by emailing a copy of Richard Stallman’s Free Software, Free Society as a PDF attachment. Of course you can do anything you want when your app receives an email but this is the little sample I dreamed up. And this is live by the way, so go ahead and try sending an email to stallmanbook@opencfsummit2011.appspotmail.com to get your free gift. 😉
In mailhandler.cfm we first check to see if the recipients array (which is the ArrayList of strings that we created in MailReceiverServlet and then put in the request object) contains the appropriate email address for this particular action:
<cfif request.recipients.contains("stallmanbook@opencfsummit2011.appspotmail.com")>
Remember that when we configured the servlet mapping for MailReceiverServlet we used a wildcard in the URL pattern (/_ah/mail/*), so all inbound mail to any address will hit MailReceiverServlet. We only want to our app to send a copy of Stallman’s book when one of the “to” addresses on the inbound email is stallmanbook@opencfsummit2011.appspotmail.com
After we check to see if the correct email address is one of the “to” addresses, we loop over the “from” addresses and use our old friend CFMAIL to send an email with the PDF attachment to each “from” address:
<cfloop index="i" from="1" to="#ArrayLen(request.senders)#">
<cfmail from="stallmanbook@opencfsummit2011.appspotmail.com"
to="#request.senders[i]#"
subject="Stallman Book!"
mimeattach="#ExpandPath('/pdfs/fsfs-ii-2.pdf')#">
Here is the Stallman book you asked for. Enjoy!
</cfmail>
</cfloop>
Pretty slick, huh?
Now that we’re well-versed in receiving mail, let’s move on to XMPP. Luckily since many of the basic concepts are the same we’ll be able to pick up the pace a bit.
Google App Engine XMPP Service
The GAE XMPP service works very similarly to the mail service, with a few more options specific to XMPP. If you’re not familiar with XMPP, XMPP stands for “Extensible Messaging and Presence Protocol” and while it can be used in a variety of ways, the usage people are most familiar with is for instant messaging. If you use Google Talk, you use XMPP. Also you may be running something like your own Jabber or OpenFire server in your organizaion for non-public IM, and these are based on XMPP.
As a quick reminder, when we configured inbound services for XMPP on our app we enabled:
- xmpp_message: allows the app to receive XMPP messages
- xmpp_presence: allows other XMPP users to see your applicaton’s IM user presence (online, offline, busy, etc.)
- xmpp_subscribe: allows other XMPP users to “friend” your application’s IM user as well as exchange message and status data
As with inbound mail, in order to get this running on your app all you need to do is specify XMPP inbound services in appengine-web.xml. With that in place once you deploy your app it is XMPP enabled.
Also as with inbound mail, your application gets its own XMPP IDs automatically. So once you enable XMPP your application can interact with the XMPP world using one of these two ID formats:
- YOUR_APP_ID@appspot.com
- ANYTHING@YOUR_APP_ID.appspotchat.com
(And yes, the domain is different between these two ID formats. The first is appspot.com, the second is appspotchat.com.)
Yet again as with inbound mail (I hope you’re sensing a pattern), when an XMPP message hits your application it generates an HTTP POST to /_ah/xmpp/message/chat so all you have to do is write whatever code you want to handle incoming XMPP messages.
The XmppReceiverServlet
As discussed above it isn’t really possible (yet) to have a POST to /_ah/xmpp/message/chat hit CFML directly, so we need a servlet that will handle POSTS to this URL pattern. The code for XmppReceiverServlet in the sample app projects is in the src directory, in the org.opencfsummit package.
First we declare some static final service type stuff in our servlet starting on line 21:
private static final Logger log = Logger.getLogger(XmppReceiverServlet.class.getName());
private static final XMPPService xmppService = XMPPServiceFactory.getXMPPService();
This gives us a logger so we can log incoming messages, as well as the service we’ll be using to parse the inbound XMPP message. Side note: logging on GAE is a bit quirky so make sure and check my post about configuring logging on GAE if you run into issues.
In the doPost method in XmppReceiverServlet we’ll parse the message, do some logging, parrot the message back to the sender (just to show we can send XMPP), and then we’ll again forward the request on to a CFML page to do some additional stuff.
Message message = xmppService.parseMessage(request);
JID jid = message.getFromJid();
String body = message.getBody();
log.info(jid.getId() + " --> opencfsummit2011: " + body);
message = new MessageBuilder().withRecipientJids(jid).withBody(body).build();
xmppService.sendMessage(message);
request.setAttribute("toJID", "opencfsummit2011@appspot.com");
request.setAttribute("fromJID", jid.getId());
request.setAttribute("messageType", message.getMessageType().toString());
request.setAttribute("body", body);
getServletContext().getRequestDispatcher("/xmppdump.cfm").forward(request, response);
That’s a bit of a mouthful so let’s walk through this.
First we use the xmppService to parse the XMPP message out of the request object. This gives us an object of type com.google.appengine.api.xmpp.Message and represents the incoming XMPP message. Next, we get the user ID (JID) of the XMPP user who sent the message as an object of type com.google.appengine.api.xmpp.JID, and get the body of the message as a string. We then log this info which I was simply using for debugging purposes as I was working on the sample app.
Next, we build a new XMPP message using the MessageBuilder, and use the xmppService to send the user’s message back to them. Obviously this isn’t a great real-world example but getting the message you send bounced back to you is a nice way to make sure things are working properly.
After that, we put all the relevant information from the XMPP message into the request object so we’ll have it available in the request scope on the CFML side. We then forward the request to xmppdump.cfm.
As with any servlet we need to add this to our web.xml and set up a servlet mapping so this servlet will respond to POSTs to /_ah/xmpp/message/chat:
<servlet>
<servlet-name>xmppReceiver</servlet-name>
<servlet-class>org.opencfsummit.XmppReceiverServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>xmppReceiver</servlet-name>
<url-pattern>/_ah/xmpp/message/chat/</url-pattern>
</servlet-mapping>
xmppdump.cfm
Once the request is forwarded to xmppdump.cfm (which is in the war directory of the project) we can do anything we want with this information on the CFML side. In this case I’m first emailing myself the contents of CFDUMP for debugging/testing purposes:
<cfsavecontent variable="foo">
<cfdump />
</cfsavecontent>
<cfmail from="mpwoodward@gmail.com"
to="matt@mattwoodward.com"
subject="foo" type="html">
#foo#
</cfmail>
One thing I didn’t mention earlier about sending mail from your GAE app is that the from address must be either:
- An administrator/developer on the application, or
- The address of the user who’s currently signed in with their Google Account, or
- A valid “receiving” email address for the application (e.g. whatever@myappid.appspotmail.com)
Next, I decided just for fun I’d leverage CFHTTP to create a bit of an XMPP-to-SMS service. Any IMs that come into the application are turned into an SMS and sent to a (in this case hard-coded) mobile number using Twilio.
<cfhttp result="smsTestSend"
method="post"
url="https://MY_TWILIO_URL"
username="bar"
password="baz">
<cfhttpparam type="formfield" name="From" value="5551234567" />
<cfhttpparam type="formfield" name="To" value="5559876543" />
<cfhttpparam type="formfield" name="Body" value="#request.body#" />
</cfhttp>
As you can see Twilio just takes an HTTP POST and turns it into an SMS message, and here since I have the body of the inbound XMPP message available as request.body, that’s what becomes the body of the SMS message.
Yet another example that’s of dubious use in the real world, but I hope it illustrates the points.
Try It Out!
This sample is live as well. I removed the part that sends text messages but you can log into Google Talk and add opencfsummit2011@appspot.com as a buddy, and any IMs you send will be bounced back to you. OK, enough testing. Stop talking to yourself. 😉 As a next step I was going to try to implement an Eliza-type bot but as with most things these days just didn’t yet have the time.
Conclusion
In this post you learned how to leverage two more services available to you in the amazing Google App Engine platform, specifically the inbound mail and XMPP services. They’re extremely easy to set up, incredibly powerful, and with just a bit of servlet code in the middle this all plays very nicely with Open BlueDragon for GAE.
If you have comments or questions fire away, and if there are specific aspects of GAE you’d like to know more about (I have some additional posts in mind) please let me know. Thanks!
