Tuesday, May 19, 2015

My First Lambda Function

I thought is was about time I got familiar with AWS Lambda.

AWS Lambda is an event-driven computing service, which at that time of writing is available in US-EAST (Virginia) , US-WEST (Oregon) and EU (Ireland).

Lambda allows custom functions, written in NodeJs (server side JavaScript), to be executed, on-demand and in response to particular triggers, or events.

My Problem: We have an application which sends out thousands of emails a day. Our application uses another AWS Service SES (Simple Email Service).

When sending hundreds of emails, it's not un-common to get a lot of delivery failures, or bounces. These bounces will eventually make their way back to a service mail box which someone or something can trawl through to build up a list of dead email addresses.

This to me seemed like a perfect candidate process for a bit of "cloudification", and so we begin.

My project uses the following services,

NOTE: I'm using region N.Virginia for all non-global services.

SES (N.Virginia)
SNS (N.Virginia)
DynamoDB (N.Virginia)
Lambda (N.Virginia)
IAM (Global)

SES (Simple Email Service)

The first thing we need to do is setup SES to forward bounce and complaint notifications to an SNS topic.  You'll need to go to your verified domain or email address to modify the notification settings.

If you don't have an SNS topic setup, you can click on the "Click here to create a new Amazon SNS topic" click.

Simply enter a Topic Name and Display Name

Once you've done that, pick the SNS topic from the drop down list. I didn't bother with Deliveries notifications because it would generate a lot to notifications.


The next step in my little project was to create a new DynamoDB table in which to store the information I'm going to gather from my bounce notifications.

The details of DynamoDb table creation are beyond the scope of this article, but I've included some high level steps below.

Create a new table, I called mine sesNotifications. I set a hash and range primary key type for which I'll be using the SNS Topic Arn (Amazon Resource Name) as the hash and the snsPublishTime stamp as the range. This will give me a nice sorted range index based based on the time the notification was received.

I also added the messageId as a global secondary index so that in the future, I could search for notifications based on the SES Message IDs (another project we're working on).

You'll next need to specific the read and write capacity for the database ... I'll leave that one to you.

Click continue through the remainder of the screens and your DynamoDB table will be created for you.


Whilst we wait for our DynamoDB table to create, we can now move on to the exciting part of the project. Lambda.

Jump to the Lambda management consoleand click on the massive blue button which says "Get Started Now" to begin building your first Lambda function.

Some points to note, Lambda functions are written in javascript, they can be developed locally and uploaded as Zip files, including all of the necessary packages. I'm not there yet, so I'm going to do everything inline, via the GUI.

First things first, give the function a name and a description:

In the function code window, you can choose from a number of templates, or simply create your own:

My function was initially based on the SNS Message template, and here it is.

var aws = require('aws-sdk');
var ddb = new aws.DynamoDB({params: {TableName: 'sesNotification'}});
exports.handler = function(event, context) {
  var SnsMessageId = event.Records[0].Sns.MessageId;
  var SnsPublishTime = event.Records[0].Sns.Timestamp;
  var SnsTopicArn = event.Records[0].Sns.TopicArn;
  var SnsMessage = event.Records[0].Sns.Message;
  var LambdaReceiveTime = new Date().toString();
  var MessageContent = JSON.parse(SnsMessage);
  var SesNotify = MessageContent['notificationType'];
  var SesFailedTarget = MessageContent['bounce']['bouncedRecipients'][0]['emailAddress'];
  var SesFailedCode = MessageContent['bounce']['bouncedRecipients'][0]['diagnosticCode'];
  var SesMessageId = MessageContent['mail']['messageId'];

  var itemParams = {Item: {SnsTopicArn: {S: SnsTopicArn},
  SnsPublishTime: {
        S: SnsPublishTime}, 
        SnsMessageId: {S: SnsMessageId},
        LambdaReceiveTime: {S: LambdaReceiveTime},
        SnsMessage: {S: SnsMessage},
        SesNotificationType: {S: SesNotify}, 
        SesTarget: {S: SesFailedTarget},
        SesError: {S: SesFailedCode},
        SesError: {S: SesMessageId}
}}; ddb.putItem(itemParams, function() { context.done(null,''); }); };

With code in place, we need to assign a role to the Lambda function to allow it to interact with the DynamoDB table we've created.

To keep things simple at this stage, you could choose "Basic with Dynamo", this role will allow the function all of the rights it needs to interact with DynamoDB.

If you want to be a little more granular, you can use the "Basic execution role" and add a role policy that looks a bit like this ...

  "Version": "2012-10-17",
      "Resource":["arn:aws:dynamodb:us-east-1:<blahblahblah>:table/sesNotification"]    }

Once the role / policy has been created, assign it to the function and click the big blue "Create Lambda Function" button at the bottom of the screen.

SNS (Simple Notification Service) 

The final step in the project to bring it all together is to push the SNS notifications to our newly created Lambda function.

Hop on over to the SNS management console  and track down the SNS topic we created earlier.

Click on the highlighted ARN to view the topic details. Click on the "Create Subscription" button

The topic ARN will be auto populated, choose AWS Lambda from the protocol list and choose the new Lambda Function from the Endpoint drop list.

Click Create Subscription.

And that's it. We should now be able to send a few emails to addresses which don't exist and wait for the bounces to start showing up in DynamoDb

I have a small PHP script which I use to test sending emails via SES, I modified a few parameters with a non-existent email address and hey presto, this is what we get.

We've got the table items containing the SES Message ID, the failed target address and the Error Code. All very useful.


Rich Gopstein said...
This comment has been removed by the author.
Rich Gopstein said...

First, thanks for this! I was looking to do exactly the same thing.

I was curious, in itemParams, did you intend to have one of the SesError labels be SesMessageId?

SesError: {S: SesFailedCode},

SesError: {S: SesMessageId}

Eric Hammond said...

Nice application for the AWS services you used!

tip: In general, it's best to pick a DynamoDB hash key that has more widely distributed values, not a limited set of values. That way it can scale without having a performance bottleneck.

Unknown said...

Thanks for this write-up, this is exactly what I needed. The only thing I changed is setting the primary key to be the failed target email address. This way we don't get duplicates added to the database.

Anonymous said...

Thanks for the guide Mitch.

I actually re-visited your guide after reading through the AWS documentation trying to find the JSON object template so I can build my var's. However this was unsuccessful as I cant find this anywhere.

How would I export or find the full JSON object of a SES bounce notification which I can then choose my key/values ?

Anonymous said...

Never mind just found it! (still learning node.js)

console.log('Loading event');
var aws = require('aws-sdk');
exports.handler = function(event, context) {
console.log("event: ", JSON.stringify(event, null, 4));

This will output the raw json object to cloudwatch logs for that function run.

Unknown said...

Thanks for a great how-to. I've been using this setup slightly modified in production for the past year. However I have modified the script slightly and wanted to share that back.
SES bounces/complaints can actually contain more than one email address so I now iterate through those. Also, if you send an email to "'Firstname Lastname' " that entire string gets returned as the bounced recipient. To make it easier for a subsequent suppressionlist lookup I've added code to strip the human friendly name and <>s from the email address before storing it in DynamoDB.
My code can be found in my github below, and again thanks for the excellent idea.

aws jobs in hyderabad said...

Great... Excellent sharing.. This is very helpful for beginners. Read that provide me more enthusiastic. This helps me get a more knowledge about this topic. Thanks for this.hunt aws jobs in hyderabad

A little about Me

My photo
My name is Mitch Beaumont and I've been a technology professional since 1999. I began my career working as a desk-side support engineer for a medical devices company in a small town in the middle of England (Ashby De La Zouch). I then joined IBM Global Services where I began specialising in customer projects which were based on and around Citrix technologies. Following a couple of very enjoyable years with IBM I relocated to London to work as a system operations engineer for a large law firm where I responsible for the day to day operations and development of the firms global Citrix infrastructure. In 2006 I was offered a position in Sydney, Australia. Since then I've had the privilege of working for and with a number of companies in various technology roles including as a Solutions Architect and Technical team leader.