Wednesday, May 27, 2015

Installing the AWS SDK for PHP onto my Raspberry Pi


Just some notes for my own reference on getting the AWS SDK for PHP working on my RaspberryPi.

1. Installed PHP and Apache (needed for this particular project)

pi@raspberrypi ~ $ sudo apt-get install apache2 php5 libapache2-mod-php5

2. Moved my project folder and created the composer.json file (as per the SDK installation instructions).

nano composer.json

3. Popped in the required JSON.

{
    "require": {
        "aws/aws-sdk-php": "2.*"
    }

}

4. Ran the installer and ... bop-bow! ....

pi@raspberrypi ~/phpscripts/greenmode $ php composer.phar install
Loading composer repositories with package information
Installing dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1

    - aws/aws-sdk-php 2.4.0 requires guzzle/guzzle ~3.7.0 -> satisfiable by guzzle/guzzle[v3.7.0, v3.7.1, v3.7.2, v3.7.3, v3.7.4].

... and

 guzzle/guzzle v3.9.3 requires ext-curl * -> the requested PHP extension curl is missing from your system.

4. Loaded up the php5-curl package ....

pi@raspberrypi /var/www $ sudo apt-get install php5-curl

5. Everything working!

pi@raspberrypi ~/phpscripts/greenmode $ php composer.phar install
Loading composer repositories with package information
Installing dependencies (including require-dev)
  - Installing symfony/event-dispatcher (v2.6.8)
    Downloading: 100%         

  - Installing guzzle/guzzle (v3.9.3)
    Downloading: 100%         

  - Installing aws/aws-sdk-php (2.8.7)
    Downloading: 100%    

Just to be sure ... let's do something .... 

A quick Ec2 iterator script ...

<?php 

        require 'vendor/autoload.php';

        $ec2Client = \Aws\Ec2\Ec2Client::factory(array(
                'profile' => 'dev',
                'region'  => 'ap-southeast-2'
        ));


        function allInstances(){

                $iterator = $GLOBALS['ec2Client']->getIterator('describeInstances',array(
                        'Filters' => array(
                                        array(

                                                'Name' => 'instance-state-name',
                                                'Values' => array('running')

                                                )

                                        ))
                                );

                foreach($iterator as $object){

                        echo $object['InstanceId'] . PHP_EOL;
                }

        }


echo print_r(allInstances(),true);

... and we get our instances back. Sweet!

pi@raspberrypi ~/phpscripts/greenmode $ /usr/bin/php sdktest.php
i-4fcc3a81
i-4585708b
i-b4a6697a

i-960efa58

Tuesday, May 19, 2015

My First Lambda Function

I thought is was about time I got familiar with AWS Lambda.

AWS Lambda is an event-driven computing service, which at that time of writing is available in US-EAST (Virginia) , US-WEST (Oregon) and EU (Ireland).

Lambda allows custom functions, written in NodeJs (server side JavaScript), to be executed, on-demand and in response to particular triggers, or events.

My Problem: We have an application which sends out thousands of emails a day. Our application uses another AWS Service SES (Simple Email Service).

When sending hundreds of emails, it's not un-common to get a lot of delivery failures, or bounces. These bounces will eventually make their way back to a service mail box which someone or something can trawl through to build up a list of dead email addresses.

This to me seemed like a perfect candidate process for a bit of "cloudification", and so we begin.

My project uses the following services,

NOTE: I'm using region N.Virginia for all non-global services.

SES (N.Virginia)
SNS (N.Virginia)
DynamoDB (N.Virginia)
Lambda (N.Virginia)
IAM (Global)

SES (Simple Email Service)


The first thing we need to do is setup SES to forward bounce and complaint notifications to an SNS topic.  You'll need to go to your verified domain or email address to modify the notification settings.



If you don't have an SNS topic setup, you can click on the "Click here to create a new Amazon SNS topic" click.


Simply enter a Topic Name and Display Name



Once you've done that, pick the SNS topic from the drop down list. I didn't bother with Deliveries notifications because it would generate a lot to notifications.


DynamoDB 


The next step in my little project was to create a new DynamoDB table in which to store the information I'm going to gather from my bounce notifications.

The details of DynamoDb table creation are beyond the scope of this article, but I've included some high level steps below.



Create a new table, I called mine sesNotifications. I set a hash and range primary key type for which I'll be using the SNS Topic Arn (Amazon Resource Name) as the hash and the snsPublishTime stamp as the range. This will give me a nice sorted range index based based on the time the notification was received.


I also added the messageId as a global secondary index so that in the future, I could search for notifications based on the SES Message IDs (another project we're working on).



You'll next need to specific the read and write capacity for the database ... I'll leave that one to you.



Click continue through the remainder of the screens and your DynamoDB table will be created for you.

Lambda 


Whilst we wait for our DynamoDB table to create, we can now move on to the exciting part of the project. Lambda.

Jump to the Lambda management consoleand click on the massive blue button which says "Get Started Now" to begin building your first Lambda function.

Some points to note, Lambda functions are written in javascript, they can be developed locally and uploaded as Zip files, including all of the necessary packages. I'm not there yet, so I'm going to do everything inline, via the GUI.

First things first, give the function a name and a description:



In the function code window, you can choose from a number of templates, or simply create your own:



My function was initially based on the SNS Message template, and here it is.

var aws = require('aws-sdk');
var ddb = new aws.DynamoDB({params: {TableName: 'sesNotification'}});
 
exports.handler = function(event, context) {
  var SnsMessageId = event.Records[0].Sns.MessageId;
  var SnsPublishTime = event.Records[0].Sns.Timestamp;
  var SnsTopicArn = event.Records[0].Sns.TopicArn;
  var SnsMessage = event.Records[0].Sns.Message;
  var LambdaReceiveTime = new Date().toString();
  
  var MessageContent = JSON.parse(SnsMessage);
  var SesNotify = MessageContent['notificationType'];
  var SesFailedTarget = MessageContent['bounce']['bouncedRecipients'][0]['emailAddress'];
  var SesFailedCode = MessageContent['bounce']['bouncedRecipients'][0]['diagnosticCode'];
  var SesMessageId = MessageContent['mail']['messageId'];

  
  var itemParams = {Item: {SnsTopicArn: {S: SnsTopicArn},
  SnsPublishTime: {
        S: SnsPublishTime}, 
        SnsMessageId: {S: SnsMessageId},
        LambdaReceiveTime: {S: LambdaReceiveTime},
        SnsMessage: {S: SnsMessage},
        SesNotificationType: {S: SesNotify}, 
        SesTarget: {S: SesFailedTarget},
        SesError: {S: SesFailedCode},
        SesError: {S: SesMessageId}
}}; ddb.putItem(itemParams, function() { context.done(null,''); }); };

With code in place, we need to assign a role to the Lambda function to allow it to interact with the DynamoDB table we've created.



To keep things simple at this stage, you could choose "Basic with Dynamo", this role will allow the function all of the rights it needs to interact with DynamoDB.

If you want to be a little more granular, you can use the "Basic execution role" and add a role policy that looks a bit like this ...

{
  "Version": "2012-10-17",
  "Statement":[
    {
      "Sid":"AllowDynamoDbAccess",
      "Effect":"Allow",
      "Action":["dynamodb:*"],
      "Resource":["arn:aws:dynamodb:us-east-1:<blahblahblah>:table/sesNotification"]    }
  ]
}

Once the role / policy has been created, assign it to the function and click the big blue "Create Lambda Function" button at the bottom of the screen.

SNS (Simple Notification Service) 


The final step in the project to bring it all together is to push the SNS notifications to our newly created Lambda function.

Hop on over to the SNS management console  and track down the SNS topic we created earlier.



Click on the highlighted ARN to view the topic details. Click on the "Create Subscription" button


The topic ARN will be auto populated, choose AWS Lambda from the protocol list and choose the new Lambda Function from the Endpoint drop list.



Click Create Subscription.

And that's it. We should now be able to send a few emails to addresses which don't exist and wait for the bounces to start showing up in DynamoDb

I have a small PHP script which I use to test sending emails via SES, I modified a few parameters with a non-existent email address and hey presto, this is what we get.



We've got the table items containing the SES Message ID, the failed target address and the Error Code. All very useful.


A little about Me

My photo
My name is Mitch Beaumont and I've been a technology professional since 1999. I began my career working as a desk-side support engineer for a medical devices company in a small town in the middle of England (Ashby De La Zouch). I then joined IBM Global Services where I began specialising in customer projects which were based on and around Citrix technologies. Following a couple of very enjoyable years with IBM I relocated to London to work as a system operations engineer for a large law firm where I responsible for the day to day operations and development of the firms global Citrix infrastructure. In 2006 I was offered a position in Sydney, Australia. Since then I've had the privilege of working for and with a number of companies in various technology roles including as a Solutions Architect and Technical team leader.