Monday, July 09, 2018

Building a CD Pipeline for Fargate from the Command-Line

In an earlier post, AWS Fargate from the Command-Line, we packaged up a simple Go HTTP web server as a Docker container image and deployed it in to Amazon ECS as an AWS Fargate backed service ... using only the command-line. Pretty sweet. 

In this post, we're going to take our project to the next-level and provision a Docker powered pipeline that'll take our simple Go HTTP web server, build it and prepare it to be shipped to Amazon ECS.

But why command-line?


Everything I've done in this post, and my previous posts, could just as easily be done using the AWS management console. I'd definitely encourage you to run through this process using the management console to get a feel for the differences. The management console, and more specifically the first-run wizards, do a good job of abstracting away a lot of the implementation details, which is great if you're looking to get started quickly.

That being said, I've found that by assembling the Lego blocks using the command-line (or any of the SDKs), you end up with a much deeper understanding of the relationships between those components.


 

TL;DR

By the end of this post you'll have a pipeline that can be used to build and deploy projects to an ECS cluster written in most languages using most tools.

Let's get started!


In the previous post, you'll recall that we had a simple HTTP server written in Go. I'd placed the main.go file and a Dockerfile in to a standard directory on my computer. 

What does every good software project need?


As with any good software project, the source code should be stored in a version control system. As it happens, AWS has fully managed VCS, AWS CodeCommit.

Let's start by creating ourselves a repository for storing our project.


aws codecommit create-repository --repository-name go-frontend

Git, Git, Git, Git ... 


Now, we need to turn our vanilla working directory in to a mean, green git-tracked machine by running a few simple commands.

# Initialise my workspace as a git tracked repo
git init

# Add the dockerfile and the main.go
git add .

# Do an initial commit
git commit -m 'initial commit'

Cool. So, we've now converted our existing folder in a git-tracked repo. Albeit locally tracked and stored. The first step in addressing that shortcoming is setting up some authentication (auth) for the hosted CodeCommit repository we just created.

I'm going to use public/private key-pair for auth. It's also possible to setup some git credentials using the AWS IAM console, but I prefer SSH keys. That's all I have to say about that.

I'm going to start by generating a new key-pair as follows:

ssh-keygen

During the key-pair creation process you'll be prompted to choose a file in which to store the private key. I opted to create a new file specifically for this new private key so that my existing private key (which I use for other things) did not get overwritten.

The corresponding public key will be written to a file with the same name but with the .pub extension.

The next step in the process requires us to copy the public key material, from the file with the .pub extension, and upload it up to AWS. To do this, cat the file and copy it's content to your clipboard.

cat Users/mitch/.ssh/codecommit_rsa.pub


Next, we use the AWS CLI to upload the public key material and associate it with an IAM user. In my case, my IAM has a name of mitch.beaumont, so that who I'm going to attach the public key to.

Paste the public key material you copied earlier as the value for the --ssh-public-key-body argument, don't forget to paste between quotes " ".

aws iam upload-ssh-public-key \
--user-name mitch.beaumont \
--ssh-public-key-body "ssh-rsa ..."

The response from this step will include an SSHPublicKeyId. You'll need this for the next step so take a note.

We're now going to hack our SSH config file (~/.ssh/config) to help make using the custom SSH key we generated for auth, a little more seamless.

Host git-codecommit.*.amazonaws.com
  User <whatever_your_SSHPublicKeyId_was>
  IdentityFile ~/.ssh/<whatever_your_private_key_file_is_called>

If you had to create the SSH config file then you'll need to chmod the permissions. Otherwise, you should be all set.

chmod 600 ~/.ssh/config

We now need to update the configuration of our local repo and add a "remote". This is the URL of the remote location were we want the contents of our repo to be stored. In our case, this is CodeCommit.

CodeCommit, much like GitHub, support two different kinds of URLs for repos. HTTPS and SSH. Because we're auth'ing using SSH, we need to use the SSH URL.

git remote add origin ssh://git-codecommit.us-west-2.amazonaws.com/v1/repos/go-frontend

Let's check that the remote URL took ...

git remote -v

Now we're ready to push our local repo up to our remote.

git push --set-upstream origin master

The stage is set!

We now find ourselves in a great position. As we get new ideas for killer features that we want to add to our simple Go HTTP web server, we hack away, commit the changes and push them to a remote location for safe-keeping.

For some people however, that might not be enough. WE ARE THOSE PEOPLE! We WANT continuous delivery.

Each time we commit a change, we want those changes to trigger a pipeline that takes our updated code, mixes it with some fairy-dust and spits our a fully functional container image that we can do something with, like scan, test or deploy to an environment.

 Something that looks a bit like this ...

Bill of Materials

The diagram above looks fairly straightforward. HOWEVER. There are a few things we're going to need in place to make our pipeline a reality. So let's take a look at what they are:

  1. CodeCommit Repoistory (we already have this)
  2. An S3 bucket to store various artifacts produced and consumed by the stages of the pipeline.
  3. An IAM role with the necessary permissions to to execute the various stages of our pipeline. 
  4. An IAM role with the necessary permissions to execute our build processes.
  5. A CodeBuild project.
  6. A CodePipeline pipeline.
  7. A CloudWatch event that fires each time a change is made to the code stored in the CodeCommit repository.

A place to call artifactStore

We begin the next exciting chapter of our adventure, but creating an S3 bucket. This bucket will be used by our pipeline for storing artifacts.

What's an artifact?

Good question! Simply put, Artifacts are files that are created by a stage in the pipeline. You can choose to discard the artifacts, or keep them and share them with other stages in the pipeline. (Which is what we'll be doing)

The S3 bucket we're creating in this step is where those artifacts we choose to keep will be stored. Keep a note of the name of the bucket. You'll need it a little later.

aws s3 mb s3://codepipeline-${REGION}-${ACCOUNT_ID}

 

Show me the permissions!

If you look back at our diagram, you can see that CodePipeline kind of runs the show. It's job is to share artifacts, trigger builds and generally keep things moving forwards. Because of that, we need to pass some privileges to CodePipeline. We do that via an IAM role.

If we want an AWS service, such as CodePipline, to assume an IAM role in our AWS account, we have to create a trust relationship. In our case, we are the trusting party and our policy says, we trust <service_whatever> to perform some actions within the boundaries of our AWS account. 

Let's start by defining that trust relationship and creating a file called pipeline_trust_policy.json.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "codepipeline.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Now we'll create a new IAM role called CodePipelineExecutionRole and configure it to use the trust policy we defined in pipeline_trust_policy.json.

Be sure to record the role Arn from the output of this step. You'll be needing that a little later.

aws iam create-role \ 
--role-name CodePipelineExecutionRole \
--assume-role-policy-document \
file://./pipeline_trust_policy.json

Next we need to define the actions we want our "trustee" to be able to perform. Create another file and called it pipeline_permissions_policy.json.

This policy file gives the trustee, the CodePipeline service, permissions to perform a bunch of actions, including allowing it to retrieve code from our CodeCommit repository, artifacts from our artifact store and some additional rights, which we may tap in to later, for deploying in to ECS:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "codecommit:GetBranch",
                "codecommit:GetCommit",
                "codecommit:UploadArchive",
                "codecommit:GetUploadArchiveStatus",
                "codecommit:CancelUploadArchive",
                "codebuild:BatchGetBuilds",
                "codebuild:StartBuild"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::codepipeline*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ecs:DescribeServices",
                "ecs:DescribeTaskDefinition",
                "ecs:DescribeTasks",
                "ecs:ListTasks",
                "ecs:RegisterTaskDefinition",
                "ecs:UpdateService"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:iam:PassRole",
            "Resource": "*"
        }
    ]
}

Lets use the pipeline_permissions_policy.json file to create a policy and attach it to the CodePipelineExecutionRole IAM role we created earlier.

Here we are creating an inline-policy. A better way to do this to help with future reuse might be to create a separate policy object and attach it. That way the same policy object can be reused by other roles in the future.

aws iam put-role-policy \
--role-name CodePipelineExecutionRole \
--policy-name CodePipeline-Permissions-Policy \
--policy-document file://./pipeline_permissions_policy.json

Now that we have an IAM role for our pipeline to use when executing, the next thing we need to do is create a similar IAM role with which to execute our CodeBuild project.

Another trust policy is needed first:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "codebuild.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Then the policy that defines the actions we need our CodeBuild project to be able to perform.

A few things worth calling out here, firstly, you'll notice that there are some "Allow" actions for CloudWatch Logs. This is required for CodeBuild to output the logs from the build project. Extremely useful when trying to debug our build process.

There are also a number of "Allow" actions for ECR. Cast your mind back to the original goal of this walk-through. We want to be at a point where we have an artifact that can be deployed following each push / merge to our master branch. (Continuous delivery).

The artifact we're producing is a container image. This is why our CodeBuild project needs access to ECR. It needs to be able to push the container image there.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:/aws/codebuild/*:log-stream:*"
            ]
        },
        {
            "Effect": "Allow",
            "Action":[
                "s3:GetObject",
                "s3:PutObject",
                "s3:GetObjectVersion",
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability",
                "ecr:PutImage",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload",
                "ecr:GetAuthorizationToken"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Let's create our IAM role for CodeBuild and attach the trust policy ...

aws iam create-role \
--role-name CodeBuildExecutionRole \
--assume-role-policy-document file://./build_trust_policy.json \

... and add the permissions policy.

aws iam put-role-policy \
--role-name CodeBuildExecutionRole \
--policy-name CodeBuild-Permissions-Policy \
--policy-document file://./build_permissions_policy.json

Buildy build build build.

What we're going to do now is create and configure a CodeBuild project. This project will take, as an input, the source-code from our CodeCommit repo, compile-it and spit out a container image.

I've taken a slightly different approach to this step in that I'm passing the configuration of my pipeline in as command line arguments.

You'll need to the Arn of the CodeBuildExecution role and the URI of the CodeCommit repository for this step.

Something of note here is the "image" property. Code build uses containers to execute the steps in the build project ...

Hang-on, aren't you building a container?

So, we're using a container, to build a container ... interesting. This approach is often referred to as DinD or Docker-in-Docker. (The everything-pipeline starts to take shape!)

Check out a full breakdown of what goes in to the aws/codebuild/docker:17.09.0 image here:

https://github.com/aws/aws-codebuild-docker-images/tree/master/ubuntu/docker/17.09.0

Let's create our build project. Don't forget to grab the name of the CodeBuild project as you'll need that for the next step.

aws codebuild create-project \
--name "goFrontend" \
--description "Build project for GoFrontend" \
--source type="CODEPIPELINE" \
--service-role "<Arn_of_codepbuildexecutionrole_role>" \
--environment type="LINUX_CONTAINER",\
image="aws/codebuild/docker:17.09.0",\
computeType="BUILD_GENERAL1_SMALL",\
environmentVariables="[{name=AWS_DEFAULT_REGION,value='${REGION}'},{name=AWS_ACCOUNT_ID,value='${ACCOUNT_ID}'},{name=REPOSITORY_URI,value=<Uri_of_ECR_repository>}]" \
--artifacts type="CODEPIPELINE"

buildspec.yml

How do we tell CodeBuild what to do as part of the build project? For that, we need to add a file called buildspec.yml to out project folder. This file defines all of the steps that we need CodeBuild to execute on our behalf. There is a lot going on in this file, but most of what you can see relates to extracting the metadata from our build process and using it to tag the resulting container image correctly, then pushing it to ECR.

You can grab the buildspec.yml file from a gist I've crested here:

Place the file in the root of your project and be sure to run git add / git commit / git push.

Your project directory should be looking something like this once you're done.


.
├── Dockerfile
├── buildspec.yml
└── main.go

We're almost there!

Most of the pieces of our puzzle are in-place. We now need to apply the glue that's going to hold everything in place: Our Pipeline.

We'll start by creating a new file pipeline_structure.json. This file defines the attributes and structure of our pipeline, including things like the role used to execute the pipeline and the stages that make up the pipeline.

You can see I've defined 3 stages in my pipeline

Source

This stage retrieves the source code of my project from CodeCommit. It then bundles up that source code as an output artifact and drops it in to our artifactStore. In the configuration section you'll notice that I've set PollForSourceChanges to "false". I've done this because I'm going to use CloudWatch Events to trigger my pipeline each time a change is made.

Build

This stage triggers the build pipeline that we created earlier. The input artifact of this stage is the output artifact of the Source stage (the bundles up source code). You'll notice I've created and output artifact called imagedefinitions. This file contains a simple JSON structure that includes the name of the container my container image represents and the container image tag (which is the build Id).

Release: 

The final stage is the release stage. This stage has two actions. Firstly, a manual approval, which is a blocking action. Assuming the previous stage completes successfully, the pipeline will halt at this point waiting for intervention from the operator. Once the operator "approves" or "rejects, the pipeline will continue, or fail respectively.

The second action in this stage deploys the changes to our ECS cluster. Notice the configuration items here the name of the cluster, the name of the service and the imagedefinitions.json file.


{
        "roleArn": "<Arn_of_codepipelineexecutionrole_role>",
        "stages": [
            {
                "name": "Source",
                "actions": [
                    {
                        "inputArtifacts": [],
                        "name": "Source",
                        "actionTypeId": {
                            "category": "Source",
                            "owner": "AWS",
                            "version": "1",
                            "provider": "CodeCommit"
                        },
                        "outputArtifacts": [
                            {
                                "name": "MyApp"
                            }
                        ],
                        "configuration":{
                            "RepositoryName": "<name_of_codecommit_repository>",
                            "BranchName": "master",
                            "PollForSourceChanges": "false"
                        },
                        "runOrder": 1
                    }
                ]
            },
            {
                "name": "Build",
                "actions": [
                    {
                        "inputArtifacts": [
                            {
                                "name": "MyApp"
                            }
                        ],
                        "name": "Build",
                        "actionTypeId": {
                            "category": "Build",
                            "owner": "AWS",
                            "version": "1",
                            "provider": "CodeBuild"
                        },
                        "outputArtifacts": [
                            {
                                "name": "imagedefinition"
                            }
                        ],
                        "configuration":{
                            "ProjectName": "goFrontend"
                        },
                        "runOrder": 1
                    }
                ]
            },
            {
                "name": "Release",
                "actions": [
                    {
                        "name": "Approval",
                        "actionTypeId": {
                            "category": "Approval",
                            "owner": "AWS",
                            "version": "1",
                            "provider": "Manual"
                        },
                        "runOrder": 1
                    },
                    {
                        "inputArtifacts": [
                            {
                                "name": "imagedefinition"
                            }
                        ],
                        "name": "DeployToECS",
                        "actionTypeId": {
                            "category": "Deploy",
                            "owner": "AWS",
                            "version": "1",
                            "provider": "ECS"
                        },
                        "outputArtifacts": [],
                        "configuration":{
                            "ClusterName": "fargate-cluster",
                            "ServiceName": "go-http-server",
                            "FileName": "imagedefinitions.json"
                        },
                        "runOrder": 2
                    }
                ]
            }
        ],
        "artifactStore": {
            "type": "S3",
            "location": "<name_of_the_s3_bucket_artifact>"
          },
        "name": "GoFrontend",
        "version": 1
}

Let's create the pipeline now, using the pipeline_structure.json file as out input.

aws codepipeline create-pipeline \
--pipeline file://./pipeline_structure.json

Test our pipeline

Assuming nothing has imploded, we're now at a point where we can test out our pipeline. Guess what? We can also do that using the command-line!

aws codepipeline start-pipeline-execution \
--name GoFrontend

The previous command will output an pipelineExecitionId. We will use that to check on the status of our pipeline as follows.


aws codepipeline get-pipeline-state --name GoFrontend \
| jq '.stageStates[] | select (.latestExecution.pipelineExecutionId == "<pipelineExecutionId>")'

Approving the pipeline

After <___> minutes we should see our pipeline is paused at the "Approval" action of the "Release" stage

You'll notice a token value. We're going to use this to approve the pipeline action so that it can move to the next action in this state. Which happens to be: Release it to our ECR cluster.

{
            "stageName": "Release",
            "inboundTransitionState": {
                "enabled": true
            },
            "actionStates": [
                {
                    "actionName": "Approval",
                    "latestExecution": {
                        "status": "InProgress",
                        "lastStatusChange": 1234567.123,
                        "token": "cxxxxxd-8xxx5-4xx2-9xxc-9xxxxxxxxe"
                    }
                }

Let's approve the stage using the put-approval-result action.


aws codepipeline put-approval-result \
--pipeline-name GoFrontend \
--stage-name Release \
--action-name Approval \
--result summary="Approving",status="Approved" \
--token cxxxxxd-8xxx5-4xx2-9xxc-9xxxxxxxxe

If we check the state of our pipeline again, we should now see that it has moved forward.

Wrapping it in a bow.

Let's recap what were we are. We have a CodeCommit repository that is storing our project. An S3 bucket that is going to be used for storing artifacts shared across the stages of our pipeline. We have a CodeBuild project that takes our source code as an input, starts a Docker container to compile the simple Go HTTP web server in to a statically linked binary. Finally, we have a CodePipeline that pulls all of these disparate parts in to a nice cohesive workflow.

That last thing we need to do is configure a trigger for our pipeline each time a change is made to the master branch of our project. For this, we're going to use CloudWatch Events.

Just like every good service, we need to trust the CloudWatch Events service to perform actions on our behalf.

(Click on the files to check out an example gist for each policy)

Capture the Arn of the role, you'll need it a little later.

aws iam create-role \
--role-name Role-for-MyRule \
--assume-role-policy-document file://./event_trust_policy.json

Next, give our role some permissions.

aws iam put-role-policy \
--role-name Role-for-MyRule \
--policy-name CodePipeline-Permissions-Policy-For-CWE \
--policy-document file://event_permissions_policy.json

Now we need to create a rule. This includes a pattern to the kind of events that would trigger this event.

aws events put-rule \
--cli-input-json file://./event_put_rule.json 

Finally, we need to give our rule some targets. Which in our case, is CodePipeline.

aws events put-targets \
--rule CodeCommiteRule \
--targets Id=1,Arn=<arn_for_code_pipeline>,RoleArn=<arn_for_the_event_rule>

Change is good

Go a head and make a change to you main.go file. Perhaps change the message that gets displayed when you hit the we service. Once you've made the changes, commit and push them to CodeCommit. Then sit back and watch the magic happen.



Things to Fix

This approach, just like our software, has room for improvement. For example, we may some to be a little more explicit with some of the policies we created for the services. Naming the specific resources that actions can be performed against.

It would also be cool to setup a better way of responding to approval notifications. Check out this awesome post on the AWS blog for some ideas on how this might look with Slack.

https://aws.amazon.com/blogs/devops/use-slack-chatops-to-deploy-your-code-how-to-integrate-your-pipeline-in-aws-codepipeline-with-your-slack-channel/

As always, I'd love your feedback and thoughts.

~mitch

No comments:

A little about Me

My photo
My name is Mitch Beaumont and I've been a technology professional since 1999. I began my career working as a desk-side support engineer for a medical devices company in a small town in the middle of England (Ashby De La Zouch). I then joined IBM Global Services where I began specialising in customer projects which were based on and around Citrix technologies. Following a couple of very enjoyable years with IBM I relocated to London to work as a system operations engineer for a large law firm where I responsible for the day to day operations and development of the firms global Citrix infrastructure. In 2006 I was offered a position in Sydney, Australia. Since then I've had the privilege of working for and with a number of companies in various technology roles including as a Solutions Architect and Technical team leader.