Saturday, August 25, 2018

Some talks I've given

From time to time I'm fortunate enough to have the opportunity to share some of my experiences at conferences and events.






I thought it might be useful for anyone who follows my blog to start compiling a list of some of the talks I've given. Hopefully you'll find some of the information I share in these talks useful.


AWS Dev Day Australia 2018 - Containers, Containers, Containers
https://youtu.be/mBiOBwy-UiE


AWS Dev Day Australia 2018 - Cleaning Up the Mesh With a Service Mesh!

https://youtu.be/d8iPOsdz1K4

AWS re:Invent 2017 - Architecting Container Infrastructure for Security and Compliance

https://youtu.be/Ofu22X7qHn

Monday, July 09, 2018

Building a CD Pipeline for Fargate from the Command-Line

In an earlier post, AWS Fargate from the Command-Line, we packaged up a simple Go HTTP web server as a Docker container image and deployed it in to Amazon ECS as an AWS Fargate backed service ... using only the command-line. Pretty sweet. 

In this post, we're going to take our project to the next-level and provision a Docker powered pipeline that'll take our simple Go HTTP web server, build it and prepare it to be shipped to Amazon ECS.

But why command-line?


Everything I've done in this post, and my previous posts, could just as easily be done using the AWS management console. I'd definitely encourage you to run through this process using the management console to get a feel for the differences. The management console, and more specifically the first-run wizards, do a good job of abstracting away a lot of the implementation details, which is great if you're looking to get started quickly.

That being said, I've found that by assembling the Lego blocks using the command-line (or any of the SDKs), you end up with a much deeper understanding of the relationships between those components.


 

TL;DR

By the end of this post you'll have a pipeline that can be used to build and deploy projects to an ECS cluster written in most languages using most tools.

Let's get started!


In the previous post, you'll recall that we had a simple HTTP server written in Go. I'd placed the main.go file and a Dockerfile in to a standard directory on my computer. 

What does every good software project need?


As with any good software project, the source code should be stored in a version control system. As it happens, AWS has fully managed VCS, AWS CodeCommit.

Let's start by creating ourselves a repository for storing our project.


aws codecommit create-repository --repository-name go-frontend

Git, Git, Git, Git ... 


Now, we need to turn our vanilla working directory in to a mean, green git-tracked machine by running a few simple commands.

# Initialise my workspace as a git tracked repo
git init

# Add the dockerfile and the main.go
git add .

# Do an initial commit
git commit -m 'initial commit'

Cool. So, we've now converted our existing folder in a git-tracked repo. Albeit locally tracked and stored. The first step in addressing that shortcoming is setting up some authentication (auth) for the hosted CodeCommit repository we just created.

I'm going to use public/private key-pair for auth. It's also possible to setup some git credentials using the AWS IAM console, but I prefer SSH keys. That's all I have to say about that.

I'm going to start by generating a new key-pair as follows:

ssh-keygen

During the key-pair creation process you'll be prompted to choose a file in which to store the private key. I opted to create a new file specifically for this new private key so that my existing private key (which I use for other things) did not get overwritten.

The corresponding public key will be written to a file with the same name but with the .pub extension.

The next step in the process requires us to copy the public key material, from the file with the .pub extension, and upload it up to AWS. To do this, cat the file and copy it's content to your clipboard.

cat Users/mitch/.ssh/codecommit_rsa.pub


Next, we use the AWS CLI to upload the public key material and associate it with an IAM user. In my case, my IAM has a name of mitch.beaumont, so that who I'm going to attach the public key to.

Paste the public key material you copied earlier as the value for the --ssh-public-key-body argument, don't forget to paste between quotes " ".

aws iam upload-ssh-public-key \
--user-name mitch.beaumont \
--ssh-public-key-body "ssh-rsa ..."

The response from this step will include an SSHPublicKeyId. You'll need this for the next step so take a note.

We're now going to hack our SSH config file (~/.ssh/config) to help make using the custom SSH key we generated for auth, a little more seamless.

Host git-codecommit.*.amazonaws.com
  User <whatever_your_SSHPublicKeyId_was>
  IdentityFile ~/.ssh/<whatever_your_private_key_file_is_called>

If you had to create the SSH config file then you'll need to chmod the permissions. Otherwise, you should be all set.

chmod 600 ~/.ssh/config

We now need to update the configuration of our local repo and add a "remote". This is the URL of the remote location were we want the contents of our repo to be stored. In our case, this is CodeCommit.

CodeCommit, much like GitHub, support two different kinds of URLs for repos. HTTPS and SSH. Because we're auth'ing using SSH, we need to use the SSH URL.

git remote add origin ssh://git-codecommit.us-west-2.amazonaws.com/v1/repos/go-frontend

Let's check that the remote URL took ...

git remote -v

Now we're ready to push our local repo up to our remote.

git push --set-upstream origin master

The stage is set!

We now find ourselves in a great position. As we get new ideas for killer features that we want to add to our simple Go HTTP web server, we hack away, commit the changes and push them to a remote location for safe-keeping.

For some people however, that might not be enough. WE ARE THOSE PEOPLE! We WANT continuous delivery.

Each time we commit a change, we want those changes to trigger a pipeline that takes our updated code, mixes it with some fairy-dust and spits our a fully functional container image that we can do something with, like scan, test or deploy to an environment.

 Something that looks a bit like this ...

Bill of Materials

The diagram above looks fairly straightforward. HOWEVER. There are a few things we're going to need in place to make our pipeline a reality. So let's take a look at what they are:

  1. CodeCommit Repoistory (we already have this)
  2. An S3 bucket to store various artifacts produced and consumed by the stages of the pipeline.
  3. An IAM role with the necessary permissions to to execute the various stages of our pipeline. 
  4. An IAM role with the necessary permissions to execute our build processes.
  5. A CodeBuild project.
  6. A CodePipeline pipeline.
  7. A CloudWatch event that fires each time a change is made to the code stored in the CodeCommit repository.

A place to call artifactStore

We begin the next exciting chapter of our adventure, but creating an S3 bucket. This bucket will be used by our pipeline for storing artifacts.

What's an artifact?

Good question! Simply put, Artifacts are files that are created by a stage in the pipeline. You can choose to discard the artifacts, or keep them and share them with other stages in the pipeline. (Which is what we'll be doing)

The S3 bucket we're creating in this step is where those artifacts we choose to keep will be stored. Keep a note of the name of the bucket. You'll need it a little later.

aws s3 mb s3://codepipeline-${REGION}-${ACCOUNT_ID}

 

Show me the permissions!

If you look back at our diagram, you can see that CodePipeline kind of runs the show. It's job is to share artifacts, trigger builds and generally keep things moving forwards. Because of that, we need to pass some privileges to CodePipeline. We do that via an IAM role.

If we want an AWS service, such as CodePipline, to assume an IAM role in our AWS account, we have to create a trust relationship. In our case, we are the trusting party and our policy says, we trust <service_whatever> to perform some actions within the boundaries of our AWS account. 

Let's start by defining that trust relationship and creating a file called pipeline_trust_policy.json.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "codepipeline.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Now we'll create a new IAM role called CodePipelineExecutionRole and configure it to use the trust policy we defined in pipeline_trust_policy.json.

Be sure to record the role Arn from the output of this step. You'll be needing that a little later.

aws iam create-role \ 
--role-name CodePipelineExecutionRole \
--assume-role-policy-document \
file://./pipeline_trust_policy.json

Next we need to define the actions we want our "trustee" to be able to perform. Create another file and called it pipeline_permissions_policy.json.

This policy file gives the trustee, the CodePipeline service, permissions to perform a bunch of actions, including allowing it to retrieve code from our CodeCommit repository, artifacts from our artifact store and some additional rights, which we may tap in to later, for deploying in to ECS:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "codecommit:GetBranch",
                "codecommit:GetCommit",
                "codecommit:UploadArchive",
                "codecommit:GetUploadArchiveStatus",
                "codecommit:CancelUploadArchive",
                "codebuild:BatchGetBuilds",
                "codebuild:StartBuild"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::codepipeline*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ecs:DescribeServices",
                "ecs:DescribeTaskDefinition",
                "ecs:DescribeTasks",
                "ecs:ListTasks",
                "ecs:RegisterTaskDefinition",
                "ecs:UpdateService"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:iam:PassRole",
            "Resource": "*"
        }
    ]
}

Lets use the pipeline_permissions_policy.json file to create a policy and attach it to the CodePipelineExecutionRole IAM role we created earlier.

Here we are creating an inline-policy. A better way to do this to help with future reuse might be to create a separate policy object and attach it. That way the same policy object can be reused by other roles in the future.

aws iam put-role-policy \
--role-name CodePipelineExecutionRole \
--policy-name CodePipeline-Permissions-Policy \
--policy-document file://./pipeline_permissions_policy.json

Now that we have an IAM role for our pipeline to use when executing, the next thing we need to do is create a similar IAM role with which to execute our CodeBuild project.

Another trust policy is needed first:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "codebuild.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Then the policy that defines the actions we need our CodeBuild project to be able to perform.

A few things worth calling out here, firstly, you'll notice that there are some "Allow" actions for CloudWatch Logs. This is required for CodeBuild to output the logs from the build project. Extremely useful when trying to debug our build process.

There are also a number of "Allow" actions for ECR. Cast your mind back to the original goal of this walk-through. We want to be at a point where we have an artifact that can be deployed following each push / merge to our master branch. (Continuous delivery).

The artifact we're producing is a container image. This is why our CodeBuild project needs access to ECR. It needs to be able to push the container image there.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:/aws/codebuild/*:log-stream:*"
            ]
        },
        {
            "Effect": "Allow",
            "Action":[
                "s3:GetObject",
                "s3:PutObject",
                "s3:GetObjectVersion",
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability",
                "ecr:PutImage",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload",
                "ecr:GetAuthorizationToken"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Let's create our IAM role for CodeBuild and attach the trust policy ...

aws iam create-role \
--role-name CodeBuildExecutionRole \
--assume-role-policy-document file://./build_trust_policy.json \

... and add the permissions policy.

aws iam put-role-policy \
--role-name CodeBuildExecutionRole \
--policy-name CodeBuild-Permissions-Policy \
--policy-document file://./build_permissions_policy.json

Buildy build build build.

What we're going to do now is create and configure a CodeBuild project. This project will take, as an input, the source-code from our CodeCommit repo, compile-it and spit out a container image.

I've taken a slightly different approach to this step in that I'm passing the configuration of my pipeline in as command line arguments.

You'll need to the Arn of the CodeBuildExecution role and the URI of the CodeCommit repository for this step.

Something of note here is the "image" property. Code build uses containers to execute the steps in the build project ...

Hang-on, aren't you building a container?

So, we're using a container, to build a container ... interesting. This approach is often referred to as DinD or Docker-in-Docker. (The everything-pipeline starts to take shape!)

Check out a full breakdown of what goes in to the aws/codebuild/docker:17.09.0 image here:

https://github.com/aws/aws-codebuild-docker-images/tree/master/ubuntu/docker/17.09.0

Let's create our build project. Don't forget to grab the name of the CodeBuild project as you'll need that for the next step.

aws codebuild create-project \
--name "goFrontend" \
--description "Build project for GoFrontend" \
--source type="CODEPIPELINE" \
--service-role "<Arn_of_codepbuildexecutionrole_role>" \
--environment type="LINUX_CONTAINER",\
image="aws/codebuild/docker:17.09.0",\
computeType="BUILD_GENERAL1_SMALL",\
environmentVariables="[{name=AWS_DEFAULT_REGION,value='${REGION}'},{name=AWS_ACCOUNT_ID,value='${ACCOUNT_ID}'},{name=REPOSITORY_URI,value=<Uri_of_ECR_repository>}]" \
--artifacts type="CODEPIPELINE"

buildspec.yml

How do we tell CodeBuild what to do as part of the build project? For that, we need to add a file called buildspec.yml to out project folder. This file defines all of the steps that we need CodeBuild to execute on our behalf. There is a lot going on in this file, but most of what you can see relates to extracting the metadata from our build process and using it to tag the resulting container image correctly, then pushing it to ECR.

You can grab the buildspec.yml file from a gist I've crested here:

Place the file in the root of your project and be sure to run git add / git commit / git push.

Your project directory should be looking something like this once you're done.


.
├── Dockerfile
├── buildspec.yml
└── main.go

We're almost there!

Most of the pieces of our puzzle are in-place. We now need to apply the glue that's going to hold everything in place: Our Pipeline.

We'll start by creating a new file pipeline_structure.json. This file defines the attributes and structure of our pipeline, including things like the role used to execute the pipeline and the stages that make up the pipeline.

You can see I've defined 3 stages in my pipeline

Source

This stage retrieves the source code of my project from CodeCommit. It then bundles up that source code as an output artifact and drops it in to our artifactStore. In the configuration section you'll notice that I've set PollForSourceChanges to "false". I've done this because I'm going to use CloudWatch Events to trigger my pipeline each time a change is made.

Build

This stage triggers the build pipeline that we created earlier. The input artifact of this stage is the output artifact of the Source stage (the bundles up source code). You'll notice I've created and output artifact called imagedefinitions. This file contains a simple JSON structure that includes the name of the container my container image represents and the container image tag (which is the build Id).

Release: 

The final stage is the release stage. This stage has two actions. Firstly, a manual approval, which is a blocking action. Assuming the previous stage completes successfully, the pipeline will halt at this point waiting for intervention from the operator. Once the operator "approves" or "rejects, the pipeline will continue, or fail respectively.

The second action in this stage deploys the changes to our ECS cluster. Notice the configuration items here the name of the cluster, the name of the service and the imagedefinitions.json file.


{
        "roleArn": "<Arn_of_codepipelineexecutionrole_role>",
        "stages": [
            {
                "name": "Source",
                "actions": [
                    {
                        "inputArtifacts": [],
                        "name": "Source",
                        "actionTypeId": {
                            "category": "Source",
                            "owner": "AWS",
                            "version": "1",
                            "provider": "CodeCommit"
                        },
                        "outputArtifacts": [
                            {
                                "name": "MyApp"
                            }
                        ],
                        "configuration":{
                            "RepositoryName": "<name_of_codecommit_repository>",
                            "BranchName": "master",
                            "PollForSourceChanges": "false"
                        },
                        "runOrder": 1
                    }
                ]
            },
            {
                "name": "Build",
                "actions": [
                    {
                        "inputArtifacts": [
                            {
                                "name": "MyApp"
                            }
                        ],
                        "name": "Build",
                        "actionTypeId": {
                            "category": "Build",
                            "owner": "AWS",
                            "version": "1",
                            "provider": "CodeBuild"
                        },
                        "outputArtifacts": [
                            {
                                "name": "imagedefinition"
                            }
                        ],
                        "configuration":{
                            "ProjectName": "goFrontend"
                        },
                        "runOrder": 1
                    }
                ]
            },
            {
                "name": "Release",
                "actions": [
                    {
                        "name": "Approval",
                        "actionTypeId": {
                            "category": "Approval",
                            "owner": "AWS",
                            "version": "1",
                            "provider": "Manual"
                        },
                        "runOrder": 1
                    },
                    {
                        "inputArtifacts": [
                            {
                                "name": "imagedefinition"
                            }
                        ],
                        "name": "DeployToECS",
                        "actionTypeId": {
                            "category": "Deploy",
                            "owner": "AWS",
                            "version": "1",
                            "provider": "ECS"
                        },
                        "outputArtifacts": [],
                        "configuration":{
                            "ClusterName": "fargate-cluster",
                            "ServiceName": "go-http-server",
                            "FileName": "imagedefinitions.json"
                        },
                        "runOrder": 2
                    }
                ]
            }
        ],
        "artifactStore": {
            "type": "S3",
            "location": "<name_of_the_s3_bucket_artifact>"
          },
        "name": "GoFrontend",
        "version": 1
}

Let's create the pipeline now, using the pipeline_structure.json file as out input.

aws codepipeline create-pipeline \
--pipeline file://./pipeline_structure.json

Test our pipeline

Assuming nothing has imploded, we're now at a point where we can test out our pipeline. Guess what? We can also do that using the command-line!

aws codepipeline start-pipeline-execution \
--name GoFrontend

The previous command will output an pipelineExecitionId. We will use that to check on the status of our pipeline as follows.


aws codepipeline get-pipeline-state --name GoFrontend \
| jq '.stageStates[] | select (.latestExecution.pipelineExecutionId == "<pipelineExecutionId>")'

Approving the pipeline

After <___> minutes we should see our pipeline is paused at the "Approval" action of the "Release" stage

You'll notice a token value. We're going to use this to approve the pipeline action so that it can move to the next action in this state. Which happens to be: Release it to our ECR cluster.

{
            "stageName": "Release",
            "inboundTransitionState": {
                "enabled": true
            },
            "actionStates": [
                {
                    "actionName": "Approval",
                    "latestExecution": {
                        "status": "InProgress",
                        "lastStatusChange": 1234567.123,
                        "token": "cxxxxxd-8xxx5-4xx2-9xxc-9xxxxxxxxe"
                    }
                }

Let's approve the stage using the put-approval-result action.


aws codepipeline put-approval-result \
--pipeline-name GoFrontend \
--stage-name Release \
--action-name Approval \
--result summary="Approving",status="Approved" \
--token cxxxxxd-8xxx5-4xx2-9xxc-9xxxxxxxxe

If we check the state of our pipeline again, we should now see that it has moved forward.

Wrapping it in a bow.

Let's recap what were we are. We have a CodeCommit repository that is storing our project. An S3 bucket that is going to be used for storing artifacts shared across the stages of our pipeline. We have a CodeBuild project that takes our source code as an input, starts a Docker container to compile the simple Go HTTP web server in to a statically linked binary. Finally, we have a CodePipeline that pulls all of these disparate parts in to a nice cohesive workflow.

That last thing we need to do is configure a trigger for our pipeline each time a change is made to the master branch of our project. For this, we're going to use CloudWatch Events.

Just like every good service, we need to trust the CloudWatch Events service to perform actions on our behalf.

(Click on the files to check out an example gist for each policy)

Capture the Arn of the role, you'll need it a little later.

aws iam create-role \
--role-name Role-for-MyRule \
--assume-role-policy-document file://./event_trust_policy.json

Next, give our role some permissions.

aws iam put-role-policy \
--role-name Role-for-MyRule \
--policy-name CodePipeline-Permissions-Policy-For-CWE \
--policy-document file://event_permissions_policy.json

Now we need to create a rule. This includes a pattern to the kind of events that would trigger this event.

aws events put-rule \
--cli-input-json file://./event_put_rule.json 

Finally, we need to give our rule some targets. Which in our case, is CodePipeline.

aws events put-targets \
--rule CodeCommiteRule \
--targets Id=1,Arn=<arn_for_code_pipeline>,RoleArn=<arn_for_the_event_rule>

Change is good

Go a head and make a change to you main.go file. Perhaps change the message that gets displayed when you hit the we service. Once you've made the changes, commit and push them to CodeCommit. Then sit back and watch the magic happen.



Things to Fix

This approach, just like our software, has room for improvement. For example, we may some to be a little more explicit with some of the policies we created for the services. Naming the specific resources that actions can be performed against.

It would also be cool to setup a better way of responding to approval notifications. Check out this awesome post on the AWS blog for some ideas on how this might look with Slack.

https://aws.amazon.com/blogs/devops/use-slack-chatops-to-deploy-your-code-how-to-integrate-your-pipeline-in-aws-codepipeline-with-your-slack-channel/

As always, I'd love your feedback and thoughts.

~mitch

Wednesday, July 04, 2018

AWS Fargate from the Command-Line

We all love a good command-line demo. Here is one I put together recently for demonstrating a few things, firstly Docker multi-stage builds, and secondly how a simple web service written in Go could be deployed to AWS Fargate using nothing but the command line.

 

What's cool about that I hear you ask?

What's cool about it is that at no point during this demo am I deploying, configuring or going to have to manage ANY servers.

 

Is it "serverless" ... is it containers? YES!

Let’s take a look:

Here is the simple Go HTTP server.

package main

import (
 "log"
 "net/http"

 "github.com/gorilla/mux"
)

func YourHandler(w http.ResponseWriter, r *http.Request) {
 w.Write([]byte("<h1>Hello Mitch Beaumont!</h1>\n"))
}

func main() {
 r := mux.NewRouter()

 r.HandleFunc("/", YourHandler)

 log.Fatal(http.ListenAndServe(":8000", r))
}

As I mentioned, I'm going to use Docker's multi-stage build process to compile my HTTP server and then build a lightweight container image.

 

Why are you using multi-stage builds?  

When it comes to containers, there aren't many people who'd argue that the smaller, the better! 

Multi-stage builds help us optimise the size of our container image by allowing us to, from a software development perspective, clear the wheat from the chaff. 

In the case of our simple Go application we require Go to be installed so that we can compile the application in to a binary. Once compiled, these dependencies are not required for the binary to run our simple web service. By using multi-stage builds, we easily use one container image, with the required dependencies, to build our binary. We can then move our artifact (the binary) in to a new container image, with a minimal footprint (in our case, scratch). 
The result is an optimised container image.
 
Here is my how my multi-stage Dockerfile looks.

# STEP 1 build executable binary

FROM golang:alpine As builder 
COPY . $GOPATH/src/github.com/mitchybawesome/http-server/
WORKDIR $GOPATH/src/github.com/mitchybawesome/http-server/

RUN apk add --no-cache git mercurial

#get dependancies
RUN go get -d -v

#build the binary
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o /go/bin/httpserver

# STEP 2 build a small image

# start from scratch
FROM scratch

# Copy our static executable from the builder
COPY --from=builder /go/bin/httpserver /go/bin/httpserver
ENTRYPOINT ["/go/bin/httpserver"]

Now I have my container image, I need to push it in to a repository. It just so happens that I have an Amazon ECR repo setup and ready to go. I'll use the following commands to build the image, tag it, login to ECR and push the image. (The account ID is a dummy)

ACCOUNT_ID="000000000"
REGION="us-west-2" 
REPO="go-http-server"
 
# Login to ecr
$(aws ecr get-login --no-include-email --region ${REGION})

# Build docker image
docker build -t ${REPO} .

# Tag docker image
docker tag ${REPO}:latest ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${REPO}:latest

# Push docker image
docker push ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${REPO}:latest

Now that we have our container image available. The next step is to create an ECS cluster in which to run our container.

 

BUT YOU SAID NO SERVERS! 

I know, I know. Cool your jets man!

Whilst it's true, I am creating a cluster, there are no actual instances being provisioned in my AWS account. The cluster construct is used purely as a management construct and security boundary.

# Create Cluster
aws ecs create-cluster --cluster-name fargate-cluster --region ${REGION}

Next, we need to create a load balancer that will route requests to our running service. In order to create the load balancer, we need few details about the environment, specifically, we need to know the VPC in to which we will be deploying the load balancer, and the subnets that we're going to connect our load balancer to.

vpcid=$(aws ec2 describe-vpcs | jq '.Vpcs[] | select (.IsDefault == true) | .VpcId' | sed -e 's/^"//' -e 's/"$//')

The $vpcid variable now contains our default VPC ID. We will use this to parse the list of available subnets. For the purpose of this demo, I'll be creating all of my resources in the default VPC.

Notice the filter I've applied in the jq query to select the VPC which has the "IsDefault" flag set to "true".

subnets=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=${vpcid}" | jq '.Subnets[].SubnetId') && echo $subnets

Run the above command to output a list of subnets. Keep these safe, we'll need them later.

The next step in our process is to create a security group that will be attached to the load balancer. We use this security group to control traffic ingress from the public Internet.

If we did not create / assign a security group, the VPCs default security group would be assigned to the load balancer and we'd have a hard time accessing our service.

# Create ELB security group
aws ec2 create-security-group \
--description "security group for ALB" \
--group-name "security-group-for-alb" \
--region ${REGION} | jq '.GroupId'

Grab the "GroupId" output from the previous command, and use it to define the ingress rules for the security group. You will also need the load balancer's GroupId for a later step, so keep it close.

In this example, we're allowing all TCP traffic from any source IP address to connect to the load balancer.

# Configure ELB security group ingress rule to allow ELB to connect to tasks. 
aws ec2 authorize-security-group-ingress \
--group-id <GroupId_from_previous_command> \
--protocol tcp \
--port 80 \
--cidr 0.0.0.0/0

We're now ready to create our load balancer. This command creates an ALB (Application Load Balancer) and outputs the ARN (Amazon Resource Name) to the command line.

 

Why ALB?

We're using an ALB because of the support for dynamic port mappings and path based routing. Both of these features translate to a more optimized deployment model in terms of infrastructure and ultimately cost.

 

Were you paying attention?

In an earlier step, I asked you to record some information. Do you remember what it was? 
I hope so, because you'll need that information to complete this command.  (Hint: It's the subnets!)

# Create a load balancer and get the ELB ARN.
aws elbv2 create-load-balancer \ 
--name go-http-server \ 
--subnets subnet-111111 \
subnet-222222 \
subnet-333333 \ 
--security-groups <ALB_Security_GroupId> \
| jq '.LoadBalancers[].LoadBalancerArn'

Now that we have a load balancer created, we need to create a target group. The load balancer uses the target group to route requests to one or more registered targets. Which in our case will be containers (or tasks) running our simple Go web service.

# Create a target group
aws elbv2 create-target-group \
--name fargate-targets \
--target-type ip \
--protocol HTTP \
--port 80 \
--vpc-id ${vpcid} | jq '.TargetGroups[].TargetGroupArn'

 

Joining the dots! 

Our target group now needs to be attached to a listener and the listener needs to be attached the load balancer to complete the setup. You'll need the load balancer ARN and the target group ARN, both of which were outputs from the previous commands, to complete this step.

# Create a listener
aws elbv2 create-listener \
--load-balancer-arn <load_balancer_arn> \
--protocol HTTP \
--port 80 \
--default-actions Type=forward,TargetGroupArn=<target_group_arn>

 

How do we get our application deployed in to the cluster?

Great question! We have an image that we pushed in to ECR. But we somehow need to tell ECS that we want it to launch a container based off of that image.  We do that by creating task definition.

The task definition is a set of properties that allows us to model the run-time environment for our containerised go web service. Within the task definition we specify, among other things, how much memory and CPU we want to allocate to our task.

Task definitions are written in JSON. I've taken the liberty of dropping an example of the task definition I'm using in to a gist, which can be found here.

If you use this task definition, don't forget to update the image path!

# Register Task Definition
aws ecs register-task-definition \
--cli-input-json file://./go-http-server.json \
--region ${REGION} --query 'taskDefinition.taskDefinitionArn'

Our task needs a security group assigned to it so that we can control the kind and the sources of traffic that are allowed to reach it.

Record the "GroupId". You'll need it later.

# Create security group for the tasks
aws ec2 create-security-group \
--description "security group for fargate task" \
--group-name "security-group-for-fargate-task" \
--region ${REGION} | jq '.GroupId'

I mentioned earlier that we we'd need the GroupId of the load balancers security group. Now is the time. Use that group ID to define and attach an ingress rule to the task security group.

Basically what we're doing here is telling the task that it can accept TCP connections over port 8000 from the load balancer.

Keeping our traffic flow rules tight!

# Configure security group ingress rule to allow ELB to connect to tasks.
aws ec2 authorize-security-group-ingress \
--group-id \
--protocol tcp \
--port 8000 \
--source-group <alb_security_group_id>

 

Wrapping up

The final step is creating our service. You'll need some of the outputs from the previous commands to complete this step, including: the name of the task definition, a comma separated list of the subnets in the VPC to which the tasks needs to be connected, the ID of the task security group, the ARN of the target group and the name of the container. (Which you can get from the task definition you created earlier).

# Create Service
aws ecs create-service --cluster fargate-cluster --service-name go-http-server \
--task-definition <task_definition> --desired-count 2 --launch-type "FARGATE" \
--network-configuration "awsvpcConfiguration={subnets=[ <command_separated_list_subnets> ],securityGroups=[<security_group>],assignPublicIp=ENABLED}" \
--load-balancers targetGroupArn=<target_group_arn>,containerName=<container_name>,containerPort=<container_port> \
--region ${REGION}

Let's now make sure we can connect to our service. Grab the DNSName of the load balancer by querying the load balancer ARN.


url=$(aws elbv2 describe-load-balancers \
--load-balancer-arns <load_balancer_arn> \
| jq '.LoadBalancers[].DNSName' | sed -e 's/^"//' -e 's/"$//') && curl $url


Hopefully you get a response back! If you don't perhaps check your security group rules.

 

 Drum roll please

So, using nothing but the command line and NO SERVERS, we've packaged up a simple web service,  pushed it to a secure repository and created a framework for scaling, securing, deploying and serving the web service.

In the next few posts, I'm going to explore some of the finer details around security, scaling and updating our simple web service.

As always, I'd love your feedback and thoughts.

~mitch




Saturday, June 23, 2018

Amazon EKS and IAM Authentication

This past month, AWS launched Amazon EKS. Elastic Container Service for Kubernetes.


What is Amazon EKS?

Amazon EKS is a managed service aimed at making it easier to deploy and operate Kubernetes clusters on AWS. 

What is IAM?

IAM, or Identity and Access Management, is the AWS authn and authz service. Access to and interaction with all AWS services is gated through IAM. 

What does IAM have to do with Kubernetes?

As part of Amazon EKS, authentication of requests made to the Kubernetes API is handled by IAM. That is to say that when you make a call to the Kubernetes API, like this:

kubectl get pods

The Kubernetes API server will firstly pass a bearer token, by way of a webhook, over to IAM to ensure that the request is coming from a valid identity. Once the validity of the identity has been confirmed, Kubernetes then uses it's own RBAC capability (Role Based Access Control) to approve of deny the specific action the requester tried to perform.

The super smart folks over at heptio created an awesome tool called the heptio authenticator for AWS which enables an integration between Kubernetes and IAM. 

https://github.com/heptio/aws-iam-authenticator/blob/master/README.md

How does this work? 

Lets run through the process of creating a new IAM user and then granting that user access to the Kubernetes API.

We begin by creating a new IAM user. Once the user has been created, the response should include the ARN (Amazon Resource Name) of the user. Record that. We'll need it later.

aws iam create-user \
--user-name curly \
--output text \
--query 'User.Arn'

We now need to update the AWS Auth configmap to map the IAM user to a kubernetes user object. The basic template can be found here.

mapUsers: |
    - userarn: arn:aws:iam::000000000000:user/curly
      username: curly

The diff should look something like this. (Don't forget to add in your instance role ARN!)

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: <ARN of instance role (not instance profile)>
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
  mapUsers: |
    - userarn: arn:aws:iam::000000000000:user/curly
      username: curly

Now it's time to apply the updated config map to your cluster.

kubectl apply -f aws-auth-cm.yaml


Quickly check that the user mapping was created by running this command.

kubectl describe configmap aws-auth -n kube-system 


Once we have our IAM user mapped to Kubernetes user object, lets see what roles we can bind to that user.

kubectl get clusterroles
 

One of the roles you'll see near to the top of the list is the edit role. Let's create a clusterrolebinding for the user curly to the edit cluster role.

kubectl create clusterrolebinding curly-edit --clusterrole=edit --user=curly 


POP QUIZ: What's wrong with this?
Everything I'm doing here involves imperative commands. Ideally, all of this should be applied as declarative configurations. Which might look a bit like this:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: curly-edit
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: edit
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
name: curly 

We'll now test if curly can now create a deployment.

kubectl auth can-i create deployments --as=curly

Hopefully you got a "Yes" response back! If not, maybe go back and check the steps above one more time.

Check Point.

We've proven at this point, that a new user object within Kubernetes can do some stuff. Let us now create access key / secret access key for the user curly and make sure that the IAM user curly can actually use the Kubernetes API to do some stuff.

aws iam create-access-key --user-name curly

 

Using that secret access key and access key, update the local credentials file to use  curlys access key and secret access key.

aws configure --profile curly

I've added the --profile to make it easy to flip back and fourth between our admin user and our curly user in later steps.

No you should be able to run a simple kubectl command and get a response back. To force the heptio authenticator to use the credentials from the curly profile, set the AWS_PROFILE envar in line with the kubectl command.

Let's try:

AWS_PROFILE=curly kubectl get pods

Curly should now be able to see a list of running pods within the cluster.  

Excellent, good job! 


Do I have to create a mapping for each user?

No! In addition to creating a mapping for a specific user, we can also create a mapping for an IAM role. This role can be assumed by multiple users via group membership.

Let's start by deleting the existing cluster role binding we created for curly.

kubectl delete clusterroldebinding curly-edit

Next we'll set some envars to help with our IAM role creation process.

(I've borrowed these steps from the Heptio github repo!)

ACCOUNT_ID=$(aws sts get-caller-identity --output text --query 'Account')

POLICY=$(echo -n '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::'; echo -n "$ACCOUNT_ID"; echo -n ':root"},"Action":"sts:AssumeRole","Condition":{}}]}')

Now lets create a role, record the ARN as per the previous example. 

aws iam create-role \
--role-name KubernetesAdmin \
--description "Kubernetes administrator role" \
--assume-role-policy-document "$POLICY" \
--output text \
--query 'Role.Arn'

Next we need to modify our config map again to include the new role ARN. The diff should look a bit like this:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: <ARN of instance role (not instance profile)>
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: arn:aws:iam::000000000000:role/KubernetesAdmin 
      username: admin-curly:{{SessionName}}
      groups:
       -  mitchyb:editors
  mapUsers: |
    - userarn: arn:aws:iam::000000000000:user/curly
      username: curly 

Let's apply the updated config map to the cluster.

kubectl apply -f aws-auth-cm.yaml

Quickly check that the role mapping was created by running this command.

kubectl describe configmap aws-auth -n kube-system


Hopefully you see the the role mapping in the output. 

Next we can create a cluster role binding to the edit cluster role, for those users who assume the arn:aws:iam::000000000000:role/KubernetesAdmin role.

kubectl create clusterrolebinding curly-edit --clusterrole=edit --group=mitchyb:editors

ACCOUNT_ID=$(aws sts get-caller-identity --output text --query 'Account')

POLICY=$(echo -n '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Resource": "arn:aws:iam::'; echo -n "$ACCOUNT_ID"; echo -n ':role/KubernetesAdmin","Action":"sts:AssumeRole"}]}')


Create a policy that enables the role to be assumed:
 
aws iam create-policy \
--policy-name kubernetes-admin-policy \
--policy-document "$POLICY" \
--output text \
--query 'Policy.Arn' 


Create a group:

aws iam create-group \
--group-name kubernetes-admins 


Attach a policy to allows the members of the group to assume the policy:
 
aws iam attach-group-policy \
--group-name kubernetes-admins \
--policy-arn arn:aws:iam::
000000000000:policy/kubernetes-admin-policy

Put curly in the new group:

aws iam add-user-to-group \
--group-name kubernetes-admins \
--user-name curly

 

Configuring the AWS CLI

The only thing left for us to do is make a few slight tweaks to the configuration of the AWS CLI to so that when we make a call using kubectl, our session assumes the arn:aws:iam::000000000000:role/KubernetesAdmin role.

[profile adminaccess]
role_arn: arn:aws:iam::000000000000:role/KubernetesAdmin
source_profile: curly

Armed with our new profile, lets request a list of pods, we can force the heptio authenticator to use curly profiles (and associated credentials) by setting the AWS_PROFILE envar inline with the kubectl, as follows:

AWS_PROFILE=adminaccess kubectl get pods

Just to prove that this is all working as expected, let's remove curly from the
kubernetes-admins group. 

Easy to do via the management console.
or using the AWS CLI, without setting the AWS_PROFILE envar inline.

aws iam remove-user-from-group \
--group-name kubernetes-admins \
--user-name curly


Given our current setup this should lead to curly no longer be able to assume the arn:aws:iam::000000000000:role/KubernetesAdmin role which in turn means as far as RBAC is concerned, curly is not authorized to perform any action against the Kubernetes API.

AWS_PROFILE=adminaccess kubectl get pods

Expect to see something like this:

could not get token: AccessDenied: User: arn:aws:iam::000000000000:user/curly is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::000000000000:role/KubernetesAdmin  

 

And that's a wrap.

That's about all there is to it. As you can see, with this configuration, you can easily create groups of users (via IAM) and authorize the members of those groups to perform specific actions within the Kubernetes cluster (thanks to RBAC).

Revoking those privileges is then as easy as removing those users from the given groups.


A little about Me

My photo
My name is Mitch Beaumont and I've been a technology professional since 1999. I began my career working as a desk-side support engineer for a medical devices company in a small town in the middle of England (Ashby De La Zouch). I then joined IBM Global Services where I began specialising in customer projects which were based on and around Citrix technologies. Following a couple of very enjoyable years with IBM I relocated to London to work as a system operations engineer for a large law firm where I responsible for the day to day operations and development of the firms global Citrix infrastructure. In 2006 I was offered a position in Sydney, Australia. Since then I've had the privilege of working for and with a number of companies in various technology roles including as a Solutions Architect and Technical team leader.