Friday, December 21, 2018

Meshing around with Istio and EKS

eks-istio-deploy

Meshing around with Istio and Amazon EKS

elastic-band-ball

In this post, as you may have gathered from the title, I'm going to be playing around with Istio and Amazon EKS. Earlier this year (2018) I presented a demo of deploying Istio to EKS.

Picture of Mitch

Cleaning up the mesh with a service mesh

However, at that time, EKS was missing support for dynamic admission controllers. This meant that a few additional steps had to be taken in order for Istio to be deployed correctly in to an EKS based Kubernetes cluster.

In October of this year support for Dynamic Admission Controllers was added to EKS. I wanted to revisit the demo I put together previously to understand if this new functionality simplified the process of deploying and operating Istio on EKS.

So, let's go!

Create a Cluster Using eksctl

I'm going to start by creating a new EKS cluster. I'm happy to report that as of December 20th 2018 Amazon EKS is now supported in my home region on ap-southeast-2 (Sydney). I'll be creating my clusters in that region using the awseome command line tool eksctl.

eksctl create cluster --region ap-southeast-2

Testing 1 ... 2 ... 3

After a few minutes, the EKS control plane and nodes should be up and running. Let's run a few simple tests to ensure that everything is looking healthy.

aws eks list-clusters --region ap-souteast-2
{
    "clusters": [
        "extravagant-monster-1545334691"
    ]
}

kubectl get nodes

NAME                                                STATUS    ROLES     AGE       VERSION
ip-192-168-38-190.ap-southeast-2.compute.internal   Ready     <none>    3m        v1.11.5
ip-192-168-93-109.ap-southeast-2.compute.internal   Ready     <none>    3m        v1.11.5

Looking good so far!

Installing Helm

We're going to setup Istio within the EKS cluster using Helm[1], so for my next trick, I am going to install Helm in to my EKS cluster.

Begin by creating the service account required by the Tiller[2] service for doing stuff.


kubectl create -f https://raw.githubusercontent.com/istio/istio/master/install/kubernetes/helm/helm-service-account.yaml

[1]: Helm is the package manager for Kubernetes. As an operating system package manager makes it easy to install tools on an OS, Helm makes it easy to install applications and resources into Kubernetes clusters.

[2]: Tiller is the in-cluster component of Helm. It interacts directly with the Kubernetes API server to install, upgrade, query, and remove Kubernetes resources. It also stores the objects that represent releases.

Next, we will initialise Helm within the EKS cluster using the serice account which we created earlier.

helm init --service-account tiller

Let's do a quick check to ensure that things were installed as expected

kubectl get pods --all-namespaces

You should expect to see a pod with a NAME that looks something like this tiller-deploy-64ddd985c9-qcw2g.

Download the latest Istio release

Before we can install Istio, we need to download the Istio components.

curl -L https://git.io/getLatestIstio | sh -

Switch to the newly created directory.

cd istio-1.0.x

Installing the custom resource definitions

Istio requires a number of customer resources[1] to be created within the Kubernetes API to support it's various modes of operation and features. We can use the pre-provided CRD[2] (Custom resource definitions) to create these custom resources within the Kubernetes API.

kubectl apply -f install/kubernetes/helm/istio/templates/crds.yaml

[1]: A custom resource is an object that extends the Kubernetes API or allows you to introduce your own API into a project or a cluster.

[2]: A custom resource definition (CRD) file defines your own object kinds and lets the API Server handle the entire lifecycle. Deploying a CRD into the cluster causes the Kubernetes API server to begin serving the specified custom resource.

Installing Istio using Helm

In 2018 BDAC (Before Dynamic Admission Controllers) in order to successfully install and run Istio on EKS you need to disable some specific features using parameter flags.

sidecarInjectorWebhook.enabled=false and global.configValidation=false

However, in 2018 ADAC (After Dynamic Admission Controllers) we can deloy simply, using the following command:

helm install install/kubernetes/helm/istio --name istio --namespace istio-system \
--set servicegraph.enabled=true \
--set tracing.enabled=true \
--set grafana.enabled=true

Label the default namespace for sidecar injection

In order for the automatic sidecar injection to function, we need to add some metadata to the Kubernetes namespace in to which we plan to deploy our pods. We'll use a simple label with the key "istio-injection" and the value "enabled".

kubectl label namespace default istio-injection=enabled

Deploying the demo application

We're now all set to deploy our application. This particular demo application is made up of two discrete services. The gopher-requester and the gopher-distrubutor.

What does this application do?

It's incredibly sophisticated. When a call is made to the gopher-requester, using curl for example, that call is passed along to the gopher-distributor.

gopher-solution

The gopher-distributor constructs a response and sends it back along the calls stack.

Let's start with the gopher-requester

kubectl apply -f ../gopher-requester/deployment.yaml

And now the gopher-distributor

kubectl apply -f ../gopher-distributor/deployment.yaml

Once these pods have been deployed, you should be able to check if the sidecar container has been injected

kubectl get pods

The output should look something like this

NAME                                  READY     STATUS    RESTARTS   AGE
gopher-distributor-694fd4f4db-24csn   2/2       Running   0          1m
gopher-distributor-694fd4f4db-4hk9t   2/2       Running   0          1m
gopher-distributor-694fd4f4db-gnn4g   2/2       Running   0          1m
gopher-requester-7d5cbc7989-bw8zc     2/2       Running   0          52s
gopher-requester-7d5cbc7989-tw69r     2/2       Running   0          43s
gopher-requester-7d5cbc7989-whxcm     2/2       Running   0          36s

If you unpack one of those pods using some JQ-foo ...

kubectl get po/gopher-distributor-694fd4f4db-24csn -o json | jq '.spec.containers[].name'

... we should see that there are infact two containers running within the context of the pod. One for our core application, which in this case is our gopher-distributor and another, our Istio proxy sidecar (Envoy)

"gopher-distributor"
"istio-proxy"

Testing our application is responding

We'll use curl to make sure that we're getting a response back from our application.


curl $(kubectl get svc gopher-requester -o json | jq '.status.loadBalancer.ingress[0].hostname' | tr -d '"')

Hopeully we now see something like this:


Request >> Gopher Version 1 from pod gopher-distributor-694fd4f4db-gnn4g

Let's now open up another terminal emulator window or tab. We will use the following while statement to generate some traffic to our application.


while true;do curl $(kubectl get svc -l app=gopher -o jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}');sleep 1;done

We should now be seeing a fairly even distribution of responses back from different instances of the gopher-distributor service running within the cluster.


Request >> Gopher Version 1 from pod gopher-distributor-694fd4f4db-gnn4g
Request >> Gopher Version 1 from pod gopher-distributor-694fd4f4db-4hk9t
Request >> Gopher Version 1 from pod gopher-distributor-694fd4f4db-24csn
Request >> Gopher Version 1 from pod gopher-distributor-694fd4f4db-gnn4g
Request >> Gopher Version 1 from pod gopher-distributor-694fd4f4db-4hk9t
Request >> Gopher Version 1 from pod gopher-distributor-694fd4f4db-24csn
Request >> Gopher Version 1 from pod gopher-distributor-694fd4f4db-gnn4g

Laying down some ground rules

Before Istio can start doing anything useful for us, we need to tell it about our application and the services that make up our application. Let's start by creating VirtualServices (these are API resources) that represent out gopher-requester and gopher-distribtor services


cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: gopher-requester-virtual-service
spec:
  hosts:
  - "*"
  gateways:
  - gopher-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: gopher-requester.default.svc.cluster.local
        port:
          number: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: gopher-distributor-virtual-service
spec:
  hosts:
  - gopher-distributor.default.svc.cluster.local
  http:
  - route:
    - destination:
        host: gopher-distributor.default.svc.cluster.local
        subset: v1
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: gopher-distributor-route-rule
spec:
  host: gopher-distributor.default.svc.cluster.local
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
EOF

We should see some output confirming that our routing rules have been created. Sweet!


virtualservice.networking.istio.io "gopher-requester-virtual-service" created
virtualservice.networking.istio.io "gopher-distributor-virtual-service" created
destinationrule.networking.istio.io "gopher-distributor-route-rule" created

Dark Launch

Great, so now we have our application, Istio and some Istio rules that help route (amongst other things) our service calls.

Let's now experiment with a particularly cool feature of Istio called traffic mirroring. This feature lets us launch a new version of our service and mirror production traffic to it, without actually making it live.

We'll start by deploying a new version of our gopher-distributor service:


kubectl apply -f ../gopher-distributor-v2/deployment.yaml

Checking back in with our test call in the other terminal windoe/tab, we see everything looks pretty much normal. We are still getting responses from v1 of our distributor.


Request >> Gopher Version 1 from pod gopher-distributor-694fd4f4db-gnn4g
Request >> Gopher Version 1 from pod gopher-distributor-694fd4f4db-4hk9t
Request >> Gopher Version 1 from pod gopher-distributor-694fd4f4db-24csn
...
...

Let's now up the VirtualService for the gopher-distributor to include some traffic mirror magic.


cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: gopher-distributor-virtual-service
spec:
  hosts:
  - gopher-distributor.default.svc.cluster.local
  http:
  - route:
    - destination:
        host: gopher-distributor.default.svc.cluster.local
        subset: v1
      weight: 100
    mirror:
        host: gopher-distributor.default.svc.cluster.local
        subset: v2
EOF

If we deconstruct the updated rule, all it's really saying is for 100 percent of requests to the hostname gopher-distributor.default.svc.cluster.local send requests to the v1 subset of that service, but also mirror the requests to the v2 subset (which is the version we just deployed).

Observability included

Okay, so we've done that, but nothing really changed. How do we know traffic is actually going to v2? Enter Grafana. We instructed Istio to setup Grafana when we ran the helm Installer for Istio earlier.

We're going to tap in to the awesome port-forward function of kubectl to access the Grafana instance that Istio very kindly configured for us.


kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o jsonpath='{.items[0].metadata.name}') 3000:3000 &

Once that's done, up your browser of choice and then navigate to the following URL:

http://localhost:3000/d/1/istio-mesh-dashboard

You should now be able to see two instances of the gopher-distributor service. v1 and v2.

Image of Grafana

The metrics should look pretty identical and that's down to the mirroring.

The cutover

Okay, so now that we are happy that the new version of our gopher-distributor service is behaving, let's start an incremental cutover.

This next rule us going to update the gopher-distributor VirtualServce and split requests 50/50 between v1 and v2.


cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: gopher-distributor-virtual-service
spec:
  hosts:
  - gopher-distributor.default.svc.cluster.local
  http:
  - route:
    - destination:
        host: gopher-distributor.default.svc.cluster.local
        subset: v1
      weight: 50
    - destination:
        host: gopher-distributor.default.svc.cluster.local
        subset: v2
      weight: 50
EOF

If we check back in with our test call in the other emulator window/tab, we should now see something a little different. Roughly 50 percent of the responses are coming from the v2 instance of the gopher-distributor service:


Request >> Gopher Version 2 from pod gopher-distributor-v2-6d5887bb96-vzc4x (817b00a0b7cca7a5)
Request >> Gopher Version 1 from pod gopher-distributor-694fd4f4db-24csn
Request >> Gopher Version 2 from pod gopher-distributor-v2-6d5887bb96-xjh45 (4246faf2c16d238a)
Request >> Gopher Version 1 from pod gopher-distributor-694fd4f4db-gnn4g
Request >> Gopher Version 2 from pod gopher-distributor-v2-6d5887bb96-xcrxh (6f33515876dfa2eb)
Request >> Gopher Version 1 from pod gopher-distributor-694fd4f4db-24csn
...

COOOL!

Let's now wrap things up with a full cutover and send 100 percent of requests to the new v2 instance of the gopher-distributor service.


cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: gopher-distributor-virtual-service
spec:
  hosts:
  - gopher-distributor.default.svc.cluster.local
  http:
  - route:
    - destination:
        host: gopher-distributor.default.svc.cluster.local
        subset: v2
      weight: 100
EOF

You'll see all of responses are coming back from the new v2 instance of the gopher-distributor service.


Request >> Gopher Version 2 from pod gopher-distributor-v2-6d5887bb96-xcrxh (353130990d3d3bad)
Request >> Gopher Version 2 from pod gopher-distributor-v2-6d5887bb96-vzc4x (93ec1cdc162261e4)
Request >> Gopher Version 2 from pod gopher-distributor-v2-6d5887bb96-xjh45 (6986a4496f15c931)
Request >> Gopher Version 2 from pod gopher-distributor-v2-6d5887bb96-xcrxh (c6d4ab12d05ab45f)
Request >> Gopher Version 2 from pod gopher-distributor-v2-6d5887bb96-vzc4x (67d56a7d02b15803)
Request >> Gopher Version 2 from pod gopher-distributor-v2-6d5887bb96-xjh45 (c46eebd43f7eb07b)

That's a wrap

As you can see, it's pretty easy to get Istio up and running within EKS. With the introduction of Dynamic Admission Controllers it got even easier to add you services in to the Istio service mesh. Istio offers a lot of really awesome features that remove so much of the heavy lifting that development teams traditionally had to do in order to monitor and operate complex distributed applications.

Saturday, August 25, 2018

Some talks I've given

From time to time I'm fortunate enough to have the opportunity to share some of my experiences at conferences and events.






I thought it might be useful for anyone who follows my blog to start compiling a list of some of the talks I've given. Hopefully you'll find some of the information I share in these talks useful.


AWS Dev Day Australia 2018 - Containers, Containers, Containers
https://youtu.be/mBiOBwy-UiE


AWS Dev Day Australia 2018 - Cleaning Up the Mesh With a Service Mesh!

https://youtu.be/d8iPOsdz1K4

AWS re:Invent 2017 - Architecting Container Infrastructure for Security and Compliance

https://youtu.be/Ofu22X7qHn

Monday, July 09, 2018

Building a CD Pipeline for Fargate from the Command-Line

In an earlier post, AWS Fargate from the Command-Line, we packaged up a simple Go HTTP web server as a Docker container image and deployed it in to Amazon ECS as an AWS Fargate backed service ... using only the command-line. Pretty sweet. 

In this post, we're going to take our project to the next-level and provision a Docker powered pipeline that'll take our simple Go HTTP web server, build it and prepare it to be shipped to Amazon ECS.

But why command-line?


Everything I've done in this post, and my previous posts, could just as easily be done using the AWS management console. I'd definitely encourage you to run through this process using the management console to get a feel for the differences. The management console, and more specifically the first-run wizards, do a good job of abstracting away a lot of the implementation details, which is great if you're looking to get started quickly.

That being said, I've found that by assembling the Lego blocks using the command-line (or any of the SDKs), you end up with a much deeper understanding of the relationships between those components.


 

TL;DR

By the end of this post you'll have a pipeline that can be used to build and deploy projects to an ECS cluster written in most languages using most tools.

Let's get started!


In the previous post, you'll recall that we had a simple HTTP server written in Go. I'd placed the main.go file and a Dockerfile in to a standard directory on my computer. 

What does every good software project need?


As with any good software project, the source code should be stored in a version control system. As it happens, AWS has fully managed VCS, AWS CodeCommit.

Let's start by creating ourselves a repository for storing our project.


aws codecommit create-repository --repository-name go-frontend

Git, Git, Git, Git ... 


Now, we need to turn our vanilla working directory in to a mean, green git-tracked machine by running a few simple commands.

# Initialise my workspace as a git tracked repo
git init

# Add the dockerfile and the main.go
git add .

# Do an initial commit
git commit -m 'initial commit'

Cool. So, we've now converted our existing folder in a git-tracked repo. Albeit locally tracked and stored. The first step in addressing that shortcoming is setting up some authentication (auth) for the hosted CodeCommit repository we just created.

I'm going to use public/private key-pair for auth. It's also possible to setup some git credentials using the AWS IAM console, but I prefer SSH keys. That's all I have to say about that.

I'm going to start by generating a new key-pair as follows:

ssh-keygen

During the key-pair creation process you'll be prompted to choose a file in which to store the private key. I opted to create a new file specifically for this new private key so that my existing private key (which I use for other things) did not get overwritten.

The corresponding public key will be written to a file with the same name but with the .pub extension.

The next step in the process requires us to copy the public key material, from the file with the .pub extension, and upload it up to AWS. To do this, cat the file and copy it's content to your clipboard.

cat Users/mitch/.ssh/codecommit_rsa.pub


Next, we use the AWS CLI to upload the public key material and associate it with an IAM user. In my case, my IAM has a name of mitch.beaumont, so that who I'm going to attach the public key to.

Paste the public key material you copied earlier as the value for the --ssh-public-key-body argument, don't forget to paste between quotes " ".

aws iam upload-ssh-public-key \
--user-name mitch.beaumont \
--ssh-public-key-body "ssh-rsa ..."

The response from this step will include an SSHPublicKeyId. You'll need this for the next step so take a note.

We're now going to hack our SSH config file (~/.ssh/config) to help make using the custom SSH key we generated for auth, a little more seamless.

Host git-codecommit.*.amazonaws.com
  User <whatever_your_SSHPublicKeyId_was>
  IdentityFile ~/.ssh/<whatever_your_private_key_file_is_called>

If you had to create the SSH config file then you'll need to chmod the permissions. Otherwise, you should be all set.

chmod 600 ~/.ssh/config

We now need to update the configuration of our local repo and add a "remote". This is the URL of the remote location were we want the contents of our repo to be stored. In our case, this is CodeCommit.

CodeCommit, much like GitHub, support two different kinds of URLs for repos. HTTPS and SSH. Because we're auth'ing using SSH, we need to use the SSH URL.

git remote add origin ssh://git-codecommit.us-west-2.amazonaws.com/v1/repos/go-frontend

Let's check that the remote URL took ...

git remote -v

Now we're ready to push our local repo up to our remote.

git push --set-upstream origin master

The stage is set!

We now find ourselves in a great position. As we get new ideas for killer features that we want to add to our simple Go HTTP web server, we hack away, commit the changes and push them to a remote location for safe-keeping.

For some people however, that might not be enough. WE ARE THOSE PEOPLE! We WANT continuous delivery.

Each time we commit a change, we want those changes to trigger a pipeline that takes our updated code, mixes it with some fairy-dust and spits our a fully functional container image that we can do something with, like scan, test or deploy to an environment.

 Something that looks a bit like this ...

Bill of Materials

The diagram above looks fairly straightforward. HOWEVER. There are a few things we're going to need in place to make our pipeline a reality. So let's take a look at what they are:

  1. CodeCommit Repoistory (we already have this)
  2. An S3 bucket to store various artifacts produced and consumed by the stages of the pipeline.
  3. An IAM role with the necessary permissions to to execute the various stages of our pipeline. 
  4. An IAM role with the necessary permissions to execute our build processes.
  5. A CodeBuild project.
  6. A CodePipeline pipeline.
  7. A CloudWatch event that fires each time a change is made to the code stored in the CodeCommit repository.

A place to call artifactStore

We begin the next exciting chapter of our adventure, but creating an S3 bucket. This bucket will be used by our pipeline for storing artifacts.

What's an artifact?

Good question! Simply put, Artifacts are files that are created by a stage in the pipeline. You can choose to discard the artifacts, or keep them and share them with other stages in the pipeline. (Which is what we'll be doing)

The S3 bucket we're creating in this step is where those artifacts we choose to keep will be stored. Keep a note of the name of the bucket. You'll need it a little later.

aws s3 mb s3://codepipeline-${REGION}-${ACCOUNT_ID}

 

Show me the permissions!

If you look back at our diagram, you can see that CodePipeline kind of runs the show. It's job is to share artifacts, trigger builds and generally keep things moving forwards. Because of that, we need to pass some privileges to CodePipeline. We do that via an IAM role.

If we want an AWS service, such as CodePipline, to assume an IAM role in our AWS account, we have to create a trust relationship. In our case, we are the trusting party and our policy says, we trust <service_whatever> to perform some actions within the boundaries of our AWS account. 

Let's start by defining that trust relationship and creating a file called pipeline_trust_policy.json.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "codepipeline.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Now we'll create a new IAM role called CodePipelineExecutionRole and configure it to use the trust policy we defined in pipeline_trust_policy.json.

Be sure to record the role Arn from the output of this step. You'll be needing that a little later.

aws iam create-role \ 
--role-name CodePipelineExecutionRole \
--assume-role-policy-document \
file://./pipeline_trust_policy.json

Next we need to define the actions we want our "trustee" to be able to perform. Create another file and called it pipeline_permissions_policy.json.

This policy file gives the trustee, the CodePipeline service, permissions to perform a bunch of actions, including allowing it to retrieve code from our CodeCommit repository, artifacts from our artifact store and some additional rights, which we may tap in to later, for deploying in to ECS:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "codecommit:GetBranch",
                "codecommit:GetCommit",
                "codecommit:UploadArchive",
                "codecommit:GetUploadArchiveStatus",
                "codecommit:CancelUploadArchive",
                "codebuild:BatchGetBuilds",
                "codebuild:StartBuild"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::codepipeline*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ecs:DescribeServices",
                "ecs:DescribeTaskDefinition",
                "ecs:DescribeTasks",
                "ecs:ListTasks",
                "ecs:RegisterTaskDefinition",
                "ecs:UpdateService"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:iam:PassRole",
            "Resource": "*"
        }
    ]
}

Lets use the pipeline_permissions_policy.json file to create a policy and attach it to the CodePipelineExecutionRole IAM role we created earlier.

Here we are creating an inline-policy. A better way to do this to help with future reuse might be to create a separate policy object and attach it. That way the same policy object can be reused by other roles in the future.

aws iam put-role-policy \
--role-name CodePipelineExecutionRole \
--policy-name CodePipeline-Permissions-Policy \
--policy-document file://./pipeline_permissions_policy.json

Now that we have an IAM role for our pipeline to use when executing, the next thing we need to do is create a similar IAM role with which to execute our CodeBuild project.

Another trust policy is needed first:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "codebuild.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Then the policy that defines the actions we need our CodeBuild project to be able to perform.

A few things worth calling out here, firstly, you'll notice that there are some "Allow" actions for CloudWatch Logs. This is required for CodeBuild to output the logs from the build project. Extremely useful when trying to debug our build process.

There are also a number of "Allow" actions for ECR. Cast your mind back to the original goal of this walk-through. We want to be at a point where we have an artifact that can be deployed following each push / merge to our master branch. (Continuous delivery).

The artifact we're producing is a container image. This is why our CodeBuild project needs access to ECR. It needs to be able to push the container image there.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:/aws/codebuild/*:log-stream:*"
            ]
        },
        {
            "Effect": "Allow",
            "Action":[
                "s3:GetObject",
                "s3:PutObject",
                "s3:GetObjectVersion",
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability",
                "ecr:PutImage",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload",
                "ecr:GetAuthorizationToken"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Let's create our IAM role for CodeBuild and attach the trust policy ...

aws iam create-role \
--role-name CodeBuildExecutionRole \
--assume-role-policy-document file://./build_trust_policy.json \

... and add the permissions policy.

aws iam put-role-policy \
--role-name CodeBuildExecutionRole \
--policy-name CodeBuild-Permissions-Policy \
--policy-document file://./build_permissions_policy.json

Buildy build build build.

What we're going to do now is create and configure a CodeBuild project. This project will take, as an input, the source-code from our CodeCommit repo, compile-it and spit out a container image.

I've taken a slightly different approach to this step in that I'm passing the configuration of my pipeline in as command line arguments.

You'll need to the Arn of the CodeBuildExecution role and the URI of the CodeCommit repository for this step.

Something of note here is the "image" property. Code build uses containers to execute the steps in the build project ...

Hang-on, aren't you building a container?

So, we're using a container, to build a container ... interesting. This approach is often referred to as DinD or Docker-in-Docker. (The everything-pipeline starts to take shape!)

Check out a full breakdown of what goes in to the aws/codebuild/docker:17.09.0 image here:

https://github.com/aws/aws-codebuild-docker-images/tree/master/ubuntu/docker/17.09.0

Let's create our build project. Don't forget to grab the name of the CodeBuild project as you'll need that for the next step.

aws codebuild create-project \
--name "goFrontend" \
--description "Build project for GoFrontend" \
--source type="CODEPIPELINE" \
--service-role "<Arn_of_codepbuildexecutionrole_role>" \
--environment type="LINUX_CONTAINER",\
image="aws/codebuild/docker:17.09.0",\
computeType="BUILD_GENERAL1_SMALL",\
environmentVariables="[{name=AWS_DEFAULT_REGION,value='${REGION}'},{name=AWS_ACCOUNT_ID,value='${ACCOUNT_ID}'},{name=REPOSITORY_URI,value=<Uri_of_ECR_repository>}]" \
--artifacts type="CODEPIPELINE"

buildspec.yml

How do we tell CodeBuild what to do as part of the build project? For that, we need to add a file called buildspec.yml to out project folder. This file defines all of the steps that we need CodeBuild to execute on our behalf. There is a lot going on in this file, but most of what you can see relates to extracting the metadata from our build process and using it to tag the resulting container image correctly, then pushing it to ECR.

You can grab the buildspec.yml file from a gist I've crested here:

Place the file in the root of your project and be sure to run git add / git commit / git push.

Your project directory should be looking something like this once you're done.


.
├── Dockerfile
├── buildspec.yml
└── main.go

We're almost there!

Most of the pieces of our puzzle are in-place. We now need to apply the glue that's going to hold everything in place: Our Pipeline.

We'll start by creating a new file pipeline_structure.json. This file defines the attributes and structure of our pipeline, including things like the role used to execute the pipeline and the stages that make up the pipeline.

You can see I've defined 3 stages in my pipeline

Source

This stage retrieves the source code of my project from CodeCommit. It then bundles up that source code as an output artifact and drops it in to our artifactStore. In the configuration section you'll notice that I've set PollForSourceChanges to "false". I've done this because I'm going to use CloudWatch Events to trigger my pipeline each time a change is made.

Build

This stage triggers the build pipeline that we created earlier. The input artifact of this stage is the output artifact of the Source stage (the bundles up source code). You'll notice I've created and output artifact called imagedefinitions. This file contains a simple JSON structure that includes the name of the container my container image represents and the container image tag (which is the build Id).

Release: 

The final stage is the release stage. This stage has two actions. Firstly, a manual approval, which is a blocking action. Assuming the previous stage completes successfully, the pipeline will halt at this point waiting for intervention from the operator. Once the operator "approves" or "rejects, the pipeline will continue, or fail respectively.

The second action in this stage deploys the changes to our ECS cluster. Notice the configuration items here the name of the cluster, the name of the service and the imagedefinitions.json file.


{
        "roleArn": "<Arn_of_codepipelineexecutionrole_role>",
        "stages": [
            {
                "name": "Source",
                "actions": [
                    {
                        "inputArtifacts": [],
                        "name": "Source",
                        "actionTypeId": {
                            "category": "Source",
                            "owner": "AWS",
                            "version": "1",
                            "provider": "CodeCommit"
                        },
                        "outputArtifacts": [
                            {
                                "name": "MyApp"
                            }
                        ],
                        "configuration":{
                            "RepositoryName": "<name_of_codecommit_repository>",
                            "BranchName": "master",
                            "PollForSourceChanges": "false"
                        },
                        "runOrder": 1
                    }
                ]
            },
            {
                "name": "Build",
                "actions": [
                    {
                        "inputArtifacts": [
                            {
                                "name": "MyApp"
                            }
                        ],
                        "name": "Build",
                        "actionTypeId": {
                            "category": "Build",
                            "owner": "AWS",
                            "version": "1",
                            "provider": "CodeBuild"
                        },
                        "outputArtifacts": [
                            {
                                "name": "imagedefinition"
                            }
                        ],
                        "configuration":{
                            "ProjectName": "goFrontend"
                        },
                        "runOrder": 1
                    }
                ]
            },
            {
                "name": "Release",
                "actions": [
                    {
                        "name": "Approval",
                        "actionTypeId": {
                            "category": "Approval",
                            "owner": "AWS",
                            "version": "1",
                            "provider": "Manual"
                        },
                        "runOrder": 1
                    },
                    {
                        "inputArtifacts": [
                            {
                                "name": "imagedefinition"
                            }
                        ],
                        "name": "DeployToECS",
                        "actionTypeId": {
                            "category": "Deploy",
                            "owner": "AWS",
                            "version": "1",
                            "provider": "ECS"
                        },
                        "outputArtifacts": [],
                        "configuration":{
                            "ClusterName": "fargate-cluster",
                            "ServiceName": "go-http-server",
                            "FileName": "imagedefinitions.json"
                        },
                        "runOrder": 2
                    }
                ]
            }
        ],
        "artifactStore": {
            "type": "S3",
            "location": "<name_of_the_s3_bucket_artifact>"
          },
        "name": "GoFrontend",
        "version": 1
}

Let's create the pipeline now, using the pipeline_structure.json file as out input.

aws codepipeline create-pipeline \
--pipeline file://./pipeline_structure.json

Test our pipeline

Assuming nothing has imploded, we're now at a point where we can test out our pipeline. Guess what? We can also do that using the command-line!

aws codepipeline start-pipeline-execution \
--name GoFrontend

The previous command will output an pipelineExecitionId. We will use that to check on the status of our pipeline as follows.


aws codepipeline get-pipeline-state --name GoFrontend \
| jq '.stageStates[] | select (.latestExecution.pipelineExecutionId == "<pipelineExecutionId>")'

Approving the pipeline

After <___> minutes we should see our pipeline is paused at the "Approval" action of the "Release" stage

You'll notice a token value. We're going to use this to approve the pipeline action so that it can move to the next action in this state. Which happens to be: Release it to our ECR cluster.

{
            "stageName": "Release",
            "inboundTransitionState": {
                "enabled": true
            },
            "actionStates": [
                {
                    "actionName": "Approval",
                    "latestExecution": {
                        "status": "InProgress",
                        "lastStatusChange": 1234567.123,
                        "token": "cxxxxxd-8xxx5-4xx2-9xxc-9xxxxxxxxe"
                    }
                }

Let's approve the stage using the put-approval-result action.


aws codepipeline put-approval-result \
--pipeline-name GoFrontend \
--stage-name Release \
--action-name Approval \
--result summary="Approving",status="Approved" \
--token cxxxxxd-8xxx5-4xx2-9xxc-9xxxxxxxxe

If we check the state of our pipeline again, we should now see that it has moved forward.

Wrapping it in a bow.

Let's recap what were we are. We have a CodeCommit repository that is storing our project. An S3 bucket that is going to be used for storing artifacts shared across the stages of our pipeline. We have a CodeBuild project that takes our source code as an input, starts a Docker container to compile the simple Go HTTP web server in to a statically linked binary. Finally, we have a CodePipeline that pulls all of these disparate parts in to a nice cohesive workflow.

That last thing we need to do is configure a trigger for our pipeline each time a change is made to the master branch of our project. For this, we're going to use CloudWatch Events.

Just like every good service, we need to trust the CloudWatch Events service to perform actions on our behalf.

(Click on the files to check out an example gist for each policy)

Capture the Arn of the role, you'll need it a little later.

aws iam create-role \
--role-name Role-for-MyRule \
--assume-role-policy-document file://./event_trust_policy.json

Next, give our role some permissions.

aws iam put-role-policy \
--role-name Role-for-MyRule \
--policy-name CodePipeline-Permissions-Policy-For-CWE \
--policy-document file://event_permissions_policy.json

Now we need to create a rule. This includes a pattern to the kind of events that would trigger this event.

aws events put-rule \
--cli-input-json file://./event_put_rule.json 

Finally, we need to give our rule some targets. Which in our case, is CodePipeline.

aws events put-targets \
--rule CodeCommiteRule \
--targets Id=1,Arn=<arn_for_code_pipeline>,RoleArn=<arn_for_the_event_rule>

Change is good

Go a head and make a change to you main.go file. Perhaps change the message that gets displayed when you hit the we service. Once you've made the changes, commit and push them to CodeCommit. Then sit back and watch the magic happen.



Things to Fix

This approach, just like our software, has room for improvement. For example, we may some to be a little more explicit with some of the policies we created for the services. Naming the specific resources that actions can be performed against.

It would also be cool to setup a better way of responding to approval notifications. Check out this awesome post on the AWS blog for some ideas on how this might look with Slack.

https://aws.amazon.com/blogs/devops/use-slack-chatops-to-deploy-your-code-how-to-integrate-your-pipeline-in-aws-codepipeline-with-your-slack-channel/

As always, I'd love your feedback and thoughts.

~mitch

Wednesday, July 04, 2018

AWS Fargate from the Command-Line

We all love a good command-line demo. Here is one I put together recently for demonstrating a few things, firstly Docker multi-stage builds, and secondly how a simple web service written in Go could be deployed to AWS Fargate using nothing but the command line.

 

What's cool about that I hear you ask?

What's cool about it is that at no point during this demo am I deploying, configuring or going to have to manage ANY servers.

 

Is it "serverless" ... is it containers? YES!

Let’s take a look:

Here is the simple Go HTTP server.

package main

import (
 "log"
 "net/http"

 "github.com/gorilla/mux"
)

func YourHandler(w http.ResponseWriter, r *http.Request) {
 w.Write([]byte("<h1>Hello Mitch Beaumont!</h1>\n"))
}

func main() {
 r := mux.NewRouter()

 r.HandleFunc("/", YourHandler)

 log.Fatal(http.ListenAndServe(":8000", r))
}

As I mentioned, I'm going to use Docker's multi-stage build process to compile my HTTP server and then build a lightweight container image.

 

Why are you using multi-stage builds?  

When it comes to containers, there aren't many people who'd argue that the smaller, the better! 

Multi-stage builds help us optimise the size of our container image by allowing us to, from a software development perspective, clear the wheat from the chaff. 

In the case of our simple Go application we require Go to be installed so that we can compile the application in to a binary. Once compiled, these dependencies are not required for the binary to run our simple web service. By using multi-stage builds, we easily use one container image, with the required dependencies, to build our binary. We can then move our artifact (the binary) in to a new container image, with a minimal footprint (in our case, scratch). 
The result is an optimised container image.
 
Here is my how my multi-stage Dockerfile looks.

# STEP 1 build executable binary

FROM golang:alpine As builder 
COPY . $GOPATH/src/github.com/mitchybawesome/http-server/
WORKDIR $GOPATH/src/github.com/mitchybawesome/http-server/

RUN apk add --no-cache git mercurial

#get dependancies
RUN go get -d -v

#build the binary
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o /go/bin/httpserver

# STEP 2 build a small image

# start from scratch
FROM scratch

# Copy our static executable from the builder
COPY --from=builder /go/bin/httpserver /go/bin/httpserver
ENTRYPOINT ["/go/bin/httpserver"]

Now I have my container image, I need to push it in to a repository. It just so happens that I have an Amazon ECR repo setup and ready to go. I'll use the following commands to build the image, tag it, login to ECR and push the image. (The account ID is a dummy)

ACCOUNT_ID="000000000"
REGION="us-west-2" 
REPO="go-http-server"
 
# Login to ecr
$(aws ecr get-login --no-include-email --region ${REGION})

# Build docker image
docker build -t ${REPO} .

# Tag docker image
docker tag ${REPO}:latest ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${REPO}:latest

# Push docker image
docker push ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${REPO}:latest

Now that we have our container image available. The next step is to create an ECS cluster in which to run our container.

 

BUT YOU SAID NO SERVERS! 

I know, I know. Cool your jets man!

Whilst it's true, I am creating a cluster, there are no actual instances being provisioned in my AWS account. The cluster construct is used purely as a management construct and security boundary.

# Create Cluster
aws ecs create-cluster --cluster-name fargate-cluster --region ${REGION}

Next, we need to create a load balancer that will route requests to our running service. In order to create the load balancer, we need few details about the environment, specifically, we need to know the VPC in to which we will be deploying the load balancer, and the subnets that we're going to connect our load balancer to.

vpcid=$(aws ec2 describe-vpcs | jq '.Vpcs[] | select (.IsDefault == true) | .VpcId' | sed -e 's/^"//' -e 's/"$//')

The $vpcid variable now contains our default VPC ID. We will use this to parse the list of available subnets. For the purpose of this demo, I'll be creating all of my resources in the default VPC.

Notice the filter I've applied in the jq query to select the VPC which has the "IsDefault" flag set to "true".

subnets=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=${vpcid}" | jq '.Subnets[].SubnetId') && echo $subnets

Run the above command to output a list of subnets. Keep these safe, we'll need them later.

The next step in our process is to create a security group that will be attached to the load balancer. We use this security group to control traffic ingress from the public Internet.

If we did not create / assign a security group, the VPCs default security group would be assigned to the load balancer and we'd have a hard time accessing our service.

# Create ELB security group
aws ec2 create-security-group \
--description "security group for ALB" \
--group-name "security-group-for-alb" \
--region ${REGION} | jq '.GroupId'

Grab the "GroupId" output from the previous command, and use it to define the ingress rules for the security group. You will also need the load balancer's GroupId for a later step, so keep it close.

In this example, we're allowing all TCP traffic from any source IP address to connect to the load balancer.

# Configure ELB security group ingress rule to allow ELB to connect to tasks. 
aws ec2 authorize-security-group-ingress \
--group-id <GroupId_from_previous_command> \
--protocol tcp \
--port 80 \
--cidr 0.0.0.0/0

We're now ready to create our load balancer. This command creates an ALB (Application Load Balancer) and outputs the ARN (Amazon Resource Name) to the command line.

 

Why ALB?

We're using an ALB because of the support for dynamic port mappings and path based routing. Both of these features translate to a more optimized deployment model in terms of infrastructure and ultimately cost.

 

Were you paying attention?

In an earlier step, I asked you to record some information. Do you remember what it was? 
I hope so, because you'll need that information to complete this command.  (Hint: It's the subnets!)

# Create a load balancer and get the ELB ARN.
aws elbv2 create-load-balancer \ 
--name go-http-server \ 
--subnets subnet-111111 \
subnet-222222 \
subnet-333333 \ 
--security-groups <ALB_Security_GroupId> \
| jq '.LoadBalancers[].LoadBalancerArn'

Now that we have a load balancer created, we need to create a target group. The load balancer uses the target group to route requests to one or more registered targets. Which in our case will be containers (or tasks) running our simple Go web service.

# Create a target group
aws elbv2 create-target-group \
--name fargate-targets \
--target-type ip \
--protocol HTTP \
--port 80 \
--vpc-id ${vpcid} | jq '.TargetGroups[].TargetGroupArn'

 

Joining the dots! 

Our target group now needs to be attached to a listener and the listener needs to be attached the load balancer to complete the setup. You'll need the load balancer ARN and the target group ARN, both of which were outputs from the previous commands, to complete this step.

# Create a listener
aws elbv2 create-listener \
--load-balancer-arn <load_balancer_arn> \
--protocol HTTP \
--port 80 \
--default-actions Type=forward,TargetGroupArn=<target_group_arn>

 

How do we get our application deployed in to the cluster?

Great question! We have an image that we pushed in to ECR. But we somehow need to tell ECS that we want it to launch a container based off of that image.  We do that by creating task definition.

The task definition is a set of properties that allows us to model the run-time environment for our containerised go web service. Within the task definition we specify, among other things, how much memory and CPU we want to allocate to our task.

Task definitions are written in JSON. I've taken the liberty of dropping an example of the task definition I'm using in to a gist, which can be found here.

If you use this task definition, don't forget to update the image path!

# Register Task Definition
aws ecs register-task-definition \
--cli-input-json file://./go-http-server.json \
--region ${REGION} --query 'taskDefinition.taskDefinitionArn'

Our task needs a security group assigned to it so that we can control the kind and the sources of traffic that are allowed to reach it.

Record the "GroupId". You'll need it later.

# Create security group for the tasks
aws ec2 create-security-group \
--description "security group for fargate task" \
--group-name "security-group-for-fargate-task" \
--region ${REGION} | jq '.GroupId'

I mentioned earlier that we we'd need the GroupId of the load balancers security group. Now is the time. Use that group ID to define and attach an ingress rule to the task security group.

Basically what we're doing here is telling the task that it can accept TCP connections over port 8000 from the load balancer.

Keeping our traffic flow rules tight!

# Configure security group ingress rule to allow ELB to connect to tasks.
aws ec2 authorize-security-group-ingress \
--group-id \
--protocol tcp \
--port 8000 \
--source-group <alb_security_group_id>

 

Wrapping up

The final step is creating our service. You'll need some of the outputs from the previous commands to complete this step, including: the name of the task definition, a comma separated list of the subnets in the VPC to which the tasks needs to be connected, the ID of the task security group, the ARN of the target group and the name of the container. (Which you can get from the task definition you created earlier).

# Create Service
aws ecs create-service --cluster fargate-cluster --service-name go-http-server \
--task-definition <task_definition> --desired-count 2 --launch-type "FARGATE" \
--network-configuration "awsvpcConfiguration={subnets=[ <command_separated_list_subnets> ],securityGroups=[<security_group>],assignPublicIp=ENABLED}" \
--load-balancers targetGroupArn=<target_group_arn>,containerName=<container_name>,containerPort=<container_port> \
--region ${REGION}

Let's now make sure we can connect to our service. Grab the DNSName of the load balancer by querying the load balancer ARN.


url=$(aws elbv2 describe-load-balancers \
--load-balancer-arns <load_balancer_arn> \
| jq '.LoadBalancers[].DNSName' | sed -e 's/^"//' -e 's/"$//') && curl $url


Hopefully you get a response back! If you don't perhaps check your security group rules.

 

 Drum roll please

So, using nothing but the command line and NO SERVERS, we've packaged up a simple web service,  pushed it to a secure repository and created a framework for scaling, securing, deploying and serving the web service.

In the next few posts, I'm going to explore some of the finer details around security, scaling and updating our simple web service.

As always, I'd love your feedback and thoughts.

~mitch




A little about Me

My photo
My name is Mitch Beaumont and I've been a technology professional since 1999. I began my career working as a desk-side support engineer for a medical devices company in a small town in the middle of England (Ashby De La Zouch). I then joined IBM Global Services where I began specialising in customer projects which were based on and around Citrix technologies. Following a couple of very enjoyable years with IBM I relocated to London to work as a system operations engineer for a large law firm where I responsible for the day to day operations and development of the firms global Citrix infrastructure. In 2006 I was offered a position in Sydney, Australia. Since then I've had the privilege of working for and with a number of companies in various technology roles including as a Solutions Architect and Technical team leader.