Saturday, June 23, 2018

Amazon EKS and IAM Authentication

This past month, AWS launched Amazon EKS. Elastic Container Service for Kubernetes.


What is Amazon EKS?

Amazon EKS is a managed service aimed at making it easier to deploy and operate Kubernetes clusters on AWS. 

What is IAM?

IAM, or Identity and Access Management, is the AWS authn and authz service. Access to and interaction with all AWS services is gated through IAM. 

What does IAM have to do with Kubernetes?

As part of Amazon EKS, authentication of requests made to the Kubernetes API is handled by IAM. That is to say that when you make a call to the Kubernetes API, like this:

kubectl get pods

The Kubernetes API server will firstly pass a bearer token, by way of a webhook, over to IAM to ensure that the request is coming from a valid identity. Once the validity of the identity has been confirmed, Kubernetes then uses it's own RBAC capability (Role Based Access Control) to approve of deny the specific action the requester tried to perform.

The super smart folks over at heptio created an awesome tool called the heptio authenticator for AWS which enables an integration between Kubernetes and IAM. 

https://github.com/heptio/aws-iam-authenticator/blob/master/README.md

How does this work? 

Lets run through the process of creating a new IAM user and then granting that user access to the Kubernetes API.

We begin by creating a new IAM user. Once the user has been created, the response should include the ARN (Amazon Resource Name) of the user. Record that. We'll need it later.

aws iam create-user \
--user-name curly \
--output text \
--query 'User.Arn'

We now need to update the AWS Auth configmap to map the IAM user to a kubernetes user object. The basic template can be found here.

mapUsers: |
    - userarn: arn:aws:iam::000000000000:user/curly
      username: curly

The diff should look something like this. (Don't forget to add in your instance role ARN!)

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: <ARN of instance role (not instance profile)>
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
  mapUsers: |
    - userarn: arn:aws:iam::000000000000:user/curly
      username: curly

Now it's time to apply the updated config map to your cluster.

kubectl apply -f aws-auth-cm.yaml


Quickly check that the user mapping was created by running this command.

kubectl describe configmap aws-auth -n kube-system 


Once we have our IAM user mapped to Kubernetes user object, lets see what roles we can bind to that user.

kubectl get clusterroles
 

One of the roles you'll see near to the top of the list is the edit role. Let's create a clusterrolebinding for the user curly to the edit cluster role.

kubectl create clusterrolebinding curly-edit --clusterrole=edit --user=curly 


POP QUIZ: What's wrong with this?
Everything I'm doing here involves imperative commands. Ideally, all of this should be applied as declarative configurations. Which might look a bit like this:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: curly-edit
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: edit
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
name: curly 

We'll now test if curly can now create a deployment.

kubectl auth can-i create deployments --as=curly

Hopefully you got a "Yes" response back! If not, maybe go back and check the steps above one more time.

Check Point.

We've proven at this point, that a new user object within Kubernetes can do some stuff. Let us now create access key / secret access key for the user curly and make sure that the IAM user curly can actually use the Kubernetes API to do some stuff.

aws iam create-access-key --user-name curly

 

Using that secret access key and access key, update the local credentials file to use  curlys access key and secret access key.

aws configure --profile curly

I've added the --profile to make it easy to flip back and fourth between our admin user and our curly user in later steps.

No you should be able to run a simple kubectl command and get a response back. To force the heptio authenticator to use the credentials from the curly profile, set the AWS_PROFILE envar in line with the kubectl command.

Let's try:

AWS_PROFILE=curly kubectl get pods

Curly should now be able to see a list of running pods within the cluster.  

Excellent, good job! 


Do I have to create a mapping for each user?

No! In addition to creating a mapping for a specific user, we can also create a mapping for an IAM role. This role can be assumed by multiple users via group membership.

Let's start by deleting the existing cluster role binding we created for curly.

kubectl delete clusterroldebinding curly-edit

Next we'll set some envars to help with our IAM role creation process.

(I've borrowed these steps from the Heptio github repo!)

ACCOUNT_ID=$(aws sts get-caller-identity --output text --query 'Account')

POLICY=$(echo -n '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::'; echo -n "$ACCOUNT_ID"; echo -n ':root"},"Action":"sts:AssumeRole","Condition":{}}]}')

Now lets create a role, record the ARN as per the previous example. 

aws iam create-role \
--role-name KubernetesAdmin \
--description "Kubernetes administrator role" \
--assume-role-policy-document "$POLICY" \
--output text \
--query 'Role.Arn'

Next we need to modify our config map again to include the new role ARN. The diff should look a bit like this:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: <ARN of instance role (not instance profile)>
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: arn:aws:iam::000000000000:role/KubernetesAdmin 
      username: admin-curly:{{SessionName}}
      groups:
       -  mitchyb:editors
  mapUsers: |
    - userarn: arn:aws:iam::000000000000:user/curly
      username: curly 

Let's apply the updated config map to the cluster.

kubectl apply -f aws-auth-cm.yaml

Quickly check that the role mapping was created by running this command.

kubectl describe configmap aws-auth -n kube-system


Hopefully you see the the role mapping in the output. 

Next we can create a cluster role binding to the edit cluster role, for those users who assume the arn:aws:iam::000000000000:role/KubernetesAdmin role.

kubectl create clusterrolebinding curly-edit --clusterrole=edit --group=mitchyb:editors

ACCOUNT_ID=$(aws sts get-caller-identity --output text --query 'Account')

POLICY=$(echo -n '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Resource": "arn:aws:iam::'; echo -n "$ACCOUNT_ID"; echo -n ':role/KubernetesAdmin","Action":"sts:AssumeRole"}]}')


Create a policy that enables the role to be assumed:
 
aws iam create-policy \
--policy-name kubernetes-admin-policy \
--policy-document "$POLICY" \
--output text \
--query 'Policy.Arn' 


Create a group:

aws iam create-group \
--group-name kubernetes-admins 


Attach a policy to allows the members of the group to assume the policy:
 
aws iam attach-group-policy \
--group-name kubernetes-admins \
--policy-arn arn:aws:iam::
000000000000:policy/kubernetes-admin-policy

Put curly in the new group:

aws iam add-user-to-group \
--group-name kubernetes-admins \
--user-name curly

 

Configuring the AWS CLI

The only thing left for us to do is make a few slight tweaks to the configuration of the AWS CLI to so that when we make a call using kubectl, our session assumes the arn:aws:iam::000000000000:role/KubernetesAdmin role.

[profile adminaccess]
role_arn: arn:aws:iam::000000000000:role/KubernetesAdmin
source_profile: curly

Armed with our new profile, lets request a list of pods, we can force the heptio authenticator to use curly profiles (and associated credentials) by setting the AWS_PROFILE envar inline with the kubectl, as follows:

AWS_PROFILE=adminaccess kubectl get pods

Just to prove that this is all working as expected, let's remove curly from the
kubernetes-admins group. 

Easy to do via the management console.
or using the AWS CLI, without setting the AWS_PROFILE envar inline.

aws iam remove-user-from-group \
--group-name kubernetes-admins \
--user-name curly


Given our current setup this should lead to curly no longer be able to assume the arn:aws:iam::000000000000:role/KubernetesAdmin role which in turn means as far as RBAC is concerned, curly is not authorized to perform any action against the Kubernetes API.

AWS_PROFILE=adminaccess kubectl get pods

Expect to see something like this:

could not get token: AccessDenied: User: arn:aws:iam::000000000000:user/curly is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::000000000000:role/KubernetesAdmin  

 

And that's a wrap.

That's about all there is to it. As you can see, with this configuration, you can easily create groups of users (via IAM) and authorize the members of those groups to perform specific actions within the Kubernetes cluster (thanks to RBAC).

Revoking those privileges is then as easy as removing those users from the given groups.


Sunday, June 17, 2018

Amazon ECS Daemon Scheduling Strategy

So this past week AWS launched a new daemon scheduling strategy for ECS (Elastic Container Service).

 

 

What is a Daemon? 

There are plenty of excellent definitions out there so I'm not going to attempt to reinvent the wheel.

"A daemon is a type of program on Unix-like operating systems that runs unobtrusively in the background, rather than under the direct control of a user, waiting to be activated by the occurance of a specific event or condition" - Linfo.org

What Wikipedia says: Daemon

 

What is a Daemon Scheduling Stratergy?

Imagine you have a cluster of N instances. Lets say you want to ensure that each instance in your cluster runs a copy of a given container (or task in ECS parlance). Well, this is what the daemon scheduling strategy does for you.

But, could I not just create a service that has the same number of tasks as there are instances in my cluster and use a distinct instance placement constraint to ensure that each instance is running a copy of a given task. Well, yes, you could do that. But what happens when your cluster scales? You'd need to go and update the service to reflect the changes in cluster size, which doesn't sound like much fun.

This is where the daemon scheduling stratergy really shines. It does not care about how many nodes you have, instead, it just makes sure that for *every node, there is a copy of a particular task running on it. Simples!

#Tasks == #Nodes

 

*It is possible to scope down the instances to which tasks within a daemon service are scheduled using placement constraints. See towards the bottom of this post for more details.

More information about this new capability can be found here: https://aws.amazon.com/about-aws/whats-new/2018/06/amazon-ecs-adds-daemon-scheduling/

 

What kind of things would I use a Daemon Scheduling Strategy for?

A common use case is agents. Monitoring, log collection and security to name a few. These are all things that we generally want running in the background and that wait to be activated by a specific event, some metrics being emitted or log files being appended to.

A lot of third-party monitoring solutions that require an agent to be deployed, ship that agent as a Docker container. Datadog is a good example. This is a great use case for the daemon scheduling strategy.

 

So how do I use this new Daemon Scheduling Stratergy?

I'm going to deploy the Datadog agent to my small ECS cluster to give a practical example of how and why the Daemon scheduling strategy is so cool.

Let's start by creating a new task definition for the Datadog agent. The template task definition can be found here

The task definition can be created using the ECS management console, like so: 

  

Or using the AWS CLI, like so

aws ecs register-task-definition \
--cli-input-json file://path/to/datadog-agent-ecs.json

You'll need to mod the task definition slightly in include your specific Datadog API key.

The next step is to use this task definition to create a new service. If you're using the management console, click on the Actions menu and choose Create Service:



Fill out the details on the service configuration screen as you like. But what's important here is that where we would typically choose a set number of replicas, we're instead going to choose DAEMON.



I'm then going to click through the remaining configuration screens because my daemon does not need to be load balanced and daemon services don't support auto scaling. 

The last step is for me to click Create Service. 

This whole process can be distilled down to a simple CLI call with a few arguments like this:

aws ecs create-service \ 
--service-name datadog-agent \
--cluster daemonset \
--scheduling-strategy DAEMON \
--task-definition datadog-agent-task:1 \
--region us-west-2 

You'll need to make sure you're running at least version 1.15.37 of the AWS CLI for this to work.

 

Did it work? 

Good question. Given what we now know about the job of the daemon scheduling strategy we can hypothesize the if we have a five instance cluster, then there should be five tasks running. One for each of the running instances.

First lets check how many container instances we have:

aws ecs list-container-instances \
--cluster daemonset \
--region us-west-2

Next, lets check how many tasks we have running: 

aws ecs list-tasks \
--cluster daemonset \
--region us-west-2

If everything went according to plan, we should see the same number of tasks as we have running container instances.

 

Selective Daemon Placement

What if I don't want my daemon service to run on all of my container instances, only the ones which are use for a specific purposes. For example, we may want our Datadog agent to only run on container instances that are in a production fleet. No problemo!

Placement constraints and custom attributes are our friend.

We start by adding  a custom attribute to a certain subset of container instances in our cluster. I'm going to add the attribute to a single container instance. A custom attribute is nothing more than metadata that we can attach to container instances.

aws ecs put-attributes \
--cluster daemonset \
--attributes "name=env,value=prod,targetType=container-instance,targetId=4914bc6c-9f21-4a28-bcd0-5b29e210ac79" \
--region us-west-2

This command adds a custom attribute with a name of "env" with a value of "prod" to the container instance "4914bc6c-9f21-4a28-bcd0-5b29e210ac79". 

Next, let's use some jq-fu to see if it worked as expected:

aws ecs describe-container-instances \
--container-instance 4914bc6c-9f21-4a28-bcd0-5b29e210ac79 \
--cluster daemonset \
--region us-west-2 \
| jq '.containerInstances[].attributes[] | select (.name == "env")'

We should see a custom attribute with the name of "env" and the value of "prod" returned.

Now let's create a new daemon service that includes a placement constraint for container instances with an env attribute that has a value of prod:

aws ecs create-service \
--service-name datadog-agent \
--cluster daemonset \
--scheduling-strategy DAEMON \
--task-definition datadog-agent-task:1 \
--placement-constraints type=memberOf,expression=attribute:env==prod \
--region us-west-2

If we take a look at the tasks which are running, we should now only see one, and, at least in my case, it will have been scheduled on instance 4914bc6c-9f21-4a28-bcd0-5b29e210ac79:

aws ecs list-tasks \
--cluster daemonset \
--region us-west-2

And that is about all there is to it. This is a welcome addition to ECS and helps address a great many use cases. I'm really interested to see how this new capability gets used.

Enjoy!


Saturday, June 09, 2018

Amazon Container Services Learning Resources

Amazon Container Services Learning Resources

I love speaking with customers and colleagues about containers and microservices. There are so many great resources out there to help you get started out with these exciting technologies and patterns on AWS. I've started to compile a list of some of my favorites.

 If you know of any others, please comment below and I'll get them added. Enjoy.

Catsndogs.lol

This is self-paced workshop designed to allow developers and system administrators to get hands on with Amazon Elastic Container Service concepts such as service and container-instance auto-scaling, spot-fleet integration, container placement strategies, service discovery, secrets management with AWS Systems Manager Parameter Store, time-based and event-based scheduling, and automated deployment pipelines.


https://github.com/aws-samples/amazon-ecs-catsndogs-workshop 




ECS Workshop

This workshop is designed to educate engineers that might not be familiar with Fargate, ECS, and possibly even Docker container workflow.

https://ecsworkshop.com



Kubernetes the AWSome Way!

This is a self-paced workshop designed for Development and Operations teams who would like to leverage Kubernetes on Amazon Web Services (AWS).


https://github.com/aws-samples/aws-workshop-for-kubernetes



Deploying Microservices on AWS Cloud

An exploration, with examples, of a few of the different options avaialble for deploying microservices on AWS. These include Amazon ECS, Kubernetes and AWS Lambda.

https://github.com/aws-samples/aws-microservices-deploy-options 







A little about Me

My photo
My name is Mitch Beaumont and I've been a technology professional since 1999. I began my career working as a desk-side support engineer for a medical devices company in a small town in the middle of England (Ashby De La Zouch). I then joined IBM Global Services where I began specialising in customer projects which were based on and around Citrix technologies. Following a couple of very enjoyable years with IBM I relocated to London to work as a system operations engineer for a large law firm where I responsible for the day to day operations and development of the firms global Citrix infrastructure. In 2006 I was offered a position in Sydney, Australia. Since then I've had the privilege of working for and with a number of companies in various technology roles including as a Solutions Architect and Technical team leader.