Saturday, June 23, 2018

Amazon EKS and IAM Authentication

This past month, AWS launched Amazon EKS. Elastic Container Service for Kubernetes.


What is Amazon EKS?

Amazon EKS is a managed service aimed at making it easier to deploy and operate Kubernetes clusters on AWS. 

What is IAM?

IAM, or Identity and Access Management, is the AWS authn and authz service. Access to and interaction with all AWS services is gated through IAM. 

What does IAM have to do with Kubernetes?

As part of Amazon EKS, authentication of requests made to the Kubernetes API is handled by IAM. That is to say that when you make a call to the Kubernetes API, like this:

kubectl get pods

The Kubernetes API server will firstly pass a bearer token, by way of a webhook, over to IAM to ensure that the request is coming from a valid identity. Once the validity of the identity has been confirmed, Kubernetes then uses it's own RBAC capability (Role Based Access Control) to approve of deny the specific action the requester tried to perform.

The super smart folks over at heptio created an awesome tool called the heptio authenticator for AWS which enables an integration between Kubernetes and IAM. 

https://github.com/heptio/aws-iam-authenticator/blob/master/README.md

How does this work? 

Lets run through the process of creating a new IAM user and then granting that user access to the Kubernetes API.

We begin by creating a new IAM user. Once the user has been created, the response should include the ARN (Amazon Resource Name) of the user. Record that. We'll need it later.

aws iam create-user \
--user-name curly \
--output text \
--query 'User.Arn'

We now need to update the AWS Auth configmap to map the IAM user to a kubernetes user object. The basic template can be found here.

mapUsers: |
    - userarn: arn:aws:iam::000000000000:user/curly
      username: curly

The diff should look something like this. (Don't forget to add in your instance role ARN!)

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: <ARN of instance role (not instance profile)>
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
  mapUsers: |
    - userarn: arn:aws:iam::000000000000:user/curly
      username: curly

Now it's time to apply the updated config map to your cluster.

kubectl apply -f aws-auth-cm.yaml


Quickly check that the user mapping was created by running this command.

kubectl describe configmap aws-auth -n kube-system 


Once we have our IAM user mapped to Kubernetes user object, lets see what roles we can bind to that user.

kubectl get clusterroles
 

One of the roles you'll see near to the top of the list is the edit role. Let's create a clusterrolebinding for the user curly to the edit cluster role.

kubectl create clusterrolebinding curly-edit --clusterrole=edit --user=curly 


POP QUIZ: What's wrong with this?
Everything I'm doing here involves imperative commands. Ideally, all of this should be applied as declarative configurations. Which might look a bit like this:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: curly-edit
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: edit
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
name: curly 

We'll now test if curly can now create a deployment.

kubectl auth can-i create deployments --as=curly

Hopefully you got a "Yes" response back! If not, maybe go back and check the steps above one more time.

Check Point.

We've proven at this point, that a new user object within Kubernetes can do some stuff. Let us now create access key / secret access key for the user curly and make sure that the IAM user curly can actually use the Kubernetes API to do some stuff.

aws iam create-access-key --user-name curly

 

Using that secret access key and access key, update the local credentials file to use  curlys access key and secret access key.

aws configure --profile curly

I've added the --profile to make it easy to flip back and fourth between our admin user and our curly user in later steps.

No you should be able to run a simple kubectl command and get a response back. To force the heptio authenticator to use the credentials from the curly profile, set the AWS_PROFILE envar in line with the kubectl command.

Let's try:

AWS_PROFILE=curly kubectl get pods

Curly should now be able to see a list of running pods within the cluster.  

Excellent, good job! 


Do I have to create a mapping for each user?

No! In addition to creating a mapping for a specific user, we can also create a mapping for an IAM role. This role can be assumed by multiple users via group membership.

Let's start by deleting the existing cluster role binding we created for curly.

kubectl delete clusterroldebinding curly-edit

Next we'll set some envars to help with our IAM role creation process.

(I've borrowed these steps from the Heptio github repo!)

ACCOUNT_ID=$(aws sts get-caller-identity --output text --query 'Account')

POLICY=$(echo -n '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::'; echo -n "$ACCOUNT_ID"; echo -n ':root"},"Action":"sts:AssumeRole","Condition":{}}]}')

Now lets create a role, record the ARN as per the previous example. 

aws iam create-role \
--role-name KubernetesAdmin \
--description "Kubernetes administrator role" \
--assume-role-policy-document "$POLICY" \
--output text \
--query 'Role.Arn'

Next we need to modify our config map again to include the new role ARN. The diff should look a bit like this:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: <ARN of instance role (not instance profile)>
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: arn:aws:iam::000000000000:role/KubernetesAdmin 
      username: admin-curly:{{SessionName}}
      groups:
       -  mitchyb:editors
  mapUsers: |
    - userarn: arn:aws:iam::000000000000:user/curly
      username: curly 

Let's apply the updated config map to the cluster.

kubectl apply -f aws-auth-cm.yaml

Quickly check that the role mapping was created by running this command.

kubectl describe configmap aws-auth -n kube-system


Hopefully you see the the role mapping in the output. 

Next we can create a cluster role binding to the edit cluster role, for those users who assume the arn:aws:iam::000000000000:role/KubernetesAdmin role.

kubectl create clusterrolebinding curly-edit --clusterrole=edit --group=mitchyb:editors

ACCOUNT_ID=$(aws sts get-caller-identity --output text --query 'Account')

POLICY=$(echo -n '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Resource": "arn:aws:iam::'; echo -n "$ACCOUNT_ID"; echo -n ':role/KubernetesAdmin","Action":"sts:AssumeRole"}]}')


Create a policy that enables the role to be assumed:
 
aws iam create-policy \
--policy-name kubernetes-admin-policy \
--policy-document "$POLICY" \
--output text \
--query 'Policy.Arn' 


Create a group:

aws iam create-group \
--group-name kubernetes-admins 


Attach a policy to allows the members of the group to assume the policy:
 
aws iam attach-group-policy \
--group-name kubernetes-admins \
--policy-arn arn:aws:iam::
000000000000:policy/kubernetes-admin-policy

Put curly in the new group:

aws iam add-user-to-group \
--group-name kubernetes-admins \
--user-name curly

 

Configuring the AWS CLI

The only thing left for us to do is make a few slight tweaks to the configuration of the AWS CLI to so that when we make a call using kubectl, our session assumes the arn:aws:iam::000000000000:role/KubernetesAdmin role.

[profile adminaccess]
role_arn: arn:aws:iam::000000000000:role/KubernetesAdmin
source_profile: curly

Armed with our new profile, lets request a list of pods, we can force the heptio authenticator to use curly profiles (and associated credentials) by setting the AWS_PROFILE envar inline with the kubectl, as follows:

AWS_PROFILE=adminaccess kubectl get pods

Just to prove that this is all working as expected, let's remove curly from the
kubernetes-admins group. 

Easy to do via the management console.
or using the AWS CLI, without setting the AWS_PROFILE envar inline.

aws iam remove-user-from-group \
--group-name kubernetes-admins \
--user-name curly


Given our current setup this should lead to curly no longer be able to assume the arn:aws:iam::000000000000:role/KubernetesAdmin role which in turn means as far as RBAC is concerned, curly is not authorized to perform any action against the Kubernetes API.

AWS_PROFILE=adminaccess kubectl get pods

Expect to see something like this:

could not get token: AccessDenied: User: arn:aws:iam::000000000000:user/curly is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::000000000000:role/KubernetesAdmin  

 

And that's a wrap.

That's about all there is to it. As you can see, with this configuration, you can easily create groups of users (via IAM) and authorize the members of those groups to perform specific actions within the Kubernetes cluster (thanks to RBAC).

Revoking those privileges is then as easy as removing those users from the given groups.


No comments:

A little about Me

My photo
My name is Mitch Beaumont and I've been a technology professional since 1999. I began my career working as a desk-side support engineer for a medical devices company in a small town in the middle of England (Ashby De La Zouch). I then joined IBM Global Services where I began specialising in customer projects which were based on and around Citrix technologies. Following a couple of very enjoyable years with IBM I relocated to London to work as a system operations engineer for a large law firm where I responsible for the day to day operations and development of the firms global Citrix infrastructure. In 2006 I was offered a position in Sydney, Australia. Since then I've had the privilege of working for and with a number of companies in various technology roles including as a Solutions Architect and Technical team leader.