Skip to main content

Get NodeAuthorization working in Kubernetes with acs-engine

Node Authorization in k8s

I'm starting to get into the container world and I'm loving it. Recently we helped a client build out and deploy a micro-services application in Kubernetes. We created the cluster in Azure using the open source project acs-engine. After we got the cluster set up, our client asked for some updates to the cluster for security reasons. One of those updates was to enable Node Authorization.

What is Node Authorization?

Node Authorization locks down each Node in the cluster to only be able to do actions on itself. If this is not turned on, its possible for a malicious pod to take actions on any other node, including reading secrets, deleting pods, etc. There is an excellent post by Antoine Cotten that explains this very well ( as well as RBAC, which is a different subject altogether).

How do I set it up?

Based on the current documentation, it looks like setting up Node Authorization should be easy. Basically follow these steps
  1. Turn on TLS Boot Strapping
  2. Edit the api-server yaml file
  3. Sit back and relax knowing your cluster is more secure
The trouble is that #1 takes you down a very deep rabbit hole of links starting here. The rest of this post will walk you through how I got it set up.

Turning on TLS Bootstrapping

The first step in enabling TLS bootstrapping is to create a token csv file on the master node and update the apiserver yaml file to use it.

Create Token file

  1. SSH into the master node
  2. change directory to /etc/kubernetes/
  3. run the following command to create a token string 
    1. head -c 16 /dev/urandom | od -An -t x | tr -d ' '
  4. create a bootstrap.csv file
    1. touch bootstraptoken.csv
  5. edit the file
    1. vi bootstraptoken.csv
  6. Add the following text, substituting in your token created above
    1. [token],kubelet-bootstrap,10001,"system:bootstrappers"
    2. example: c0e89sdfasdfasdfff3b8718ab7868cc989,kubelet-bootstrap,10001,"system:bootstrappers"

Edit API Server

After the CSV file is created, we need to update the api server configuration to use it. 
  1. Make sure you are still SSH'ed into the master
  2. edit the api server configuration at :/etc/kubernetes/manifests/kube-apiserver.yaml
    1. vi /etc/kubernetes/manifests/kube-apiserver.yaml
  3. Add "--token-auth-file=/etc/kubernetes/bootstraptoken.csv" to the commands
Great! Now your master node is set up to allow TLS bootstrapping. Next up is to get your agents set up.

Configure Agent Node

Note: the following instructions are to configure a single agent. You will need to do this for each agent in your cluster.

  1. SSH to the Agent node. 
    • Typically this is done by SSH'ing to the master, uploading your private key to the master and then SSH'ing into the master.

Edit kubeConfig file

We need to edit the create a kubeconfig file that uses the token string created above.

  1. Change directory to /var/lib/kublet
    1. cd /var/lib/kubelet
  2. Create a bootstrap.kubeconfig file
    1. touch bootstrap.kubeconfig
  3. Edit the bootstrap.kubeconfig file with the following text, substituting in the IP address of the master node and the token string created above
    1. apiVersion: v1
      kind: Config
      - name: localcluster
          certificate-authority: /etc/kubernetes/certs/ca.crt
          server: https://MASTERNODEIP:443
      - name: kubelet-bootstrap
          token: TOKENSTRING
      - context:
          cluster: localcluster
          user: kubelet-bootstrap
        name: localclustercontext
      current-context: localclustercontext

Edit Kublet

The next step is to edit the Kublet configuration to point to the bootstrap.kubeconfig file created directly above
  1. Change Directory to /etc/systemd/system
  2. Edit kubelet.service
    1. Add the following to the startup section, making sure to include the forward slash

Restart Kubelet

After those edits, let's restart the Kubelet to make sure that it picks up the latest changes. Run the following commands: 

systemctl daemon-reload
systemctl stop kubelet
docker kill $(docker ps -q)
systemctl start kubelet

Validate it's working

Now that the master and agents are updated, lets make sure its working

  1. run "kubectl get nodes"
    1. the node you just updated should say "NotReady". This is because we have not approved the CSR for the node. Lets do that now
  2. run "kubectl get csr"
    1. this will show the CSR's that have been submitted. You should see one in a pending state for the node was just set up
  3. run "kubectl describe certificate to review the request. The Common Name should be "System:nodes:
  4. Run "kubectl get nodes" again. This time the node should be back to "Ready" state

Edit API Server (again)

Now TLS Bootstrapping is set up, but NodeAuthorization isn't enabled yet. In order to do that, we need to go back to the master and edit the api server.

  • Don't do these steps as part of editing the masters above as your cluster will become unstable since the API server will expect the agents to be configured with TLS Bootstrapping.
  • While the documentation on NodeAuthorization doesn't mention needing RBAC enabled, i was unable to schedule pods on nodes until I turned it on. There is most likely some reliance on it, or an additional configuration I missed. However enabling RBAC is probably a good idea from a security standpoint anyway.

  1. SSH Into the Master
  2. Change directory to /etc/kubernetes/manifests
  3. edit kube-apiserver.yaml and add the following commands:
    1. "--authorization-mode=RBAC,Node"
    2. Add "NodeRestriction" to the existing admission-control command
    3. example:
  4. Restart all docker images on the master
    1. docker kill $(docker ps -q)


In summary, the steps above aren't daunting and but aren't easy to figure out from the docs. Adding this capability into acs-engine wouldn't be too difficult, even if you just customized it for your own cluster.

A big thanks to Lachie Evenson for helping me get this running. He's got some great videos around Kubernetes that I've found very useful. 

 Hopefully this saves someone an hour (or day) or two. Enjoy!


    Popular posts from this blog

    Build/Deploy Windows service with TFS

    Building and deploying a web service or website via TFS is pretty straight forward, but automatically deploying a windows service or console application takes a b it of setup. I’ve outlined the steps below to get your service deployed. Step 1: Set up Automated Build and MSDeploy on the Target server. If you are doing any sort of automated build/deploy I recommend using MSDeploy. You can follow steps 1-3 from a previous post here.Step 2: Edit the .csproj fileThe project file for the app you are trying to deploy needs to be edited so that TFS will create a directory for the output of the project. Otherwise the project will be built, but the assemblies will be placed in the code drop location with no organization to it.To edit the file, go to the open file dialog and locate the csproj file. Select it but don’t click “Open. Instead click on the dropdown arrow next to the Open button, select “Open with”, and choose the XML (Text) editor as shown:Once the file is open, add the following “Pr…

    Repository Pattern with Cross Table Objects

    In a recent post, I mentioned one of the advantages of using Entity Framework in your application is that you can build a generic repository class very easily. Of course, its not realistic to think that all access to the data will be a single table at a time. Most often it’s the case where you need to return data that spans multiple tables. I’m  going to show you how I created a simple repository class that spans tables.Creating the Summary/DTO ObjectThe first thing I like to start with is to create the simple POCO object that will be used to transport the data. This is essential to define first so that you do not get caught up in data structures, but instead define the data as the application is going to need it. In the case with my database, I have a table called “Avail” that contains a ton of foreign keys to a contact table. I needed to display this data, but instead of a bunch of foreign keys, I needed to display the actual names of people. etc. I ended up defining the object as f…

    Quickly and Easily Deploy Websites/Web Services with TFS Build via Web Deploy (MSDeploy)

    When I first started deploying code from TFS I took the simple approach and created a batch file and deployed the websites via RoboCopy. I’m a very “Keep it simple” kind of guy, so this worked for us for a long time and so nothing was changed. With my most recent project however, we were deploying code over a slow VPN tunnel from our servers in Chicago to servers located in Europe. Due to this, the RoboCopy was taking over 4.5 hours to complete. I needed a better/faster way so I started looking into Web Deploy (MSDeploy). I was able to get it working fairly easily and I was pleasantly surprised how easy it was to get it working, and how much time its saved me! I can now deploy the code in less than 20 minutes!I’ve outlined the process in detail below, but in general you only need to do this:Add MSBuild parameters to your automated buildCustomize the deployment parameters for the websiteCreate a batch file to an auto-generated MSDeploy scriptExecute the batch file from the automated bu…