{"id":43166,"date":"2020-10-08T00:00:00","date_gmt":"2020-10-08T00:00:00","guid":{"rendered":"https:\/\/www.trendmicro.com\/en_us\/devops\/20\/j\/basics-of-keeping-kubernetes-clusters-secure-pt-2.html"},"modified":"2020-10-08T00:00:00","modified_gmt":"2020-10-08T00:00:00","slug":"basics-of-keeping-kubernetes-clusters-secure-part-2","status":"publish","type":"post","link":"https:\/\/www.threatshub.org\/blog\/basics-of-keeping-kubernetes-clusters-secure-part-2\/","title":{"rendered":"Basics of Keeping Kubernetes Clusters Secure Part 2"},"content":{"rendered":"

<\/p>\n

\n

By Magno Logan Trend Micro Research<\/span><\/p>\n

We outline security mitigations and settings that should be prioritized in a clustered environment. The second part of our security guide on Kubernetes clusters covers best practices related to worker nodes, the kubelet, pods, and audit logs.<\/p>\n<\/p><\/div>\n

\n

In our previous article<\/a>, we talked about the different ways developers can protect control plane components, including Kube API server configurations, RBAC authorization, and limitations in the communication between pods through network policies.<\/p>\n

This time, we focus on best practices that developers can implement to protect worker nodes and their components. We will talk about the kubelet, the pods, and how to set up audit logs to have better visibility into your cluster. At the end, we include a few basic tips that, though they might seem like common sense, still need to be highlighted.<\/p>\n

For most of our examples, it should be noted that we will use a kubeadm cluster setup with Kubernetes v1.18.<\/p>\n

\n

The worker nodes<\/h2>\n

If the control plane is the brains of the operation, the worker nodes are the muscles. They run and control all the pods and containers from your cluster. You can have zero or more worker nodes on your cluster, although it is not recommended to run your pods on the same node as the control plane. The main components of a Worker are the kubelet, the container runtime interface (which is Docker by default, but could be another), and the kube-proxy. You can see all your nodes, including the primary node, by using this kubectl command: kubectl get nodes<\/i>. The output should be something similar to the following image:<\/p>\n

\"The<\/a><\/p>\n

Figure 1. The output of the kubectl get nodes<\/i> command<\/p>\n

Let’s start with the recommended worker nodes configuration files’ ownership and permissions. These are the main files you should monitor for any changes on your Workers, according to the CIS Kubernetes Benchmark v1.5.1:<\/p>\n

\n\n\n\n\n\n\n\n\n
\n

File or directory<\/b><\/p>\n<\/td>\n

\n

Recommended ownership<\/b><\/p>\n<\/td>\n

\n

Recommended permission (or more restrictive)<\/b><\/p>\n<\/td>\n<\/tr>\n

\n

\/etc\/systemd\/system\/kubelet.service.d\/10-kubeadm.conf<\/span><\/p>\n<\/td>\n

\n

root:root<\/p>\n<\/td>\n

\n

644<\/p>\n<\/td>\n<\/tr>\n

\n

kube-proxy config file<\/p>\n<\/td>\n

\n

root:root<\/p>\n<\/td>\n

\n

644<\/p>\n<\/td>\n<\/tr>\n

\n

\/etc\/kubernetes\/kubelet.conf<\/p>\n<\/td>\n

\n

root:root<\/p>\n<\/td>\n

\n

644<\/p>\n<\/td>\n<\/tr>\n

\n

certificate authorities file<\/span><\/p>\n<\/td>\n

\n

root:root<\/p>\n<\/td>\n

\n

644<\/p>\n<\/td>\n<\/tr>\n

\n

\/var\/lib\/kubelet\/config.yaml<\/p>\n<\/td>\n

\n

root:root<\/p>\n<\/td>\n

\n

644<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Table 1. CIS Kubernetes Benchmark v1.5.1 recommendations<\/p>\n

The kubelet<\/h2>\n

The kubelet is the agent that runs on each node of your cluster and makes sure that all containers are running in a pod. It is also the agent that makes any configuration changes on the nodes. Although it is not shown on the main Kubernetes architecture diagram<\/a>, even the Master Node has a kubelet (and a kube-proxy) agent running in case you want to run other pods there, although it is not recommended.<\/p>\n

There are two main matters that you need to worry about with regard to your kubelet security settings: restricting the kubelet permissions and rotating the kubelet certificates. Restricting the kubelet permissions can prevent attackers from reading your kubelet credentials after they break out of the container and can do other things in your cluster.<\/p>\n

To check for these settings, you can run ps -ef | grep kube-apiserver<\/i> on any of your master node and see what is enabled by default, such as in the image here (Note that we are using the default settings of a kubeadm cluster setup with v1.18.5, so the results here might differ from yours).<\/p>\n

\"Command<\/a><\/p>\n

Figure 2. Command output of the ps -ef | grep kube-apiserver<\/i> showing the kubelet security settings<\/p>\n

RBAC for Kubelet<\/h3>\n

From the preceding image, it is observable that the –authorization-mode<\/i> parameter is already set with the \u201cNode, RBAC\u201d values, which are the ones that are enabled by default. However, this might be different on your cluster, so it is always important to double-check it.<\/p>\n

We have not mentioned admission controllers before, but they are pieces of code that can be called during the requests to the Kube API server after the authentication and authorization phase of the request for performing certain validations or changes (also known as mutations). Validation and mutation are two types of admission controllers in Kubernetes. Admission controllers can also be a combination of both. The only difference between them is that mutation controllers can modify the objects while validation cannot.<\/p>\n

To check which admission controllers are enabled on your cluster, check for the \u2013enable-admission-plugin<\/i> parameter on your kube-apiserver settings. You can do this by typing this command on your master node: ps -ef | grep kube-apiserver | grep enable-admission-plugins<\/i>. On our testing cluster, the only admission controller enabled by default is the NodeRestriction. This limits the objects that a kubelet can modify, so it limits what secrets can be read by which pods. In doing so, it does not allow the pods to read any secrets in the cluster, except the ones that are attached to their node or the secrets that are allowed to be seen.<\/p>\n

To learn more about admission controllers and which ones you can enable, please check the official Kubernetes documentation<\/a>.<\/p>\n

The kubelet agent authenticates the API server using certificates, which are valid for one year. You should also configure the kubelet certification rotation to generate a new key and request a new certificate from the Kubernetes API when the current certificate is close to expiring.<\/p>\n

You can find more information on certificate rotation here<\/a>.<\/p>\n

The pods<\/h3>\n

After protecting the control plane, the worker nodes, and most of the components, it might seem that we\u2019re now safe \u2014 but this isn\u2019t so. As discussed in the first article<\/a>, we always try to apply a defense-in-depth approach to reduce our attack surface and make it harder for attackers to exploit our clusters.<\/p>\n

There are three main actions that you can take to ensure the basic level of security for your pods:<\/p>\n

    \n
  1. Limit resources<\/b>. The last thing you would want to happen on your cluster is for one of your pods to start consuming all the resources available on your node, leaving the other pods with performance issues and possibly causing a denial of service (DoS) on the node. Kubernetes has a solution for that, called ResourceQuotas. You can also define hard and soft limits on the PodSpec, but that becomes hard to manage quickly. Meanwhile, ResourceQuota is an object that allows you to set hard and soft resource limits in a namespace. To enable ResourceQuota, you need to have it set on the kube-apiserver flag \u2013enable-admission-plugins=ResourceQuota<\/i>. You can check your kube-apiserver settings using this command: ps \u2013ef | grep kube-apiserver<\/i>. Here\u2019s a sample of a ResourceQuota object:\n

    <\/a><\/p>\n

    Figure 3. Sample ResourceQuota object<\/p>\n

    Source: Kubernetes Documentation on Resource Quotas<\/a><\/p>\n<\/li>\n

  2. Create and apply a security context<\/b>. A security context allows you to define privilege and access control permissions for a pod or a container. Here are a few security controls that you should always use whenever possible:\n