Protect Kubernetes Clusters with Admission Controller Solution Engineer

Security can be a tricky prospect with containers. Unlike virtual machines, containers draw resources from the same shared pool and operating system, meaning a security vulnerability on a single node can easily spiral into a cluster-wide disaster.

Runtime security measures should be just one component of your in-depth and layered security defense. Catching a potential security issues at runtime means it managed to slip through the cracks during the development and build stages, costing you resources.

This prompts the question: what could I have done to preemptively prevent this? The answer: screen containers before they even initialize. That is what a Kubernetes admission controller does. Let’s explore how an admission controller works and what kind of threats it guards against.

Kubernetes Admission Controller: A Gatekeeper to Your Kubernetes Cluster

The workflow of a Kubernetes cluster is straightforward: authenticated requests route to the Kubernetes API server, which deploys the image and assigns resources to the cluster based on its needs.

This is where admission controller comes into play. Before the container is actually initialized and added as a pod, the controller analyzes the request to ensure the image is safe for deployment.

An admission controller might consider various parameters. Some reject requests from unrecognized namespaces, while others prevent containers from running as root and obtaining privileged access. Others scan the images themselves, ensuring their integrity before approving deployment.

Most controllers only require you to enable them to start working. Some are even enabled by default, regulating container requests across the cluster without any input.

The command to activate a controller is quite simple:

kube-apiserver –enable-admission-plugins=%plugin_name%

Where %plugin_name% is replaced with an actual plugin like CertificateSigning or AlwaysPullImages.

The admission controller immediately takes effect throughout the cluster, rejecting admission requests that don’t meet the criteria in their definitions.

Disabling a controller is just as easy:

kube-apiserver –disable-admission-plugins=%plugin_name%

Now that we’ve seen how to enable and disable controllers, let’s look at the different types we can implement.

Admission controllers, like most Kubernetes functions, are hardcoded into the Kubernetes API server itself. Most plugins like EventRateLimit or NamespaceExists apply to specific scenarios and cannot be extended in any way.

The more useful controllers are those you can modify to suit your needs. There are two such classes of plugins: policies and webhooks.

Pod Security Policies

A Pod Security Policy (PSP) is a slightly different kind of admission controller. By default, enabling it blocks pod creation entirely. To enable pod deployments again, you need to describe and authorize a pod security policy.

A policy is nothing more than a set of constraints that a pod must comply with before deploying on the cluster. A host of fields control various technical aspects of a pod’s functions, all of which you can customize individually.

In practice, a PodSecurityPolicy object is just a .yaml configuration file detailing the default values the policy expects in each of the fields. The one below, for example, simply blocks the creation of privileged pods:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
   name: block_privileged
spec:
   privileged: false

seLinux:
  rule: RunAsAny
supplementalGroups:
  rule: RunAsAny
runAsUser:
  rule: RunAsAny
fsGroup:
  rule: RunAsAny
volumes:
– ‘*’

If you authorize this policy (via role-based access control [RBAC]) and enable the PSP admission controller, attempting to create a privileged pod fails. You can define complex policies in this way, controlling exactly how pods deploy throughout your cluster.

Dynamic Admission Controllers with Webhooks

ValidatingAdmissionWebhook, MutatingAdmissionWebhook, and ImagePolicyWebhook are the only three admission controllers that can be extended with custom logic. You do this using webhooks.

Webhooks are simply representational state transfer (REST) endpoints, supplied by a service running separately from the Kubernetes API Server. Usually, you deploy this webhook on the cluster itself, though it is certainly possible (though not recommended) to deploy it on another system.

The admission request passes onto these webhooks through an HTTP callback, which can then apply the webhook’s parameters to process the request and either accept or deny it, or, with a mutating admission webhook, perhaps even modify the request.

Unlike the other plugins, webhook-based admission controllers are completely flexible. You can code them in any language or framework, provided they interact with the Kubernetes API through a pre-defined format.

For example, this is how a power-on self-test (POST) request sent to a webhook looks:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
…
webhooks:
– name: security_test.webhooks.com
  admissionReviewVersions: [“v1”, “v1beta1”]
  …

For some requests, there might be additional data in JavaScript Object Notation (JSON) format, included under the request field. The response is much less verbose:

{
  “apiVersion”: “admission.k8s.io/v1”,
  “kind”: “AdmissionReview”,
   “response”: {
      “uid”: “<value from request.uid>”,
      “allowed”: false
 }
}

How do the three types of webhook plugins differ?

Validating Admission Webhook

Most admission controllers fall under the standard validating admission webhook category, which can only accept or decline a request. Like other webhooks, it receives a POST request from the Kubernetes API, including all the relevant container parameters.

The validating admission webhook can link to multiple registered webhooks working in parallel. This means the admission requests face an all-or-nothing scenario, where rejection from even one webhook scuttles the request.

This plugin is the last line of defense, enforcing rules that are not meant to be bypassed in any situation.

Mutating Admission Webhook

Sometimes, an admission request only deviates slightly from the expected standard. Maybe it requests too many resources, or perhaps it just needs different parameters to be accepted.

A mutating admission webhook is the answer to such quandaries. Unlike validating the admission controller, it can modify the requests, bringing them in line with the norm. This modification takes the form of a standard JSON patch, which is applied to the admission request before deployment.

This is how a response from a mutating admission webhook appears:

{
   “apiVersion”: “admission.k8s.io/v1”,
   “kind”: “AdmissionReview”,
   “response”: {
      “uid”: “<value from request.uid>”,
      “allowed”: true,
      “patchType”: “JSONPatch”,
      “patch”: “W3sib3AiOiAiYWRkIiwgInBhdGgiOiAiL3NwZWMvcmVwbGljYXMiLCAidmFsdWUiOiAzfV0=”
  }
}

As the webhook returns a mutated request, multiple mutating admission controllers cannot run in parallel. Instead, the callbacks are chained back-to-back, with the final output reflecting all the changes.

In standard practice, mutating admission controllers are applied first, followed by a callback to validating admission controllers. This ensures the maximum number of requests is approved while ensuring security.

Image Policy Webhook

So far, the plugins have focused on analyzing the technical parameters of the request, like the resources requested or the namespace invoked. However, what about the images?

Vetting the container image with the image policy webhook ascertains the risk the image poses to the cluster. The webhook receives the container images as an ImageReview object, containing the data in a JSON serialized format. A request looks like this:

{
  “apiVersion”:”imagepolicy.k8s.io/v1alpha1″,
   “kind”:”ImageReview”,
   “spec”:{
    “containers”:[
      {
       “image”:”repo/untested:v3″
      }
   ],
   “annotations”:{},
   “namespace”:”production”
  }
}

The annotations field can enclose additional information by users. For example, you may configure your image scanning algorithm to bypass the scan if the string break-glass is annotated.

While the webhook determines the implementation specifics, the response must follow a standard format:

{
  “apiVersion”: “imagepolicy.k8s.io/v1alpha1”,
  “kind”: “ImageReview”,
  “status”: {
    “allowed”: true
 }
}

The flag becomes false for request denial, along with an optional reason field to include a message to the user.

Now all that is great, but how do you go about implementing your own image scanner? You don’t.

While you certainly can pour time and resources into implementing a robust image policy webhook backend, we wouldn’t recommend it. Breaking apart a container image to spot vulnerabilities like API keys or exposed secrets isn’t easy. Locating malware is even more difficult. Instead of reinventing the wheel, take advantage of a dedicated image scanning tool.

Trend Micro Cloud One™ – Container Security, for example, is based entirely around comprehensive image scanning. As one of seven security solutions that make up the Trend Micro Cloud One™ platform, it enables you to set up image scanning policies that only allow secure containers to deploy after detailed scans seek out any vulnerabilities.

The dashboard flags security vulnerabilities before runtime, giving developers a chance to fix these issues. It also provides you with visibility into what your team is sending for deployment, stopping rogue Ops practices, such as using public images in a monitored cluster.

Container Security only allows images to be deployed from approved registries and specifically tagged images. The solution lets you have it all thanks to built-in runtime security built that prevents common attacks such as a remote code execution and illegal file access. Also, the runtime component can provide continuous monitoring on deployed containers that looks for container drift and policy violations. Once detected, it can isolate the container from the rest of the network or spin down/terminate the container.

Lastly, you can seamlessly integrate the solution with your existing Kubernetes cluster through an image policy webhook, leaving the rest of your framework untouched.

Admission Controllers in Microsoft Azure™ and Amazon EKS

For the most part, admission controllers work just as you would expect on both the Azure and Amazon Elastic Kubernetes Service (EKS) platforms. The only area where they differ is policies.

Azure Kubernetes Service (AKS) completely revamped the PSP system, creating a graphical user interface (GUI) that performs the same functions as a PSP. Azure Policy enables you to apply policies on your cluster (or parts of it) without messing around with code or configuration files. You can choose a policy from a large list of built-in policy definitions or define your own in the JSON format.

There is no comparable feature in Amazon EKS. Sure, you can use Open Policy Agent (OPA) to easily define new policies, but you can do that with vanilla Kubernetes too.

On both the Azure and Amazon EKS platforms, you need webhooks to implement a robust image scanning-based policy.

Next Steps

Admission controllers are an indispensable tool in the arsenal of any Kubernetes administrator. Applying cluster-wide rules on each and every deployed container helps secure your operations.

Now that you know more about the Kubernetes admission controller, take a multi-pronged approach. Use standard controllers and PSPs to regulate pod resource use and security access, and webhooks to analyze container images before deployment. A specialized image scanning tool, like Container Security, identifies security vulnerabilities and eliminates false positives.

Properly configured admission controllers go a long way in securing your Kubernetes cluster against security threats and resource misallocation. Get your 30-day free trial of Container Security and take control of your Kubernetes security today.

Read More HERE