security

Security Primitives:

Controlling access to kube-apiserver

Authorization:

RBAC Authorization ABAC Authorization Node Authorization Webhook Mode

All communication with the cluster, between the various components such as ETCD cluster Kube controller-manager scheduler Kube-apiserver as well as those working on nodes kubelet and kube-proxy work using TLS encryption.

By default all applications, i.e, PODs can access all other PODs within the same cluster.

Different users that need access to the cluster: Admins Developers Application end users(Managed by application itself) Bots

Kubernetes can create service accounts

Users are managed by the kube-apiserver by authenticate either through kubectl or APIS like curl

Different mechanism:

Static Password Files Static Token File Certificates Identity Services(Third Party)

TLS TLS certificates ensure that the communication between the server and user is encrypted and the identity of the server is true.

Asymmetric encryption: Instead of using a single key to encrypt and decrypt data, asymmetric encryption uses a pair of keys, a private and a public key.

Genarally, public keys do not have term key in their name: server.crt server.pem client.crt client.pem

Private keys have key, keyword in their name:

server.key server-key.pem client.key client-key.pem

There are three types of certificate of interest Root certificate for Certificate Authority Server Certificates Client Certificates

Kubernetes server which requires server certificates

Kube-apiserver: apiserver.crt, apiserver.key Etcd-server: etcdserver.crt, etcdserver.key Kubelet server: kubeletcrt, kubelet.key

Kubernetes Client certificates

Admin: admin.crt, admin.key Kube-scheduler: scheduler.crt, scheduler.key Kube-controller-manager: controller-manager.crt, controller-manager.key Kube-proxy: kube-proxy.crt, kube-proxy.key

All four communicate with the kube-apiserver

Etcd is also a client for kube-apiserver and it can either use apiserver key pairs or a sepearte client key pairs can be used.

apiserver-etc-client.crt, apiserver-etc-client.key

Kube-apiserver also needs to serve to kubelet server, it can either use apiserver key pairs or new key pairs can be generated

apiserver-kubelet-client.key, apiserver-kubelet-client.crt

To generate certificate, following tools can be used

EasyRSA OpenSSL CFSSL

CA: //Generate key //openssl genrsa -out ca.key 2048

//Certificate signing request // openssl req -new -key ca.key -subj "/CN=KUBERNETES-CA" -out cas.csr

//Sign certificates //openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt

Admin user //Generate key //openssl genrsa -out admin.key 2048

//Certificate signing request

//openssl req -new -key admin.key -sunj "/CN=kube-admin" -out admin.csr

//mentioning group details for the admin previledges //openssl req -new -key admin.key -sunj "/CN=kube-admin/O=system:masters" -out admin.csr

//Sign certificates //openssl x509 -req -in admin.csr -signkey admin.key -out admin.crt

Kube Scheduler

//the name of the certificate should be prefixed with system //SYSTEM: KUBE-SCHEDULER

Kube controller manager

//The name of the certificate should be prefixed with system //SYSTEM: KUBE-CONTROLLER-MANAGER

Kubeproxy

//Kube-proxy name remains as it is //KUBE-PROXY

//Enter the details of these certificate into kube-config file

Serverside Certificates:

ETCD Server //name: ETCD-SERVER

//In a high availability enviroment when multiple ETCD servers are available, each should generate additional peer certificate. //name:ETCD-PEER1, ETCDPEER2 //specify them in etcd configuration etcd.yaml

Kube-api server:

//name: KUBE-API SERVER Kubeapi server certificate is also called kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local

//openssl genrsa -out apiserver.key 2048

//openssl req -new -key apiserver.key -subj "/CN=kube-apiserver" -out apiserver.csr

//If alternate name, make this file and pass to csr gen command

//openssl.cnf
[req]
req_extension = v3_req
distinguished_name = req_distinguished_name
[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation,
sunjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.defualt.svc
DNA.4 = kubernetes.default.svc.cluster.local
IP.1 = 10.96.0.1
IP.2 = 172.17.0.87

//openssl req -new -key apiserver.key -subj "/CN=kube-apiserver" -out apiserver.csr -config openssl.cnf

//openssl x509 -req -in apiserver.csr =CA ca.crt =CAkey ca.key -CAcreateserial -out apiserver.crt -extension v3_req -extfile openssl.cnf -days 1000

Kubectl Nodes (server certrs)

The certificate pair will be named after the node names, node01, node02 etc.

Kubectl Nodes (client cert)

The certificate pair are named as: system:node:nodename,

It should be added to the node group as well SYSTEM:NODES

decrypt certs openssl x509 -in /etc/kubernetes/pki/ca.crt -text

Troubleshooting a TLS service //crictl ps -a | grep kube-apiserver //crictl logs –tail=2 e505f6123373e

Certificate API

//Create CertificateSigningRequest

////gen a key //openssl genrsa -out jane.key 2048 ////admin creates a csr //openssl -req -new -key jane.key -subj "/CN=jane" -out jane.csr ////Admin creates certficatesigningrequest object using yaml //

apiVersion: certificates.k8s.io/v1beta1
king: CertificateSigningRequest
metadata:
    name: jane
spec:
    groups:
    - system:authenticated
    usages:
    - digital signature
    - key encipherment
    - server auth
    requests:
        //out put of //cat jane.csr | base64 | tr -d "\n"

//kubectl get csr //kubectl certificate approve jane //kubectl get csr jane -o yaml //decode it and share with the user //echo "ecoded text" | base64 –decode

The controller manager takes care of the certificate related operations.

The controller manager has controllers like CSR-APPROVING, CSR-SIGNING

//Review Request

//Approve Request

//Share Certs to users //kubectl approve csr agent-smith //kubectl deny csr agent-smith //kubectl delete csr agent-smith

KubeConfig

//$HOME/.kube/config

apiVersion: v1
kind: Config

//default context
current-context: my-kube-admin@my-kube-playground
clusters:
- name: my-kube-playgroud
  cluster:
    certificate-authority: ca.crt
    server: https://my-kube-playground:6443

contexts:
- name: my-kube-admin@my-kube-playground
  context:
    cluster: my-kube-playground
    user: my-kube-admin

users:
- name: my-kube-admin
  user: 
    client-certificate: admin.crt
    client: key

//No need to create object through this file //to view the config file /kubectl config view // view a kube config file that is not present in the default location //kubectl config view –kubeconfig=my-custom-config

//change the current context //kubectl config use-context prod-user@production

//namespace

context:
    cluster: production
    user: admin
    namespace: finance

//get current context //kubectl config current-context –kubeconfig my-kube-config

//set context kubectl config –kubeconfig=/root/my-kube-config use-context research

API Groups

//core api //v1 //namespace,events,bindings,configmap //pods, endpoint, PV, secrets //rc, nodes, PVC, services

//named apis //apps->/v1-> deployments/replicasets/statefulsets //extensions //networking.k8s.io->/v1->networkpolicies //storage.k8s.io //authentication.k8s.io //certificates.k8s.io->/v1/certificatesigningrequests

//actions that can be performed on above resources: //list, get, create, delete, update, watch

//list available api groups //curl http://localhost:6443 -k

//list supported resource groups //curl http://localhost:6443/apis -k | grep "name"

//several apis will require authentication //curl http;//localhost:6443 -k –key admin.key –cert admin.crt –cacert ca.crt

//alternatively, start a kubectl-proxy client //kubectl proxy

Node Authorization

//Any request coming with a request "system:node" should be handled by node authorization. This includes tasks like

Raed: Service Endpoint Nodes Pods Write: Node Status Pod Status Events

ABAC Authorization

The policy is defined in JSON format to define access. The attribute-based access control are difficlut to manage.

RBAC Authorization

//role definition file
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
    name: developer
rules:
- apigroups: [""]
  resources: ["pods"]
  verbs: ["list", "get","update","create","delete"]
  //control acess to few resources
  reaourceName: ["blue", "red"]

//kubectl create -f developer-role.yaml

//role binding definition
apiVersion: rbac.authorization.k8s.io/v1
kind:RoleBinding
metadata:
    name: devuser-developer-binding
subjects: //user details
- kind: User
  name: dev-user
  apiGroup: rbac.authorization.k8s.io
roleRef: //role details
- kind: Role
  name: developer
  apiGroup: rbac.authorization.k8s.io

//kubectl create -f devuser-developer-binding.yaml

//view roles //kubectl get roles

//view role bindings //kubectl get rolebindings

//details of rbac //kubectl describe role developer

//details of rolebindings //kubectl describle rolebindings devuser-developer-binding

//check access //kubectl auth can-i create deployments //kubectl auth can-i delete node

//check access for a user //kubectl auth can-i create deployments –as dev-user //kubectl auth can-i delete node –as dev-user

//check access for a user in a namspace //kubectl auth can-i create deployments –as dev-user –namespace test //kubectl auth can-i delete node –as dev-user –namespace test

//comandline way to create role //kubectl create role developer –namespace=default –verb=list,create,delete –resource=pods //commanline way to create binding //kubectl create role developer –namespace=default –verb=list,create,delete –resource=pods

Webhook

Open Polci Agent is a third party tool that helps control and authorization

Always allow, always deny

Can be set in kube-apiserver config file // –authirization-mode=AlwayAllow

// –authirization-mode=Node,RBAC, Webhook //Node mode follows first and webhook at last.

Cluster Roles:

These roles are cluster wide scoped Ex. nodes, PV, clusterroles, clusterrolebindings, certificatesigningrequest, namespace

// cluster-role-definition.yaml
apiVersion rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
    name: cluster-administrator
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list","create","get","delete"]

//cluster-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
    name: cluster-admin-role-binding
subjects:
- kind: User
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
roleRef:
    kind: ClusterRole
    name: cluster-administrator
    apiGroup: rbac.authorization.k8s.io

By default cluster rules are cluster wide and not part of any namespace kubectl api-resources –namespaced=false

Service Accounts

A service account is used by the machine.

//kubectl create serviceaccount dashboard-sa //kubectl create token dashboard-sa //kubectl get serviceaccount //kubectl describe serviceaccount dashboard-sa //the token described above should be added to the application which needs access. //kubectl describe secret dashboard-sa- token-kbbdm

Image security

Private registry //kubectl create secret docker-registry regcred –docker-server=private-registry.io –docker-username=registry –docker-password=password –docker-email=my@email.com

apiVersion: v1
kind Pod
metadata:
    name:
spec:
    conatainers:
    - name: nginx
      image: private-repository.io/apps/internal-app
    imagePullSecrets:
    - name: regcred

Security in Docker

//docker run ubuntu sleep 3600 //ps aux //we see sleep process with process id 1 //use –cap-add command to add priveledge to root user inside a container using coker //docker run –cap-add CHOWN ubuntu //to remove previliedge use: //docker run –cap-drop CHOWN ubuntu

//Kubernetes Security context

//Security capability at pod level
apiversion: v1
kind: Pod
metadata:
    name: web-pod
spec:
    securityContext:
        runAsUser: 1000
    
    containers:
        - name: ubuntu
          image: ubuntu
          command: ["sleep", 3600]

//Security capability at container level
apiversion: v1
kind: Pod
metadata:
    name: web-pod
spec:  
    containers:
        - name: ubuntu
          image: ubuntu
          command: ["sleep", 3600]
              securityContext:
                runAsUser: 1000
                capabilities:
                    add: ["CHOWN"]

//kubectl exec ubuntu-sleeper – whoami //kubectl exec pod-name – linux-command

Network Policies

Ingress: Incoming traffic Egress: Outgoing traffic //Direction in which the traffic originated //Once an ingress rule is applied, automatically egress response can be sent. No need to add a new rule for responses.

apiVersion networking/kubernetes.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-policy
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  ingress:
  - from:
    - poSelector:
        matchLabels:
          name:api-pod
      namespaceSelector:
        matchLabels:
          name: prod
    //ipBlock selector to allow ingress from outside ip
    - ipBloack:
      cidr: 192.168.0.12/32
    ports:
    - protocol : TCP
      port: 3306

//egress

egress:
- to: 
  - ipBlock:
      cidr:192.168.10.25/32
  ports:
  - protocol: TCP
    port:80

Note: for quick switching between context and shorter commands, use https://github.com/ahmetb/kubectx