This time we have a slightly lighter reading.

Now you may have a k8s cluster set up, and you use

kubectl get pods -o wide -A
kubectl get nodes -o wide -A

A lot to monitor what's going on with your services and the health of the cluster.

But you don't know how the servers are running, how much CPU it is consuming currently, and how much memory is left. Bear in mind that Kubernetes cluster do not have swap...

And you may want to simplify it a bit, using a web dashboard to oversee everything, and may be administer a bit there as well.

Kubernetes Dashboard and metric-server are the 2 things you may want to set up quickly.

What is Kubernetes Dashboard?

Kubernetes Dashboard is a web-based GUI that allows you to:

  • Understand your clusters' current construction and health information
  • Review logs from various pods
  • Administer cluster configuration
  • Execute scripted commands in pods if the containers support it.

However, it does not have capability to:

  • Perform long-term trend analysis using time-series data.

It relies on metric-server to gather (scrap) information from all the nodes and pods before they are presented in a user friendly format.

What is metric-server?

The architects and developers for Kubernetes soon realise that they need a reliable way to monitor a k8s cluster. In the beginning, heapster was developed as a stop-gap solution until after a while, the official metrics API was defined.

metric-server implemnts the metrics API to collect and provide cluster-wide metrics to consumers who wants to monitor the cluster status. Apart from monitoring, you also cannot use scaling features in Kubernetes unless you have a service that implements the metrics API. Thus if you want to use the various autoscalers (like Horizontal Pod Autoscaler, which creates additional pods to serve API requests when the original running pod is too busy, based on CPU / memory utilization), you have to install metric-server (or other similar service that implements metrics API).

However, metric-server does not retain historical data. This is also why Kubernetes Dashboard cannot offer such information to you to conduct high level trend analysis for long term capacity planning.

Installing Kubernetes Dashboard and metric-server

Installation of Kubernetes Dashboard and metric-server are really straightforward, even for my Raspberry Pi cluster. All container images are x86_64 and aarch64 compatible.

I'd suggest you refer to the official websites for the latest installation guides. Also, always use manifest file and kubectl apply -f to deploy the latest versions of the service. Sometimes helm charts are out-of-date. Initially, I installed Kubernetes Dashboard using helm only to realise that the installation was done in the default namespace and the guides in the official websites does not directly work with them, leading to quite some investigation time.

Kubernetes Dashboard requires rather extensive permissions in order to work because it can be used to administer the cluster. Hence, it needs to be secured properly. Although it provides a web-based GUI, it is by default not exposed. The service was defined as a ClusterIP and we should not change it.

NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
dashboard-metrics-scraper   ClusterIP   10.21.133.191   <none>        8000/TCP   5h35m
kubernetes-dashboard        ClusterIP   10.26.48.179    <none>        443/TCP    5h35m
Do not change the service from ClusterIP to anything else!

There are multiple ways to expose the services to your administrator. I have developed a script that can automate and lightly secure access to the dashboard. It will help you create a SSH tunnel to access the dashboard.

On the other hand, metric-server simply work by itself. There is no need to do anything with it.

Accessing the Web UI for Kubernetes Dashboard, via SSH tunnel

The Web UI of Kubernetes Cluster is very powerful, so powerful that it can:

  • Define, remove any resources in your cluster
  • Start, stop any deployments.
  • Access the pods in your cluster, and gather sensitive information within it.

Hence, by default, it is only accessible via localhost in your master node(s), which means that you have to be able to access your cluster in order to use it. It makes perfect sense because if you can access your cluster's master node, then you can do anything as well.

So it is not advised to change this configuration in a permanent way. Instead, I have developed a script (in Github) that will:

  1. Create a temporary service account with random name for accessing the web UI.
  2. Assign a proper cluster role to this account.
  3. Generate an access token
  4. Start the kubernetes proxy to proxy the service to localhost.

With SSH tunnelling, you can then access the web UI accordingly. A example SSH tunnel command in Mac OS will look like this:

ssh -L 8001:localhost:8001 <master-node> "<path-to-script>/start.sh"

It will automatically establish the tunnel and start the script. Youu will get a very long access token from your terminal window. Copy & paste it into your web browser once the following page is launched:

http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
Enter your access token (from terminal window) to login

Enter the token and you will be logged into the dashboard.

When you are done, enter "STOP" in the terminal window, it will:

  1. Stop the Kubernetes proxy
  2. Delete the temporary service account and the cluster role binding.

Using this method,  you will only be exposing the dashboard and create a authenticated user when you need it, which is safer.

Using the dashboard

Accessing information

Once login, you will be presented with some of the most important information in your cluster, such as the status of your deployments (services that are running).

Dashboard once login

You will be in the "default" namespace. I find it useful to switch to other namespaces to see the health of a few important things, such as cert-manager, where the SSL certificate renewals are done, or ingress-nginx, to see if the Ingress resource is having problem.

You can browse around using the hyperlinks and review the deployment and pod health. One very useful feature is that you can review the logs generated from the pods. To do so, select a pod (not a deployment), and select "View logs" in top right corner.

View in a pod, you can see the CPU / memory usage. On the top right corner there are a few buttons, one of them is "View logs"
Viewing logs

Administration

Almost all configurations could be changed using the dashboard. The example belows shows the configuration editor for a service.

This homebridge service can be edited through the top right corner's pen button.

You can also change the auto scaling setting for your deployment, which comes in really handy.

Scaling is done by clicking the dot-and-line button on the top right corner.

Also, it is extremely useful to be able to take a look at the configuration files actually constructed inside the pod. This could be done by using the "Exec into pod" button on the top right corner.

Run a shell inside a pod.

Other features

As you can see in some of the above screenshots, you can also take a glimpse of the CPU / memory utilisations by pods or by nodes, or delete resources easily using the web UI.

Using an Ingress for Kubernetes Dashboard

Alternatively, you can also setup an Ingress for Kubernetes Dashboard using the sample below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: kubernetes-dashboard
  name: dashboard-example-com
  annotations:
    kubernetes.io/ingress.class: "nginx"    
# Backend is TLS enabled
    nginx.ingress.kubernetes.io/backend-protocol: "https"
# The following line ensures that nginx will do redirection correctly
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/whitelist-source-range: <IP range to be white listed, comma separated>
# This line ensures that we use the correct certificate issuer
    cert-manager.io/cluster-issuer: "letsencrypt-prod"

spec:
  tls:
  - hosts:
    - dashboard.example.com
    secretName: dashboard-example-com-tls
  rules:
  - host: dashboard.example.com
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: kubernetes-dashboard
            port:
              number: 8443
Sample manifest for defining Ingress for Kubernetes Dashboard.

I have also provided some shell scripts to help you create a service account and generate a permanent token for accessing the dashboard via Ingress in my Github repository.

Wrapping it up

The combination of Kubernetes Dashboard and metric-server provides a beginner's level administration tool that can allow you to easily monitor and administer the cluster. It is easy to setup, runs automatically and use minimum resources. If you want to start a small cluster, it is definitely worth a try before moving to something more complex like prometheus.

Readouts