GCP-Post-1-Kubernetes Monitoring using InfluxDB & Grafana in Google Cloud Platform

Spread the love

 

 

 

 

In this blog post i would like to share with you all on the proof of concept we did for kubernetes monitoring. We wanted to monitor our kubernetes environment for various custom metric parameters like cpu,memory,disk I/O etc at a node & pod level in a beautiful dashboard. We opted for the open source analytics platform Grafana which has inbuilt plugin to various data sources like influxDB , cloudwatch, stackdriver, datadog, appdynamics etc. Even though you can still do the kubernetes monitoring using stackdriver monitoring, grafana boasts a better UI visualization capability.

 

So the idea is simple , Heapster aggregates the metrics from the kubernetes nodes and pods , which will sink into InfluxDB’s database , which can be later used by grafana to visualize the metrics. Lets see how this was done in GCP.

 

Helm package manager was used to deploy the InfluxDB & grafana charts into this kubernetes environment. We had taken a sample app from kubernetes-engine-samples from google cloud platform github link, The link is provided below

GoogleCloudPlatform/kubernetes-engine-samples
Sample applications for Google Kubernetes Engine (GKE) - GoogleCloudPlatform/kubernetes-engine-samplesgithub.com

git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples

cd kubernetes-engine-samples/hello-app

export PROJECT_ID="$(gcloud config get-value project -q)"

docker build -t gcr.io/${PROJECT_ID}/hello-app:v1 .

docker images

gcloud auth configure-docker

docker push gcr.io/${PROJECT_ID}/hello-app:v1

 

Created a kubernetes cluster with 3 nodes and deployed the sampleapp with the hello-app container image, we had 3 pod replicas for testing purpose.

 

https://cdn-images-1.medium.com/max/800/1*eHy3s1mrzxvkZWntJF-HZg.jpeg

 

Exposed the sampleapp with a loadbalancer service,

https://cdn-images-1.medium.com/max/800/1*7eEiyVVlfE3q5tRZii20zQ.jpeg

 

Tested the connectivity

https://cdn-images-1.medium.com/max/800/1*cmlTlWrnQNoQCrCnabCLfw.jpeg

Helm

Helm is a package manager used for deploying applications in Kubernetes , all the related services, rolebindings, serviceaccounts , configmaps, secrets etc will be taken care when we deploy through helm, More information is available in the below link

 

Helm -The Kubernetes Package Manager
The Kubernetes Package Mana
gerhelm.sh

 

 

wget https://storage.googleapis.com/kubernetes-helm/helm-v2.9.1-linux-amd64.tar.gz

tar zxfv helm-v2.9.1-linux-amd64.tar.gz

gcloud container clusters get-credentials sample --zone us-central1-a --project​​ projectname

kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value account)

kubectl create serviceaccount tiller --namespace kube-system

kubectl create clusterrolebinding tiller-admin-binding --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

./helm init --service-account=tiller

./helm update

./helm version

 

If you want to initialize the helm with RBAC , you need to provide certificate and key details,

 

helm init \

— override ‘spec.template.spec.containers[0].command’=’{/tiller, — storage=secret}’ \

— tiller-tls \

— tiller-tls-verify \

— tiller-tls-cert=cert.pem \

— tiller-tls-key=key.pem \

— tls-ca-cert=ca.pem \

— service-account=accountname

 

Persistent Volumes

Before we went ahead with the installation of the influxdb and grafana, we created the GCE disks for persistence,

gcloud compute disks create influxdbdisk grafanadisk --zone us-cenrtral1-a --size=10gi

The yaml file for persistent volume & persistent volume claim creation is given in the below link,

 

rajathithan/kubernetesmonitoring
YAML for PV-PVC github.com

 

kubectl create -f pv-pvc.yaml

kubectl get pv

kubectl get pvc

https://cdn-images-1.medium.com/max/800/1*7S1BOr6PI30VQu-dN09ktQ.jpeg

Under Kubernetes storage , you can view these new volumes.

 

InfluxDB

Next we proceeded with the installation of InfluxDB, using helm package manager. We had edited the default values.yaml file to given the persistent volume claim

 

InfluxDB helm/charts
Curated applications for Kubernetes. Contribute to helm/charts development by creating an account on GitHub.github.com

persistence:
 ​​​​ enabled: true
 ​​​​ useExisting: true
 ​​​​ name: influxdbclaim
 ​​​​ storageClass: ""
 ​​​​ accessMode: ReadWriteMany
 ​​​​ size: 2Gi

rajathithan/kubernetesmonitoring
InfluxDBgithub.com

./helm install --name sampledb -f influxdbvalues.yaml stable/influxdb

All the related service , configmap and workload will be deployed automatically.

 

https://cdn-images-1.medium.com/max/800/1*eFcr6trDqNU-pSElDP_JHA.jpeg

 

InfluxDB can be accessed via port 8086 on the following DNS name from within your cluster:

http://sampledb-influxdb.default:8086

 

You can easily connect to the remote instance with your local influx cli. To forward the API port to localhost:8086 run the following:

kubectl port-forward — namespace default $(kubectl get pods — namespace default -l app=sampledb-influxdb -o jsonpath=’{ .items[0].metadata.name }’) 8086:8086

 

You can also connect to the influx cli from inside the container. To open a shell session in the InfluxDB pod run the following:

 

kubectl exec -i -t — namespace default $(kubectl get pods — namespace default -l app=sampledb-influxdb -o jsonpath=’{.items[0].metadata.name}’) /bin/sh

 

To tail the logs for the InfluxDB pod run the following:

 

kubectl logs -f — namespace default $(kubectl get pods — namespace default -l app=sampledb-influxdb -o jsonpath=’{ .items[0].metadata.name }’)

 

Grafana

Next we installed grafana using helm package manager, run the below command to install grafana, the default values.yaml was edited to enable the persistence volume claim and to set the userid and password.

 

persistence:
 ​​​​ enabled: true
 ​​​​ storageClassName: ""
 ​​​​ accessModes:
 ​​ ​​ ​​​​ - ReadWriteMany
 ​​​​ size: 2Gi
 ​​​​ annotations: {}
 ​​​​ subPath: ""
 ​​​​ existingClaim: grafanaclaim

adminUser: admin
adminPassword: password

 

In the newer grafana versions >5.1 , the Docker container are owned by id/gid 472 (the grafana user/group), the container crashes when persistence is enabled , so we have modified the security context , to run the process as a root user.

 

securityContext:
 ​​​​ runAsUser: 0

grafanavalues.yaml file is available in the below

rajathithan/kubernetesmonitoring
Grafanagithub.com

./helm install --name samplegrafana -f grafanavalues.yaml stable/grafana

https://cdn-images-1.medium.com/max/800/1*Q_QadfhcIvvHyOU1suLxAw.jpeg

 

You can get your ‘admin’ user password by running:

kubectl get secret — namespace default samplegrafana -o jsonpath=”{.data.admin-password}” | base64 — decode ; echo

 

The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

samplegrafana.default.svc.cluster.local

you can get the Grafana URL to visit by running these commands in the same shell:

 

export POD_NAME=$(kubectl get pods — namespace default -l “app=grafana,release=samplegrafana” -o jsonpath=”{.items[0].metadata.name}”)
kubectl — namespace default port-forward $POD_NAME 3000

 

Heapster

Both the grafana and influxdb deployments should be up and running before making the changes to the heapster, the heapster workload will be running under the kube-system namespace, export the heapster deployment’s yaml file and edit it to sink the details to the influxdb database.

 

kubectl get deployments -n kube-system
kubectl get deployment heapster-v1.5.3 -n kube-system -o yaml > heapster.yaml

 

Right after the spec , make the below changes

spec:

containers:

- command:

- /heapster

- --source=kubernetes.summary_api:""

- --sink=influxdb:http://sampledb-influxdb.default:8086

name: heapster

 

apply the changes to the heapster deployment

kubectl apply -f heapster.yaml

 

Verified the data in influxDB by connecting to the container, a new database called k8s will be created in influxdb

https://cdn-images-1.medium.com/max/800/1*OTQILUzGlFwn8Ta8ImRCSA.png

 

Used port-forwarding to access the grafana console.

https://cdn-images-1.medium.com/max/800/1*78pXfPybrpjf0PrQoB3HEQ.jpeg

Logged in with the admin credentials, ( you might have to use a strong password )

 

https://cdn-images-1.medium.com/max/800/1*m5neJ3gvK3dvamYIvWYiGA.jpeg

 

Add the influxDB data source,

https://cdn-images-1.medium.com/max/800/1*SLrqkMvb_geH65MsGC0LEQ.jpeg

configure the data source and test it

https://cdn-images-1.medium.com/max/800/1*izMmMlSgcuCntwudsbt96w.png

 

Add new Dashboard and add a graph, under the metrics section select the query on metrics name , container name & pod name. Give the title name under the general section and finally save the dashboard.

https://cdn-images-1.medium.com/max/800/1*hpcCCpPeUpWz0L99bAz0yw.jpeg

https://cdn-images-1.medium.com/max/800/1*5_czyRZbxDWMmoQTKzdYvw.jpeg

 

So now to check , whether the data is persistent on the grafana and influxdb , delete the pods , you can see that the data on the influxdb and grafana are still available on the new pod that was created . This​​ concludes the setup , Thanks for reading through this post, Hope you find this useful.

 

Thanks

 

Related posts

Leave a Comment