GCP-Post-1-Kubernetes Monitoring using InfluxDB & Grafana in Google Cloud Platform

Spread the love





In this blog post i would like to share with you all on the proof of concept we did for kubernetes monitoring. We wanted to monitor our kubernetes environment for various custom metric parameters like cpu,memory,disk I/O etc at a node & pod level in a beautiful dashboard. We opted for the open source analytics platform Grafana which has inbuilt plugin to various data sources like influxDB , cloudwatch, stackdriver, datadog, appdynamics etc. Even though you can still do the kubernetes monitoring using stackdriver monitoring, grafana boasts a better UI visualization capability.


So the idea is simple , Heapster aggregates the metrics from the kubernetes nodes and pods , which will sink into InfluxDB’s database , which can be later used by grafana to visualize the metrics. Lets see how this was done in GCP.


Helm package manager was used to deploy the InfluxDB & grafana charts into this kubernetes environment. We had taken a sample app from kubernetes-engine-samples from google cloud platform github link, The link is provided below

Sample applications for Google Kubernetes Engine (GKE) - GoogleCloudPlatform/kubernetes-engine-samplesgithub.com

git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples

cd kubernetes-engine-samples/hello-app

export PROJECT_ID="$(gcloud config get-value project -q)"

docker build -t gcr.io/${PROJECT_ID}/hello-app:v1 .

docker images

gcloud auth configure-docker

docker push gcr.io/${PROJECT_ID}/hello-app:v1


Created a kubernetes cluster with 3 nodes and deployed the sampleapp with the hello-app container image, we had 3 pod replicas for testing purpose.




Exposed the sampleapp with a loadbalancer service,



Tested the connectivity



Helm is a package manager used for deploying applications in Kubernetes , all the related services, rolebindings, serviceaccounts , configmaps, secrets etc will be taken care when we deploy through helm, More information is available in the below link


Helm -The Kubernetes Package Manager
The Kubernetes Package Mana



wget https://storage.googleapis.com/kubernetes-helm/helm-v2.9.1-linux-amd64.tar.gz

tar zxfv helm-v2.9.1-linux-amd64.tar.gz

gcloud container clusters get-credentials sample --zone us-central1-a --project​​ projectname

kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value account)

kubectl create serviceaccount tiller --namespace kube-system

kubectl create clusterrolebinding tiller-admin-binding --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

./helm init --service-account=tiller

./helm update

./helm version


If you want to initialize the helm with RBAC , you need to provide certificate and key details,


helm init \

— override ‘spec.template.spec.containers[0].command’=’{/tiller, — storage=secret}’ \

— tiller-tls \

— tiller-tls-verify \

— tiller-tls-cert=cert.pem \

— tiller-tls-key=key.pem \

— tls-ca-cert=ca.pem \

— service-account=accountname


Persistent Volumes

Before we went ahead with the installation of the influxdb and grafana, we created the GCE disks for persistence,

gcloud compute disks create influxdbdisk grafanadisk --zone us-cenrtral1-a --size=10gi

The yaml file for persistent volume & persistent volume claim creation is given in the below link,


YAML for PV-PVC github.com


kubectl create -f pv-pvc.yaml

kubectl get pv

kubectl get pvc


Under Kubernetes storage , you can view these new volumes.



Next we proceeded with the installation of InfluxDB, using helm package manager. We had edited the default values.yaml file to given the persistent volume claim


InfluxDB helm/charts
Curated applications for Kubernetes. Contribute to helm/charts development by creating an account on GitHub.github.com

 ​​​​ enabled: true
 ​​​​ useExisting: true
 ​​​​ name: influxdbclaim
 ​​​​ storageClass: ""
 ​​​​ accessMode: ReadWriteMany
 ​​​​ size: 2Gi


./helm install --name sampledb -f influxdbvalues.yaml stable/influxdb

All the related service , configmap and workload will be deployed automatically.




InfluxDB can be accessed via port 8086 on the following DNS name from within your cluster:



You can easily connect to the remote instance with your local influx cli. To forward the API port to localhost:8086 run the following:

kubectl port-forward — namespace default $(kubectl get pods — namespace default -l app=sampledb-influxdb -o jsonpath=’{ .items[0].metadata.name }’) 8086:8086


You can also connect to the influx cli from inside the container. To open a shell session in the InfluxDB pod run the following:


kubectl exec -i -t — namespace default $(kubectl get pods — namespace default -l app=sampledb-influxdb -o jsonpath=’{.items[0].metadata.name}’) /bin/sh


To tail the logs for the InfluxDB pod run the following:


kubectl logs -f — namespace default $(kubectl get pods — namespace default -l app=sampledb-influxdb -o jsonpath=’{ .items[0].metadata.name }’)



Next we installed grafana using helm package manager, run the below command to install grafana, the default values.yaml was edited to enable the persistence volume claim and to set the userid and password.


 ​​​​ enabled: true
 ​​​​ storageClassName: ""
 ​​​​ accessModes:
 ​​ ​​ ​​​​ - ReadWriteMany
 ​​​​ size: 2Gi
 ​​​​ annotations: {}
 ​​​​ subPath: ""
 ​​​​ existingClaim: grafanaclaim

adminUser: admin
adminPassword: password


In the newer grafana versions >5.1 , the Docker container are owned by id/gid 472 (the grafana user/group), the container crashes when persistence is enabled , so we have modified the security context , to run the process as a root user.


 ​​​​ runAsUser: 0

grafanavalues.yaml file is available in the below


./helm install --name samplegrafana -f grafanavalues.yaml stable/grafana



You can get your ‘admin’ user password by running:

kubectl get secret — namespace default samplegrafana -o jsonpath=”{.data.admin-password}” | base64 — decode ; echo


The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:


you can get the Grafana URL to visit by running these commands in the same shell:


export POD_NAME=$(kubectl get pods — namespace default -l “app=grafana,release=samplegrafana” -o jsonpath=”{.items[0].metadata.name}”)
kubectl — namespace default port-forward $POD_NAME 3000



Both the grafana and influxdb deployments should be up and running before making the changes to the heapster, the heapster workload will be running under the kube-system namespace, export the heapster deployment’s yaml file and edit it to sink the details to the influxdb database.


kubectl get deployments -n kube-system
kubectl get deployment heapster-v1.5.3 -n kube-system -o yaml > heapster.yaml


Right after the spec , make the below changes



- command:

- /heapster

- --source=kubernetes.summary_api:""

- --sink=influxdb:http://sampledb-influxdb.default:8086

name: heapster


apply the changes to the heapster deployment

kubectl apply -f heapster.yaml


Verified the data in influxDB by connecting to the container, a new database called k8s will be created in influxdb



Used port-forwarding to access the grafana console.


Logged in with the admin credentials, ( you might have to use a strong password )




Add the influxDB data source,


configure the data source and test it



Add new Dashboard and add a graph, under the metrics section select the query on metrics name , container name & pod name. Give the title name under the general section and finally save the dashboard.




So now to check , whether the data is persistent on the grafana and influxdb , delete the pods , you can see that the data on the influxdb and grafana are still available on the new pod that was created . This​​ concludes the setup , Thanks for reading through this post, Hope you find this useful.




Related posts

Leave a Comment