Using Jenkins pipeline to add Grafana Annotations for release monitoring

For the Plivo tech team, DevOps is not just a more efficient way of building products - It’s a way of life. To make life more interesting, and efficient, we are always trying out new things to see what works, and what doesn’t. Release process is one area where we see a lot of scope for innovation. Today we are sharing one such best practice that’s working well for us.

At Plivo, we use Jenkins as our CI/CD tool. Specifically, we use Jenkins Pipeline DSL to write build jobs as code. Using CI/CD, we deploy code several times a day across many micro-services. To ensure that our services are performing optimally within the defined system parameters, we monitor a variety of metrics across releases immediately after the deployment.

Today, we will show you how we annotate Grafana panels via Jenkins pipeline and Grafana.

Our CI/CD pipeline looks like a standard pipeline with stages -


During the Deploy stage, we annotate Grafana panels via annotate REST API more specifically, using the POST /api/annotations API.

The pipeline code looks like -

echo "Creating annotation in Grafana with tags: ${tags} and description: ${text} ..."

withCredentials([[$class: 'StringBinding', credentialsId: 'GRAFANA_APIKEY', variable: 'GRAFANA_APIKEY']]) {
	apiKey = env.GRAFANA_APIKEY
		acceptType: 'APPLICATION_JSON',
		consoleLogResponseBody: false,
		contentType: 'APPLICATION_JSON',
		customHeaders: [[maskValue: true, name: 'Authorization', value: 'Bearer ' + apiKey]],
		httpMode: 'POST',
		requestBody: toJson(payload),
		responseHandle: 'NONE',
		url: grafana + '/api/annotations',
		validResponseCodes: '200'

The Grafana API credentials (GRAFANA_APIKEY) is used from Jenkins credentials store. The payload to the /api/annotations API is sent as a JSON object as shown in the Grafana docs example as reproduced below

  "text":"Annotation Description"

Once the annotations are sent to Grafana, we add them to the Grafana panels manually. We will use the optional dashboardId and panelId fields in the future to build dashboards on the fly with Grafana 5.0 removing the need to manually add it.

The above code will produce Grafana annotation like the one shown below -

Grafana Annotation

As you can see above, the Grafana panel for an imaginary service has an annotation between 12:16 and 12:22 which is the duration of the deployment, and it contains description and tags. The description has the version and the person responsible for the deployment as well as the approver for the release. The description and the tags are defined using various variables earlier in our pipeline code.

As you could see, we showed you a peek of how pipeline could be used to add additional capabilities to CI/CD that will help DevOps teams improve the release process. We will post an update to this blog with improvements with Grafana 5.0. We will also publish more blogs on other areas of our CI/CD pipeline in the future. Do subscribe to our blog to receive updates as we make them available.

comments powered by Disqus