For the last few years I have been very interested in what we can learn about our software development processes by looking at the data generated by the tools that we use to manage and deploy our work. This work started the Kuona project (Kuona is the shona word "To see.." http://kuona.io. All the code is open source on github https://github.com/kuona

Initial work focused on creating a dashboard from gathered metrics. Building full stack meant great flexibility but a lot of work and finding time for that work challenging with busy schedules. Over the past couple of weeks I have been working on feeding data from the collectors into elasticsearch and then using kibana to visualise the collected data.

Cutting to the chase heres and example chart:

spring boot commits by day

I am not close enough to the project to draw significant insight from this chart but as an outside observer I can see that the project has been continually active since 2013 in this repository.

Here’s another metric drawn from Spring Boot history:

spring boot kuona metrics

Putting these metrics together in Kibana is really straightforward. But how do we get the data there?

Setting up the Git Kuona Collectors

Kuona and the collectors are still in development so you will need to build/run from source.

Pre-requisites

Before running you will need to write a configuration file for the collector to use.

properties.json
{"workspace": "<workspace directory>",
  "git": [{ "url": "git@github.com:spring-projects/spring-framework.git" },
                { "url": "git@github.com:spring-projects/spring-boot.git" }]
}

Change <workspace directory> to some temporary space. On first run the git collector clones the repositories into the workspace. Subsequent runs update the repository before scanning history.

The collector only supports history on a single branch (master). Still need to work out how to handle multiple branches well.

  • Install ElasticSearch. If you are using a mac brew install elasticseach works nicely

  • Install Kibana. Again homebrew has a recipe brew install kibana

  • Make sure Elastic search is running http://localhost:9200

  • Make sure that Kibana is running http://localhost:5601

Elasticsearch does not have any of or data so we will need to postpone Kibana configuration until we have at least one record.

Running the collector:

In the git-collector directory

  • Create your properties file (see above)

lein run

Sit back and relax - collecting revision history takes a while.

Once the initial clone is complete records will appear in Elasticsearch so we can configure Kibana to use the generated data:

kuona kibana index setup

Once set up the index should look something like this:

kuona kibana index

It would be great to get feedback on this and the other tools being developed for Kuona. Currently working on a collector for Jenkins https://github.com/kuona/jenkins-collector that works like the git collector but for Jenkins build jobs.

comments powered by Disqus