How To Measure Code Coverage Using SonarQube and Jacoco

How to measure code coverage using SonarQube and Jacoco

Last Updated on September 25, 2020

Code coverage is a metric that many teams use to check the quality of their tests, as it represents the percentage of production code that has been tested. Discover how to apply the Gradle Jacoco plugin to your project and run a SonarQube scan to generate a code coverage report.

1. Overview

SonarQube is a tool which aims to improve the quality of your code using static analysis techniques to report:

  • code coverage
  • bugs
  • code smells
  • security vulnerabilities

The SonarQube server is a standalone service which allows you to browse reports from all the different projects which have been scanned. To scan a specific codebase you run the SonarQube scanner. This is a local process that analyses your code then sends reports to the SonarQube server.

With SonarQube, the code coverage metric has to be computed outside of SonarQube. The tool we’ll be looking at today to calculate code coverage for a Java project is called Jacoco. It analyses the code and generates a report, which later gets ingested by SonarQube.

2. A worked example

We’re going to run through an example of exactly how this works. To do this, I’ve put together a GitHub project which you can check out to see this working with your own eyes, if you like.

We’ll be setting up:

  1. an instance of SonarQube running in Docker
  2. a Java project with a class and some unit tests
  3. a Gradle build which runs the SonarQube scanner against the Java project

The project has 2 classes:

  1. A MathService class which has 2 methods, multiply and subtract
  2. A MathServiceTest class which only tests 1 out of the 2 methods in MathService

So we’re hoping that SonarQube will highlight the fact that we’re missing a test here i.e. only one of the methods in MathService has been tested. Any guesses for what percentage code coverage SonarQube will report in this case? 🤔

2.1. Setting up a SonarQube instance

An official Docker image exists for SonarQube, making this really easy to get up and running using Docker Compose. Just add the following docker-compose.yml file to your project:

version: "3"
services:
  sonarqube:
    image: sonarqube:lts
    ports:
      - 9000:9000

This will use the lts (long term support) version of the SonarQube Docker image, which is currently SonarQube version 7 (for version 8 instructions see the later section). SonarQube is configured to start on port 9090.

To be able to run Docker Compose from Gradle, just add this plugin to your build.gradle:

plugins {
    id "com.palantir.docker-compose" version "0.22.1"
    // any other plugins
}

Info: if you’re following along with the example GitHub repository rather than applying this to your own project, the previous and following steps are already configured for you.

Now we can run:

./gradlew dockerComposeUp

That was successful, but we can double check everything is OK by seeing what Docker processes are running:

docker ps

Here we can see SonarQube is running on localhost:9000. It might take a minute to fully start up, but eventually we’ll see this screen:

This is correctly reporting we currently have 0 projects analysed. Let’s fix that! 👌

2.2. Using the Jacoco Gradle plugin

Before we get onto actually scanning our code with SonarQube, let’s set up the Jacoco Gradle plugin. This will generate the test coverage statistics for our Java code. Just add the following plugin definition to build.gradle:

plugins {
    id 'jacoco'
    // any other plugins
}

Now let’s run ./gradlew test. Notice we have a file jacoco/test.exec output in our build directory.

This contains the code coverage information that SonarQube will pick up during it’s scan. It’s in binary format, so unfortunately we can’t take a look inside.

2.3. Using the SonarQube Gradle plugin

Now that we’ve got our test code coverage data being generated by Jacoco, it’s time to hook all this up by running a SonarQube scan. This will report on the code coverage as well as run a full scan of our code. To do this we’ll use the SonarQube Gradle plugin which adds the sonarqube task to our build. We can include it in our build.gradle like this:

plugins {
    id 'org.sonarqube' version '3.0'
    // any other plugins
}

We also need to include a configuration to tell the SonarQube scanner where to find the SonarQube server that we have running:

sonarqube {
    properties {
        property 'sonar.host.url', 'http://localhost:9000'
    }
}

Lastly, to ensure the Jacoco test report will always be created when we run the sonarqube task let’s setup the following dependsOn relationship:

tasks['sonarqube'].dependsOn test

Now we just need to run the sonarqube task to run a scan:

./gradlew sonarqube

Success!

2.4. Viewing the SonarQube report details

We can head back to SonarQube at localhost:9000 to see the test code coverage report:

Click on the 1 project analysed link to see the report overview:

We can see a reported code coverage of 66.7%. Awesome! Click on the sonarqube-jacoco-code-coverage link and we’ll try to drill into exactly how this was calculated.

This is a more detailed view of the report. Click on the 66.7% link.

We now see information about what class has been analysed, in this case the MathService. Click on the link to see even more details:

We can now see the class itself, where green highlights code that is properly tested and red code that isn’t. Let’s zoom in a bit:

We can see that SonarQube is telling us that:

  • the class MathService is covered by tests (green mark)
  • the multiply method is covered by tests (green mark)
  • the subtract method is not covered by tests (red mark)

That makes 2 out of 3, hence the 66.7% being reported by SonarQube. So there’s definitely room for improvement!

3. Test coverage with SonarQube 8

The steps discussed in this article to generate a jacoco.exec file and then use it during a SonarQube scan to generate a coverage report work well for SonarQube 7. With SonarQube 8 the jacoco.exec file is no longer compatible, and instead we have to create a report in xml format.

Fortunately with the Gradle Jacoco plugin this is straightforward, and can be achieved with this small configuration in build.gradle:

jacocoTestReport {
    reports {
        xml.enabled true
    }
}
test.finalizedBy jacocoTestReport

Now when we run ./gradlew test we’ll get an xml report at build/reports/jacoco/test/jacocoTestReport.xml:

Jacoco XML report

And./gradlew sonarqube can be run as normal against a SonarQube 8 server.

SonarQube 8 test coverage

For an example of this setup, check out the sonarqube-8 branch on GitHub.

4. Conclusion

You’ve seen that it’s really easy to setup code coverage reporting in a Gradle project using Jacoco and SonarQube. Try it out on your own project to see how you measure up.

It’s worth mentioning that this metric isn’t the only metric you should use to measure your test quality, but it can be a helpful indicator. Code may have a high code coverage percentage, but it might be brittle and difficult to maintain.

I suggest also having a look at the other reports within SonarQube, such as bugs, vulnerabilities, and code smells. Maybe you’ll learn something new about your codebase and how to improve it?

5. Resources

GITHUB REPOSITORY
Follow along with this article by checking out the accompanying GitHub repository. This uses the LTS version of SonarQube (currently version 7).

View the sonarqube-8 branch if you want to see an example with the latest SonarQube version.

GRADLE PLUGINS
Jacoco Plugin docs
SonarQube Plugin docs

SONARQUBE
General documentation

VIDEO
If you prefer to learn in video format, check out this accompanying video to this post on the Tom Gregory Tech YouTube channel.

How To Measure Code Coverage Using SonarQube and Jacoco

11 thoughts on “How To Measure Code Coverage Using SonarQube and Jacoco

  1. Dear Tom,
    Nice and easy explained.
    However i get 0% coverage, 100% unit test
    If i run the same example against an external sonarqube scanner i have also 0 %.
    This is the logging:

    build 24-Mar-2020 18:13:42 INFO: parsing [/ec/local/citnet/bamboo-agent-home/xml-data/build-dir/EACDEVOPS-EACDEVOPSPLAN1-CHEC/sonarqube-jacoco-code-coverage/build/test-results/test]
    build 24-Mar-2020 18:13:42 INFO: Sensor SurefireSensor [java] (done) | time=31ms
    build 24-Mar-2020 18:13:42 INFO: Sensor JaCoCoSensor [java]
    build 24-Mar-2020 18:13:42 INFO: Sensor JaCoCoSensor [java] (done) | time=1ms
    build 24-Mar-2020 18:13:42 INFO: Sensor JavaXmlSensor [java]
    build 24-Mar-2020 18:13:42 INFO: Sensor JavaXmlSensor [java] (done) | time=1ms
    build 24-Mar-2020 18:13:42 INFO: Sensor HTML [web]
    build 24-Mar-2020 18:13:42 INFO: Sensor HTML [web] (done) | time=26ms
    build 24-Mar-2020 18:13:42 INFO: Sensor JaCoCo XML Report Importer [jacoco]
    build 24-Mar-2020 18:13:42 INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=3ms
    build 24-Mar-2020 18:13:42 INFO: ————- Run sensors on project
    build 24-Mar-2020 18:13:42 INFO: Sensor Dependency-Check [dependencycheck]
    build 24-Mar-2020 18:13:42 INFO: Process Dependency-Check report
    build 24-Mar-2020 18:13:42 INFO: Dependency-Check XML report does not exists. Please check property sonar.dependencyCheck.reportPath:…
    build 24-Mar-2020 18:13:42 INFO: Analysis skipped/aborted due to missing report file
    build 24-Mar-2020 18:13:42 INFO: Dependency-Check HTML report does not exists. Please check property sonar.dependencyCheck.htmlReportPath:…
    build 24-Mar-2020 18:13:42 INFO: HTML-Dependency-Check report does not exist.
    build 24-Mar-2020 18:13:42 INFO: Process Dependency-Check report (done) | time=4ms
    build 24-Mar-2020 18:13:42 INFO: Sensor Dependency-Check [dependencycheck] (done) | time=4ms
    build 24-Mar-2020 18:13:42 INFO: Sensor Zero Coverage Sensor
    build 24-Mar-2020 18:13:42 INFO: Sensor Zero Coverage Sensor (done) | time=11ms
    build 24-Mar-2020 18:13:42 INFO: Sensor Java CPD Block Indexer
    build 24-Mar-2020 18:13:42 INFO: Sensor Java CPD Block Indexer (done) | time=19ms
    build 24-Mar-2020 18:13:42 INFO: SCM Publisher is disabled
    build 24-Mar-2020 18:13:42 INFO: 1 file had no CPD blocks
    build 24-Mar-2020 18:13:42 INFO: Calculating CPD for 0 files
    build 24-Mar-2020 18:13:42 INFO: CPD calculation finished
    build 24-Mar-2020 18:13:42 INFO: Analysis report generated in 122ms, dir size=78 KB
    build 24-Mar-2020 18:13:42 INFO: Analysis report compressed in 11ms, zip size=13 KB
    build 24-Mar-2020 18:13:42 INFO: Analysis report uploaded in 28ms
    build 24-Mar-2020 18:13:42 INFO: ANALYSIS SUCCESSFUL, you can browse https://webgate.ec.europa.eu/CITnet/sonarqube/dashboard?id=EACDEVOPS-SRCKEY
    build 24-Mar-2020 18:13:42 INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
    build 24-Mar-2020 18:13:42 INFO: More about the report processing at https://webgate.ec.europa.eu/CITnet/sonarqube/api/ce/task?id=AXENiSBOgY0MYh9regFH
    build 24-Mar-2020 18:13:42 INFO: Analysis total time: 5.861 s
    build 24-Mar-2020 18:13:42 INFO: ————————————————————————
    build 24-Mar-2020 18:13:42 INFO: EXECUTION SUCCESS
    build 24-Mar-2020 18:13:42 INFO: ————————————————————————
    build 24-Mar-2020 18:13:42 INFO: Total time: 13.805s
    build 24-Mar-2020 18:13:42 INFO: Final Memory: 33M/349M
    build 24-Mar-2020 18:13:42 INFO: ————————————————————————
    simple 24-Mar-2020 18:13:42 Finished task ‘sonarqube source scanning’ with result: Success
    s

    Kind Rg
    Stefan

    1. Hi Stefan. Thanks for emailing this question to me. I’m adding my response here in case it’s useful for anyone.

      I think the problem is with the latest version of Sonarqube, as specified in docker-compose.yml. For some reason it’s not generating the code coverage stats correctly. I have updated the GitHub repository and blog post to specify the version of lts (long term support) instead of latest

  2. Could it be related to this:
    Property ‘sonar.jacoco.reportPath’ is no longer supported. Use JaCoCo’s xml report and sonar-jacoco plugin.
    Property ‘sonar.jacoco.reportPaths’ is no longer supported. Use JaCoCo’s xml report and sonar-jacoco plugin.

  3. could not see the code coverage as of running through this today. Is sonarqube or jacoco broken? I was trying to fix why it wasn’t working in a pipeline for work, but I can’t even get it to work using this demo.

    1. Hi Kevin. Sorry you couldn’t get the example working. I tried it a few weeks ago without issue. I will be taking a look later today, so please bare with me.

        1. Hi again Kevin. I’ve just tried running the example from the GitHub repository and I’m getting the 66.7% test coverage as shown in this article.

          Can you please provide some more details about the problem you’re having? Are you managing to log into the SonarQube UI? If so, are you seeing that the project has been analysed?

          The version of SonarQube used in the project is the lts (long term support version) and the Jacoco plugin comes with the version of Gradle in the project (6.4.1).

          1. Tom,

            I was able to get it to work on my end. It was partly user error! It had to do with the java that I was using. I got it working in the end. Thank you for running through it again and verifying though! I’m currently trying to integrate the xml reporting in as that’s what broke code coverage for a work project. Thanks for providing this tutorial. It has been helpful in me figuring out how all of this works!

  4. Hi @Tom,

    I ran your example. It is working fine and you explained it very nice.
    So how do we generate pdf report using sonar result? Is there any plugin?

    Thanks.

    1. Hi Erandika. Thanks for the feedback. I know that Gradle Enterprise offers PDF reporting. Another option might be to use the Web API to get the information you need then format it into a report.

Leave a Reply

Your email address will not be published. Required fields are marked *

Scroll to top

To keep up to date with all things to do with scaling developer productivity, subscribe to my monthly newsletter!