Reqover: easily measure your API tests coverage like a ninja

When it comes to test coverage, many people would probably imagine something like code coverage by unit tests. Or a traceability matrix with user stories, test cases and checkboxes next to them, shedding light on what’s automated and what’s not. But it seems to me, there was always a grey area in between. Functional API tests aren’t something that can be easily mapped with end-to-end user stories or measured just as straightforwardly as code coverage, by tools like JaCoCo or SonarQube.

I’ve been thinking about this and digging into the topic for a while now. Recently, my wandering around brought me to the open-source project called Reqover. It seemed like a tool aimed to solve challenges similar to mine. So, I decided to give it a try, build an example project with functional API tests and measure their coverage against the chosen API specification. Here is the tech stack that I will be working with:

  • Programming language: Kotlin
  • Testing framework: Kotest
  • REST API client: RestAssured
  • API specification standard: OAS3
  • API test coverage tool: Reqover
  • Build tool: Maven (may the Gradle fans forgive me)

You might be wondering – why Kotest instead of the good old JUnit 5? Well, I have been exploring Kotest for a few days already and found it quite flexible, with concise syntax that utilises Kotlin perks. Hence, I decided to kill two birds with one bullet and play with 2 new tools at the same time. I bet you are going to try them after this article as well.

API under test

One of my specific goals was to figure out how to measure API test coverage for services whose API is documented with the OpenAPI specification standard, version 3 (OAS3). Finding a good candidate took some time. There are plenty of publicly available services but many of them have limitations on free use, and not that many of the last have their specifications in OAS3.

Eventually, I chose Canada Holidays API for the following reasons:

  1. It’s public and has no limitations on free usage. It doesn’t even have a paid plan, so a perfect fit.
  2. Its API specification is documented in OAS3. Again, exactly what I need.
  3. It has path and query parameters that I also wanted to have for the proof of concept.
  4. It’s also friendly for parameterized tests that I wanted to try with Kotest.

I exported the Canada Holidays API specification to a JSON file and put it in the root folder of my test project. Will use it later, when the time to generate a report comes.

Let’s draw a goal: to create a simple framework with a few API tests, measure their coverage against the Canada Holidays API specification and have it visualised. So, it can be used for analysing what’s covered, and what’s not and eventually give a good understanding of what other API tests need to be written to increase the coverage.

It’s all about POM

Since I created my project from IntelliJ IDEA’s Kotlin/Maven template, all required dependencies were added to the pom.xml automatically. However, there are a few more dependencies still to be added to get all we need for starting using Kotest:

<dependency>
    <groupId>io.kotest</groupId>
    <artifactId>kotest-runner-junit5-jvm</artifactId>
    <version>5.5.5</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>io.kotest</groupId>
    <artifactId>kotest-assertions-core-jvm</artifactId>
    <version>5.5.5</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>io.kotest</groupId>
    <artifactId>kotest-assertions-json-jvm</artifactId>
    <version>5.5.5</version>
</dependency>

For this experiment, I needed the Kotest JVM framework itself for writing and running tests, Kotest core assertions to validate response codes and other stuff, and Kotest JSON assertions to be able to validate some parts of the response bodies. Please keep in mind that Kotest’s JUnit 5 runner requires the maven-surefire-plugin to be configured (not shown in the picture above).

And finally, I added the Reqover dependency to be able to intercept communication with the Canada Holidays service (more on that later):

<dependency>
    <groupId>io.reqover</groupId>
    <artifactId>reqover-java</artifactId>
    <version>0.2.6</version>
    <scope>test</scope>
</dependency>

Please keep in mind that it’s not my complete pom.xml. RestAssured and other dependencies were left behind for the sake of keeping this article concise. You can check the full pom.xml out if needed.

How Reqover works

The Reqover tool consists of several components that analyse traffic between your tests and a server, compare it with an API specification and generate a coverage report with insights into what is covered and what is not. In my case (my chosen tech stack), these components are:

  • the Reqover’s RestAssured filter that captures requests/responses sent to/from the Canada Holidays server;
  • Reqover command-line tool which takes results from the filter, compares them with the specification and produces a report.

Sounds good! Let’s start by adding the Reqover’s RestAssured filter in my custom-made REST client:

object RestClient {
    fun createBaseSpecification(): RequestSpecification {
        RestAssured.baseURI = BASE_URI
        RestAssured.basePath = BASE_PATH
        return RestAssured.given().filters(RequestLoggingFilter(), ResponseLoggingFilter(), SwaggerCoverage())
    }
}

Readers who are familiar with RestAssured must have recognized its standard filters for logging requests and responses (RequestLoggingFilter and ResponseLoggingFilter). The SwaggerCoverage filter is the one that comes from the before-mentioned reqover-java dependency and will be capturing backend communication.

This is basically it regarding the filter. Let’s produce some tests, run them and then check what the SwaggerCoverage filter captured.

Kotest fatality

I would be lying if I say I didn’t really like Kotest. After a couple of days of exploring it, I began to realize it suits my needs more and more. This topic definitely deserves a separate article (or two), and I’m going to work on it later. But what’s Kotest in a nutshell?

It’s a multi-platform framework for writing tests, powerful assertions and is friendly for data-driven testing. I’d say that it took the best from JUnit, Cucumber, JavaScript and wrapped everything into Kotlin’s flexibility. Unlike JUnit Jupiter (where everything is packed in 2 libraries), Kotest ships its different components as separate dependencies, and you import only what you need.

There are different styles of writing tests (Cucumber-like, traditional JUnit, JavaScript-like or even pure strings). For my experiment, I chose the “describe / it” style where a typical test looks like this:

describe("When a logged in user opens their profile page") {
    it("should show their shopping lists") {
        // test and asserts logic
    }

    it("should allow creating a new shopping list") {
        // test and asserts logic
    }
}

Moreover, there can be “describe” inside another “describe”, your custom code inside each of these containers, lifecycle callbacks and so on.

Okay, let’s write the very first test for one of the Canada Holidays endpoints, for example, GET /provinces/{provinceId}:

class CanadaHolidaysTest : DescribeSpec({
    isolationMode = IsolationMode.InstancePerLeaf

    val specification = RestClient.createBaseSpecification()

    describe("When user requests a province by unknown ID") {
        it("should return 404 response code") {
            specification.get("/provinces/XYZ").statusCode.shouldBe(HttpStatus.SC_BAD_REQUEST)
        }
    }
}

A couple of key points here:

  • the IsolationMode.InstancePerLeaf means that for every test case inside a test class, a new instance of that test class will be instantiated. In very simplified words: for every “it” inside your “describe” your code up to the leaf test path will be executed;
  • the shouldBe assertion comes from the previously imported kotest-assertions-core-jvm library out of the box. In the next test, I will be adding a few custom assertions just to be able to validate the response body in a more elegant way.

Okay, what about the happy case? Canada has 10 provinces and 3 territories, and users should be able to fetch their info and holidays of each of them. Looks like this is a perfect fit for a data-driven test. And here is how it looks in Kotest:

val provincesIds = listOf("AB", "BC", "MB", "NB", "NL", "NS", "NT", "NU", "ON", "PE", "QC", "SK", "YT")

describe("When user requests a province by ID") {
    provincesIds.forEach { id ->
        val response = specification.get("/provinces/$id")

        it("should return the $id province with associated holidays of the current year") {
            response.statusCode.shouldBe(HttpStatus.SC_OK)
            response.shouldContainPathWithValue("province.id", id)
            response.shouldNotHaveEmptyList("province.holidays")
            // extra assertions for the holidays in the province
        }
    }
}

Just by using language features available in Kotlin, I was able to enumerate through the list of provinces’ IDs and parameterise a single test to cover all of them. By the way, it’s just the simplest example – for more complex parameterization, Kotest offers other ways to handle this.

The output in IntelliJ IDEA then looked the following:

If you paid attention to the code snippet above, you might have noticed a couple of funny assertions: shouldContainPathWithValue() and shouldNotHaveEmptyList(). They didn’t come out of the box, even despite the fact that Kotest does have a library with JSON matches.

Since I consciously decided to omit data models in this PoC, a few extension functions for the RestAssured’s Response class saved the day:

fun Response.shouldContainPathWithValue(path: String, value: String) {
    this.body.asString().shouldContainJsonKeyValue(path, value)
}

fun Response.shouldNotHaveEmptyList(path: String) {
    this.jsonPath().getList<String>(path).shouldNotBeEmpty()
}

fun Response.shouldHaveListWithSize(path: String, expectedSize: Int) {
    this.jsonPath().getList<String>(path).size.shouldBe(expectedSize)
}

Again, no rocket science here. But they made the test more readable and, I believe, can be reused in the upcoming scenarios.

Following the same approach, I wrote a few more tests for the other API endpoints to increase test coverage to… what? Good question! It’s time to find out.

And here Reqover comes…

After the API tests are run, the reqover-results folder with a bunch of JSON files appears at the root of the test project. These are the results of the SwaggerCoverage filter work. Each of these JSONs contains intercepted request-response info from a particular test:

P.S. It is possible to specify another location of a folder where to save Reqover results. Just pass the desired path to the SwaggerCoverage filter’s constructor when configuring your ResrAssured client.

So, what do we do with these results? Apparently, they need to be merged together and visualised to produce some meaningful insights. This can be done with the Reqover command-line tool (see the link for installation via npm).

When you have it installed, the simplest option would be to navigate to the root of the test project and execute the following command to generate a Reqover report:

reqover generate -f swagger_canadaHolidays.json -d reqover-results --html

where:

  • f flag specifies the location of the Canada Holidays API specification (downloaded and saved to the project’s root);
  • -d flag specifies where a folder with JSON files produced by the SwaggerCoverage filter is.

If needed, the -o flag can be used for specifying the output folder where the final report should be saved. If not specified, the current folder will be used.

When Reqover command-line tool is done, the results appear in the specified location:

The newly created coverage.json file contains the results of the test coverage calculation. The .reqover folder instead contains a generated HTML report that can be either opened in the default browser or (for lazy like me) Reqover can do it after hitting the following command:

reqover serve

Just as simple as that! Time for the final and the most beautiful part…

Cover me, Reqover!

In total, I wrote 6 test cases and was really curious – what % of the API specification do they cover? When Reqover served the HTML, I observed the following:

At first sight, 4 endpoints should be covered completely, 1 endpoint should be partially covered, and 1 endpoint – is not covered at all. Such distribution led to 67% test coverage, while 17% of the API spec were covered partially, and 16% were not covered at all.

As you can see, the sections in the report reflect the structure of the Canada Holidays API specification: info, holidays and provinces. Let’s have a look at what is fully covered and see what is inside those sections. For instance, provinces:

The key takeaway from the picture above is that my tests for the GET /provinces/{provinceId} endpoint cover both documented response codes, and all provinces IDs that I used in my data-driven test are also displayed in the report. Super convenient.

If you had a look at the API specification, you might have noticed that some endpoints of the Canada Holidays API have optional query parameters, such as {year} or {optional}. I didn’t cover them in my tests intentionally, and that’s what you can also observe in the report. According to the creator of Reqover, only response codes are currently used for coverage calculation.

Conclusions

  • I definitely like Reqover quite a lot! It works perfectly with RestAssured which, de facto, is the standard REST API client in the Java/Kotlin world.
  • What I’ve seen so far is that Reqover can work well in the pipeline, and its HTML test coverage report can be easily hosted if needed (for example, on Gitlab Pages). At the same time, there is no straightforward way to fail a pipeline if the test coverage percentage is lower than the desired number. In that case, you’d have to parse the generated coverage.json file, extract the full coverage value and fail your job from your bash script, if needed. Perhaps, this can be implemented out of the box as well.
  • I have a feeling that query parameters should also contribute to the coverage % value. Getting back to my example where I didn’t test the {year} query parameter – there is no guarantee that if tested GET /provinces/MB, another GET /provinces/MB?year=2022 call will work as expected, isn’t it?
  • There are other possible features that I have in mind and will try to share them with the Reqover team. Nevertheless, I found Reqover a great and promising API test coverage tool that can help quality engineers to solve their problems.

That’s it for today, folks! Stay in tune for the next interesting articles!