Test Architects Hub

Long time no see, ladies and gentlemen! I hope you’ve had or are still going to have good summer holidays. I’ve been quite busy lately, so this blog has got less priority than usual. But even without actual writing, many ideas of potential research and articles were still buzzing in my head.

When looking into the blog’s statistics after the vacation, I was surprised to discover that my recent JUnit 5 vs Kotest article gained popularity via organic search. It was even featured in the official Kotest blog (scroll to the end of the list).

I never aspired to write for the sake of SEO. My primary goal in this blog is to spread knowledge and help people to overcome their daily challenges. Thus, I sincerely believe that the JUnit 5 vs Kotest article does have some value for the community, and it’s time to dig further into the JUnit 5 vs Kotest comparison.

In this part 2, I’ll assess frameworks’ capabilities regarding test parameterization. As an application under test, I’ll still be using the good old Canada Holidays service which API is described here, in case you need a reference.

Table of contents:

Parameterised tests

In my opinion, this feature has always had a little bit more hype around it than any other capability of unit test frameworks. People like to ask about parameterised tests at job interviews, write articles about them and… sometimes, exaggerate their complexity.

In fact, test parameterisation in JUnit 4 used to be indeed cumbersome and not always straightforward. But not anymore. Let’s figure that out.

JUnit 5: Parameterising a test with a list of values

JUnit Jupiter has got a lot of architecture rework and made lots of improvements regarding parameterised tests as well. In most cases, we just need to use a couple of annotations to complete the work.

Let’s write a very simple test that fetches holidays for each of the 13 Canadian provinces and territories and asserts the response code:

class JunitExamplesPart2Test {

   private val specification = RestClient.createBaseSpecification()

   @ParameterizedTest
   @DisplayName("Requesting known province by ID should result in 200 error code")
   @ValueSource(strings = ["AB", "BC", "MB", "NB", "NL", "NS", "NT", "NU", "ON", "PE", "QC", "SK", "YT"])
   fun checkAllProvincesCanBeFetchedById(provinceId: String) {
       val response = specification.get("/provinces/$provinceId")
       assertEquals(response.statusCode, HttpStatus.SC_OK)
   }
}

You’ve seen most of this code in the previous article, so let’s just talk about the differences:

• There is no need to specify a different runner for parameterized tests in JUnit 5 anymore. All we need to do is to replace the default @Test annotation with @ParameterizedTest one (line 5). It lets the test runner know that the annotated method should be called multiple times with different arguments;
• In addition, at least one arguments provider must be declared. I went for the @ValueSource one and passed my list of the Canadian provinces’ IDs (line 7). For now, JUnit 5 supports 9 different argument providers (we’ll see a few of them later).
• And, apparently, the test method should have the corresponding number of parameters (in my case, it’s provinceId), so JUnit can resolve them properly.

This is basically it. After executing the above-mentioned parameterised test, the output is pretty much predictable:

JUnit 5: Parameterising a test with an enum

If by design you have your test input data stored in an enum (like in my case), parameterising tests becomes even more elegant:

@ParameterizedTest
@DisplayName("Requesting known enum province by ID should result in 200 error code")
@EnumSource
fun checkAllEnumProvincesCanBeFetchedById(province: Provinces) {
   val response = specification.get("/provinces/${province.name}")
   assertEquals(response.statusCode, HttpStatus.SC_OK)
}

In the code snippet above, I replaced the @ValueSource annotation with the @EnumSource one (line 3). That freed me up from specifying the source directly there. It will just simply be resolved from the enum parameter I provided.

I personally find this much more convenient than it used to be with JUnit 4. It’s more manageable and enables more tweaks. For instance, specifying different matching rules, like included and excluded values:

@EnumSource(mode = EnumSource.Mode.EXCLUDE, names = ["AB", "BC"])

JUnit 5: Parameterising a test with multiple parameters

And the last (but not least) JUnit 5 case for comparison is multiple test parameters.

Some improvements have been made in the fifth version of JUnit under the hood, and now it can be done by the @MethodSource annotation. For example, let’s say we want to extend our Canada Holidays test and fetch holidays per province for a specific year:

@ParameterizedTest(name = "{index} - Check the {0} province holidays for {1} year can be fetched")
@MethodSource("provincesAndYearsProvider")
fun checkMethodProvincesCanBeFetchedById(province: String, year: String) {
   val response = specification
       .queryParam("year", year)
       .get("/provinces/${province}")
   assertEquals(response.statusCode, HttpStatus.SC_OK)
}


companion object {
   @JvmStatic
   fun provincesAndYearsProvider(): Stream<Arguments> {
       return Stream.of(
           arguments("AB", "2023"),
           arguments("BC", "2022")
       )
   }
}

The key point in the code snippet above is the provincesAndYearsProvider() factory method (line 13) that returns the stream of arguments. It must always be static or your test class should be annotated with @TestInstance(Lifecycle.PER_CLASS). For illustration purposes, I put it in the companion object { .. } block.

It is already more flexible and readable than it used to be with JUnit 4. And by the way, for the sake of improving the test output, I changed the pattern of how my parameterized tests are named:

Looks pretty cool and convenient so far. Let’s see what Kotest brings to the table for solving the same challenges.

---

Kotest: Parameterising a test with a list of values

In order to maintain a fair competition spirit, I am going to try to replicate the same cases but with Kotest. The first one is easy, and I need nothing more but to define the list of my provinces’ IDs and write a simple test like this one:

class KotestExamplesPart2Test : DescribeSpec( {
   val specification = createBaseSpecification()
   val provinces = listOf("AB", "BC", "MB", "NB", "NL", "NS", "NT", "NU", "ON", "PE", "QC", "SK", "YT")


   describe("When user requests a province by ID") {
       provinces.forEach { id ->
           it("should return the $id province with associated holidays of the current year") {
               val response = specification.get("/provinces/$id")
               response.statusCode shouldBe HttpStatus.SC_OK
           }
       }
   }
}

You are already familiar with the Describe spec already from the previous article. The only part that is worth explaining here is the forEach { .. } method, called on the provinces collection (line 7).

The power of Kotlin & Kotest syntax let me specify it() TestScope in the lambda expression body inside the forEach { .. } block. Hence, for each province ID in my provinces list, I will have a separate test that will grab the next ID from the list, do a backend call and validate the status code.

After running the above test, I got the following output:

It is more readable than my first JUnit 5 test without name customisation but it’s not a big deal. What about enum values?

Kotest: Parameterising a test with an enum

In order to solve that, another Kotest dependency for property-based testing needs to be introduced:

<dependency>
   <groupId>io.kotest</groupId>
   <artifactId>kotest-property-jvm</artifactId>
   <version>${kotest.version}</version>
   <scope>test</scope>
</dependency>

In a nutshell, the purpose of kotest-property library is to generate ranges of different data that can be used as input for tests. While this Kotest subproject is meant mostly for unit testing, it can actually be beneficial for QA engineers who usually deal with more high-level tests such as end-2-end.

For example, with property-based testing, it is easy to “feed” a range of valid and invalid inputs to a test that validates a login form.

The kotest-property library contains a lot of different generators for various data types. For solving my problem, I went with the Exhaustive.enum<T>() one that iterates through all the constants defined in the provided enum:

describe("When user requests a province by ID (enum)") {
   Exhaustive.enum<Provinces>().checkAll { id ->
       it("should return the $id province with associated holidays of the current year") {
           val response = specification.get("/provinces/$id")
           response.statusCode shouldBe HttpStatus.SC_OK
       }
   }
}

The checkAll() method (line 2) ensures that the number of iterations is equal to the number of combinations in the exhaustive, which is in my case 13. And the rest of the test is exactly the same as in the previous example.

If you for whatever reason prefer not to deal with generics, the same result can be achieved with:

Exhaustive.collection(provinces).checkAll { .. }

Kotest: Parameterising a test with multiple parameters

Now let’s move on to the most exciting part. For solving the 2-parameters challenge, I had to step aside from the property-based testing. Not sure if I would still be able to find a solution with kotest-property, if I’d spend more time. Please let me know if you found one.

There’s another Kotest subproject called kotest-framework-dataset. Its purpose is to facilitate data-driven testing, in particular, testing many combinations of parameters. Exactly what I needed!

In particular, I discovered the withData() methods that take sets of arguments as parameters and register tests for each provided argument within the given context.

First, I created a data class for my combination of parameters:

data class ProvinceByYear(val province: String, val year: Int)

I’ll be passing instances of the ProvinceByYear data class to the above-mentioned withData() method. (line 6) The test logic then can be placed inside the withData() method’s lambda body and executed as usual:

class MyDoubleParameterisedTests : FunSpec({
   val specification = createBaseSpecification()


   context("User requests a province by ID for the selected year") {
       withData(
           ProvinceByYear("AB", 2022),
           ProvinceByYear("BC", 2023)
       ) { (province, year) ->
           specification.queryParam("year", year)
               .get("/provinces/${province}")
               .statusCode shouldBe HttpStatus.SC_OK
       }
   }
})

Let’s run the test and see what the output looks like:

Oops! The first test for the combination of “AB” province and 2022 year succeeded. But the second test for the “BC” province and 2023 year failed with the 400 response code which is (spoiler alert!) “Bad request”.

After a short investigation, I found out that the “year” query parameter got doubled in the second test’s request:

It happened because I didn’t specify any isolation mode in my test class, and by default, the same spec state was shared between my parameterised tests. In simple words, the “year” query parameter was appended to my existing RestAssured request specification again and again in every new test (look at specification.queryParam(“year”, year) line).

I’ll dive into isolation modes in more detail in a different article. For now, this issue can be easily fixed by adding the following line of code at the beginning of the test class:

class MyDoubleParameterisedTests : FunSpec({
   isolationMode = IsolationMode.InstancePerTest
   // the rest of test code
})

Voila! The second test is now passing, and the 3rd comparison case is done:

Kotest: Bonus topic: nested data tests

After fixing the above-mentioned problem, I couldn’t stop and decided to dig further. Canada has 10 provinces + 3 territories, and the Canada Holidays API supports returning holiday data starting from 2016. How can I parameterise a test with the list of provinces and years in a more efficient way than supplying dozens of instances of the ProvinceByYear data class? The way I’ve done it before is apparently not efficient.

Several years ago, I was doing test automation with C# and became a big fan of the NUnit test framework. One of my favourite NUnit features was the Combinatorial attribute that allows creating tests for all possible combinations of supplied parameters. It was very easy and very straightforward.

Unfortunately, I wasn’t able to find the same straightforward way of solving this challenge with JUnit 5. Yes, people employ other third-party libraries in combination with JUnit’s TestFactory and dynamic tests. But this all is not trivial and definitely not what I would prefer.

Then what about Kotest? Well, it’s supported out of the box!

The already mentioned withData() methods can be nested, and every next nesting layer will add more and more combinations of input parameters. For example, let’s fetch holidays for only 3 provinces (Ontario, Quebec, Saskatchewan) but for the last 3 years (2021, 2022, 2023). With the previous method, I’d have to pass 9 instances of the ProvinceByYear data class. If I’d want to test all provinces and territories for the last 3 years, I’d have to pass 13 * 3 = 39 instances, which is a nightmare.

With Nested data tests, I just did the following:

class MyCombinatorialTests : FunSpec({
   isolationMode = IsolationMode.InstancePerTest
   val specification = createBaseSpecification()

   val provinces = listOf("ON", "QC", "SK")
   val years = listOf("2021", "2022", "2023")

   context("User requests a province by ID for the selected year") 
       withData(provinces) { province ->
           withData(years) { year ->
               specification.queryParam("year", year)
                   .get("/provinces/${province}")
                   .statusCode shouldBe HttpStatus.SC_OK
           }
       }
   }
})

The main magic happens in lines 9-10 where the nested withData() methods are specified. As a result, I got 9 tests (3 per each province):

If you want a nicer output, it can be done easily via custom test names:

context("User requests a province by ID for the selected year") {
   withData<String>({ "for the $it province" }, provinces) { province ->
       withData<String>({ "should return holidays for $it year" }, years) {year ->
           specification.queryParam("year", year)
               .get("/provinces/${province}")
               .statusCode shouldBe HttpStatus.SC_OK
       }
   }
}

Cool, isn’t it?

Conclusions

• When writing tests for this article, I noticed that every time I did something with JUnit Jupiter, I also unconsciously compared it with the “old” JUnit 4 way. Which was not that great and convenient for me. Thus, I was already quite happy with the improvements made in JUnit 5, and that softened my attitude towards the good old framework. At the same time, Kotest as a relatively new framework had plenty of time to learn from other’s mistakes and align with the community’s needs. Which probably makes such a comparison not entirely fair.
• Nevertheless, I find the JUnit way of parameterising tests a bit more concise than Kotest’s one for higher-level tests (acceptance, end-2-end). The idea of passing different matching modes as @EnumSource annotation parameters is also good and can come in handy in different scenarios.
• For lower-level tests (unit, integration) Kotest with its property-based testing is way ahead of JUnit thanks to its advanced data generators that can make our lives much easier. That comes with a little price, however, as we have to introduce and manage extra dependencies. But I don’t think it’s weighty enough to dissuade people from trying Kotest.
• Another feature that bribes me to invest more into testing with Kotest is the nested data tests I’ve just shown. It literally brought back warm memories about NUnit combinatorics and it’s just so simple to use. I think it has great potential for backend and frontend test automation engineers that need to cover various combinations of test inputs.

To summarise, Kotest won the second round not by knockout but by points. I can’t say it’s unconditional victory, not at all. There are things that I don’t like in Kotest, or like their JUnit alternatives more. But we’ll talk about them in the next articles.

Until then – let’s keep in touch!

Viktor