Spock maintainer Rob Fletcher describes the current state of the Kotlin testing framework and compares it to Spek, which is a variant of JetBrains’ standard framework. As an experienced tester, Rob highlights the strengths and weaknesses of both libraries, as well as what Spek is good at and what he expects from Spek to improve. With Kotlin, you can greatly simplify some of the repetitive testing and piling.
Test it with Kotlin(0:00)
Hi, my name is Rob Fletcher, and we’re here to compare some of the interesting use cases we’ve found in different test frameworks and how we’ve implemented similar things in Spek.
I’m a development engineer at Netflix, and I don’t use Kotlin in my daily work, as much as I’d like to. Right now I’m writing a book about Spock, a testing framework developed for Groovy. Some of the things I’m going to show you are some really cool features of Spock, and then I’m going to show you if we can do something similar in Spek, or even better.
Let’s get started. I will cover three important sections: iterative testing, piling, and TCKs.
Iterative testing(1:05)
Iterative – Spock(which)
First, we’ll look at some iterative tests. They are very nice nested BDD constructs that you can use in Spock. Let’s see how these parameterized tests are handled, like this:
def "the diamond function rejects illegal characters"() { when: diamond.of(c) then: thrown IllegalArgumentException where: c << generator.chars() .findAll { ! ('A'.. 'Z').contains(it) } .take(100) }Copy the codeThis is a typical Spock structure. We have a block of code called WHERE that takes one parameter. It uses this snazzy left-indent syntax to get the iterated data source.
What will happen is that the test will run once on every value retrieved from the variable, and that value will be used throughout the test. This is great for property-based testing; Or the case where you have a set of values that are used by the boundary use case, which is also very helpful.
Iterative – Spek2:05 ()
Now let’s look at an example of a similar situation in Spek:
describe("handling invalid characters") { chars().filter { it ! in 'A'.. 'Z' } .take(100) .forEach { c -> it("throws an exception") assertFailsWith(IllegalArgumentException::class) { diamond.of(c) } } }Copy the codeI like the simplicity of the style because it reminds me of Jasmine. I do a lot of front-end work in my job, I’ve always been partial to Jasmine and I haven’t found anything like that structure on the JVM since. Spek can do that.
We used for-each iterations between our Describe code blocks. We can have a describe, or a context, or any number of iterations. So the test defined on the IT block runs once on every value that comes up, and if you look at it inside the IDE or on the command line, you’ll see that each value has an independent test result.
Mark the test name – Spock(3:07)
When you do this, obviously, you’ll get the same IT every time, which of course is not optimal. One really cool thing Spock allows you to do is to add this Unroll annotation to your tests, and then you can replace their values with the names of those tests, so that when your report is generated, or your IDE test runner is working, if a single use case fails, You can tell which one it is.
@Unroll def "the diamond function rejects #c"() { when: diamond.of(c) then: thrown IllegalArgumentException where: c << generator.chars() .findAll { ! ('A'.. 'Z').contains(it) } .take(100) }Copy the codeAnnotate test names and nesting – Spek(property)
In Spek, this is easier because you can use string interpolation in your describe or IT code blocks. Very easy:
describe("handling invalid characters") { chars().filter { it ! in 'A'.. 'Z' } .take(100) .forEach { c -> it("throws an exception for '$c'") assertFailsWith(IllegalArgumentException::class) { diamond.of(c) } } }Copy the codeOne really cool thing that YOU can do with Spek that Spock can’t do is this useless little diamond cutter, and it’s something I’ve been struggling to do. For those of you familiar with this conundrum, implement nested iterations: you have a set of tests that need to run on a level of iteration, and the rest of the tests need to run within the iteration.
('B'.. 'Z').forEach { c -> describe("the diamond of '$c'") { val result = diamond.of(c) it("is square") { assert(result.all { it.length == rows.size }) } ('A'.. c).forEach { rowChar -> val row = rowChar - 'A' val col = c - rowChar it("has a '$rowChar' at row $row column $col") { assertEquals(rowChar, result[row][col]) } } } }Copy the codeHere we want to test the production of different characters, but in it we want to verify that there is a pattern to mashing other characters. So you need to nest an iteration that you can’t do with the Spock structure I just showed you, because you need to get a hierarchy of parameters.
Tabular iteration – Spock(now)
A Spock does fine, but Spek doesn’t do tabular data. In addition to the left-indent operator we saw, Spock lets you define multiple variables at the head of a column, under which you can provide tables of data.
@Unroll
def "the diamond of #c has #height rows"() {
expect:
diamond.of(c).size == height
where:
c | height
'A' | 1
'B' | 3
'C' | 5
}
Copy the codeAnother cool feature is that IntelliJ IDEA will help you align these tables correctly. You can then enter multiple parameters for your test. Spek can’t support this right now, but I know it’s on their list.
The best thing you can do now is:
for ((c, height) in mapOf('A' to 1, 'B' to 2, 'C' to 3)) {
describe("the diamond of '$c") {
val result = diamond(c)
it("has $height rows") {
assertEquals(height, result.size)
}
}
}
Copy the codeThe reason we can get around this is because our data has only two dimensions and we can iterate over a map to form our Describe and IT blocks. If you have three or more dimensions of data, this becomes more troublesome, although you can define some data classes and then iterate over the collection of those classes. You can do that, but it’s not as clean as what you do in Spock.
Pile driving – Mockito meets Kotlin(5:45)
Mocks was an interesting topic for Kotlin, and even more so for Spek for several reasons. So let’s look at how to implement the Spek class using Mockito.
describe("publishing events") {
val eventBus = EventBus()
val subscriber = mock(Subscriber::class.java) as Subscriber<ExampleEvent> // can't infer generic type
beforeEach {
eventBus.post(ExampleEvent("test running"))
}
it("should send the event to the subscriber") {
verify(subscriber)
.onEvent(argThat { // returns null so fails at runtime
(it as ExampleEvent).description == "test running" // can't infer generic matcher argument
})
}
}
Copy the codeHere we have some difficult things. First of all, we’re doing a clunky, Java-like conversion, because we only have these generic types that we expect to pile, and Kotlin doesn’t know the runtime types because they’re removed. So we need to re-convert in our assertions. We re-validate on the Mockito staked object.
We don’t have type inference support, so we lose type information for all generic types, which isn’t good.
Because Mockito’s matches all return NULL, none of the top layers work, and the runtime fails because Kotlin checks for NULL rigorously.
As far as I know, Kotlin doesn’t have a piling framework. Of course, there must be some Mockito that supports Kotlin, so if you add this library to your Gradle build, you will get a Kotlin version of the Mockito structure. Now this is a very neat combination.
Repositories {jCenter ()} dependencies {compile "com.nhaarman:mockito-kotlin:0.4.1"}Copy the codedescribe("publishing events") {
val eventBus = EventBus()
val subscriber: Subscriber<Event> = mock() // reified generics allow type inference
beforeEach {
eventBus.post(Event("test running"))
}
it("should send the event to the subscriber") {
verify(subscriber).onEvent(argThat { // returns dummy value
description == "test running" // type inference on receiver
})
}
}
Copy the codeWe now have type inference on the pile, so the variable subscriber is declared as a generic. Kotlin’s embodied generics enables us to figure out what type is in the pile factory. You don’t have to specify classes, you don’t throw away any type information for generics, and you don’t have to convert anything.
We have a defined receiver in the validation block so that we don’t need it. . In the argument passed to the matcher, we do type inference again, so we don’t need to convert this argument.
This is really much better, the matcher returns an invalid value, so we no longer have that null problem. This is a very small library that simplifies using Mockito in Kotlin and Spek. If you are using any kind of double test, I highly recommend you take a look at this library.
TCKs – A complete set of tests(“)
TCKs is a technology compatibility test package. The idea behind it is that many different implementations of the same thing need to follow a particular set of standards.
A simple example is Java lists. We have arrayLists, we have linkedLists, and there are a lot of different implementations. But they all need to follow a specific set of rules, like their return values need to be in insert order, some nulls are allowed, some nulls are not, but they can all be compared and their hash values need to be consistent. There is a consistent set of rules to follow.
In this way, the test cases for these rules only need to be written once, and then your implementation needs to test the same set of tests without allowing changes to the tests.
TCK – Spock (9:01)
Let’s take a look at the Spock is how to deal with this thing, of course, you will define an abstract test class, there are some abstract factory method to generate the test need to class, write a set of test cases to be able to access the object, and then you can use a specific class to extend these test interface classes, these specific class will overload test factory method.
abstract class PubSubSpec extends Specification {
@Subject abstract EventBus factory()
// ... tests appear here
}
class SyncPubSubSpec extends PubSubSpec {
@Override EventBus factory() {
new EventBus()
}
}
class AsyncPubSubSpec extends PubSubSpec {
@Override EventBus factory() {
new AsyncEventBus(newSingleThreadExecutor())
}
}
Copy the codeThey can also add their own test cases if they have extensions for an implementation. Anyway, when you run your set of tests, all of these tests defined in the abstract base class, they’re going to need subclasses or subclasses of subclasses or something.
You can quickly build a set of acceptance tests by implementing interfaces.
TCK – Spek (therefore)
Now one of the interesting things about Spek is that the tests themselves are defined in the static initialization function of a Java class. You don’t have any functions that you can override to do anything like that.
abstract class PubSubTck(val eventBus: EventBus) : Spek ({
describe("publishing events") {
// ... tests appear here
}
})
class SyncPubSubSpec
: PubSubTck(EventBus())
class AsyncPubSubSpec
: PubSubTck(AsyncEventBus(newSingleThreadExecutor()))
Copy the codeAs mentioned above, you can define class-level methods in the Spek class, but these tests can’t see those methods, it can only see what’s in the companion object. Obviously, companion objects such as Java static methods cannot be inherited.
One solution is that you can extend Spek’s base classes with an abstract class, but define a property for your eventBus.
You can define a property, then its implementation needs to provide constructors, and then write all your test DSLS normally. This way you have specific extensions to these implementations, and you can simply pass values to their constructors, which can be accessed in subsequent tests.
This is fine, as long as you can express the construction process of your test class in a concise enough form, as you do in constructors. If you can’t, you probably need to use some kind of factory method. Lambda is also an obvious solution you can use. Once again, you have an abstract Spek class that has a factory method to pass empty arguments to your test class, and then if you want, Each test can create a new instance and has a value that is passed in when the factory method is called, which is then provided by the specific test implementation of those extension classes.
abstract class PubSubTck(val factory: () -> EventBus) : Spek ({
val eventBus = factory()
describe("publishing events") {
// ... tests appear here
}
})
class SyncPubSubFactorySpec
: PubSubTck(::EventBus)
class AsyncPubSubFactorySpec
: PubSubTck({ AsyncEventBus(newSingleThreadExecutor()) })
Copy the codeHere we can use Kotlin’s snappy method reference syntax to reference the standard EventBus constructor. If it’s a zero-argument constructor, you can do this.
We used a lambda closure here because it has a more complex constructor and some more complex Settings. It works very well. This lets you bypass the fact that you can’t use inheritance and virtual functions in Spek classes, but you can pass factories and properties to constructors.
The illustration assertion(now)
This is my biggest hope for Spek: graphical assertions. This is a cool feature that Spock introduced, and the Groovy language has adopted it all in its Assert keyword.
In Spock’s expected code block, anything is considered Boolean and is treated as an assertion, so this is an assertion.
expect:
order.user.firstName == "Rob"
Copy the codeIf this assertion fails, Spock gives us really nice output like this:
order.user.firstName == "Rob"
| | | |
| | rob false
| | 1 difference (66% similarity)
| | (r)ob
| | (R)ob
| [firstName:rob, lastName:fletcher]
Order<#3243627>
Copy the codeEach step on both sides of the binary operator of an expression is broken down, and you can see each individual object. You can figure out what’s going on. That way, you can scratch your head and take a look and think, “OK, why is this name wrong? Is it the wrong user, in the wrong order?”
It’s hard to see why without this nice graph breakdown to help clarify things, but you still need a good implementation of toString() on these objects.
In this regard, Spock’s implementation is more tidy, and if the toString() implementation is the same, it can also point out problems when evaluating equality. If you have a string that’s a number one, and you compare it to an integer one, even though they print the same, they don’t compare equally, it will tell you exactly why it failed because of the type difference.
When this powerful assertion feature is adopted by the Groovy language, the Groovy assertion keyword has it by default. ScalaTest also has a feature that you can use to implement a similar set of features, a similar set of features.
This is also the number one feature I wish Spek had added, because I’ve been writing Spock tests for years, and not having this feature would have been one of my toughest things. So I love Spek, and for me, it’s the number one feature Speck is missing right now.
This article covers some interesting and useful test cases for both testing frameworks, which I particularly want to highlight. Thank you.