Want to write good unit tests in go? Don’t panic… or should you?
Over the last few weeks I had some free time to improve my graphql parsing library. There were a lot of inefficiencies in the lexer and in the parser as well so I had to refactor the complete codebase. Test coverage was important to me from the beginning, so refactoring to a state where everything still works as expected was no problem. Here’s a link if you are interested in parsing, lexing or graphql:
While refactoring, I came across many situations where I was fighting against my tests. Due to the way I’ve implemented my unit tests it wasn’t always easy to find the actual issue.
In the end I discovered/developed a style which I call “check based panic testing”. I’m confident this way of testing will make further refactoring a lot easier.
To get everyone on the same page I’d like to outline all the methods and styles of unit testing I’ve used, what their pro’s and con’s are and why I finally chose check based panic testing over the others. I hope you can learn something from my golang testing adventures.
Plain and simple tests
func TestSomething(t *testing){
parser := NewParser()
parser.l.SetInput(`
"describes direction"
enum Direction {
NORTH
}`)
output,err := parser.doSomeThing()
if err != nil {
t.Fatal(err)
}
if output != "Direction" {
t.Fatal("unexpected output")
}
}
func TestAnother(t *testing){
parser := NewParser()
parser.l.SetInput(`
enum Direction {
NORTH
}`)
output,err := parser.doAnother()
if err != nil {
t.Fatal(err)
}
if output != "NORTH" {
t.Fatal("unexpected output")
}
}Plain go tests are a good start. It’s better than nothing yet when writing a lexer and a parser you will face quite a lot of repetitiveness when trying to test extensively. You’ll have to provide different inputs to the same method and verify all the results. What is good about plain go tests is that you’ll easily find a test when there’s a problem (which is not the case for table driven tests) as the stack-trace directly leads you to the issue.
Pros: no dependencies, easy setup
Cons: a lot of copy paste for extensive testing
Table driven tests
The holy grail of unit testing or so I thought.
func TestSomething(t *testing.T) {
for _,tt := range [...]struct{
name string
input string
expectedOutput string
}{
{
name: "test something with some input",
input: `
"describes direction"
enum Direction {
NORTH
}`,
expectedOutput: "Direction",
},
{
name: "test something with some input",
input: `
"describes direction"
enum Direction {
NORTH
}`,
expectedOutput: "Direction",
},
} {
t.Run(tt.name, func(t *testing.T) {
parser := NewParser()
parser.l.SetInput(tt.input)
output,err := parser.doSomeThing()
if err != nil {
t.Fatal(err)
}
if output != tt.expectedOutput {
t.Fatal("unexpected output")
}
})
}
}A typical table driven test might look like this. As you can see it’s a lot less repetitive compared to plain old tests. Table driven tests have better structure yet they seem a bit messy. It’s not easy to grasp everything at once. Self made table driven tests are a bit hard to read because they don’t read top down, like a story. But that’s not all. Imagine you, like me, have to refactor a lot of code and many tests are red. What happens is that the stack-trace, starting at “t.Fatal” (in the loop) will not indicate which test failed. This is no problem for two or three test cases but once you fill many pages you’ll have to manually search for the case definition by its name. This process is time consuming and not a very desirable developer experience. Another con is the inflexibility of handwritten table driven tests, especially when it comes to assertions. What if you wanted to test cases where err must not be nil? You’d have to write another test for that one.
Pros: A lot less code to write
Cons: messy, bad developer experience when something went red, inflexible
Table driven tests with goblin & gomega
func TestTypeParser(t *testing.T) {
g := Goblin(t)
RegisterFailHandler(func(m string, _ ...int) { g.Fail(m) })
g.Describe("parseType", func() {
tests := []struct {
it string
input string
expectErr types.GomegaMatcher
expectValues types.GomegaMatcher
}{
{
it: "should parse a simple Named Type",
input: "String",
expectErr: BeNil(),
expectValues: Equal(document.NamedType{
Name: "String",
}),
},
{
it: "should parse a NonNull Named Type",
input: "String!",
expectErr: BeNil(),
expectValues: Equal(document.NamedType{
Name: "String",
NonNull: true,
}),
},
}
for _, test := range tests {
test := test
g.It(test.it, func() {
parser := NewParser()
parser.l.SetInput(test.input)
val, err := parser.parseType()
Expect(err).To(test.expectErr)
Expect(val).To(test.expectValues)
})
}
})
}I wasn’t happy with those messy table driven tests, goblin & gomega to the rescue! Thanks to the creator for these awesome libraries.
As you can see with the help of these two libraries our tests look a lot less messy. You still cannot read top down but thats okay because with gomega we have great flexibility when it comes to assertions. This makes testing for the error path easy and we don’t have to write additional tests. On the other hand we have to setup our dependencies and initialise goblin with the testing context. While the overall experience is better than self made table driven tests there’s still the issue with stack-traces. Goblin & gomega offer beautiful test reports but you’ll have to search the test cases by hand. This is still not the developer experience I’m looking for.
Pros: beautiful reporting, good structure
Cons: you’ll have to setup external dependencies, stack-traces don’t guide you to the test definition
Will ginkgo make us happy?
func TestLexer(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Lexer")
}var _ = Describe("Lexer.Read", func() {
type Case struct {
in string
out token.Token
expectErr types.GomegaMatcher
}
var lexer *Lexer
BeforeEach(func() {
lexer = NewLexer()
})
DescribeTable("Read Single Token", func(c Case) {
lexer.SetInput(c.in)
tok, err := lexer.Read()
if c.expectErr != nil {
Expect(err).To(c.expectErr)
} else {
Expect(err).To(BeNil())
}
Expect(tok).To(Equal(c.out))
},
Entry("should read integer", Case{
in: "1337",
out: token.Token{
Keyword: keyword.INTEGER,
Literal: "1337",
Position: position.Position{
Line: 1,
Char: 1,
},
},
}),
Entry("should read integer with comma at the end", Case{
in: "1337,",
out: token.Token{
Keyword: keyword.INTEGER,
Literal: "1337",
Position: position.Position{
Line: 1,
Char: 1,
},
},
}),
...
When you started using goblin & gomega you’ll love ginkgo.
Ginkgo makes your tests much more expressive. You’ll have to go all in with the DSL but this pays off big times. While it’s still a bit of a burden to setup everything you get beautifully structured tests that read top down in return. Good things aside, there are two aspects which I’m not quite happy about. First, ginkgo doesn’t report as beautiful as gomega but I could live with that. Second you’ll have to lookup tests by hand again. I really hate doing this! The Assertion stack-traces always end in the execution loop of the table. I’m really unhappy about this, especially when it costs me hours! The third con is another story.
Pros: beautiful test structure
Cons: setup, stack-traces are still off (which has more to do with the usage than with the library itself)
All these approaches have one problem in common. This has nothing to do with the tooling, it’s more about the structure of the tests itself. When it comes to testing large structs it’s often a lot of work to define exact assertions. If you have to refactor a lot of your code you might have to update all these structs. Here’s one such case:
{
it: "should parse an ObjectTypeDefinition with multiple FieldDefinition",
input: `Person {
name: [String]!
age: [ Int ]
}`,
expectErr: BeNil(),
expectIndex: Equal([]int{0}),
expectParsedDefinitions: Equal(ParsedDefinitions{
FieldDefinitions: document.FieldDefinitions{
document.FieldDefinition{
Name: "name",
Type: document.ListType{
Type: document.NamedType{
Name: "String",
},
NonNull: true,
},
},
document.FieldDefinition{
Name: "age",
Type: document.ListType{
Type: document.NamedType{
Name: "Int",
},
},
},
},
ObjectTypeDefinitions: document.ObjectTypeDefinitions{
{
Name: "Person",
FieldsDefinition: []int{0, 1},
},
},
}.initEmptySlices()),
},Did you notice initEmptySlices() at the end? It’s a helper function that init’s all nil slices as empty slice so that I don’t have to add all those initialisations by hand and gomega (assertions) is still happy. How to solve the problem?
Check based panic testing
Yep, sounds badass doesn’t it? Let’s see some code:
func TestParser_parseTypeSystemDefinition(t *testing.T) {
type checkFunc func(definition document.TypeSystemDefinition, parsedDefinitions ParsedDefinitions, err error)
checks := func(checkFuncs ...checkFunc) []checkFunc { return checkFuncs }
type Case struct {
input string
checks []checkFunc
}
run := func(t *testing.T, c Case) {
parser := NewParser()
parser.l.SetInput(c.input)
definition, err := parser.parseTypeSystemDefinition()
for _, check := range c.checks {
check(definition, parser.ParsedDefinitions, err)
}
}
hasError := func() checkFunc {
return func(definition document.TypeSystemDefinition, parsedDefinitions ParsedDefinitions, err error) {
if err == nil {
panic(fmt.Errorf("hasError: expected error, got nil"))
}
}
}t.Run("Schema", func(t *testing.T) {
run(t, Case{
input: `
schema {
query: Query
mutation: Mutation
}
`,
checks: checks(
hasNoError(),
hasSchema("Query", "Mutation", ""),
),
})
})
t.Run("ScalarTypeDefinitions", func(t *testing.T) {
run(t, Case{
input: `
"this is a scalar"
scalar JSON
scalar testName @fromTop(to: "bottom")
"this is another scalar" scalar XML
`,
checks: checks(
hasNoError(),
hasScalar("JSON", "this is a scalar"),
hasScalar("testName", "", "fromTop"),
hasScalar("XML", "this is another scalar"),
),
})
})
The beginning looks a bit messy, especially in this grey box, but I’m absolutely okay with that. You’ll define a few helper functions. Then you setup a run method and the “checks” which might be reusable across tests. The test case declarations are pure gold. Top down readable in pure and simple english words. 100% flexibility without too much repetition. Some checks might be a bit copy pasty but this enables easy to read tests which makes it worth it. In addition to that you probably noticed the keyword panic. “t.Run” gives every test case its unique entry point so all tests will run through even if one is panicking. But that’s not what’s this about. “panic()” in this case helps building a valid stack-trace back to the test declaration. This means that any capable IDE, like Goland, will happily guide you directly to the failing test by clicking on the link. No more copy paste, no more manually searching. This is it! Thanks for the inspiration to this guy:
…who got the idea from the awesome standard library:
In my eyes the subtraction of the table and the addition of panics make this idea a really great developer experience. Reporting isn’t comparable to gomega but I can live with that when stack-traces guide me to the issue. In the end you’ll have to make some tradeoffs when deciding how you structure your tests.
Pros: no dependencies, the best test case reading experience, great flexibility, great developer experience when tests fail
Cons: It’s a good bit of work to setup all checks (still you might reuse them)
Conclusion
Why do we need to take the time to write meaningful tests? Tests document the expected behaviour of our applications. Tests help the next guy working on the project understand your intentions. If you manage to write your code and tests in a way that enables easy refactoring the next guy will thank you! He’ll add the next feature, get all lamps green and business will be happy.
I hope you were able to learn something about testing from this article. If you feel this can be further improved I’m open to your suggestions. Other than that I’m interested in your opinion on my graphql parser/lexer. If you find a bug or think I could improve something I’d be happy hearing from you. Here’s the link again:
