TLDR: customizing zig test output is easy, if you’re willing to get your hands dirty
Using custom Zig test runners (part 1)
Lastly, the Zig build system is relatively simple and self-contained, and reading its source code will allow you to master it.
– comments from
zig init’sbuild.zig
Zig has testing support builtin to the language, butttt…
it’s pretty minimal and comes with some interesting choices.
Note: Zig is an evolving language. This article is written for Zig 0.16.0.
Table of Contents
- The default Zig testing experience
- How Zig runs tests
- What's a test runner?
- A minimal test runner
- TAP
- Next time
The default Zig testing experience
When you create a new zig project, you get tests by default.
$ zig init
info: created build.zig
info: created build.zig.zon
info: created src/main.zig
info: created src/root.zig
info: see `zig build --help` for a menu of options
If we look in the build.zig, we can see the tests are configured explicitly.
Here’s the full build.zig it generates, stripped of its noisy comments:
const std = @import("std");
pub fn build(b: *std.Build) void {
const target = b.standardTargetOptions(.{});
const optimize = b.standardOptimizeOption(.{});
const mod = b.addModule("test_exmaple", .{
.root_source_file = b.path("src/root.zig"),
.target = target,
});
const exe = b.addExecutable(.{
.name = "test_exmaple",
.root_module = b.createModule(.{
.root_source_file = b.path("src/main.zig"),
.target = target,
.optimize = optimize,
.imports = &.{
.{ .name = "test_exmaple", .module = mod },
},
}),
});
b.installArtifact(exe);
const run_step = b.step("run", "Run the app");
const run_cmd = b.addRunArtifact(exe);
run_step.dependOn(&run_cmd.step);
run_cmd.step.dependOn(b.getInstallStep());
if (b.args) |args| {
run_cmd.addArgs(args);
}
const mod_tests = b.addTest(.{
.root_module = mod,
});
const run_mod_tests = b.addRunArtifact(mod_tests);
const exe_tests = b.addTest(.{
.root_module = exe.root_module,
});
const run_exe_tests = b.addRunArtifact(exe_tests);
const test_step = b.step("test", "Run tests");
test_step.dependOn(&run_mod_tests.step);
test_step.dependOn(&run_exe_tests.step);
}
Aside:
Yes, the default starting template includes a typo (“test_exmaple”).
No, Zig doesn’t want your PRs to fix it (example).
From that, we can see that there’s a test step configured, so we can run tests like so:
zig build test
Surprisingly, while you can see things compiling here on the first run, this results in no output.
Which is a bit weird for a test runner.
There’s also no options for other typical test runner things, like running tests in random order, failing fast, etc.
In short, it’s good enough for some use-cases, but wayy too minimal for most use-cases.
Turns out, what we’re actually talking about here isn’t Zig itself, but the default Zig test runner.
How Zig runs tests
The comments from the build.zig that zig init generated say this:
The Zig build system is entirely implemented in userland, which means that it cannot hook into private compiler APIs.
All compilation work orchestrated by the build system will result in other Zig compiler subcommands being invoked with the right flags defined.
You can observe these invocations when one fails (or you pass a flag to increase verbosity) to validate assumptions and diagnose problems.
Lastly, the Zig build system is relatively simple and self-contained, and reading its source code will allow you to master it.
So, let’s do that:
mkdir -p ~/code/forks
git clone -b 0.16.0 https://codeberg.org/ziglang/zig ~/code/forks/zig
If we look for how tests work
rg 'fn addTest' ~/code/forks/zig
we find that it’s pretty straightforward (lib/std/Build.zig)
/// Creates an executable containing unit tests.
///
/// Equivalent to running the command `zig test --test-no-exec ...`.
///
/// **This step does not run the unit tests**. Typically, the result of this
/// function will be passed to `addRunArtifact`, creating a `Step.Run`. These
/// two steps are separated because they are independently configured and
/// cached.
pub fn addTest(b: *Build, options: TestOptions) *Step.Compile {
return .create(b, .{
.name = options.name,
.kind = if (options.emit_object) .test_obj else .@"test",
.root_module = options.root_module,
.max_rss = options.max_rss,
.filters = b.dupeStrings(options.filters),
.test_runner = options.test_runner,
.use_llvm = options.use_llvm,
.use_lld = options.use_lld,
.zig_lib_dir = options.zig_lib_dir,
});
}
So that immediately tells us
- This just delegates to
zig test - We can provide our own test runner
What's a test runner?
When we ask zig test for help:
$ zig test --help
it unhelpfully spams us with a page or two of help for every zig command.
We tirelessly sift through all that to find the test stuff:
... so much output...
...
Test Options:
--test-filter [text] Skip tests that do not match any filter
--test-cmd [arg] Specify test execution command one arg at a time
--test-cmd-bin Appends test binary path to test cmd args
--test-no-exec Compiles test binary without running it
--test-runner [path] Specify a custom test runner
--test-execve Runs the test binary with execve if available instead of as a child process
...
So the --test-runner command sounds like the thing we want.
Back in our zig init example, we can pass --verbose to see what zig build test actually does:
$ zig build test --verbose
And the magic is (mostly) revealed, it’s just simple zig test commands like the comment said:
/path/to/your/zig/0.16.0/zig test -Mroot=/tmp/test-exmaple/src/root.zig --cache-dir .zig-cache --global-cache-dir /home/your-user/.cache/zig --name test --zig-lib-dir /path/to/your/zig/0.16.0/lib/ --listen=-
/path/to/your/zig/0.16.0/zig test -ODebug --dep test_exmaple -Mroot=/tmp/test-exmaple/src/main.zig -Mtest_exmaple=/tmp/test-exmaple/src/root.zig --cache-dir .zig-cache --global-cache-dir /home/your-user/.cache/zig --name test --zig-lib-dir /path/to/your/zig/0.16.0/lib/ --listen=-
./.zig-cache/o/8b59c61312f32614313e27f50c701b2a/test --cache-dir=./.zig-cache --seed=0x8965f4b4 --listen=-
./.zig-cache/o/5764db233d7c4e8d8dc6c0c018445a6d/test --cache-dir=./.zig-cache --seed=0x8965f4b4 --listen=-
We can sanity-check that ourselves:
$ zig test src/main.zig
All 2 tests passed.
1 fuzz tests found.
$ zig test src/main.zig
All 2 tests passed.
1 fuzz tests found.
But what does the test runner do, exactly?
A minimal test runner
If we look around in Zig’s source code, we can find this test/standalone/test_runner_path/test_runner.zig example:
const std = @import("std");
const builtin = @import("builtin");
pub fn main() void {
var ok_count: usize = 0;
var skip_count: usize = 0;
var fail_count: usize = 0;
for (builtin.test_functions) |test_fn| {
if (test_fn.func()) |_| {
ok_count += 1;
} else |err| switch (err) {
error.SkipZigTest => skip_count += 1,
else => fail_count += 1,
}
}
if (ok_count != 1 or skip_count != 1 or fail_count != 1) {
std.process.exit(1);
}
}
and its test/standalone/test_runner_path/build.zig:
const std = @import("std");
pub fn build(b: *std.Build) void {
const test_step = b.step("test", "Test the program");
b.default_step = test_step;
const test_exe = b.addTest(.{ .root_module = b.createModule(.{
.target = b.graph.host,
.root_source_file = b.path("test.zig"),
}) });
test_exe.test_runner = .{
.path = b.path("test_runner.zig"),
.mode = .simple,
};
const test_run = b.addRunArtifact(test_exe);
test_step.dependOn(&test_run.step);
}
We saw earlier that the .test_runner stuff just eventually gets passed to zig test invocations, so if we do that ourselves, it should work, right?
So let’s add a minimal test_runner.zig based on that:
const std = @import("std");
const builtin = @import("builtin");
pub fn main() void {
var fail_count: usize = 0;
for (builtin.test_functions) |test_fn| {
if (test_fn.func()) |_| {
std.debug.print("pass\n", .{});
} else |err| switch (err) {
error.SkipZigTest => {
std.debug.print("skip\n", .{});
},
else => {
fail_count += 1;
std.debug.print("fail\n", .{});
},
}
}
if (fail_count > 0) {
std.process.exit(1);
}
}
And run it ourselves:
$ zig test src/root.zig --test-runner test_runner.zig
pass
Great, we have the Hello, world of Zig test runners.
Now let’s make something more useful.
TAP
If you haven’t heard of it, TAP is a minimal test output format originally used in Perl.
The project website explains it best:
TAP, the Test Anything Protocol, is a simple text-based interface between testing modules in a test harness. It decouples the reporting of errors from the presentation of the reports.
…
Here’s what a TAP test stream looks like:
1..4 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file not ok 4 - Summarized correctly # TODO Not written yet
Since it’s super simple and only depends on stdout, we can wire that up easily:
pub fn main() void {
std.debug.print("1..{d}\n", .{builtin.test_functions.len});
var fail_count: usize = 0;
for (builtin.test_functions, 0..) |test_fn, index| {
if (test_fn.func()) |_| {
std.debug.print("ok {d} {s}\n", .{ index + 1, test_fn.name });
} else |err| switch (err) {
error.SkipZigTest => {
std.debug.print("ok {d} {s} # skip\n", .{ index + 1, test_fn.name });
},
else => {
fail_count += 1;
std.debug.print("not ok {d} {s} ({})\n", .{ index + 1, test_fn.name, err });
},
}
}
if (fail_count > 0) {
std.process.exit(1);
}
}
Let’s add some more example tests to src/root.zig first:
test "skip" {
return error.SkipZigTest;
}
test "fail" {
try std.testing.expectEqual(true, false);
}
We can see it outputs things correctly:
$ zig test src/root.zig --test-runner test_runner.zig
1..3
ok 1 root.test.basic add functionality
ok 2 root.test.skip # skip
expected true, found false
not ok 3 root.test.fail (error.TestExpectedEqual)
error: the following test command failed with exit code 1:
.zig-cache/o/65544eadd8d13020b86b4159748e0a1c/test --seed=0xbf6e8c8e
Then, just to prove it’s valid TAP format, let’s validate it against a third-party TAP consumer:
sudo apt update -y
sudo apt install -y pipx
pipx install tap.py
just pipe it in:
$ zig test src/root.zig --test-runner test_runner.zig 2>&1 | tap
.sF
======================================================================
FAIL: <file=stream>
root.test.fail (error.TestExpectedEqual)
----------------------------------------------------------------------
----------------------------------------------------------------------
Ran 3 tests in 0.000s
FAILED (failures=1, skipped=1)
To see what it looks like when it passes, comment out the failing test, then run it again:
$ zig test src/root.zig --test-runner test_runner.zig 2>&1 | tap
.s
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK (skipped=1)
Great! It may be simple, but it works.
Next time
As we’ve learned, zig test and zig build test aren’t complicated.
Instead, they’re simple enough that you can adapt them however you need to, even if that requires a bit of extra work.
Future improvements:
And in case you don’t want to write your own, feel free to use mine (this article is based on it):
(that’s bb4df010 at the time of this writing)