A better systemtest runner

Group Phase Implementation
1

Hi!

Sorry for the long post, but this may be useful.

Because I wanted to see what actually happens during the simulation, I built a small visualizer for system tests. It is just a wrapper around the built-in system test runner, with a visual overview of all system tests you have, their results, and also a small visual step-by-step runner for the simulation (based purely on your logs, so any bugs in what you log would also appear here).

I could’ve probably spent that time on something more useful for our group, but it’s done now and I want to share it here.

In some cases, this makes it easier to inspect what exactly is going on and helps you “remember” orders taken.

Screenshots

Screenshot 1
Screenshot 11920×1027 32.3 KB

Screenshot 2
Screenshot 21920×1027 279 KB

How I built this

This is a bit technical (and the writeup is still a bit short, maybe I’ll extend it in the future), but if you’re interested, this is how I built it:

How I built this

My first step was trying to recreate the original SE Lab system test library exactly, whose source code I do not have access to.

This library was written in Kotlin, and compiled to a JAR file. A JAR file contains multiple .class files, which are so-called “Java bytecode”. You can imagine these (this is extremely simplified) as similar to assembly code.

Unlike with actual assembly, there are very good “decompilers” (tools that can convert binaries/class files back to source code) for Java. These are designed for Java, so it doesn’t give me exact Kotlin code, but some decompilers, like Vineflower, can also work with Kotlin.

These helped getting “somewhat readable” code, although some could not be decompiled. The code was also not really easy to read, but it was at least somewhat close to the library. I started cleaning it up and when I only had bytecode, guessing what the method does (or trying to understand the bytecode).

After quite a bit of work, I managed to get some source code that compiles exactly to a JAR file that is byte-identical to what the SE Lab team provides us with.

At this point, I had quite readable source code I could use.

The main app is a CLI interface to trigger arbitary tests. I extended it a bit to give it exactly one new flag: --server, which starts a server mode.

This allows me to start an HTTP server that exposes a simple API to trigger tests. This API simply exposes internal pre-existing functions to run tests as HTTP endpoints.

With this API, I used Nuxt to build a small web app that can trigger tests and visualize the simulation, then I embedded that in the application so it is automatically visible when you visit the interface.

In short:

  • This is not vibecoded (otherwise, the quality would probably be higher).
  • This should behave identical to the official library if server mode is not used (I made this quite sure, but can’t guarantee it).

Features

  • View logs of individual system tests in a nice UI
  • View the config of a system test
  • Check the config of a system test against the given schemas
  • Visualize the current state during any step of a system test (based on the logs, not your actual simulation state)
  • View system test results
  • Shows crashes during test execution more clearly

What doesn’t work

  • Not all log messages are properly handled in the visualizer yet, so sometimes it shows an invalid state
  • The visualizer guesses state based on your logged messages, it does not show internal state
    • For example, table merging is not supported, because splitting them is not logged, so we can’t know when merged tables are split again
    • Pantry contents are based on purchases you log, and on logs about throwing away ingredients, and what is ordered. So whenever a dish is ordered, the ingredients for it are removed from what is shown as pantry contents. This may not reflect what your simulation actually does.
    • Dishes that are finished, but the customers leave, are not removed, so they are still shown as “finished” in Kitchen

How to use this

You can download my adjusted version, with source code for the back-end here. This is a ZIP file encrypted with password 5Ww8qbM4ZzLQZxVspa6fUbkyxEYk87yw

Because the ZIP is encrypted, you may need something like 7Zip on Windows to extract the encrypted file. Ark from KDE can extract it properly, I didn’t test anything else.

To use this, you first need to copy the better-systemtest-runner.jar from the ZIP to your project’s main directory directly, and then modify your build.gradle.kts file.

After you’ve added the file, you need to open your build.gradle.kts file and, in the dependencies block, replace the following:

    "systemtestImplementation"("selab.systemtest:systemtest-api:0.7.5") { isChanging = true }
    "systemtestImplementation"("org.jetbrains.kotlinx:kotlinx-serialization-json:1.9.0")
    "systemtestImplementation"("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.10.2")

with

    "systemtestImplementation"(files("better-systemtest-runner.jar"))
    "systemtestImplementation"("org.jetbrains.kotlinx:kotlinx-serialization-json:1.9.0")
    "systemtestImplementation"("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.10.2")
    "systemtestImplementation"(platform("org.http4k:http4k-bom:6.58.0.0"))
    "systemtestImplementation"("org.http4k:http4k-core")
    "systemtestImplementation"("org.http4k:http4k-server-netty")

Please note: This is a drop-in replacement for the original system test library, but commiting these changes could lead to failed builds/runs on the server. So this can be used for local testing, but you should not commit these changes. You can tell git to ignore this by git update-index --assume-unchanged

Also, it is not the official library. I am pretty sure it behaves identical, but if your tests pass with this, but not the official library, don’t blame me please.

With watchexec (automatically restart)

I personally use watchexec to automatically restart the visualizer. You can find install instructions on their GitHub page. Once you have it installed, you need to run two things in parallel (in two terminals):

./gradlew jar -t

and

watchexec -w libs/selab.jar -r -- java -cp libs/selab.jar de.unisaarland.cs.se.selab.systemtest.selab26.MainKt --jar libs/selab.jar --run reference_implementation --timeout 10 --server

You can then visit the visualizer at http://localhost:9000 in your browser.

Without automatic restart

If you don’t want to use watchexec, you can also just run the following commands (in order):

./gradlew jar

java -cp libs/selab.jar de.unisaarland.cs.se.selab.systemtest.selab26.MainKt --jar libs/selab.jar --run reference_implementation --timeout 10 --server

You can then visit the visualizer at http://localhost:9000 in your browser.

However, please be aware that you need to cancel & re-run this command every time you change the code or a test.

If you have any feedback on this experiment, please let me know.

Changelog

For each new version, I replaced the link above, but here you can download older versions here.

Version 2

Added keyboard shortcuts to visualizer:

CTRL + A: Proceed to next log line
CTRL + Enter: Skip to beginning of next tick
CTRL + E: Skip to beginning of tick 24
CTRL + Backspace: Resets the visualizer to the beginning

Version 1

Initial release

12 Likes