ScalaCLI.md Spec

Turn each Scala CLI snippet from your markdown documentation into a test case and each markdown document into a test suite.

Usage

Scala CLI

Create test-snippets.scala (use at least Java 11!):

//> using scala 3.3.3
//> using jvm temurin:1.11.0.23
//> using dep "com.kubuszok::scala-cli-md-spec:0.1.1"
import com.kubuszok.scalaclimdspec.*
@main def run(args: String*): Unit = testSnippets(args.toArray) { cfg =>
  new Runner.Default(cfg) // or provide your own :)
}

then run it with Scala CLI:

# run all tests
scala-cli run test-snippets.scala -- "$PWD/docs"
# run only tests from Section in my-markdown.md
scala-cli run test-snippets.scala -- --test-only="my-markdown.md#Section*" "$PWD/docs"

To see how one can customize the code to e.g. inject variables or use the newest library version in an arbitrary markdown documentation generator see Chimney's example.

Coursier

If you are not providing any modification, you can run it straight from the Coursier:

# run all tests
coursier launch com.kubuszok:scala-cli-md-spec_3:0.1.1 -M com.kubuszok.scalaclimdspec.testSnippets -- "$PWD/docs"
# run only tests from Section in my-markdown.md
coursier launch com.kubuszok:scala-cli-md-spec_3:0.1.1 -M com.kubuszok.scalaclimdspec.testSnippets -- --test-only="my-markdown.md#Section*" "$PWD/docs"

Rules of the game

1. each markdown is its own test suite

2. by default only Scala (and Java) snipets containing //> using are considered tests

   • other snippets are considered pseudocode and are ignored

     // will be tested
     //> using scala 3.3.3
     println("yolo")

     // will NOT be tested
     println("yolo")

3. by default snippets are tested for the lack of errors

   • by the lack of errors we mean that Scala CLI returns 0

     // should pass
     //> using scala 3.3.3
     println("yolo")

     // thou shall NOT pass!
     //> using scala 3.3.3
     throw Exception("yolo")

   • if there is // expected output: followed by inline comments in the immediate next lines, the snippet will be expected to succeed and its standard output will be expected to contain the content provided in these comments

     // should pass
     //> using scala 3.3.3
     println("yolo")
     // expected output:
     // yolo

     // thou shall NOT pass!
     //> using scala 3.3.3
     println("yolo")
     // expected output:
     // eee macarena!

   • if there is // expected error: followed by inline comments in the immediate next lines, the snippet will be expected to fail and its standard error will be expected to contain the content provided in these comments

     // should pass
     //> using scala 3.3.3
     throw Exception("yolo")
     // expected error:
     // yolo

     // should pass
     //> using scala 3.3.3
     summon[String]
     // expected error:
     // No given instance of type String was found

     // thou shall NOT pass!
     //> using scala 3.3.3
     println("yolo")
     // expected error:
     // yolo

4. by default each snippet is a standalone Scala snippet, it will be tested in a separate directory, containing a single snippet.sc file

   • multiple pieces of code can be combined into one multi-file snippet with: // file: [filename] - part of [example name used for grouping] syntax - e.g. // file: filename.scala - part of X example would group all X example snippets in the same directory, and use filename.scala as a filename for this particular piece of code

     // file: model.scala - part of multi-file
     case class Model(a: Int)

     // file: example.sc - part of multi-file
     println(Model(10))
     // expected output:
     // Model(10)

     With multi-file //> using is not required to consider the code as a Scala CLI test. Remember that to make it work, like with normal Scala CLI app, there should be either exactly one .sc file or only .scala files with exactly one explicitly defined main.

   • if at least one file in a multi-file snippet has a name ending with .test.scala, then scala-cli test [dirname] will be used unstead of scala-cli run [dirname] (useful for e.g. defining macros in the compile scope and showing them in the test scope since Scala CLI is NOT multi modular and you cannot demonstrate macros in another way)

     // file: macro.scala - part of macro example
     //> using scala 3.3.3
     
     object MyMacro:
       inline def apply[A](a: A): Unit = ${ applyImpl[A]('a) }
     
       import scala.quoted.*
       def applyImpl[A: Type](a: Expr[A])(using Quotes): Expr[Unit] = '{ () }

     // file: macro.test.scala - part of macro example
     //> using test.dep org.scalameta::munit::1.0.0-RC1
     
     class MacroSpec extends munit.FunSuite {
       test("Macro(a) should do thing") {
         assert(MyMacro("wololo") == ())
       }
     }

   • Java snippets should not only use java in markdown, but also define // file: filename.java - part of ...

     // file: MyEnum.java - part of java enum example
     enum MyEnum {
       ONE, TWO;
     }

     // file: snippet.sc - part of java enum example
     println(MyEnum.values())

5. if --test-only flag is used, only suites containing at least 1 matching snippet and, within them, only the matching snippets will be run and displayed (but all markdowns still need to be read to find snippets and match them against the pattern!)

   # quotes around * are needed in shell
   
   # test all snippets
   scala-cli run test-snippets.scala -- --test-only '*' "$PWD/docs"
   # test all snippets in my-markdown.md
   scala-cli run test-snippets.scala -- --test-only 'my-markdown.md#*' "$PWD/docs"
   # test all snippets in my-markdown.md, in section name starting with My Section
   scala-cli run test-snippets.scala -- --test-only 'my-markdown.md#My section*' "$PWD/docs"

Why though

Some people would ask: if you need to make sure code in your documentation compiles, why not use something like mdoc?

Well, because mdoc doesn't work for my cases:

• it requires picking a specific documentation tool whether or not its look'n'feel is what authors desires
• it makes it difficult to have different scalacOptions, different Scala versions and different libraries available in each snippet - all settings are global, so one would have to work really hard around these limitations
• since it depends on settings provided when the code was build (scalacOptions, libraries) code might not be exactly reproducibe by the users - if they just copy-paste the code from the docs, they might find the code not working since they were not aware of some flags, libraries or compiler plugins used by authors of the example

Meanwhile, there is one perfectly suitable tool for the job - Scala CLI. With its //> using directives it is very easy to create a self-contained, perfectly reproducible snippet. As a matter of the fact, it even supports running snippets in markdown files. Its markdown support has a downside, however, because this mode considers all snippets to be defined in the same scope (same Scala version, same libraries, same compiler options - different //> using classes are appended and may override conflicting options).

This library:

• extracts snippets from markdown files
• puts them into separate /tmp subdirectories
• and runs as standalone snippets

allowing each snippet to be a self-contained, reproducible example.