Generate dashboard using test results

[rw1nkler]

This commit adds an initial version of the dashboard generation feature to XLS. To demonstrate the dashboard generation process, two examples are provided - a simple dashboard for a passthrough example and a more complex dashboard for a basic RLE encoder design.

The dashboard is generated from data extracted from tests or their output. To extract the data, users may use parsers provided in this PR or their custom scripts, which are responsible for extracting the information and saving it in a Dashboard JSON format. Custom parsers should read the log from stdin and write the extracted information to stdout.

The dashboard generation is done in a few steps. The whole process is controlled by the main dashboard.py script. Generating a dashboard is as follows:

1. The dashboard.py script is invoked with a set of parameters that specify two types of operations: parsing output of a test (-p option) or parsing output files generated by the test (-f option). The arguments contain information about the test to run, the parser to use, and the potential file from which the data should be extracted. The script parses arguments and saves them in a more convenient form internally.

2. The tests are then executed and parsed using the run_and_parse function from the run_and_parse.py file. For the -p option, the parser is run on the log produced by the test, for the -f option, the parser is run on the output file created by the test. The output files are assumed to be located in $TEST_UNDECLARED_OUTPUTS_DIR, which is a default location for output files produced by Bazel tests.

3. The parsed data in JSON format is collected and then verified if it matches the Dashboard JSON schema. Both the verification mechanism and the schema are available in the validate_dashboard_json.py file.

4. Next, the correct piece of information in the Dashboard JSON format is sorted and converted to a markdown file by the json_to_markdown function from the Python file with the same name.

5. Finally, the utilities from mkdocs_creator.py are used to produce an HTML using mkdocs.

Since the dashboard relies on tests, it cannot be produced on build time. To generate the dashboard, one has to provide a path to which the HTML should be saved. For example:

bazel run -- //xls/modules/rle:rle_dashboard -o rle_dashboard

Here is a screenshot of the RLE Dashboard:

Relies on changes from #1031
Resolves #1058
CC @proppy

[proppy]

Thanks for sending this PR, other interesting metrics that would be interesting to track:

• IR schedule timing, commulative per path and per node, from https://github.com/google/xls/blob/main/xls/scheduling/pipeline_schedule.proto#L19-L30
• generated BoM breakdown from https://github.com/google/xls/blob/main/xls/codegen/module_signature.proto#L259C12-L259C27 and https://github.com/google/xls/blob/main/xls/codegen/xls_metrics.proto#L126-L128
• other metrics from https://github.com/google/xls/blob/main/xls/codegen/xls_metrics.proto
• throughput metrics post IR simulation
• synthesis gate summary and breakdown from https://github.com/google/xls/blob/main/xls/codegen/xls_metrics.proto
• sta timing data (per gate, and path commulative) from https://github.com/hdl/bazel_rules_hdl/blob/main/static_timing/sta.tcl#L22 after placement, and routing
• other sta metrics (wns, tns, etc) after placement, and routing
• layout preview after placement and/or routing https://openroad.readthedocs.io/en/latest/main/src/gui/README.html#save-screenshot-of-layout
• post placement and routing power metrics from https://github.com/hdl/bazel_rules_hdl/blob/b3d3a2a727f8950eaacddf2b02390a93e8df9ffa/place_and_route/private/report_power.bzl#L24
• area estimate from https://github.com/hdl/bazel_rules_hdl/blob/b3d3a2a727f8950eaacddf2b02390a93e8df9ffa/place_and_route/private/report_area.bzl#L26
• wire length using https://openroad.readthedocs.io/en/latest/main/src/grt/README.html#report-wirelength
• final gds preview using klayout python module, see Qt-less canvas discussion KLayout/klayout#1082 (comment)
• drc report using klayout xml report, see https://www.klayout.de/rdb_format.html
• and any other metrics you may find relevant?

[proppy]

It would be nice to integrate some of that metric generation with https://github.com/hdl/bazel_rules_hdl and https://github.com/google/xls/tree/main/xls/build_rules so that we can easily generate / configure those report from BUILD files and have them being generated by our current build infrastructure on each CL.

[rw1nkler]

Yes, it would be nice. However, can we consider adding more metrics in subsequent PRs?

[proppy]

curious if you've considered integrating with:
https://google.github.io/xls/bazel_rules_macros/#xls_benchmark_ir, https://google.github.io/xls/bazel_rules_macros/#xls_benchmark_verilog
and benchmark_synth recently refactored in 6381eb1 by @ericastor

[proppy]

what do you think of moving this (along side #1160) in separate repo similar to https://github.com/antmicro/xls-cosimulation-demonstrator ?