# REP15: New mass testing infrastructure ###### tags: `design` `mass-testing` **Status**: `✅ DONE` ## What is this? The mass-testing infrastructure is bitrotten and unused, we need to refactor it to make it work well again ## Why is it important? Having a working mass-testing installation ensures that we have up-to-date results on what are the major crash-points and slowdowns of the infrastructure. Also, we can prioritize our focus on the parts of the code that lead to most crashes/timeouts. ## Requirements * Use of `revng-test-configure`: previously a python script was in charge of generating a `build.ninja` file, drop it in favor of recycling the `revng-test-configure` infrastructure * No use of `perf-on-configure`: test only run once. The role of `perf` reports will be mostly superseeded by `--trace` traces. ## Design ### Workflow The workflow will be in the following phases: 1. `collect.sh`: this script will generate the `members` entry for `revng-test-configure` 2. `revng-test-configure` and `ninja` are run as usual, for each input executable. There'll be a `test-harness` wrapper executable that **is required** to be prefixed to the `revng ...` invocation, which will take care of producing the following files: * `test-harness.json`: json with misc. information (exit code, timeout flag, time output) * `output.log`: output of the program * `input` (optional): input file * `stacktrace.json` (optional): stacktrace in json format, created by the `llvm-symbolizer` Additionally, the test can augment the result by placing a `meta.yml` file in the top level directory, this has the format: ```yaml extra_columns: - name: <name in meta.yml> label: <label> type: int | float | str | bool downloads: - name: <filename> label: <label> crash_related: true | false hooks: pre: | <bash code to run before the actual command> post: | <bash code to run after the actual command> notes: | free form text to display in the overview ``` Typically, these additional files will be: * `sections.json`: section data * `trace.json.gz`: the output of the `--trace` option 3. The files produced in (2) will be be ingested by a `generate_report` script, this will take care of: 1. Creating flamegraphs for crashes/timeouts 2. Creating some html files (`index.html`, `crashes.html`, `timeouts.html`, `raw_data.html`) described below ### Report contents * `index.html`: Overall successes, failures, timeouts, etc. Aggregated graphs for failures and timeouts `revng` and `revng-c` version used * `timeouts.html`: * Bottom-up and top-down flame graphs * Table of each executable with links for: * Downloading the binary * Downloading the trace * `crashes.html`: * Bottom-up and top-down flame graphs * Table of each failed executable with links for: * Downloading the binary * Downloading the `trace.json` * Downloading a reproducer `.tar` * Downloading the `output.log` * `raw_data.html`: * Table of all executables, with download of raw sqlite DB ## Repo-wise changes ### revng revng will have the following commands: ```bash # Runs binary collection, revng-test-configure revng mass-testing configure "$INPUT_BINARIES_DIR" "$BUILD_DIR" # Runs ninja with the right options revng mass-testing run "$BUILD_DIR" # Runs the report collection infra revng mass-testing generate-report "$BUILD_DIR" "$REPORT_DIR" ``` ### llvm Add `llvm-symbolizer` as a symlink in `/bin` ### orchestra The following new components will be introduced: * `mass-testing`: full mass testing with all binaries * `mass-testing-regression`: mass testing with only regression binaries * `flamegraph`: `flamegraph.pl` installed in orchestra ## Input binaries Input binaries will be uploaded to S3 There will be a new repo `mass-testing-binaries` with the following structure: ``` binaries/ windows_10.yml linux.yml mass.yml regression.main.yml regression.autogenerated.yml ``` Format for `*.yml` under `binaries`: ```yaml - name: <name/path of the binary> size: <size> text_size: <text size> hash: <hash> tags: [] # optional ... ``` Format for `mass.yml`, `regression.yml`, `regression_list.yml`: ```yaml # Binary names to include, in OR include: - <regex 1> - <regex 2> # Binary names to exclude, processed after `include`, in OR exclude: - <regex 1> - <regex 2> ``` As part of the orchestra CI process, the `regression.autogenerated.yml` will be automatically updated and pushed when running `mass-testing` The files will be stored in a S3 bucket, with the prefix and credentials as private variables exported to orchestra