Skip to content

Commit

Permalink
clean up and add metadata to Fixtures
Browse files Browse the repository at this point in the history
  • Loading branch information
@kbhargava-jump
kbhargava-jump committed Oct 22, 2024
1 parent 1533faf commit d8b84c6
Show file tree
Hide file tree
Showing 17 changed files with 1,200 additions and 441 deletions.
5 changes: 4 additions & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -164,4 +164,7 @@ cython_debug/
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
scratch/
scratch/

# protosol
protosol/
84 changes: 42 additions & 42 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,69 +14,69 @@ Clone this repository and run:
source install.sh
```

### (HIGHLY Recommended) Install auto-completion
### Install auto-completion

```sh
solana-test-suite --install-completion
```
You will need to reload your shell + the `test_suite_env` venv to see the changes.

## Currently Supported Harness Types
`list-harness-types` will provide the most updated list.
```
$ solana-test-suite list-harness-types
Available harness types:
- ElfLoaderHarness
- InstrHarness
- SyscallHarness
- CpiHarness
- VmInterpHarness
- VmValidateHarness
- TxnHarness
```

## Protobuf

Each target must contain a `sol_compat_instr_execute_v1` function that takes in a `InstrContext` message and outputs a `InstrEffects` message (see [`proto/invoke.proto`](https://github.com/firedancer-io/protosol/blob/main/proto/invoke.proto)). See `utils.py:process_instruction` to see how the program interacts with shared libraries.
Each target must contain a function entrypoint that takes in a Context input message and outputs a Effects message (see [`proto/invoke.proto`](https://github.com/firedancer-io/protosol/blob/main/proto/invoke.proto) as an example).

```
Function Entrypoints:
- ElfLoaderHarness: sol_compat_elf_loader_v1
- InstrHarness: sol_compat_instr_execute_v1
- SyscallHarness: sol_compat_vm_syscall_execute_v1
- CpiHarness: sol_compat_vm_cpi_syscall_v1
- VmInterpHarness: sol_compat_vm_interp_v1
- VmValidateHarness: sol_compat_vm_validate_v1
- TxnHarness: sol_compat_txn_execute_v1
```

### Updating definitions
All message definitions are defined in [Protosol](https://github.com/firedancer-io/protosol/). You may need to periodically update these definitions with the following command:
All message definitions are defined in [protosol](https://github.com/firedancer-io/protosol/). Anytime, protofuf definitions are updated in protosol, you will need to run the following command:

```sh
./fetch_and_generate.sh
```
## Setting up Environment
To setup the `solana-conformance` environment, run the following command and you will be all set:
```
source test_suite_env/bin/activate
```

## Usage
Run `solana-test-suite --help` to see the available commands.
Or, refer to the [commands.md](commands.md) for a list of available commands.

### Note on [commands.md](commands.md)
`commands.md` is automatically generated from the `typer utils docs` module.
Help messages are dynamically generated based on the [currently set harness type](#selecting-the-correct-harness). Thus, descriptions specific to harness type (typically the Context, Effects, and Fixtures names) in `commands.md` refer to the harness type set during time of generation.

When the desired [harness type is set](#selecting-the-correct-harness), the `--help` output in the CLI will reflect the correct names. Try it!

### Selecting the correct harness
The harness type should be specified by an environment variable `HARNESS_TYPE` and supports the following values (default is `InstrHarness` if not provided):
- `InstrHarness`
- `TxnHarness`
- `SyscallHarness`
- `CpiHarness`
- `VmInterpHarness`
- `VmValidateHarness`
- `ElfLoaderHarness`

`solana-test-suite list-harness-types` will provide the most updated list.


### Data Preparation

Before running tests, Context messages may be converted into Protobuf's text format, with all `bytes` fields base58-encoded (for human readability). This can be done with [decode-protobuf](commands.md#solana-test-suite-decode-protobuf) command.

Optionally, context messages may also be left in the original Protobuf binary-encoded format.


### Debugging

For failing test cases, it may be useful to analyze what could have differed between Solana and Firedancer. You can execute a Protobuf message (human-readable or binary) through the desired client with the [`debug-instr`](commands.md#solana-test-suite-debug-instr) command.


#### Alternative (and preferred) debugging solution
Run the following to view all supported commands or refer to [commands.md](commands.md):
```
solana-test-suite --help
```

Use the following command instead if you want the ability to directly restart the program without having to restart GDB:
```sh
### Preferred Debugging
Use the following command instead if you want the ability to debug in GDB:
```
<gdb / rust-gdb> --args python3.11 -m test_suite.test_suite exec-instr --input <input_file> --target <shared_lib>
```

Refer to [`exec-instr`](commands.md#solana-test-suite-exec-instr) command for more information.

Recommended usage is opening two terminals side by side, and running the above command on both with one having `--target` for Solana (`impl/lib/libsolfuzz_agave_v2.0.so`) and another for Firedancer (`impl/lib/libsolfuzz_firedancer.so`), and then stepping through the debugger for each corresponding test case.
Recommended usage is opening two terminals side by side, and running the above command on the same output while changing the target parameter to the two desired targets and stepping through the debugger for each corresponding test case.


### Uninstalling
Expand Down
88 changes: 37 additions & 51 deletions commands.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# `solana-test-suite`

Validate effects from clients using InstrContext Protobuf messages.
Validate effects from clients using Protobuf messages.

**Usage**:

Expand All @@ -17,20 +17,19 @@ $ solana-test-suite [OPTIONS] COMMAND [ARGS]...
**Commands**:

* `create-fixtures`: Create test fixtures from a directory of...
* `debug-instr`
* `debug-mismatches`: Run tests on a set of targets with a list...
* `debug-non-repros`: Run tests on a set of targets with a list...
* `decode-protobuf`: Convert InstrContext messages to...
* `exec-instr`: Execute InstrContext message(s) and print...
* `instr-from-fixtures`: Extract InstrContext messages from fixtures.
* `decode-protobufs`: Convert Context and/or Fixture messages to...
* `execute`: Execute Context or Fixture message(s) and...
* `fix-to-ctx`: Extract Context messages from Fixtures.
* `list-harness-types`: List harness types available for use.
* `regenerate-all-fixtures`: Regenerate all fixtures in provided...
* `regenerate-fixtures`: Regenerate InstrFixture messages by...
* `regenerate-fixtures`: Regenerate Fixture messages by checking...
* `run-tests`: Run tests on a set of targets with a...

## `solana-test-suite create-fixtures`

Create test fixtures from a directory of InstrContext messages.
Create test fixtures from a directory of Context and/or Fixture messages.
Effects are generated by the target passed in with --solana-target or -s.
You can also pass in additional targets with --target or -t
and use --keep-passing or -k to only generate effects for test cases that match.
Expand All @@ -43,9 +42,10 @@ $ solana-test-suite create-fixtures [OPTIONS]

**Options**:

* `-i, --input-dir PATH`: Either a file or directory containing InstrContext messages [default: corpus8]
* `-i, --input PATH`: Input protobuf file or directory of protobuf files [default: corpus8]
* `-h, --default-harness-type TEXT`: Harness type to use for Context protobufs [default: InstrHarness]
* `-s, --solana-target PATH`: Solana (or ground truth) shared object (.so) target file path [default: impl/lib/libsolfuzz_agave_v2.0.so]
* `-t, --target PATH`: Shared object (.so) target file paths (pairs with --keep-passing). Targets must have sol_compat_instr_execute_v1 defined
* `-t, --target PATH`: Shared object (.so) target file paths (pairs with --keep-passing). Targets must have required function entrypoints defined
* `-o, --output-dir PATH`: Output directory for fixtures [default: test_fixtures]
* `-p, --num-processes INTEGER`: Number of processes to use [default: 4]
* `-r, --readable`: Output fixtures in human-readable format
Expand All @@ -54,21 +54,6 @@ $ solana-test-suite create-fixtures [OPTIONS]
* `-l, --log-level INTEGER`: FD logging level [default: 5]
* `--help`: Show this message and exit.

## `solana-test-suite debug-instr`

**Usage**:

```console
$ solana-test-suite debug-instr [OPTIONS]
```

**Options**:

* `-i, --input PATH`: Input file
* `-t, --target PATH`: Shared object (.so) target file path to debug [default: impl/lib/libsolfuzz_firedancer.so]
* `-d, --debugger TEXT`: Debugger to use (gdb, rust-gdb) [default: gdb]
* `--help`: Show this message and exit.

## `solana-test-suite debug-mismatches`

Run tests on a set of targets with a list of FuzzCorp mismatch links.
Expand All @@ -84,11 +69,11 @@ $ solana-test-suite debug-mismatches [OPTIONS]
**Options**:

* `-s, --solana-target PATH`: Solana (or ground truth) shared object (.so) target file path [default: impl/lib/libsolfuzz_agave_v2.0.so]
* `-t, --target PATH`: Shared object (.so) target file paths (pairs with --keep-passing). Targets must have sol_compat_instr_execute_v1 defined [default: impl/lib/libsolfuzz_firedancer.so]
* `-o, --output-dir PATH`: Output directory for InstrContext messages [default: debug_mismatch]
* `-t, --target PATH`: Shared object (.so) target file paths (pairs with --keep-passing). Targets must have required function entrypoints defined [default: impl/lib/libsolfuzz_firedancer.so]
* `-o, --output-dir PATH`: Output directory for messages [default: debug_mismatch]
* `-u, --repro-urls TEXT`: Comma-delimited list of FuzzCorp mismatch links
* `-s, --section-names TEXT`: Comma-delimited list of FuzzCorp section names
* `-f, --fuzzcorp-url TEXT`: Comma-delimited list of FuzzCorp section names
* `-f, --fuzzcorp-url TEXT`: Comma-delimited list of FuzzCorp section names [default: https://api.dev.fuzzcorp.asymmetric.re/uglyweb/firedancer-io/solfuzz/bugs/]
* `--help`: Show this message and exit.

## `solana-test-suite debug-non-repros`
Expand All @@ -106,62 +91,64 @@ $ solana-test-suite debug-non-repros [OPTIONS]
**Options**:

* `-s, --solana-target PATH`: Solana (or ground truth) shared object (.so) target file path [default: impl/lib/libsolfuzz_agave_v2.0.so]
* `-t, --target PATH`: Shared object (.so) target file paths (pairs with --keep-passing). Targets must have sol_compat_instr_execute_v1 defined [default: impl/lib/libsolfuzz_firedancer.so]
* `-o, --output-dir PATH`: Output directory for InstrContext messages [default: debug_mismatch]
* `-t, --target PATH`: Shared object (.so) target file paths (pairs with --keep-passing). Targets must have required function entrypoints defined [default: impl/lib/libsolfuzz_firedancer.so]
* `-o, --output-dir PATH`: Output directory for messages [default: debug_mismatch]
* `-u, --repro-urls TEXT`: Comma-delimited list of FuzzCorp mismatch links
* `-s, --section-names TEXT`: Comma-delimited list of FuzzCorp section names
* `-f, --fuzzcorp-url TEXT`: Comma-delimited list of FuzzCorp section names
* `-f, --fuzzcorp-url TEXT`: Comma-delimited list of FuzzCorp section names [default: https://api.dev.fuzzcorp.asymmetric.re/uglyweb/firedancer-io/solfuzz/bugs/]
* `--help`: Show this message and exit.

## `solana-test-suite decode-protobuf`
## `solana-test-suite decode-protobufs`

Convert InstrContext messages to human-readable format.
Convert Context and/or Fixture messages to human-readable format.

**Usage**:

```console
$ solana-test-suite decode-protobuf [OPTIONS]
$ solana-test-suite decode-protobufs [OPTIONS]
```

**Options**:

* `-i, --input PATH`: Either a InstrContext message or directory of messages [default: raw_context]
* `-o, --output-dir PATH`: Output directory for base58-encoded, human-readable InstrContext messages [default: readable_context]
* `-i, --input PATH`: Input protobuf file or directory of protobuf files [default: raw_context]
* `-o, --output-dir PATH`: Output directory for base58-encoded, Context and/or Fixture human-readable messages [default: readable_context]
* `-p, --num-processes INTEGER`: Number of processes to use [default: 4]
* `-h, --default-harness-type TEXT`: Harness type to use for Context protobufs [default: InstrHarness]
* `--help`: Show this message and exit.

## `solana-test-suite exec-instr`
## `solana-test-suite execute`

Execute InstrContext message(s) and print the effects.
Execute Context or Fixture message(s) and print the Effects.

**Usage**:

```console
$ solana-test-suite exec-instr [OPTIONS]
$ solana-test-suite execute [OPTIONS]
```

**Options**:

* `-i, --input PATH`: Input InstrContext file or directory of files
* `-i, --input PATH`: Input protobuf file or directory of protobuf files
* `-h, --default-harness-type TEXT`: Harness type to use for Context protobufs [default: InstrHarness]
* `-t, --target PATH`: Shared object (.so) target file path to execute [default: impl/firedancer/build/native/clang/lib/libfd_exec_sol_compat.so]
* `-r, --randomize-output-buffer`: Randomizes bytes in output buffer before shared library execution
* `-l, --log-level INTEGER`: FD logging level [default: 2]
* `--help`: Show this message and exit.

## `solana-test-suite instr-from-fixtures`
## `solana-test-suite fix-to-ctx`

Extract InstrContext messages from fixtures.
Extract Context messages from Fixtures.

**Usage**:

```console
$ solana-test-suite instr-from-fixtures [OPTIONS]
$ solana-test-suite fix-to-ctx [OPTIONS]
```

**Options**:

* `-i, --input-dir PATH`: Input directory containing InstrFixture messages [default: fixtures]
* `-o, --output-dir PATH`: Output directory for InstrContext messages [default: instr]
* `-i, --input PATH`: Input Fixture file or directory of Fixture files [default: fixtures]
* `-o, --output-dir PATH`: Output directory for messages [default: instr]
* `-p, --num-processes INTEGER`: Number of processes to use [default: 4]
* `--help`: Show this message and exit.

Expand Down Expand Up @@ -191,16 +178,15 @@ $ solana-test-suite regenerate-all-fixtures [OPTIONS]

**Options**:

* `-i, --input-dir PATH`: Input test-vectors directory [default: corpus8]
* `-i, --input PATH`: Input test-vectors directory [default: corpus8]
* `-o, --output-dir PATH`: Output directory for regenerated fixtures [default: /tmp/regenerated_fixtures]
* `-t, --target PATH`: Shared object (.so) target file path to execute [default: impl/lib/libsolfuzz_agave_v2.0.so]
* `-s, --stubbed-target PATH`: Stubbed shared object (.so) target file path to execute [default: impl/lib/libsolfuzz_firedancer.so]
* `--help`: Show this message and exit.

## `solana-test-suite regenerate-fixtures`

Regenerate InstrFixture messages by
checking FeatureSet compatibility with the target shared library.
Regenerate Fixture messages by checking FeatureSet compatibility with the target shared library.

**Usage**:

Expand All @@ -210,7 +196,7 @@ $ solana-test-suite regenerate-fixtures [OPTIONS]

**Options**:

* `-i, --input-dir PATH`: Either a file or directory containing InstrFixture messages [default: corpus8]
* `-i, --input PATH`: Either a file or directory containing messages [default: corpus8]
* `-t, --target PATH`: Shared object (.so) target file path to execute [default: impl/lib/libsolfuzz_agave_v2.0.so]
* `-o, --output-dir PATH`: Output directory for regenerated fixtures [default: regenerated_fixtures]
* `-d, --dry-run`: Only print the fixtures that would be regenerated
Expand All @@ -220,8 +206,7 @@ $ solana-test-suite regenerate-fixtures [OPTIONS]

## `solana-test-suite run-tests`

Run tests on a set of targets with a directory of InstrContext
or InstrFixture messages.
Run tests on a set of targets with a directory of Context and/or Fixture messages.

Note: each `.so` target filename must be unique.

Expand All @@ -233,7 +218,8 @@ $ solana-test-suite run-tests [OPTIONS]

**Options**:

* `-i, --input PATH`: Single input file or input directory containing InstrContext or InstrFixture messages [default: corpus8]
* `-i, --input PATH`: Input protobuf file or directory of protobuf files [default: corpus8]
* `-h, --default-harness-type TEXT`: Harness type to use for Context protobufs [default: InstrHarness]
* `-s, --solana-target PATH`: Solana (or ground truth) shared object (.so) target file path [default: impl/lib/libsolfuzz_agave_v2.0.so]
* `-t, --target PATH`: Shared object (.so) target file paths [default: impl/lib/libsolfuzz_firedancer.so]
* `-o, --output-dir PATH`: Output directory for test results [default: test_results]
Expand Down
Loading

0 comments on commit d8b84c6

Please sign in to comment.