Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add scripts for building/testing gcc torture tests #317

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Toizi
Copy link

@Toizi Toizi commented Jan 9, 2023

See #80

  • generate_compile_commands.py and compress_bitcode.sh have been taken from the AnghaBench repository and modified slightly for the gcc torture tests.
  • build_and_compress.sh will download the test suite using LLVM's release tags (e.g. "release/15.x") and call the other two scripts to build and compress the binaries/bitcode.
  • roundtrip.py has been modified to support recursing into subdirectories to support using it on the output generated by the lifting-tools-ci/tool_run_scripts/rellic.py script.
  • test-gcc-torture.sh is a modified version of test-angha-1k.sh with the option to run roundtrip.py on the output for the host architecture.
  • For the tests to work, the script
    lifting-tools-ci/datasets/fetch_gcc_torture.sh will need to be added analogous to the fetch_angha.sh one.

Only tested with LLVM 15 so far.
Test output (slightly trimmed) using dummy fetch_gcc_torture.sh for just amd64

~/dev/rellic/rellic-gcc-torture ~/dev/rellic
bitcode.clang-15.amd64.tar.xz                                                                                                                                                                         100% 1410KB 180.4MB/s   00:00
[+] Testing architecture amd64
Running against Rellic:
rellic-decomp version unknown
Commit Hash: 87c50e8c89b5af9d00bc129d6c032849696422d5
Commit Date: 2022-12-22 13:37:23 +0100
Last commit by: Francesco Bertolaccini [francesco.bertolaccini@trailofbits.com]
Commit Subject: [Migrate to LLVM15 (#308)]

Uncommitted changes were present during build.
Using LLVM 15.0.6


Listing files in /home/mg/dev/rellic/rellic-gcc-torture/bitcode/amd64
Found 1461 bitcode files
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1461/1461 [08:50<00:00,  2.75it/s]
Run took 0:08:50.966609
Speed of 2.75 runs/sec
Success Metrics: [1377/1461]
Success Percentage: [94.25%]
`Decl.cpp:3455`: `34` failures
`Decomp.cpp:130`: `28` failures
`sigsegv`: `9` failures
`ASTBuilder.cpp:380`: `4` failures
`ASTBuilder.cpp:158`: `4` failures
`timeout`: `4` failures
`ASTBuilder.cpp:413`: `1` failures
[!] There were [84] failures on amd64:
`Decl.cpp:3455`: `34` failures
`Decomp.cpp:130`: `28` failures
`sigsegv`: `9` failures
`ASTBuilder.cpp:158`: `4` failures
`ASTBuilder.cpp:380`: `4` failures
[!] Failed decompilation for amd64
Running against Clang:
clang version 15.0.6
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/mg/dev/cxx-common/vcpkg/installed/x64-linux-rel/tools/llvm

Listing files in /home/mg/dev/rellic/rellic-gcc-torture/decompile/amd64
Found 1379 C files
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1379/1379 [00:14<00:00, 95.92it/s]
Run took 0:00:14.381400
Speed of 95.89 runs/sec
Success Metrics: [1355/1379]
Success Percentage: [98.26%]
`use_of_undeclared_identifier`: `13` failures
`incomplete_definition_of_type`: `4` failures
`too_many_errors_emitted`: `3` failures
`expected_member_name_or`: `1` failures
`incompatible_pointer_to_integer`: `1` failures
`ISO_C_requires_a`: `1` failures
`passing_double`: `1` failures
[!] One or more failures encountered during test

generate_compile_commands.py and compress_bitcode.sh have been taken
from the AnghaBench repository and modified slightly for the gcc torture
tests.

build_and_compress.sh will download the test suite using LLVM's release
tags (e.g. "release/15.x") and call the other two scripts to build and
compress the binaries/bitcode.

roundtrip.py has been modified to support recursing into subdirectories
to support using it on the output generated by the
lifting-tools-ci/tool_run_scripts/rellic.py script.

test-gcc-torture.sh is a modified version of test-angha-1k.sh with the
option to run roundtrip.py on the output for the host architecture.

For the tests to work, the script
lifting-tools-ci/datasets/fetch_gcc_torture.sh will need to be added
analogous to the fetch_angha.sh one.
@CLAassistant
Copy link

CLAassistant commented Jan 9, 2023

CLA assistant check
All committers have signed the CLA.

@pgoodman
Copy link
Collaborator

pgoodman commented Jan 9, 2023

@Toizi This is excellent!

@frabert
Copy link
Collaborator

frabert commented Jan 18, 2023

Hi @Toizi! This is really great. Can you share what your dummy fetch_gcc_torture.sh looks like?

@Toizi
Copy link
Author

Toizi commented Jan 18, 2023

Hey @frabert, the dummy script will simply need to copy the compressed bitcode to the current directory.
Since I've compiled it on a VM, my fetch_gcc_torture.sh was a simple one-liner:
scp vmUbuntu22:dev/rellic/compiled/bitcode.clang-15.amd64.tar.xz .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants