Add option to save all traces #49
Conversation
|
Interesting. Probably a good idea (presumably for something like checking coverage?) I think this is probably the right way to do it. I'll do detailed review on Monday, but one thought: is there any reason you save out the unannotated |
|
Didn't think of trying m_traceAB. The show test looks like: While showing the m_traceAB looks like: I think I prefer showing just the test because of readability and being able to replay the test back through TestRIG. |
marnovandermaas
force-pushed
the
save_all_option
branch
from
ad3d7f9
to
ce2c644
Compare
|
Ah, yes, there might be a way to smooth out the types to get rid of the maybes and make the print nicer, but for sure it will be verbose! |
| where onDeath test = do putStrLn "Failure rerunning test" | ||
| askAndSave sourceFile (show test) Nothing testTrans | ||
| onTrace trace = askAndSave sourceFile (showAnnotatedTrace (isNothing m_implB) archDesc trace) (Just trace) testTrans | ||
| let checkTrapAndSave sourceFile test = saveOnFail sourceFile test (check_mcause_on_trap :: Test TestResult -> Test TestResult) | ||
| let checkResult = if optVerbosity flags > 1 then verboseCheckWithResult else quickCheckWithResult | ||
| let checkGen gen remainingTests = | ||
| checkResult (Args Nothing remainingTests 1 (testLen flags) (optVerbosity flags > 0) (if optShrink flags then 1000 else 0)) | ||
| (prop implA m_implB alive (checkTrapAndSave Nothing) archDesc (timeoutDelay flags) (optVerbosity flags) (optIgnoreAsserts flags) (optStrict flags) gen) | ||
| (prop implA m_implB alive (checkTrapAndSave Nothing) archDesc (timeoutDelay flags) (optVerbosity flags) (if (optSaveAll flags) then (saveDir flags) else Nothing) (optIgnoreAsserts flags) (optStrict flags) gen) |
There was a problem hiding this comment.
Slight issue: users might be surprised if they passed "save-all", forgot to pass a directory, and lose all the tests. I wonder if it's convenient to give an error early on in that case? We could also have separate options "failures-dir" and "tests-dir" that the user can set separately.
There was a problem hiding this comment.
Yeah, locally I had put that check into TestRIG run script. I might merge this PR and then see if I can get the failure and test directories separated.
This adds an option to QCVEngine to save all traces instead of just failures. I'm happy to implement any suggestions on how to do this in a cleaner way!