CTG tests are not reproducible

[trdthg]

After some testing, I found that the tests ctg (without -r option) generated each time is different. This is completely different from what I expected. I also had another person try it, and the results were also different.

I think Tim have already found this issue a few month ago: riscv-software-src/riscv-ctg#115

I'm trying to fix this, but simply appending a line random.seed(10) after all import random statements does not solve the problem. I will check it again.

Update: The reason is the use of set in the code, and I am replacing them with ordered_set

---

if you don't know what -r option does:

-r, --randomize Randomize Outputs.

If randomization is enabled we use the MinConflictsSolver solver to find solutions.

riscv-arch-test/riscv-ctg/riscv_ctg/generator.py

Lines 493 to 496 in cd94912

	if self.random:
	problem = Problem(MinConflictsSolver())
	else:
	problem = Problem()

I haven't tested the behavior of MinConflictsSolver, which will use random inside

Originally posted by @trdthg in #565 (comment)

---

Update: In summary, two steps are needed to solve the problem here

• random: <>
• set: [CTG] Replace set with OrderedSet #570

[trdthg]

But I don't know how to handle random, according to the following code, it seems intentional, or it is necessary because we need to choose one from various solutions

riscv-arch-test/riscv-ctg/riscv_ctg/cross_comb.py

Lines 223 to 225 in cd94912

	# Randomly choose an instruction
	instr = random.choice(instrs_sol)
	isa_set += (self.OP_TEMPLATE[instr]['isa'])

However, I think we can set the random seed as a parameter, which ensures reproducibility while also allowing for some randomness

[allenjbaum]

While I wish it were reproducible, what is important is really the coverage. The only reason tests are recreated is to add specific cover points, Test generation should not be adding tests that don't add new coverpoints, so as CTG runs, it must be keeping track of how much coverage has already been found. If we specify just new coverpoints, AND add the capability of pointing to existing cover points from a previous run, then new tests that match only existing coverpoints could be dropped, only tests matching new coverpoints will be added, and existing tests won't remain unchanged. That would make reviewing new tests much simpler. I don't think that's an option that CTG has, though internally it must do something like that when creating tests, so shouldn't be hard to add. (I can say this because I know I'm incapable of making that change....:-

…

On Mon, Nov 25, 2024 at 7:17 PM MingZhu Yan ***@***.***> wrote: But I don't know how to handle random, according to the following code, it seems intentional ( I haven't looked closely at why to do this way) https://github.com/riscv-non-isa/riscv-arch-test/blob/cd94912fed2aab75d7d5f115b441da0813fdce8d/riscv-ctg/riscv_ctg/cross_comb.py#L223-L225 — Reply to this email directly, view it on GitHub <#566 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHPXVJVNH4AH2ITBER3VM5L2CPR3JAVCNFSM6AAAAABSF6BSO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJZGU3DIMZVGQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[trdthg]

I roughly understand what you mean; ctg only generates new test cases for points that have not yet been covered. I will give it a try

I highly recommend adding an option to filter the instruction set, as it would make debugging much simpler, and I will send a PR later, I've already add a --filter=<PATH> for riscof riscv-software-src/riscof#127

[allenjbaum]

If you are generating tests for a specific extension, you can add a filter,
Riscof generates a test list default, so you could filter it, but there is also an option to skip generating the test list and just test using an existing test list, which probably does what you want.