Larger wordlists

[Samyak2]

What and why?

Currently, the only built-in word list is the top 250 words list. This is very limiting as words will often repeat again in the same line and multiple times throughout a test.

It would be nice to have these word lists too:

• top 500
• top 1000
• top 2500
• top 5000
• top 10000 (added in Add new word lists - top10000, top25000 and commonly-misspelled #27)
• top 25000 (added in Add new word lists - top10000, top25000 and commonly-misspelled #27)
• More?

How?

More info about the existing word list: https://docs.rs/toipe/latest/toipe/wordlists/constant.TOP_250.html

The word list needs to be added in this directory: https://github.com/Samyak2/toipe/tree/main/src/word_lists

and it needs to be listed here: https://github.com/Samyak2/toipe/blob/main/src/wordlists.rs

[Samyak2]

The source I was using had only 5000 words (for free). Added 500-5000 words lists in 7c049c5, which is coming in v0.4.0.

[benliepert]

I think the word list size is misleading - textgen.rs only looks at words between 2 and 8 characters. There are 927 words in the 5000 wordlist, for example, that didn't meet this criteria (925/927 were larger than 8 chars).
Maybe you could allow word size preference to be specified as a parameter (but default to between 2 and 8)?

[Samyak2]

I think the word list size is misleading - textgen.rs only looks at words between 2 and 8 characters. There are 927 words in the 5000 wordlist, for example, that didn't meet this criteria (925/927 were larger than 8 chars). Maybe you could allow word size preference to be specified as a parameter (but default to between 2 and 8)?

Good catch! The 2 to 8 chars filter was quite arbitrary. --min-length and --max-length flags to specify this would be nice, although that will require a bit of work to make the RawWordSelector store the ToipeConfig too.