Test report when features are in nested directories

[ansort25]

Hi! I'm having this issue that I'm trying to understand and solve. I didn't open this as a bug or feature request because I'm not sure it is one of those situations.

So I'm having some features in nested directories under features/ and using the import step workaround offered by behave. My structure is like this:

main_dir/
└── features/
| ├── steps/
| │ ├── init.py
| │ └── import_steps.py
| ├── feature1/
| │ ├── steps/
| │ │ ├── init.py
| │ │ └── feat1.py
| │ └── feat1.feature
| └── environment.py

In this way I can have subfolders under features/ organized based on component.

My issue is with behavex. I'm running behavex and it runs giving me this output:
1 feature passed, 0 failed, 0 skipped 2 scenarios passed, 0 failed, 0 skipped 9 steps passed, 0 failed, 0 skipped, 0 undefined Took 0m10.208s Exit code: 0
But if I open the html report I don't see the feature.

As you can see in the attached screenshots the feature is not present and in the metrics tab it is shown that 6 steps were executed and passed.

Do you have any input on how to tackle this?

[hrcorval]

Hi @ansort25, thanks for reporting this. Just to have more feedback on that behavior, would it be feasible to share the report.json file that was generated as part of the execution? If you have any sensitive data, maybe you can obfuscate it.
Thanks in advance.

[ansort25]

@hrcorval this is the content of the report.json:
{ "environment": [], "features": [], "steps_definition": { "3055599": "step_7", "6640229": "step_1", "17705584": "step_6", "36436750": "step_9", "61528386": "step_10", "62050047": "step_3", "62739964": "step_5", "79002220": "step_2", "90069712": "step_4", "90273715": "step_8" } }

[hrcorval]

Thanks a lot for your quick answer. We see the .JSON file is empty also, like if any of the scenarios crash for any reason. Also, as the HTML report is created based on the .JSON report, it is logically empty.
As I still need more context, have you tried executing the tests in parallel by scenario ("--parallel-processes 2") or by feature ("--parallel-processes 2 parallel-scheme feature")? Maybe any of that executions capture the error.

Regards

[ansort25]

I'll try on monday again. I remember executing once or twice in parallel but I don't remember the output. And the code is on the work computer. Have a nice weekend

[iamkenos]

could be related:

i have a similar setup (nested directories) and im getting a similar issue when I don't pass a features path argument. was meaning to file an issue for this.

@ansort25 - can you try passing a features path and see if it helps?
e.g. in your behavex command, just append feature1/ at the end.

[ansort25]

@hrcorval using both suggested commands the output looks like there are no features or scenarios. (0 features/scenarios passed, 0 failed, 0 skipped)

@iamkenos everytime I'm giving the feature path to behavex the output is an error saying: 'Context' object has no attribute 'page'

Maybe there's an error in my solution, as this is the first time I'm using playwright and behave!

The funny thing is that if I only run behavex scenarios are running, all I'm missing is the correct report.

[hrcorval]

Hi @ansort25, we released a new version with improvements in the way features/scenarios are being executed as we were experiencing some issues also on our end. Can you please give it a try? As we cannot reproduce the issue above, so maybe this was fixed with the mentioned changes.
Also, can you try running the suite with only one parallel session and also with >1 sessions? This will help a lot to see if you experience differences in the execution results.
Please, keep us posted on this.

Thanks!

[ansort25]

Hi! I've tried with the new version and now the issue is different. I'm getting the 'Context' object has no attribute 'page' error. But I think it's due to how I set up the project.

I've already stopped using Behavex as with the project I'm working on it is not possible to run the tests in parallel (there's a lock on the tab session). I still wanted to use Behavex for the hmtl reporter but I've moved towards other reporters/formatters.

Do you think it would be possible to release the html formatter/report as a library on its own?