-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
introduce JUnit tag "flaky", to exclude tests from running. #5231
Conversation
915f87a
to
b4ff9fa
Compare
ExampleTest.testClientCreation fails without real reason, mark it flaky at the moment. permit this tag in engine-tests/build.gadle.kts to separate test runs for normal integration tests, and ones which are not stable on github, for currently unknown reasons. normally of course it would be better to fix these tests - but accepting pull requests is coupled on tests running through. now two commands instead of one run all integration tests: gradle integrationTest gradle integrationTestFlaky please do not lightheartedly mark tests as flaky.
b4ff9fa
to
317ec5c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general, this looks good to me.
I made a few suggestions based on my personal preferences of wording and expression, they are not blocking though.
I split the CI stage using the new gradle task you introduced which works nicely 👍
As you can see the tests are executed by the stage matching their tagging:
org.terasology.engine.integrationenvironment.ExampleTest.testClientConnection()
failed in the last CI run. This test is one of the other flaky tests in ExampleTest
that I mentioned in the last sync. We probably want to mark at least that one "flaky", too.
ok, committed your suggestions and squashed into one. i am understanding correct, we still see the cause of any failed integration test? |
result of flaky test will be displayed in also github, result of failing normal test is viewed in jenkins.
fdc37c5
to
b0108ef
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There shouldn't be any flaky tests, however, those flaking tests have little benefit and the failures have been very disruptive. This is likely the most practical path forward right now.
Co-authored-by: BenjaminAmos <24301287+BenjaminAmos@users.noreply.github.com>
in the sense that test failures are still reported including stack trace etc.? yes |
ExampleTest.testClientCreation fails without real reason, mark it flaky at the moment. permit this tag in engine-tests/build.gadle.kts to separate test runs for normal integration tests, and ones which are not stable on github, for currently unknown reasons. normally of course it would be better to fix these tests - but accepting pull requests is coupled on tests running through. now two commands instead of one run all integration tests:
please do not lightheartedly mark tests as flaky.