Support validating test labels using the test runner #35308
Conversation
lasinicl
added
the
Team/CompilerFE
tests/ballerina-spec-conformance-tests/src/test/java/org/ballerinalang/test/TestRunner.java
Outdated
Show resolved
Hide resolved
...s/ballerina-spec-conformance-tests/src/test/java/org/ballerinalang/test/TestRunnerUtils.java
Outdated
Show resolved
Hide resolved
...s/ballerina-spec-conformance-tests/src/test/java/org/ballerinalang/test/TestRunnerUtils.java
Outdated
Show resolved
Hide resolved
...s/ballerina-spec-conformance-tests/src/test/java/org/ballerinalang/test/TestRunnerUtils.java
Outdated
Show resolved
Hide resolved
| boolean hasUnknownLabels = unknownLabels.length() > 0; | ||
| boolean hasDuplicateLabels = duplicateLabels.length() > 0; | ||
|
|
||
| String diagnosticMsg = hasUnknownLabels ? (hasDuplicateLabels ? |
There was a problem hiding this comment.
Shall we propagate the line number here and log it.
There was a problem hiding this comment.
Do you mean the line number of the test case or the line number of the label? We don't seem to keep information on the line numbers of the labels ATM though. Will need to add support for this if that's the requirement here.
There was a problem hiding this comment.
Line number in the test case. Specifically .balt file.
Isn't absLineNum argument TestRunner.test() represent that?
There was a problem hiding this comment.
Data provider provides the absLineNum. The line number of the last row of the labels will be absLineNum - 1 right?
If there's more than one row of labels, right now we can't pinpoint the row.
But for now, we can log the error to the last line in case of the label list contains duplicates/unknowns.
There was a problem hiding this comment.
Yeah. But anyway we can see which test has failed right?. Then, we can just check and correct the label there. If we can't pinpoint the row I don't think it makes any difference, even if log the error to the last line since we'll anyway have to find where the label exactly is.
There was a problem hiding this comment.
Pin-pointing the raw is not needed. Just the line number will be useful as I see it. It's matter of checking the label string in the test.
Does the failed test pin-point to the specific line on the original balt. No right?
I think logging the absLineNum would be enough for the moment and it will be really useful. Theat's what I felt trying out your changes against .balt files
There was a problem hiding this comment.
As discussed offline, added the line number in 7f9e8ab
|
@lasinicl, is there a way to figure out test failures due to invalid labels from any of the reports? Without going through the logs? If not, IMO, we need to introduce/improve a report to include this. Right now, the failed test summary is also misleading when tests fail due to unknown labels. Let's create an issue for this too. |
Created #35454 |
Purpose
$title
Fixes #33691
Approach
Validates whether the labels defined under the
Labelsheader of the test case are among the predefined set of labels or else such labels are identified as "unknown labels".Also, check whether there are duplicate labels.
We can't distinguish between the typos and the unknown labels.
Samples
Remarks
Check List