-
Notifications
You must be signed in to change notification settings - Fork 0
How to Create a FAT Project That Runs a Maven Based TCK Using Surefire and Arquillian
- Copy project com.ibm.ws.microprofile.config_fat_tck from integration
- Replace all the references to 'config' with your own project's equivalent. If you have a look at the PR for the first copy/paste of this approach by someone else, at https://github.com/OpenLiberty/open-liberty/pull/1725, there is a comment in each place you would have to change for new TCK after copy/pasting the com.ibm.ws.microprofile.config_fat_tck directory under dev.
- Replace the publish/tckRunner folder with the top level folder from your tckRunner project
- If you add any jars into the test wars using an Arquillian ApplicationArchiveProcessor put these jars in the projects's top level build.gradle file and reference them as per https://github.com/OpenLiberty/open-liberty/pull/1400/files#diff-324e430492dcf6ce548918c8407a4e7f with a path like '../../../lib/hamcrest-all-1.3.jar'
- Edit the publish/tckRunner/tck/pom.xml and replace all the systemPath element contents that refer to a specific versions of jar files as these can vary from one version of Liberty to the next, for example: '/libertyGit/open-liberty/dev/build.image/wlp/lib/com.ibm.ws.microprofile.config_1.0.18.jar' is brittle and will break when version 19 comes along. Replace it with a symbolic equivalent. e.g.:
<systemPath>${/libertyGit/open-liberty/dev/build.image/wlp/lib/com.ibm.ws.microprofile.config_1.0.18.jar</systemPath>
...won't work when version "18" becomes "19" so replace it with the equivalent:
<systemPath>${com.ibm.ws.microprofile.config.1.0.}</systemPath>
The system uses regex but in the systemPath "." is literal and will match ".", "DOT" will become a regex "." and "STAR" becomes ".*" (as does "DOTSTAR"). Underscores match literally. Other regex chars that are not part of valid environment variables (^-[] etc.) are not supported.
String have ".jar" added automatically if necessary and are bookended with ".*" in order to act like a regex 'contains'.
See method jarPathInDir in MvnUtils.jar for what strings with 'match' jars from Liberty.
The test front end's call to MvnUtils.init(server) will create an environment variable like ${com.ibm.ws.microprofile.config} with the value of the actual file path "/libertyGit/open-liberty/dev/build.image/wlp/lib/com.ibm.ws.microprofile.config_1.0.18.jar" that can be used in the pom.xml. The prefix of the absolute path location comes from the Simplicity testcase LibertyServer object's server install location that is used in the test case. The pom.xml is scanned to find the list of symbolic systemPath environment variables that need binding to real versioned jar paths for that test's server.
If you look at the arquillian.xml file you will see that the arquillian server configuration is set up in a similar fashion using environment variables. These are all set from the Simplicity framework's LibertyServer that is used in the JUnit testcase front end.
When running a tckFat in your laptop then you should include IBM's artifactory in the set of places your local mvn will look at. Update your maven settings ~/.m2/settings.xml to add artifactory... example settings.xml
The TCK's TestNg tests all output report data in XML format. These raw XML files are copied into the same place and format that the Simplicity Tests' raw reports go to BEFORE the html report is created. Thus the final Simplicity HTML report will include all the TCK tests individually reported and TCK and LibertyFat tests can be freely mixed in the same FAT bucket if desired (for example testing non TCK issues like ranges) and will all appear in one HTML report directory and index.html summary in the usual places.
A build will be failed if any TCK test fails.
Arquillian does org.jboss.arquillian.protocol.servlet.runner.SecurityActions.getThreadContextClassLoader(SecurityActions.java:61) and some tests do things like System.getProperties(). Without appropriate Java 2 security permissions you will get an error when a build is done with Java2 security switched on.
A server.xml element can ge used to grant permissions. For example:
<javaPermission className="java.security.AllPermission" name="*" actions="*" />
The automated builds will test with Java 2 security if the following property is set:
global.java2.sec=true
When running locally, i.e.:
cd ..repo/dev
./gradlew <project>:buildandrun
For low level detailed logs have a look at:
${project}/build/libs/autoFVT/results/junit.html/index.html
e.g.:
file:///libertyGit/open-liberty/dev/com.ibm.ws.microprofile.config_fat_tck/build/libs/autoFVT/results/junit.html/index.html
and
${project}/build/libs/autoFVT/results/mvnOutput_TCK for the mvn stdout
and
${project}/build/libs/autoFVT/results/junit/TEST-org.eclipse.microprofile.config.tck.FATSuite.xml
For a Liberty personal build on the EBC. Go to the Downloads tab; FAT Logs; File Name: fatoutput-myProject.zip -> myProject/results/... e.g. fatoutput-com.ibm.ws.microprofile.faulttolerance_fat_tck.zip Unzip then see {results/mvnOutput_TCK, results/junit.html/index.html, results/junit/TESTS-TestSuites.xml}
There are about 4 places/processes that one can catch with Eclipse:
https://github.com/OpenLiberty/open-liberty/wiki/FAT-tests#debugging-a-fat
./gradlew com.ibm.ws.microprofile.config_fat_tck:buildandrun -Ddebug.framework
Editing the MvnUtils.java code to run mvnDebug instead of mvn
mvnDebug -DforkCount=0 test
mvn -Dmaven.surefire.debug="-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8014" test
arquillian.xml
<property name="javaVmArguments">-agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=n</property>
Either run the server in debug mode and make sure you have.
<property name="allowConnectingToRunningServer">true</property>
in arquillian.xml
Unfortunately, I have not been able to find out how to debug the Containers deploy method.
runTests:
[echo] Base dir is: /libertyGit/open-liberty/dev/com.ibm.ws.microprofile.config_fat_tck/build/libs/autoFVT
[echo] Liberty dir is: ${liberty.dir}
[echo] libertyInstallPath is: ../../../../build.image/wlp
[echo] liberty.location is: ../../../../build.image/wlp
[echo] install.location is: /libertyGit/open-liberty/dev/com.ibm.ws.microprofile.config_fat_tck/build/libs/autoFVT/
Notes:
If you have a look at, for example,
file:///libertyGit/open-liberty/dev/com.ibm.ws.microprofile.faulttolerance_fat_tck/build/libs/autoFVT/results/tck/surefire-reports/index.html
or for a remote build
.../autoFVT/results/tck/surefire-reports/index.html#
and then click on the name of the xml file in the top left
microprofile-faulttolerance-TCK
Info
tck-suite.xml
Then the right hand pane will display all the expanded wildcards from the file, for example:
<suite name="microprofile-faulttolerance-TCK" verbose="2" configfailurepolicy="continue" >
<test name="microprofile-faulttolerance 1.0 TCK">
<packages>
<package name="org.eclipse.microprofile.fault.tolerance.tck.*"></package>
<package name="org.eclipse.microprofile.*"></package>
</packages>
</test>
</suite>
becomes
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite configfailurepolicy="continue" guice-stage="DEVELOPMENT" verbose="0" name="microprofile-faulttolerance-TCK">
<test name="microprofile-faulttolerance 1.0 TCK">
<packages>
<package name="org.eclipse.microprofile.fault.tolerance.tck.*"/>
<package name="org.eclipse.microprofile.*"/>
</packages>
<classes>
<class name="org.eclipse.microprofile.config.inject.ConfigProperty"/>
<class name="org.eclipse.microprofile.config.ConfigProvider"/>
<class name="org.eclipse.microprofile.config.Config"/>
<class name="org.eclipse.microprofile.config.spi.ConfigSource"/>
...
For deployment problems one often wishes to include the whole TCK package(s) but not include anything from a (set of) classes. If you specify individual classes, then you can include and exclude individual methods ( See http://testng.org/testng-1.0.dtd ) with:
<suite name="bulkhead-suite" verbose="2" configfailurepolicy="continue" >
<test name="bulkhead-suite">
<classes>
<class name="org.eclipse.microprofile.fault.tolerance.tck.bulkhead.BulkheadSyncRetryTest">
<methods>
<include name="testBulkheadRetryClassSynchronous55" />
But listing all classes is a lot of work and brittle.
There is very useful support for beanshell in the testng.xml files which is very rarely referenced but works. This can be used to 'parameterize' to some extent the testng.xml which (unlike the pom.xml and arquillian.xml) cannot embed environment variables.
<suite name="microprofile-config-1.1-tck" verbose="2"
configfailurepolicy="continue">
<test name="tck-package-org.eclipse.microprofile.config.tck">
<packages>
<method-selectors>
<method-selector>
<script language="beanshell">
<![CDATA[
!method.getDeclaringClass().getSimpleName().startsWith("CDIPlainInjectionTest") &&
!method.getDeclaringClass().getSimpleName().startsWith("ConfigProviderTest")
]]>
</script>
</method-selector>
</method-selectors>
<package name="org.eclipse.microprofile.config.tck.*"></package>
</packages>
</test>
</suite>
You can control the 'Mode' by annotating the test method with
@Mode(TestMode.QUARANTINE) -- to not fail builds on TCK test faulure
@Mode(TestMode.FULL) -- for 'long running' tests (> 5 minutes)
@Mode(TestMode.LITE) -- for all builds
If you are pulling 'SNAPSHOT' TCK builds with 3rd parties committing tests before they are shown to pass on Libery consider setting such a TCK FAT bucket as @Mode(TestMode.QUARANTINE) so that Libery builds are not failed due to test instability.
See http://was.pok.ibm.com/xwiki/bin/view/Liberty/Test-FAT#HLitevsFullvsQuarantinemodes for more details.
You can prototype the TCK directly with a command line that looks like the one the java JUnit testcase generates. For example:
cd ...fat_tck/publish/tckRunner/tck
mvn clean test -Dwlp=/libertyGit/open-liberty/dev/build.image/wlp -Dtck_server=FATServer -Dtck_port=8010 -DtargetDirectory=/libertyGit/open-liberty/dev/com.ibm.ws.microprofile.faulttolerance_fat_tck/build/libs/autoFVT/results/tck -Dcom.ibm.websphere.org.eclipse.microprofile.faulttolerance=/libertyGit/open-liberty/dev/build.image/wlp/dev/api/stable/com.ibm.websphere.org.eclipse.microprofile.faulttolerance.1.0_1.0.18.jar -Dcom.ibm.ws.microprofile.faulttolerance=/libertyGit/open-liberty/dev/build.image/wlp/lib/com.ibm.ws.microprofile.faulttolerance_1.0.18.jar -Dcom.ibm.ws.microprofile.config=/libertyGit/open-liberty/dev/build.image/wlp/lib/com.ibm.ws.microprofile.config_1.0.18.jar -Dcom.ibm.ws.microprofile.config.cdi=/libertyGit/open-liberty/dev/build.image/wlp/lib/com.ibm.ws.microprofile.config.cdi_1.0.18.jar -Dcom.ibm.ws.microprofile.faulttolerance.cdi=/libertyGit/open-liberty/dev/build.image/wlp/lib/com.ibm.ws.microprofile.faulttolerance.cdi_1.0.18.jar -Dcom.ibm.websphere.org.eclipse.microprofile.config=/libertyGit/open-liberty/dev/build.image/wlp/dev/api/stable/com.ibm.websphere.org.eclipse.microprofile.config.1.1_1.2.18.jar -DsuiteXmlFile=tck-suite.xml