Merge pull request #406 from sandialabs/Tribits-snapshot-09-26-2023

Tribits snapshot 09 26 2023

cmake/tribits/CHANGELOG.md

@@ -2,6 +2,43 @@
 ChangeLog for TriBITS
 ----------------------------------------
 
+## 2023-06-22:
+
+* **Added:** Packages are now determined to be missing if their dependencies
+  file `<packageDir>/cmake/Dependencies.cmake` is missing.  If the package
+  directory `<packageDir>` exists but the dependencies file is missing, the
+  package is determined to be missing but a warning is printed.  (This expands
+  behavior to gracefully deal with a situation where a package source
+  directory is only partially removed, such as with `git rm -r <packageDir>`,
+  but the base directory still exists.  Therefore, this allows the project to
+  gracefully configure with the package being considered missing and avoids a
+  fatal error in this case.)
+
+## 2023-06-02:
+
+* **Added/Deprecated:** External packages/TPLs can now be (and should be)
+  listed in the `[TEST|LIB]_[REQUIRED|OPTIONAL]_PACKAGES` arguments/lists in
+  the macro `tribits_package_define_dependencies()` and the
+  `[TEST|LIB]_[REQUIRED|OPTIONAL]_TPLS` arguments/lists are deprecated (but
+  with no deprecation warning yet).  This makes it easier to write
+  `<packageDir>/cmake/Dependencies.cmake` files for packages where the set of
+  internal and external upstream dependent packages is dynamic and changes
+  depending on the TriBITS project where these package are configured under.
+  (And conceptually, a downstream package should not care if an upstream
+  dependent package is pulled in as an external package or built as an
+  internal package.)
+
+## 2023-05-03:
+
+* **Added:** Added support for non-fully TriBITS-compatible external packages.
+  Now, a `<Package>Config.cmake` file need not define
+  `<UpstreamPkg>::all_libs` targets for all of its upstream dependencies.  The
+  updated macro `tribits_process_enabled_tpls()` will find any missing
+  upstream external packages/TPLs as needed (see updated documentation in the
+  section "TriBITS-Compliant External Packages" in the "TriBITS Users Guide"
+  and the section "Processing of external packages/TPLs and TriBITS-compliant
+  external packages" in the "TriBITS Maintainers Guide").
+
 ## 2023-02-24:
 
 * **Changed:** Upgraded minimum required CMake version from 3.17 to 3.23.

cmake/tribits/README.DIRECTORY_CONTENTS.rst

@@ -22,9 +22,10 @@ TriBITS refactorings of TriBITS.
 
 .. _TriBITS Core:
 
-**core/**: Core TriBITS package-based architecture for CMake projects. This
-only depends on raw CMake and contains just the minimal support for building,
-testing, installing, and deployment.  Only depends on CMake and nothing else.
+**core/**: Core TriBITS test support and package-based architecture for CMake
+projects. This only depends on raw CMake and contains just the minimal support
+for building, testing, installing, and deployment.  This CMake code depends
+only on CMake and nothing else.
 
 **python_utils/**: Some basic Python utilities that are not specific to
 TriBITS but are used in TriBITS CI and testing support software.  There are
@@ -86,3 +87,49 @@ subdirectory. It supports the argument ``--components`` with values ``core``,
 * ``examples`` => (external tribits installation)
 * ``doc`` => ``core``, ``ci_support``, ``examples``
 * ``devtools_install`` => ``python_utils``
+
+
+TriBITS Core Directory Contents
+...............................
+
+The TriBITS ``core/`` directory is broken down into several subdirectories of
+its own:
+
+**core/utils**: General CMake utilities that are not specific to the TriBITS
+system and can be reused in any CMake project.
+
+**core/common**: As small set of common modules that the different TriBITS
+Core module files in different directories depend on.  These include things
+like common TriBITS constants and TriBITS CMake policies.
+
+**core/test_support**: Modules that help define CTest tests using functions
+like `tribits_add_test()`_ and `tribits_add_advanced_test()`_.  These can be
+used in CMake projects that are not full-blown TriBITS projects.
+
+**core/config_tests**: Some basic configure-time tests used by the TriBITS
+package architecture framework.
+
+**core/std_tpls**: Some ``Find<tplName>.cmake`` files for key external
+dependencies handled as TriBITS TPLs but are more central to the TriBITS
+system.  (Examples include CUDA and MPI support.)
+
+**core/installation**: A collection of ``*.cmake.in`` and related Cmake code
+supporting installations.
+
+**core/package_arch**: Modules for the full-blown TriBITS package architecture
+framework including package dependency management, multi-repository support,
+installations (including the generation of ``<Package>Config.cmake`` files),
+etc.
+
+The dependencies between these different TriBITS `core` subdirectories are:
+
+* ``core/utils`` => (external CMake)
+* ``core/common`` => ``core/utils``
+* ``core/test_support`` =>   ``core/utils``, ``core/common``
+* ``core/config_tests`` => (external CMake)
+* ``core/std_tpls`` => (external CMake)
+* ``core/installation`` <=> ``core/package_arch`` (bidirectional)
+* ``core/package_arch`` => ``core/utils``, ``core/common``,
+  ``core/test_support``, ``core/config_tests``, ``core/std_tpls``,
+  ``core/installation``
+

cmake/tribits/ci_support/CDashQueryAnalyzeReport.py

@@ -40,9 +40,11 @@
 try:
   # Python 2
   from urllib2 import urlopen
+  from urllib2 import quote as urlquote
 except ImportError:
   # Python 3
   from urllib.request import urlopen
+  from urllib.parse import quote as urlquote
 
 import sys
 import hashlib
@@ -716,41 +718,45 @@ def getAndCacheCDashQueryDataOrReadFromCache(
   return cdashQueryData
 
 
+def normalizeUrlStrings(*args):
+  return [urlquote(x) for x in args]
+
+
 # Construct full cdash/api/v1/index.php query URL to pull data down given the
 # pieces
 def getCDashIndexQueryUrl(cdashUrl, projectName, date, filterFields):
+  # for legacy reasons, this function assumes we normalized projectName
+  projectName, = normalizeUrlStrings(projectName,)
   if date: dateArg = "&date="+date
   else: dateArg = ""
   return cdashUrl+"/api/v1/index.php?project="+projectName+dateArg \
-    + "&"+filterFields
+      + "&"+filterFields
 
 
 # Construct full cdash/index.php browser URL given the pieces
 def getCDashIndexBrowserUrl(cdashUrl, projectName, date, filterFields):
+  # for legacy reasons, this function assumes we normalized projectName
+  projectName, = normalizeUrlStrings(projectName,)
   if date: dateArg = "&date="+date
   else: dateArg = ""
   return cdashUrl+"/index.php?project="+projectName+dateArg \
-    + "&"+filterFields
+      + "&"+filterFields
 
 
 # Construct full cdash/api/v1/queryTests.php query URL given the pieces
 def getCDashQueryTestsQueryUrl(cdashUrl, projectName, date, filterFields):
+  # for legacy reasons, this function assumes we normalized projectName
+  projectName, = normalizeUrlStrings(projectName,)
   if date: dateArg = "&date="+date
   else: dateArg = ""
   cdashTestUrl = cdashUrl+"/api/v1/queryTests.php?project="+projectName+dateArg+"&"+filterFields
-  return replaceNonUrlCharsInUrl(cdashTestUrl)
-
-
-# Replace non-URL chars and return new URL string
-def replaceNonUrlCharsInUrl(url):
-  urlNew = url
-  urlNew = urlNew.replace(' ', '%20')
-  # ToDo: Replace other chars as needed
-  return urlNew
+  return cdashTestUrl
 
 
 # Construct full cdash/queryTests.php browser URL given the pieces
 def getCDashQueryTestsBrowserUrl(cdashUrl, projectName, date, filterFields):
+  # for legacy reasons, this function assumes we normalized projectName
+  projectName, = normalizeUrlStrings(projectName,)
   if date: dateArg = "&date="+date
   else: dateArg = ""
   return cdashUrl+"/queryTests.php?project="+projectName+dateArg+"&"+filterFields
@@ -1701,29 +1707,33 @@ def __call__(self, testDict):
     dateRangeEndDateStr = self.__date
     beginEndUrlFields = "begin="+dateRangeBeginDateStr+"&end="+dateRangeEndDateStr
 
+    # normalize names for query
+    projectName_url, buildName_url, testname_url, site_url = normalizeUrlStrings(
+      projectName, buildName, testname, site)
+
     # Define queryTests.php query filters for test history
     testHistoryQueryFilters = \
       beginEndUrlFields+"&"+\
       "filtercombine=and&filtercombine=&filtercount=3&showfilters=1&filtercombine=and"+\
-      "&field1=buildname&compare1=61&value1="+buildName+\
-      "&field2=testname&compare2=61&value2="+testname+\
-      "&field3=site&compare3=61&value3="+site
+      "&field1=buildname&compare1=61&value1="+buildName_url+\
+      "&field2=testname&compare2=61&value2="+testname_url+\
+      "&field3=site&compare3=61&value3="+site_url
 
     # URL used to get the history of the test in JSON form
     testHistoryQueryUrl = \
-      getCDashQueryTestsQueryUrl(cdashUrl, projectName, None, testHistoryQueryFilters)
+      getCDashQueryTestsQueryUrl(cdashUrl, projectName_url, None, testHistoryQueryFilters)
 
     # URL to embed in email to show the history of the test to humans
     testHistoryBrowserUrl = \
-      getCDashQueryTestsBrowserUrl(cdashUrl, projectName, None, testHistoryQueryFilters)
+      getCDashQueryTestsBrowserUrl(cdashUrl, projectName_url, None, testHistoryQueryFilters)
 
     # URL for to the build summary on index.php page
     buildHistoryEmailUrl = getCDashIndexBrowserUrl(
-      cdashUrl, projectName, None,
+      cdashUrl, projectName_url, None,
       beginEndUrlFields+"&"+\
       "filtercombine=and&filtercombine=&filtercount=2&showfilters=1&filtercombine=and"+\
-      "&field1=buildname&compare1=61&value1="+buildName+\
-      "&field2=site&compare2=61&value2="+site
+      "&field1=buildname&compare1=61&value1="+buildName_url+\
+      "&field2=site&compare2=61&value2="+site_url
       )
     # ToDo: Replace this with the the URL to just this one build the index.php
     # page.  To do that, get the build stamp from the list of builds on CDash

cmake/tribits/ci_support/TribitsDumpDepsXmlScript.cmake

@@ -100,16 +100,16 @@ endif()
 get_filename_component( ${PROJECT_NAME}_TRIBITS_DIR  "${CMAKE_CURRENT_LIST_DIR}/.."  ABSOLUTE )
 message("-- Setting ${PROJECT_NAME}_TRIBITS_DIR=${${PROJECT_NAME}_TRIBITS_DIR}")
 
+include("${CMAKE_CURRENT_LIST_DIR}/../core/common/TribitsConstants.cmake")
+tribits_asesrt_minimum_cmake_version()
+include("${CMAKE_CURRENT_LIST_DIR}/../core/common/TribitsCMakePolicies.cmake"  NO_POLICY_SCOPE)
+
 set( CMAKE_MODULE_PATH
   "${${PROJECT_NAME}_TRIBITS_DIR}/core/utils"
   "${${PROJECT_NAME}_TRIBITS_DIR}/core/package_arch"
   "${${PROJECT_NAME}_TRIBITS_DIR}/ci_support"
   )
 
-include(TribitsConstants)
-tribits_asesrt_minimum_cmake_version()
-include(TribitsCMakePolicies  NO_POLICY_SCOPE)
-
 include(TribitsGlobalMacros)
 include(TribitsPrintDependencyInfo)
 include(TribitsWriteXmlDependenciesFiles)

cmake/tribits/ci_support/TribitsGetExtraReposForCheckinTest.cmake

@@ -82,11 +82,12 @@ set(${PROJECT_NAME}_CHECK_EXTRAREPOS_EXIST  ${CHECK_EXTRAREPOS_EXIST})
 # B) Include files from TriBITS
 #
 
+include("${CMAKE_CURRENT_LIST_DIR}/../core/common/TribitsCMakePolicies.cmake"  NO_POLICY_SCOPE)
+
 set( CMAKE_MODULE_PATH
   "${${PROJECT_NAME}_TRIBITS_DIR}/core/utils"
   "${${PROJECT_NAME}_TRIBITS_DIR}/core/package_arch"
   )
-include(TribitsCMakePolicies  NO_POLICY_SCOPE)
 include(Split)
 include(AppendStringVar)
 include(SetDefaultAndFromEnv) # Used in ExtraRepositoriesList.cmake file?

cmake/tribits/ci_support/TribitsWriteXmlDependenciesFiles.cmake

@@ -45,6 +45,9 @@
 ################################################################################
 
 
+include(TribitsConfigureTiming)
+
+
 # @MACRO: tribits_write_xml_dependency_files()
 #
 # Usage::

cmake/tribits/ci_support/cdash_analyze_and_report.py

@@ -542,6 +542,10 @@ def getTestHistory(self, testLOD):
     # D.2.a) Get list of dicts of builds off cdash/index.phpp
     #
 
+    # @arghdos: note, we do not have to normalize the URLs from the input
+    # options because they are currently taken from the cdash site already
+    # (i.e., they are already in normalized form).
+
     cdashIndexBuildsBrowserUrl = CDQAR.getCDashIndexBrowserUrl(
       inOptions.cdashSiteUrl, inOptions.cdashProjectName, inOptions.date,
       inOptions.cdashBuildsFilters)

cmake/tribits/common_tpls/utils/ParseLibraryList.cmake

@@ -100,7 +100,7 @@ function(parse_library_list)
     endforeach()
 
 
-    # Now set output vairables
+    # Now set output variables
     set(${PARSE_ARGS_DEBUG}     "${_debug_libs}" PARENT_SCOPE)
     set(${PARSE_ARGS_OPT}       "${_opt_libs}"   PARENT_SCOPE)
     set(${PARSE_ARGS_GENERAL}   "${_gen_libs}"   PARENT_SCOPE)

cmake/tribits/core/package_arch/TribitsConstants.cmake → cmake/tribits/core/common/TribitsConstants.cmake

@@ -39,7 +39,7 @@
 
 # Define the TriBITS minimum required CMake version
 
-set(TRIBITS_CMAKE_MINIMUM_REQUIRED 3.22.0)
+set(TRIBITS_CMAKE_MINIMUM_REQUIRED 3.23.0)
 
 macro(tribits_asesrt_minimum_cmake_version)
 

cmake/tribits/core/config_tests/fmangle/CMakeLists.txt

@@ -37,7 +37,7 @@
 # ************************************************************************
 # @HEADER
 
-cmake_minimum_required(VERSION 3.22)
+cmake_minimum_required(VERSION 3.23)
 project(fmangle C Fortran)
 add_definitions(${COMMON_DEFS})
 set(CMAKE_VERBOSE_MAKEFILE ON)

cmake/tribits/core/package_arch/TribitsAddExecutable.cmake

@@ -38,9 +38,9 @@
 # @HEADER
 
 
-include(TribitsAddExecutableTestHelpers)
+include("${CMAKE_CURRENT_LIST_DIR}/../test_support/TribitsAddExecutableTestHelpers.cmake")
+include("${CMAKE_CURRENT_LIST_DIR}/../test_support/TribitsAddTestHelpers.cmake")
 include(TribitsCommonArgsHelpers)
-include(TribitsAddTestHelpers)
 include(TribitsGeneralMacros)
 include(TribitsLibIsTestOnly)
 include(TribitsReportInvalidTribitsUsage)

cmake/tribits/core/package_arch/TribitsAddExecutableAndTest.cmake

@@ -38,8 +38,9 @@
 # @HEADER
 
 
+include("${CMAKE_CURRENT_LIST_DIR}/../test_support/TribitsAddTest.cmake")
+
 include(TribitsAddExecutable)
-include(TribitsAddTest)
 include(TribitsDeprecatedHelpers)
 
 

cmake/tribits/core/package_arch/TribitsAddLibrary.cmake

@@ -271,7 +271,7 @@ include(TribitsSetAndIncDirs)
 # ``${CMAKE_INSTALL_PREFIX}/lib/`` (actual install directory is given by
 # ``${PROJECT}_INSTALL_LIB_DIR``, see `Setting the install prefix`_).
 # However, this install target will not get created if
-# `${PROJECT_NAME}_INSTALL_LIBRARIES_AND_HEADERS`_ is ``FASLE`` and
+# `${PROJECT_NAME}_INSTALL_LIBRARIES_AND_HEADERS`_ is ``FALSE`` and
 # ``BUILD_SHARD_LIBS=OFF``.  But when ``BUILD_SHARD_LIBS=ON``, the install
 # target will get added.  Also, this install target will *not* get added if
 # ``TESTONLY`` or ``NO_INSTALL_LIB_OR_HEADERS`` are passed in.
@@ -280,7 +280,7 @@ include(TribitsSetAndIncDirs)
 # added using ``install(FILES <h0> <h1> ...)``, but only if ``TESTONLY`` and
 # ``NO_INSTALL_LIB_OR_HEADERS`` are not passed in as well.  Also, the install
 # target for the headers will not get added if
-# `${PROJECT_NAME}_INSTALL_LIBRARIES_AND_HEADERS`_ is ``FASLE``.  If this
+# `${PROJECT_NAME}_INSTALL_LIBRARIES_AND_HEADERS`_ is ``FALSE``.  If this
 # install target is added, then the headers get installed into the flat
 # directory ``${${PROJECT_NAME}_INSTALL_INCLUDE_DIR}/`` (default is
 # ``${CMAKE_INSTALL_PREFIX}/include/``, see `Setting the install prefix`_).