Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup the DAP4 support in the tds repository #394

Merged
merged 9 commits into from
Jan 22, 2024

Conversation

DennisHeimbigner
Copy link
Collaborator

@DennisHeimbigner DennisHeimbigner commented Jul 8, 2023

Description of Changes

This PR make major changes to the DAP4 code. It also makes some small but necessary changes to non-DAP4 code, which will be described below.

WARNING: The merging of this PR needs to be synchronized with the following netcdf-java PR: Unidata/netcdf-java#1211. So after both PRs are reviewed and approved, both need to be merged one right after the other.

DAP4 Change Overview

  • Cleanup the D4TS server -- D4TS standing for Dap4 Test Server -- so that it properly operates in the remotetest.unidata.ucar.edu test server.
  • Remove all DAP4 tests; they all depended on using MockServlet, which basically does not work. The assumption is that we will test dap4 support in the process of testing DAP4 support in netcdf-java and netcdf-c.
  • The DAP4 service in TDS is left disabled until everything has been successfully merged.
  • Update gradle files to reflect the flattening of the netcdf-java/dap4 module from PR Clean up DAP4 code plus minor mods to other -- non-dap4 -- code netcdf-java#1133.

DAP4 Specific Changes

  • Remove all DSP types and replace with the CDMWrap class.
  • Move the Odometer classes from netcdf-java to tds since they are only used in that repo.
  • Remove the attempt to replace netcdf-c JNA system. Now relies only on CDM API.
  • Remove the synthetic test case generator and all supporting classes.
  • Remove the dap4/d4tests directory since there are no longer any tests.
  • Change the gradle "compile" action to the "implementation" to conform to the upcoming gradle version 7 (this change is only partially completed).
  • Move the various D4TS support files into the WEB-INF directory so the from() actions are no longer needed.
  • Create a new set of .nc files to be used for testing clients. These files come from and are consistent with those expected by netcdf-c.
  • Create a new D4TS server web page template (the so-called "FrontPage").
  • Created DSR template files to support per-dataset creation of DSRs.
  • Rebuilt the handling of the getResourcePath function to make it dependent on the specific DAP4 Servlet/Controller (i.e. under d4ts vs under thredds).
  • Added an abstract getWebContentRoot function to DapController so that each servlet can provide specific locations for the template files.
  • When a zero-size dimension is encountered, now suppress that dimension and any variable that used it. Ideally this should be an error, but we have encountered too many instances in the wild, so it is better to suppress rather than cause an error.
  • A number of classes were renamed to more accurately reflect their semantics. Specifically:
    • CDMCursor -> CDMData
  • Convert DSP LRU cache to only cache the NetcdfFile instances.
  • Remove some now unused classes:
    • URLMap
    • URLMapDefault
  • Dap4Controller was modified to enable DAP4 as a service.
  • Properly handle checksums in the DMR and DAP.

Non-DAP4 Specific Changes

  • Modify CatalogViewContextParser to map a request for file using the DAP4 service: change from .dmr.xml to .dsr.html. This may change again in the future if we try to emulate the DAP2 .ascii format.
  • Re-enable the Dap4Controller
  • Enable the DAP4 service in various test catalogs.
  • The changes to the intercept code in netcdf-java required changes to some tds tests; specificially TestFormBuilder (also converted to JUNIT4 parameterized test format).
  • Some constants and static utility functions were moved into the CDM class. Currently, this only affects one file outside of DAP4, namely the opendap file NcDAS.java.
  • Change occurrences of "static protected|public" to "protected|public static".

PR Checklist

  • Link to any issues that the PR addresses
  • Add labels
  • Open as a draft PR until ready for review
  • Make sure GitHub tests pass
  • Mark PR as "Ready for Review"

This change is Reviewable

## Description of Changes

WARNING: The merging of this PR needs to be synchronized with the following netcdf-java PR: Unidata/netcdf-java#1211 So after both PRs are reviewed and approved, both need to be merged one right
after the other.

This PR make major changes to the DAP4 code. It also makes some small but necessary changes to non-DAP4 code, which will be described below.

### DAP4 Change Overview

* Cleanup the D4TS server -- D4TS standing for Dap4 Test Server -- so that it properly operates in the remotetest.unidata.ucar.edu test server.
* Remove all DAP4 tests; they all depended on using MockServlet, which basically does not work. The assumption is that we will test dap4 support in the process of testing DAP4 support in netcdf-java and netcdf-c.
* The DAP4 service in TDS is left disabled until everything has been successfully merged.
* Update gradle files to reflect the flattening of the netcdf-java/dap4 module from PR Unidata/netcdf-java#1133.

#### DAP4 Specific Changes
* Remove all DSP types and replace with the *CDMWrap* class.
* Move the Odometer classes from *netcdf-java* to *tds* since they are only used in that repo.
* Remove the attempt to replace netcdf-c JNA system. Now relies only on CDM API.
* Remove the synthetic test case generator and all supporting classes.
* Remove the dap4/d4tests directory since there are no longer any tests.
* Change the gradle "compile" action to the "implementation" to conform to the upcoming gradle version 7 (this change is only partially completed).
* Move the various D4TS support files into the WEB-INF directory so the from() actions are no longer needed.
* Create a new set of .nc files to be used for testing clients. These files come from and are consistent with those expected by netcdf-c.
* Create a new D4TS server web page template (the so-called "FrontPage").
* Created DSR template files to support per-dataset creation of DSRs.
* Rebuilt the handling of the *getResourcePath* function to make it dependent on the specific DAP4 Servlet/Controller (i.e. under d4ts vs under thredds).
* Added an abstract *getWebContentRoot* function to *DapController* so that each servlet can provide specific locations for the template files.
* When a zero-size dimension is encountered, now suppress that dimension and any variable that used it. Ideally this should be an error, but we have encountered too many instances in the wild, so it is better to suppress rather than cause an error.
* A number of classes were renamed to more accurately reflect their semantics. Specifically:
  - *CDMCursor* -> *CDMData*
* Convert DSP LRU cache to only cache the NetcdfFile instances.
* Remove some now unused classes:
  - *URLMap*
  - *URLMapDefault*
* *Dap4Controller* was modified to enable DAP4 as a service.
* Properly handle checksums in the DMR and DAP.

### Non-DAP4 Specific Changes
* Modify CatalogViewContextParser to map a request for file using the DAP4 service: change from *.dmr.xml* to *.dsr.html*. This may change again in the future if we try to emulate the DAP2 .ascii format.
* Re-enable the Dap4Controller
* Enable the DAP4 service in various test catalogs.
* The changes to the intercept code in netcdf-java required changes to some tds tests; specificially *TestFormBuilder* (also converted to JUNIT4 parameterized test format).
* Some constants and static utility functions were moved into the CDM class.
Currently, this only affects one file outside of DAP4, namely the opendap file NcDAS.java.
* Change occurrences of "static protected|public" to "protected|public static".

## PR Checklist
<!-- This will become an interactive checklist once the PR is opened -->
- [X] Link to any issues that the PR addresses
- [ ] Add labels
- [X] Open as a [draft PR](https://github.blog/2019-02-14-introducing-draft-pull-requests/) until ready for review
- [X] Make sure GitHub tests pass
- [X] Mark PR as "Ready for Review"
@DennisHeimbigner DennisHeimbigner added bug Something isn't working enhancement New feature or request service:dap4 DAP4 protocol labels Jul 8, 2023
@DennisHeimbigner DennisHeimbigner marked this pull request as ready for review October 12, 2023 20:41
@@ -24,6 +24,10 @@ dependencies {
api 'org.apache.taglibs:taglibs-standard-spec'
api 'org.apache.taglibs:taglibs-standard-impl'

// threddsIso

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tdrwenski will this mess with the TDS and threddsISO dependencies?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, good catch-- these lines should be removed (see threddsIso removal PR)

@haileyajohnson haileyajohnson merged commit 65b9065 into Unidata:main Jan 22, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request service:dap4 DAP4 protocol
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants