2024_08_28 #192
Replies: 2 comments 5 replies
-
|
@srstsavage Meeting notes, also there are action items I wrote with your name on them. |
Beta Was this translation helpful? Give feedback.
-
Test data could also be stored using releases. I don't believe release data incurs costs the same way as LFS. I believe some IOOS repos use this pattern.
Is the plan to set up a new repo/org on Docker Hub once the image building is moved here? We could set things up so that we push to both Axiom's (if they will share the keys) and another for a while until folks switch over. I may be able help out some with docker build/delivery once automated testing (even if it is local) is figured out. I think I had a checklist of what needed to happen and a draft from a previous attempt to upstream building. |
Beta Was this translation helpful? Give feedback.
-
As long as the files are not bigger than 2G we can use releases for data. Here is an example where the CalVer releases are data and the SemVer ones the Software: https://github.com/pyoceans/pocean-core/releases We download that at test time with a script. |
Beta Was this translation helpful? Give feedback.
-
|
Chris said ~10GB of test data was currently needed for the short tests IIRC. Git LFS free tier limits are lower than I remember (1GB). Maybe we'll have to just improve the Maven download/unzip workflow after all, perhaps by moving the download/unzip plugin into a profile and only activating that profile if the target directories don't exist... |
Beta Was this translation helpful? Give feedback.
-
Single file? B/c GH allows for multiple 2G files in releases. It is kind of a loophole but one can store +10G as long as no individual files is greater than 2G. |
Beta Was this translation helpful? Give feedback.
-
|
The test data is currently served using releases: https://github.com/ERDDAP/erddapTest/releases Currently maven downloads and unzips the data (and then copies it to the build output directory). The current workflow is smart enough to only download the files if needed (that's what the download_cache folder is for), but it always unzips (which is slow). Waiting for the unzip on every test run is bad (once I have the files in a local repo I often end up commenting out the entire download section of the pom). We also weren't sure if this current flow will work well with automated test runs. The zipped files are around 5gb total, unzipped is a little under 9gb (I just checked the sizes). There's additional tests that are currently disabled that use an additional 130gb of files. Looking for better solutions is about improving the developer experience (faster test iteration cycles) and removing any road blocks for automated tests. I don't know if there's a great solution for having this much test data. Most of the test data is real datasets (or partial datasets). This is great for verifying the server works on real world data. One option is to use dedicated test files that could be much smaller while still verifying ERDDAP functionality. That would require a lot of work to generate specific test files and rewrite pretty much all of the tests. |
Beta Was this translation helpful? Give feedback.
-
|
Got it. Definitely easier to find a way to use those the way they are. I don't have any good ideas for that but I'll ask around and report back if I find anything useful. |
Beta Was this translation helpful? Give feedback.
-
1) Discuss/approve a style guide.
Currently using Google style guide. No objections to that.
Current setup maven installs a git hook to run a code formatter locally on git submit.
It's also easy to have mvn verify check all code is properly formatted (to be turned on once all code has been formatted).
We have not done a full update pass to format the code. We want code in pull requests to have been formatted, but formatting a file before review means the actual changes can be hard to see.
Seth votes for a big format, addresses the issue one time.
2) Go over the priority projects and discuss any changes we'd like to make.
Current status of projects:
Chris:
Separation of slow tests/integration tests was removed from the list because I have a change locally to do that. The slow/integration tests are now run using failsafe in the integration test step of maven. Pull request coming very soon (trying to track down some flaky tests).
XInclude support is theoretically done with the SAX parser work by GSoC student Aysuh, but I didn’t want to close the bug until the functionality was verified (soon).
Automated testing on github repo- Shane motivated to help out there.
Suggestiong for using Git LFS for test data. Need to check quotas and cost.
Official Docker support discussed. Big step is moving the repo into the ERDDAP organization on GitHub.
Can Docker hub automatically forward if the repo is moved?
3) Plan what work we are targeting for the next release (2.25) and what the target release date is.
In the past releases were quarterly.
We had a release in June, so another in Sept or Oct would be good.
Changes already in:
New standard XML parser (that supports XInclude, but XInclude not yet tested)
Zarr support (with tests, but not production tested)
Bug Fixes:
Weird interaction where making a WMS query on a dataset could cause incorrect results in other queries
Email usernames for smtp login no longer need to be a complete email address
EDDTableFromFiles can support queries derived outputs (Specifically requests with no data in the source files (only globals, jexl script, or variables) will now function.)
Performance Improvements:
EDDTableFromMultidimNcFiles can respond much faster to requests now if removeMVRows is set to false (may not be do-able for all datasets)
Chris: Additional things we'd like to include in 2.25
There is a hackathon in the second half of September which has planned work on adding localized metadata support to ERDDAP. Assuming that goes well it should be in the release.
Some kind of metrics, likely at least a basic prometheus integration, additional metrics in prometheus (like those tracked for the status page) would be good, possible also bringing Callum’s log processing+dataset into ERDDAP).
Release Cadence
Shane supports contining approximately quarterly releases. Good balance of getting work out regularly but not requiring admins to update too frequently.
Quarterly releases would put the following release (2.26) during holiday season. Tentatively scheduling it for Jan/Feb.
Are there things we want to tentatively schedule for the 2.26 release?
Seth: Java resources/hardcoded paths technical debt issues. Already working on this, needed for NERDDAP (using JBoss to run)
This would allow us to move to a standard java project directory structure.
Seth and Shane both interested in making it easier to add custom dataset types. Maybe a pluggable interface for custom EDDTable/EDDGrid implementations. Java service loading approach?
Shane recommends adding #96 to the priority projects list
Action Items:
Beta Was this translation helpful? Give feedback.
All reactions