From 39cb6bacbbf3b28813c28aada5a71885c2f13728 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Sat, 29 Apr 2023 19:03:22 +0200 Subject: [PATCH 01/10] Clarify temporal intervals #331 (#394) * The temporal intervals must always be non-empty, i.e. the second instance in time must be after the first instance in time. #331 * Add uniqueItems, remove mention of 24 as the hour, remove temporal-interval subtype from climatological_normal (as it has inclusive upper boundaries) * Improved terminology * Update load_stac * Updates according to recent discussions #331 * Apply suggestions from code review Co-authored-by: Stefaan Lippens * Remove timezone from times --------- Co-authored-by: Stefaan Lippens --- CHANGELOG.md | 10 +++- aggregate_temporal.json | 37 ++++++------ climatological_normal.json | 13 ++--- filter_temporal.json | 23 ++++---- load_collection.json | 24 ++++---- meta/subtype-schemas.json | 28 +++++---- order.json | 5 -- proposals/date_between.json | 112 ++++++++++++++++++++++++++++++++++++ proposals/load_stac.json | 58 +++++++++++++++---- sort.json | 10 ---- 10 files changed, 233 insertions(+), 87 deletions(-) create mode 100644 proposals/date_between.json diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d268dd9..acb2bbe4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - New processes in proposal state: + - `date_between` - `date_difference` - `filter_vector` - `flatten_dimensions` @@ -39,9 +40,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Allow `null` as default value for units. - Input and Output for the `process` can either be data cubes or arrays (if one-dimensional). [#387](https://github.com/Open-EO/openeo-processes/issues/387) - `run_udf`: Allow all data types instead of just objects in the `context` parameter. [#376](https://github.com/Open-EO/openeo-processes/issues/376) -- `load_collection` and `load_result`: +- `load_collection` and `load_result`/`load_stac`: - Require at least one band if not set to `null`. [#372](https://github.com/Open-EO/openeo-processes/issues/372) - Added a `NoDataAvailable` exception +- `aggregate_temporal`, `filter_temporal`, `load_collection` and `load_result`/`load_stac`: + - The temporal intervals must always be non-empty, i.e. the second instance in time must be after the first instance in time. [#331](https://github.com/Open-EO/openeo-processes/issues/331) + - `24` as the hour is not allowed anymore. [#331](https://github.com/Open-EO/openeo-processes/issues/331) - `inspect`: The parameter `message` has been moved to be the second argument. [#369](https://github.com/Open-EO/openeo-processes/issues/369) - `mask` and `merge_cubes`: The spatial dimensions `x` and `y` can now be resampled implicitly instead of throwing an error. [#402](https://github.com/Open-EO/openeo-processes/issues/402) - `save_result`: Added a more concrete `DataCubeEmpty` exception. @@ -53,6 +57,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Updated the processes based on the subtypes `raster-cube` or `vector-cube` to work with the subtype `datacube` instead. [#68](https://github.com/Open-EO/openeo-processes/issues/68) - `sort` and `order`: The ordering of ties is not defined anymore. [#409](https://github.com/Open-EO/openeo-processes/issues/409) - `quantiles`: Parameter `probabilities` provided as array must be in ascending order. [#297](https://github.com/Open-EO/openeo-processes/pull/297) +- `climatological_normal`: The `climatology_period` parameter accepts an array of integers instead of strings. [#331](https://github.com/Open-EO/openeo-processes/issues/331) ### Deprecated @@ -61,7 +66,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Removed - The `examples` folder has been migrated to the [openEO Community Examples](https://github.com/Open-EO/openeo-community-examples/tree/main/processes) repository. -- `between`: Support for temporal comparison. +- `between`: Support for temporal comparison. Use `date_between` instead. [#331](https://github.com/Open-EO/openeo-processes/issues/331) - Deprecated `GeometryCollections` are not supported any longer. [#389](https://github.com/Open-EO/openeo-processes/issues/389) - Deprecated PROJ definitions for the CRS are not supported any longer. - `load_result`: @@ -71,6 +76,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - The comparison processes `eq`, `neq`, `lt`, `lte`, `gt`, `gte` and `array_contains`: - Removed support for temporal comparison. Instead explicitly use `date_difference`. - Removed support for the input data types array and object. [#208](https://github.com/Open-EO/openeo-processes/issues/208) +- `sort` and `order`: Removed support for time-only values. [#331](https://github.com/Open-EO/openeo-processes/issues/331) ### Fixed diff --git a/aggregate_temporal.json b/aggregate_temporal.json index 1a9e4b09..ad1d560b 100644 --- a/aggregate_temporal.json +++ b/aggregate_temporal.json @@ -22,7 +22,7 @@ }, { "name": "intervals", - "description": "Left-closed temporal intervals, which are allowed to overlap. Each temporal interval in the array has exactly two elements:\n\n1. The first element is the start of the temporal interval. The specified instance in time is **included** in the interval.\n2. The second element is the end of the temporal interval. The specified instance in time is **excluded** from the interval.\n\nThe specified temporal strings follow [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html). Although [RFC 3339 prohibits the hour to be '24'](https://www.rfc-editor.org/rfc/rfc3339.html#section-5.7), **this process allows the value '24' for the hour** of an end time in order to make it possible that left-closed time intervals can fully cover the day.", + "description": "Left-closed temporal intervals, which are allowed to overlap. Each temporal interval in the array has exactly two elements:\n\n1. The first element is the start of the temporal interval. The specified time instant is **included** in the interval.\n2. The second element is the end of the temporal interval. The specified time instant is **excluded** from the interval.\n\nThe second element must always be greater/later than the first element, except when using time without date. Otherwise, a `TemporalExtentEmpty` exception is thrown.", "schema": { "type": "array", "subtype": "temporal-intervals", @@ -30,6 +30,7 @@ "items": { "type": "array", "subtype": "temporal-interval", + "uniqueItems": true, "minItems": 2, "maxItems": 2, "items": { @@ -37,24 +38,20 @@ { "type": "string", "format": "date-time", - "subtype": "date-time" + "subtype": "date-time", + "description": "Date and time with a time zone." }, { "type": "string", "format": "date", - "subtype": "date" + "subtype": "date", + "description": "Date only, formatted as `YYYY-MM-DD`. The time zone is UTC. Missing time components are all 0." }, { "type": "string", - "format": "time", - "subtype": "time" - }, - { - "type": "string", - "subtype": "year", - "minLength": 4, - "maxLength": 4, - "pattern": "^\\d{4}$" + "subtype": "time", + "pattern": "^\\d{2}:\\d{2}:\\d{2}$", + "description": "Time only, formatted as `HH:MM:SS`. The time zone is UTC." }, { "type": "null" @@ -79,12 +76,12 @@ ], [ [ - "00:00:00Z", - "12:00:00Z" + "06:00:00", + "18:00:00" ], [ - "12:00:00Z", - "24:00:00Z" + "18:00:00", + "06:00:00" ] ] ] @@ -235,6 +232,9 @@ }, "DistinctDimensionLabelsRequired": { "message": "The dimension labels have duplicate values. Distinct labels must be specified." + }, + "TemporalExtentEmpty": { + "message": "At least one of the intervals is empty. The second instant in time must always be greater/later than the first instant." } }, "links": [ @@ -242,6 +242,11 @@ "href": "https://openeo.org/documentation/1.0/datacubes.html#aggregate", "rel": "about", "title": "Aggregation explained in the openEO documentation" + }, + { + "href": "https://www.rfc-editor.org/rfc/rfc3339.html", + "rel": "about", + "title": "RFC3339: Details about formatting temporal strings" } ] } diff --git a/climatological_normal.json b/climatological_normal.json index 8d97ef5a..fa3441b6 100644 --- a/climatological_normal.json +++ b/climatological_normal.json @@ -39,20 +39,17 @@ "description": "The climatology period as a closed temporal interval. The first element of the array is the first year to be fully included in the temporal interval. The second element is the last year to be fully included in the temporal interval.\n\nThe default climatology period is from 1981 until 2010 (both inclusive) right now, but this might be updated over time to what is commonly used in climatology. If you don't want to keep your research to be reproducible, please explicitly specify a period.", "schema": { "type": "array", - "subtype": "temporal-interval", + "uniqueItems": true, "minItems": 2, "maxItems": 2, "items": { - "type": "string", - "subtype": "year", - "minLength": 4, - "maxLength": 4, - "pattern": "^\\d{4}$" + "type": "integer", + "subtype": "year" } }, "default": [ - "1981", - "2010" + 1981, + 2010 ], "optional": true } diff --git a/filter_temporal.json b/filter_temporal.json index c873645b..4a463674 100644 --- a/filter_temporal.json +++ b/filter_temporal.json @@ -22,7 +22,7 @@ }, { "name": "extent", - "description": "Left-closed temporal interval, i.e. an array with exactly two elements:\n\n1. The first element is the start of the temporal interval. The specified instance in time is **included** in the interval.\n2. The second element is the end of the temporal interval. The specified instance in time is **excluded** from the interval.\n\nThe specified temporal strings follow [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html). Also supports open intervals by setting one of the boundaries to `null`, but never both.", + "description": "Left-closed temporal interval, i.e. an array with exactly two elements:\n\n1. The first element is the start of the temporal interval. The specified time instant is **included** in the interval.\n2. The second element is the end of the temporal interval. The specified time instant is **excluded** from the interval.\n\nThe second element must always be greater/later than the first element. Otherwise, a `TemporalExtentEmpty` exception is thrown.\n\nAlso supports unbounded intervals by setting one of the boundaries to `null`, but never both.", "schema": { "type": "array", "subtype": "temporal-interval", @@ -33,19 +33,14 @@ { "type": "string", "format": "date-time", - "subtype": "date-time" + "subtype": "date-time", + "description": "Date and time with a time zone." }, { "type": "string", "format": "date", - "subtype": "date" - }, - { - "type": "string", - "subtype": "year", - "minLength": 4, - "maxLength": 4, - "pattern": "^\\d{4}$" + "subtype": "date", + "description": "Date only, formatted as `YYYY-MM-DD`. The time zone is UTC. Missing time components are all 0." }, { "type": "null" @@ -92,6 +87,9 @@ "exceptions": { "DimensionNotAvailable": { "message": "A dimension with the specified name does not exist." + }, + "TemporalExtentEmpty": { + "message": "The temporal extent is empty. The second instant in time must always be greater/later than the first instant in time." } }, "links": [ @@ -99,6 +97,11 @@ "href": "https://openeo.org/documentation/1.0/datacubes.html#filter", "rel": "about", "title": "Filters explained in the openEO documentation" + }, + { + "href": "https://www.rfc-editor.org/rfc/rfc3339.html", + "rel": "about", + "title": "RFC3339: Details about formatting temporal strings" } ] } diff --git a/load_collection.json b/load_collection.json index 49df9650..b93c879c 100644 --- a/load_collection.json +++ b/load_collection.json @@ -112,11 +112,12 @@ }, { "name": "temporal_extent", - "description": "Limits the data to load from the collection to the specified left-closed temporal interval. Applies to all temporal dimensions. The interval has to be specified as an array with exactly two elements:\n\n1. The first element is the start of the temporal interval. The specified instance in time is **included** in the interval.\n2. The second element is the end of the temporal interval. The specified instance in time is **excluded** from the interval.\n\nThe specified temporal strings follow [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html). Also supports open intervals by setting one of the boundaries to `null`, but never both.\n\nSet this parameter to `null` to set no limit for the temporal extent. Be careful with this when loading large datasets! It is recommended to use this parameter instead of using ``filter_temporal()`` directly after loading unbounded data.", + "description": "Limits the data to load from the collection to the specified left-closed temporal interval. Applies to all temporal dimensions. The interval has to be specified as an array with exactly two elements:\n\n1. The first element is the start of the temporal interval. The specified time instant is **included** in the interval.\n2. The second element is the end of the temporal interval. The specified time instant is **excluded** from the interval.\n\nThe second element must always be greater/later than the first element. Otherwise, a `TemporalExtentEmpty` exception is thrown.\n\nAlso supports unbounded intervals by setting one of the boundaries to `null`, but never both.\n\nSet this parameter to `null` to set no limit for the temporal extent. Be careful with this when loading large datasets! It is recommended to use this parameter instead of using ``filter_temporal()`` directly after loading unbounded data.", "schema": [ { "type": "array", "subtype": "temporal-interval", + "uniqueItems": true, "minItems": 2, "maxItems": 2, "items": { @@ -124,19 +125,14 @@ { "type": "string", "format": "date-time", - "subtype": "date-time" + "subtype": "date-time", + "description": "Date and time with a time zone." }, { "type": "string", "format": "date", - "subtype": "date" - }, - { - "type": "string", - "subtype": "year", - "minLength": 4, - "maxLength": 4, - "pattern": "^\\d{4}$" + "subtype": "date", + "description": "Date only, formatted as `YYYY-MM-DD`. The time zone is UTC. Missing time components are all 0." }, { "type": "null" @@ -231,6 +227,9 @@ "exceptions": { "NoDataAvailable": { "message": "There is no data available for the given extents." + }, + "TemporalExtentEmpty": { + "message": "The temporal extent is empty. The second instant in time must always be greater/later than the first instant in time." } }, "examples": [ @@ -313,6 +312,11 @@ "rel": "about", "href": "https://github.com/radiantearth/stac-spec/tree/master/extensions/eo#common-band-names", "title": "List of common band names as specified by the STAC specification" + }, + { + "href": "https://www.rfc-editor.org/rfc/rfc3339.html", + "rel": "about", + "title": "RFC3339: Details about formatting temporal strings" } ] } diff --git a/meta/subtype-schemas.json b/meta/subtype-schemas.json index b44cb8dc..1a49bf64 100644 --- a/meta/subtype-schemas.json +++ b/meta/subtype-schemas.json @@ -120,7 +120,7 @@ "subtype": "date", "format": "date", "title": "Date only", - "description": "Date only representation, as defined for `full-date` by [RFC 3339 in section 5.6](https://www.rfc-editor.org/rfc/rfc3339.html#section-5.6). The time zone is UTC." + "description": "Date only representation, as defined for `full-date` by [RFC 3339 in section 5.6](https://www.rfc-editor.org/rfc/rfc3339.html#section-5.6). The time zone is UTC. Missing time components are all 0." }, "date-time": { "type": "string", @@ -282,7 +282,8 @@ "type": "array", "subtype": "temporal-interval", "title": "Single temporal interval", - "description": "Left-closed temporal interval, represented as two-element array with the following elements:\n\n1. The first element is the start of the temporal interval. The specified instance in time is **included** in the interval.\n2. The second element is the end of the temporal interval. The specified instance in time is **excluded** from the interval.\n\nThe specified temporal strings follow [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html). Although [RFC 3339 prohibits the hour to be '24'](https://www.rfc-editor.org/rfc/rfc3339.html#section-5.7), **this process allows the value '24' for the hour** of an end time in order to make it possible that left-closed time intervals can fully cover the day. `null` can be used to specify open intervals.", + "description": "Left-closed temporal interval, represented as two-element array with the following elements:\n\n1. The first element is the start of the temporal interval. The specified time instant is **included** in the interval.\n2. The second element is the end of the temporal interval. The specified time instant is **excluded** from the interval.\n\nThe second element must always be greater/later than the first element. Otherwise, an exception is thrown.\n\nThe specified temporal strings follow [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html). Also supports unbounded intervals by setting one of the boundaries to `null`, but never both.", + "uniqueItems": true, "minItems": 2, "maxItems": 2, "items": { @@ -315,8 +316,8 @@ "2016-01-01" ], [ - "00:00:00Z", - "12:00:00Z" + "00:00:00", + "12:00:00" ], [ "2015-01-01", @@ -350,12 +351,12 @@ ], [ [ - "00:00:00Z", - "12:00:00Z" + "00:00:00", + "12:00:00" ], [ - "12:00:00Z", - "24:00:00Z" + "12:00:00", + null ] ], [ @@ -369,9 +370,9 @@ "time": { "type": "string", "subtype": "time", - "format": "time", + "pattern": "^\\d{2}:\\d{2}:\\d{2}$", "title": "Time only", - "description": "Time only representation, as defined for `full-time` by [RFC 3339 in section 5.6](https://www.rfc-editor.org/rfc/rfc3339.html#section-5.6). Although [RFC 3339 prohibits the hour to be '24'](https://www.rfc-editor.org/rfc/rfc3339.html#section-5.7), this definition allows the value '24' for the hour as end time in an interval in order to make it possible that left-closed time intervals can fully cover the day." + "description": "Time only representation, as defined for `partial-time` by [RFC 3339 in section 5.6](https://www.rfc-editor.org/rfc/rfc3339.html#section-5.6). The time zone is UTC." }, "udf-code": { "type": "string", @@ -413,13 +414,10 @@ "description": "Specifies details about cartographic projections as WKT2 string. Refers to the latest WKT2 version (currently [WKT2:2018](http://docs.opengeospatial.org/is/18-010r7/18-010r7.html) / ISO 19162:2018) unless otherwise stated by the process." }, "year": { - "type": "string", + "type": "integer", "subtype": "year", - "minLength": 4, - "maxLength": 4, - "pattern": "^\\d{4}$", "title": "Year only", - "description": "Year representation, as defined for `date-fullyear` by [RFC 3339 in section 5.6](https://www.rfc-editor.org/rfc/rfc3339.html#section-5.6)." + "description": "Year as integer, can be any number of digits and can be negative." } } } diff --git a/order.json b/order.json index cb8e1681..9b67f52d 100644 --- a/order.json +++ b/order.json @@ -29,11 +29,6 @@ "type": "string", "format": "date", "subtype": "date" - }, - { - "type": "string", - "format": "time", - "subtype": "time" } ] } diff --git a/proposals/date_between.json b/proposals/date_between.json new file mode 100644 index 00000000..e361acf4 --- /dev/null +++ b/proposals/date_between.json @@ -0,0 +1,112 @@ +{ + "id": "date_between", + "summary": "Between comparison for dates and times", + "description": "By default, this process checks whether `x` is later than or equal to `min` and before or equal to `max`.\n\nIf `exclude_max` is set to `true` the upper bound is excluded so that the process checks whether `x` is later than or equal to `min` and before `max`.\n\nLower and upper bounds are not allowed to be swapped. So `min` MUST be before or equal to `max` or otherwise the process always returns `false`.", + "categories": [ + "comparison", + "date & time" + ], + "experimental": true, + "parameters": [ + { + "name": "x", + "description": "The value to check.", + "schema": [ + { + "type": "string", + "format": "date-time", + "subtype": "date-time", + "description": "Date and time with a time zone." + }, + { + "type": "string", + "format": "date", + "subtype": "date", + "description": "Date only, formatted as `YYYY-MM-DD`. The time zone is UTC. Missing time components are all 0." + }, + { + "type": "string", + "subtype": "time", + "pattern": "^\\d{2}:\\d{2}:\\d{2}$", + "description": "Time only, formatted as HH:MM:SS. The time zone is UTC." + } + ] + }, + { + "name": "min", + "description": "Lower boundary (inclusive) to check against.", + "schema": [ + { + "type": "string", + "format": "date-time", + "subtype": "date-time", + "description": "Date and time with a time zone." + }, + { + "type": "string", + "format": "date", + "subtype": "date", + "description": "Date only, formatted as `YYYY-MM-DD`. The time zone is UTC. Missing time components are all 0." + }, + { + "type": "string", + "subtype": "time", + "pattern": "^\\d{2}:\\d{2}:\\d{2}$", + "description": "Time only, formatted as HH:MM:SS. The time zone is UTC." + } + ] + }, + { + "name": "max", + "description": "Upper boundary (inclusive) to check against.", + "schema": [ + { + "type": "string", + "format": "date-time", + "subtype": "date-time", + "description": "Date and time with a time zone." + }, + { + "type": "string", + "format": "date", + "subtype": "date", + "description": "Date only, formatted as `YYYY-MM-DD`. The time zone is UTC. Missing time components are all 0." + }, + { + "type": "string", + "subtype": "time", + "pattern": "^\\d{2}:\\d{2}:\\d{2}$", + "description": "Time only, formatted as HH:MM:SS. The time zone is UTC." + } + ] + }, + { + "name": "exclude_max", + "description": "Exclude the upper boundary `max` if set to `true`. Defaults to `false`.", + "schema": { + "type": "boolean" + }, + "default": false, + "optional": true + } + ], + "returns": { + "description": "`true` if `x` is between the specified bounds, otherwise `false`.", + "schema": { + "type": [ + "boolean", + "null" + ] + } + }, + "examples": [ + { + "arguments": { + "x": "2020-01-01", + "min": "2021-01-01", + "max": "2022-01-01" + }, + "returns": false + } + ] +} diff --git a/proposals/load_stac.json b/proposals/load_stac.json index f37160a1..c71d3a80 100644 --- a/proposals/load_stac.json +++ b/proposals/load_stac.json @@ -116,11 +116,12 @@ }, { "name": "temporal_extent", - "description": "Limits the data to load to the specified left-closed temporal interval. Applies to all temporal dimensions. The interval has to be specified as an array with exactly two elements:\n\n1. The first element is the start of the temporal interval. The specified instance in time is **included** in the interval.\n2. The second element is the end of the temporal interval. The specified instance in time is **excluded** from the interval.\n\nThe specified temporal strings follow [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html). Also supports open intervals by setting one of the boundaries to `null`, but never both.\n\nSet this parameter to `null` to set no limit for the temporal extent. Be careful with this when loading large datasets! It is recommended to use this parameter instead of using ``filter_temporal()`` directly after loading unbounded data.", + "description": "Limits the data to load to the specified left-closed temporal interval. Applies to all temporal dimensions. The interval has to be specified as an array with exactly two elements:\n\n1. The first element is the start of the temporal interval. The specified instance in time is **included** in the interval.\n2. The second element is the end of the temporal interval. The specified instance in time is **excluded** from the interval.\n\nThe second element must always be greater/later than the first element. Otherwise, a `TemporalExtentEmpty` exception is thrown.\n\nAlso supports open intervals by setting one of the boundaries to `null`, but never both.\n\nSet this parameter to `null` to set no limit for the temporal extent. Be careful with this when loading large datasets! It is recommended to use this parameter instead of using ``filter_temporal()`` directly after loading unbounded data.", "schema": [ { "type": "array", "subtype": "temporal-interval", + "uniqueItems": true, "minItems": 2, "maxItems": 2, "items": { @@ -128,19 +129,14 @@ { "type": "string", "format": "date-time", - "subtype": "date-time" + "subtype": "date-time", + "description": "Date and time with a time zone." }, { "type": "string", "format": "date", - "subtype": "date" - }, - { - "type": "string", - "subtype": "year", - "minLength": 4, - "maxLength": 4, - "pattern": "^\\d{4}$" + "subtype": "date", + "description": "Date only, formatted as `YYYY-MM-DD`. The time zone is UTC. Missing time components are all 0." }, { "type": "null" @@ -293,6 +289,46 @@ "exceptions": { "NoDataAvailable": { "message": "There is no data available for the given extents." + }, + "TemporalExtentEmpty": { + "message": "The temporal extent is empty. The second instant in time must always be greater/later than the first instant in time." + } + }, + "links": [ + { + "href": "https://openeo.org/documentation/1.0/datacubes.html", + "rel": "about", + "title": "Data Cubes explained in the openEO documentation" + }, + { + "rel": "about", + "href": "https://proj.org/usage/projections.html", + "title": "PROJ parameters for cartographic projections" + }, + { + "rel": "about", + "href": "http://www.epsg-registry.org", + "title": "Official EPSG code registry" + }, + { + "rel": "about", + "href": "http://www.epsg.io", + "title": "Unofficial EPSG code database" + }, + { + "href": "http://www.opengeospatial.org/standards/sfa", + "rel": "about", + "title": "Simple Features standard by the OGC" + }, + { + "rel": "about", + "href": "https://github.com/radiantearth/stac-spec/tree/master/extensions/eo#common-band-names", + "title": "List of common band names as specified by the STAC specification" + }, + { + "href": "https://www.rfc-editor.org/rfc/rfc3339.html", + "rel": "about", + "title": "RFC3339: Details about formatting temporal strings" } - } + ] } diff --git a/sort.json b/sort.json index 27586250..422f8d53 100644 --- a/sort.json +++ b/sort.json @@ -29,11 +29,6 @@ "type": "string", "format": "date", "subtype": "date" - }, - { - "type": "string", - "format": "time", - "subtype": "time" } ] } @@ -82,11 +77,6 @@ "type": "string", "format": "date", "subtype": "date" - }, - { - "type": "string", - "format": "time", - "subtype": "time" } ] } From bab65c2ac02a2b1391c90c39d4691c40cf1adf87 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Sat, 29 Apr 2023 19:09:09 +0200 Subject: [PATCH 02/10] New definitions for fit_curve and predict_curve (#420) * Fix description of fit_curve * Fine-tune description * New version for fit_curve and predict_curve * fit_curve: Use a labeled-array for data --- CHANGELOG.md | 1 + proposals/fit_curve.json | 64 +++++++++++++++++------------------- proposals/predict_curve.json | 19 ++--------- 3 files changed, 34 insertions(+), 50 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index acb2bbe4..0eb78e8a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -57,6 +57,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Updated the processes based on the subtypes `raster-cube` or `vector-cube` to work with the subtype `datacube` instead. [#68](https://github.com/Open-EO/openeo-processes/issues/68) - `sort` and `order`: The ordering of ties is not defined anymore. [#409](https://github.com/Open-EO/openeo-processes/issues/409) - `quantiles`: Parameter `probabilities` provided as array must be in ascending order. [#297](https://github.com/Open-EO/openeo-processes/pull/297) +- `fit_curve` and `predict_curve`: Heavily modified specifications. `fit_curve` works on arrays instead of data cubes, `predict_curve` doesn't support gap filling anymore, clarify no-data handling, ... [#425](https://github.com/Open-EO/openeo-processes/issues/425) - `climatological_normal`: The `climatology_period` parameter accepts an array of integers instead of strings. [#331](https://github.com/Open-EO/openeo-processes/issues/331) ### Deprecated diff --git a/proposals/fit_curve.json b/proposals/fit_curve.json index 5d33b652..aafb917e 100644 --- a/proposals/fit_curve.json +++ b/proposals/fit_curve.json @@ -1,38 +1,34 @@ { "id": "fit_curve", "summary": "Curve fitting", - "description": "Use non-linear least squares to fit a model function `y = f(x, parameters)` to data.\n\nThe process throws an `InvalidValues` exception if invalid values are encountered. Valid values are finite numbers (see also ``is_valid()``).", + "description": "Use non-linear least squares to fit a model function `y = f(x, parameters)` to data.", "categories": [ - "cubes", + "arrays", "math" ], "experimental": true, "parameters": [ { "name": "data", - "description": "A data cube.", + "description": "A labeled array, the labels correspond to the variable `y` and the values correspond to the variable `x`.", "schema": { - "type": "object", - "subtype": "datacube" + "type": "array", + "subtype": "labeled-array", + "items": { + "type": "number" + } } }, { "name": "parameters", "description": "Defined the number of parameters for the model function and provides an initial guess for them. At least one parameter is required.", - "schema": [ - { - "type": "array", - "minItems": 1, - "items": { - "type": "number" - } - }, - { - "title": "Data Cube with optimal values from a previous result of this process.", - "type": "object", - "subtype": "datacube" + "schema": { + "type": "array", + "minItems": 1, + "items": { + "type": "number" } - ] + } }, { "name": "function", @@ -45,7 +41,10 @@ "name": "x", "description": "The value for the independent variable `x`.", "schema": { - "type": "number" + "type": [ + "number", + "null" + ] } }, { @@ -69,26 +68,23 @@ } }, { - "name": "dimension", - "description": "The name of the dimension for curve fitting. Must be a dimension with labels that have a order (i.e. numerical labels or a temporal dimension). Fails with a `DimensionNotAvailable` exception if the specified dimension does not exist.", + "name": "ignore_nodata", + "description": "Indicates whether no-data values are ignored or not. Ignores them by default. Setting this flag to `false` considers no-data values so that `null` is passed to the model function.", "schema": { - "type": "string" - } + "type": "boolean" + }, + "default": true, + "optional": true } ], "returns": { - "description": "A data cube with the optimal values for the parameters.", + "description": "An array with the optimal values for the parameters.", "schema": { - "type": "object", - "subtype": "datacube" - } - }, - "exceptions": { - "InvalidValues": { - "message": "At least one of the values is not a finite number." - }, - "DimensionNotAvailable": { - "message": "A dimension with the specified name does not exist." + "type": "array", + "minItems": 1, + "items": { + "type": "number" + } } } } diff --git a/proposals/predict_curve.json b/proposals/predict_curve.json index 9fb5d341..479b7fec 100644 --- a/proposals/predict_curve.json +++ b/proposals/predict_curve.json @@ -1,21 +1,13 @@ { "id": "predict_curve", "summary": "Predict values", - "description": "Predict values using a model function and pre-computed parameters. The process is primarily intended to compute values for new labels, but it can also fill gaps where existing labels contain no-data (`null`) values.", + "description": "Predict values using a model function and pre-computed parameters. The process is intended to compute values for new labels.", "categories": [ "cubes", "math" ], "experimental": true, "parameters": [ - { - "name": "data", - "description": "A data cube to predict values for.", - "schema": { - "type": "object", - "subtype": "datacube" - } - }, { "name": "parameters", "description": "A data cube with optimal values, e.g. computed by the process ``fit_curve()``.", @@ -60,7 +52,7 @@ }, { "name": "dimension", - "description": "The name of the dimension for predictions. Fails with a `DimensionNotAvailable` exception if the specified dimension does not exist.", + "description": "The name of the dimension for predictions.", "schema": { "type": "string" } @@ -98,15 +90,10 @@ } ], "returns": { - "description": "A data cube with the predicted values.", + "description": "A data cube with the predicted values with the provided dimension `dimension` having as many labels as provided through `labels`.", "schema": { "type": "object", "subtype": "datacube" } - }, - "exceptions": { - "DimensionNotAvailable": { - "message": "A dimension with the specified name does not exist." - } } } From 32ec399dddf8102395277c86a0b5b7b576582980 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 4 May 2023 17:51:45 +0200 Subject: [PATCH 03/10] Handle units in vector processes #330 (#436) * Handle units in vector processes #330 Co-authored-by: Daniel Thiex <60705209+dthiex@users.noreply.github.com> --- proposals/vector_buffer.json | 7 ++++++- proposals/vector_to_regular_points.json | 7 ++++++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/proposals/vector_buffer.json b/proposals/vector_buffer.json index 9c4f86ae..05be9a58 100644 --- a/proposals/vector_buffer.json +++ b/proposals/vector_buffer.json @@ -22,7 +22,7 @@ }, { "name": "distance", - "description": "The distance of the buffer in the unit of the spatial reference system. A positive distance expands the geometries and results in outward buffering (dilation) while a negative distance shrinks the geometries and results in inward buffering (erosion).", + "description": "The distance of the buffer in meters. If the unit of the spatial reference system is not meters, a `UnitMismatch` error is thrown. Use ``vector_reproject()`` to convert the geometries to a suitable spatial reference system.\n\nA positive distance expands the geometries and results in outward buffering (dilation) while a negative distance shrinks the geometries and results in inward buffering (erosion).", "schema": { "type": "number", "not": { @@ -42,5 +42,10 @@ } ] } + }, + "exceptions": { + "UnitMismatch": { + "message": "The unit of the spatial reference system is not meters, but the given distance is in meters." + } } } diff --git a/proposals/vector_to_regular_points.json b/proposals/vector_to_regular_points.json index 2f353bdb..c7f7089c 100644 --- a/proposals/vector_to_regular_points.json +++ b/proposals/vector_to_regular_points.json @@ -23,7 +23,7 @@ }, { "name": "distance", - "description": "Defines the minimum distance in the unit of the reference system that is required between two samples generated *inside* a single geometry.\n\n- For **polygons**, the distance defines the cell sizes of a regular grid that starts at the upper-left bound of each polygon. The centroid of each cell is then a sample point. If the centroid is not enclosed in the polygon, no point is sampled. If no point can be sampled for the geometry at all, the first coordinate of the geometry is returned as point.\n- For **lines** (line strings), the sampling starts with a point at the first coordinate of the line and then walks along the line and samples a new point each time the distance to the previous point has been reached again.\n- For **points**, the point is returned as given.", + "description": "Defines the minimum distance in meters that is required between two samples generated *inside* a single geometry. If the unit of the spatial reference system is not meters, a `UnitMismatch` error is thrown. Use ``vector_reproject()`` to convert the geometries to a suitable spatial reference system.\n\n- For **polygons**, the distance defines the cell sizes of a regular grid that starts at the upper-left bound of each polygon. The centroid of each cell is then a sample point. If the centroid is not enclosed in the polygon, no point is sampled. If no point can be sampled for the geometry at all, the first coordinate of the geometry is returned as point.\n- For **lines** (line strings), the sampling starts with a point at the first coordinate of the line and then walks along the line and samples a new point each time the distance to the previous point has been reached again.\n- For **points**, the point is returned as given.", "schema": { "type": "number", "minimumExclusive": 0 @@ -54,5 +54,10 @@ } ] } + }, + "exceptions": { + "UnitMismatch": { + "message": "The unit of the spatial reference system is not meters, but the given distance is in meters." + } } } From 598f4ca04c2b97dcb6da960d3fda8f4bdf5f2f0c Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 4 May 2023 17:52:14 +0200 Subject: [PATCH 04/10] Update version numbers to v2.0.0-rc.1 (#438) --- CHANGELOG.md | 5 ++++- README.md | 5 +++-- meta/subtype-schemas.json | 2 +- 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0eb78e8a..77d2916a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## Unreleased / Draft +## [2.0.0-rc.1] - 2023-05-31 + ### Added - New processes in proposal state: @@ -363,7 +365,8 @@ First version which is separated from the openEO API. Complete rework of all pro Older versions of the processes were released as part of the openEO API, see the corresponding changelog for more information. -[Unreleased]: +[Unreleased]: +[2.0.0-rc.1]: [1.2.0]: [1.1.0]: [1.0.0]: diff --git a/README.md b/README.md index 91d2096d..24c28899 100644 --- a/README.md +++ b/README.md @@ -8,12 +8,13 @@ openEO develops interoperable processes for big Earth observation cloud processi The [master branch](https://github.com/Open-EO/openeo-processes/tree/master) is the 'stable' version of the openEO processes specification. An exception is the [`proposals`](proposals/) folder, which provides experimental new processes currently under discussion. They may still change, but everyone is encouraged to implement them and give feedback. -The latest release is version **1.2.0**. The [draft branch](https://github.com/Open-EO/openeo-processes/tree/draft) is where active development takes place. PRs should be made against the draft branch. +The latest release is version **2.0.0-rc.1**. The [draft branch](https://github.com/Open-EO/openeo-processes/tree/draft) is where active development takes place. PRs should be made against the draft branch. | Version / Branch | Status | openEO API versions | | ------------------------------------------------------------ | ------------------------- | ------------------- | | [unreleased / draft](https://processes.openeo.org/draft) | in development | 1.x.x | -| [**1.2.0** / master](https://processes.openeo.org/1.2.0/) | **latest stable version** | 1.x.x | +| [**2.0.0 RC1** / master](https://processes.openeo.org/2.0.0-rc.1/) | **upcoming version (RC)** | 1.x.x | +| [1.2.0](https://processes.openeo.org/1.2.0/) | **latest stable version** | 1.x.x | | [1.1.0](https://processes.openeo.org/1.1.0/) | legacy version | 1.x.x | | [1.0.0](https://processes.openeo.org/1.0.0/) | legacy version | 1.x.x | | [1.0.0 RC1](https://processes.openeo.org/1.0.0-rc.1/) | legacy version | 1.x.x | diff --git a/meta/subtype-schemas.json b/meta/subtype-schemas.json index 1a49bf64..347df234 100644 --- a/meta/subtype-schemas.json +++ b/meta/subtype-schemas.json @@ -1,6 +1,6 @@ { "$schema": "http://json-schema.org/draft-07/schema#", - "$id": "https://processes.openeo.org/1.2.0/meta/subtype-schemas.json", + "$id": "https://processes.openeo.org/2.0.0-rc.1/meta/subtype-schemas.json", "title": "Subtype Schemas", "description": "This file defines the schemas for subtypes we define for openEO processes.", "definitions": { From da815328222860d64facc9da74c1054e267d5b59 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 4 May 2023 18:46:04 +0200 Subject: [PATCH 05/10] Update changelog --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 77d2916a..db092021 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,7 +22,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `vector_reproject` - `vector_to_random_points` - `vector_to_regular_points` -- `add_dimension`: Added new dimension type `geometries`. [#68](https://github.com/Open-EO/openeo-processes/issues/68) +- `add_dimension`: Added new dimension type `geometry`. [#68](https://github.com/Open-EO/openeo-processes/issues/68) ### Changed From fa2da6ca23987ddf455d058be7cd5a5fbaefe2f3 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Fri, 5 May 2023 13:40:02 +0200 Subject: [PATCH 06/10] Small wording improvements from @neteler --- proposals/vector_buffer.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/vector_buffer.json b/proposals/vector_buffer.json index 05be9a58..e81e9a76 100644 --- a/proposals/vector_buffer.json +++ b/proposals/vector_buffer.json @@ -22,7 +22,7 @@ }, { "name": "distance", - "description": "The distance of the buffer in meters. If the unit of the spatial reference system is not meters, a `UnitMismatch` error is thrown. Use ``vector_reproject()`` to convert the geometries to a suitable spatial reference system.\n\nA positive distance expands the geometries and results in outward buffering (dilation) while a negative distance shrinks the geometries and results in inward buffering (erosion).", + "description": "The distance of the buffer in meters. A positive distance expands the geometries, resulting in outward buffering (dilation), while a negative distance shrinks the geometries, resulting in inward buffering (erosion).\n\nIf the unit of the spatial reference system is not meters, a `UnitMismatch` error is thrown. Use ``vector_reproject()`` to convert the geometries to a suitable spatial reference system.", "schema": { "type": "number", "not": { From def8aac03b125d3af1cef376c76257c617420128 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 8 May 2023 18:13:59 +0200 Subject: [PATCH 07/10] Improve wording --- filter_bbox.json | 2 +- proposals/filter_vector.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/filter_bbox.json b/filter_bbox.json index 5eea34ed..3e2a7485 100644 --- a/filter_bbox.json +++ b/filter_bbox.json @@ -1,7 +1,7 @@ { "id": "filter_bbox", "summary": "Spatial filter using a bounding box", - "description": "Limits the data cube to the specified bounding box.\n\n* For raster data cubes, the filter retains a pixel in the data cube if the point at the pixel center intersects with the bounding box (as defined in the Simple Features standard by the OGC). Alternatively, ``filter_spatial()`` can be used to filter by geometry.\n* For vector data cubes, the filter retains the geometry in the data cube if the geometry is fully within the bounding box (as defined in the Simple Features standard by the OGC). All geometries that were empty or get empty will be removed from the data cube. Alternatively, ``filter_vector()`` can be used to filter by geometry.", + "description": "Limits the data cube to the specified bounding box.\n\n* For raster data cubes, the filter retains a pixel in the data cube if the point at the pixel center intersects with the bounding box (as defined in the Simple Features standard by the OGC). Alternatively, ``filter_spatial()`` can be used to filter by geometry.\n* For vector data cubes, the filter retains the geometry in the data cube if the geometry is fully within the bounding box (as defined in the Simple Features standard by the OGC). All geometries that were empty or not contained fully within the bounding box will be removed from the data cube.\n\nAlternatively, ``filter_vector()`` can be used to filter by geometry.", "categories": [ "cubes", "filter" diff --git a/proposals/filter_vector.json b/proposals/filter_vector.json index 4560848e..349f8d0f 100644 --- a/proposals/filter_vector.json +++ b/proposals/filter_vector.json @@ -1,7 +1,7 @@ { "id": "filter_vector", "summary": "Spatial vector filter using geometries", - "description": "Limits the vector data cube to the specified geometries. The process works on geometries as defined in the Simple Features standard by the OGC. All geometries that were empty or get empty will be removed from the data cube. Alternatively, use ``filter_bbox()`` to filter by bounding box.", + "description": "Limits the vector data cube to the specified geometries. The process works on geometries as defined in the Simple Features standard by the OGC. All geometries that were empty or become empty will be removed from the data cube. Alternatively, use ``filter_bbox()`` to filter by bounding box.", "categories": [ "cubes", "filter", From 093723a0323d4a6037b81bc65d1e13874e1ea499 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 25 May 2023 13:18:20 +0200 Subject: [PATCH 08/10] Update release date --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index db092021..407447dc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## Unreleased / Draft -## [2.0.0-rc.1] - 2023-05-31 +## [2.0.0-rc.1] - 2023-05-25 ### Added From 7b558aca5f2f13855c35fbbddaa8205fbdeb7d4d Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 25 May 2023 13:30:59 +0200 Subject: [PATCH 09/10] clean-up --- .github/workflows/docs.yml | 1 - tests/docs.html | 4 ++-- tests/package.json | 2 +- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 4e8f4f28..29baf6db 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -5,7 +5,6 @@ on: push: branches: - draft - - draft-2.0 - master jobs: deploy: diff --git a/tests/docs.html b/tests/docs.html index 23ef98e4..04b1c192 100644 --- a/tests/docs.html +++ b/tests/docs.html @@ -113,8 +113,8 @@ props: { document: 'processes.json', categorize: true, - apiVersion: '1.1.0', - title: 'openEO processes (draft)', + apiVersion: '1.2.0', + title: 'openEO processes (2.0.0-rc.1)', notice: '**Note:** This is the list of all processes specified by the openEO project. Back-ends implement a varying set of processes. Thus, the processes you can use at a specific back-end may derive from the specification, may include non-standardized processes and may not implement all processes listed here. Please check each back-end individually for the processes they support. The client libraries usually have a function called `listProcesses` or `list_processes` for that.' } }) diff --git a/tests/package.json b/tests/package.json index 15f6d2e8..1da8693f 100644 --- a/tests/package.json +++ b/tests/package.json @@ -1,6 +1,6 @@ { "name": "@openeo/processes", - "version": "2.0.0", + "version": "2.0.0-rc.1", "author": "openEO Consortium", "contributors": [ { From 7a293786b4c464d2c7452d7b5f13104098a25232 Mon Sep 17 00:00:00 2001 From: Stefaan Lippens Date: Mon, 31 Jul 2023 13:52:05 +0200 Subject: [PATCH 10/10] aggregate_spatial_window typo fix (#446) --- proposals/aggregate_spatial_window.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/aggregate_spatial_window.json b/proposals/aggregate_spatial_window.json index 747db151..9e5dce4a 100644 --- a/proposals/aggregate_spatial_window.json +++ b/proposals/aggregate_spatial_window.json @@ -75,7 +75,7 @@ }, { "name": "boundary", - "description": "Behavior to apply if the number of values for the axes `x` and `y` is not a multiple of the corresponding value in the `size` parameter. Options are:\n\n- `pad` (default): pad the data cube with the no-data value `null` to fit the required window size.\n\n- `trim`: trim the data cube to fit the required window size.\n\nSet the parameter `align` to specifies to which corner the data is aligned to.", + "description": "Behavior to apply if the number of values for the axes `x` and `y` is not a multiple of the corresponding value in the `size` parameter. Options are:\n\n- `pad` (default): pad the data cube with the no-data value `null` to fit the required window size.\n\n- `trim`: trim the data cube to fit the required window size.\n\nUse the parameter `align` to align the data to the desired corner.", "schema": { "type": "string", "enum": [