v3.5.1

What's Changed

🩹 startup failure

A bug was detected in 3.5.0, when an event stream was created in 3.4.0 and no members were ingested yet, a row in a newly created table in the database was missing, which led to the application crashing on startup. This has been fixed in #1416

Full Changelog: v3.5.0...v3.5.1

v3.5.0

What's Changed

🔧 Configurable Delimiter on State Object ingestion

As previously available in the LDI Version Object creator, It is now possible to configure a custom version delimiter when ingesting state objects. For more details, check out the documentation.

📑 Maintenance Jobs, Compaction + Retention + Deletion combined

As a cleanup effort and to provide more transparency, the compaction, retention and deletion jobs have been merged into a maintenance job. The biggest user impact is the simplified config through the maintenance cron found in the config table.

📚 Modular restructure

Although this has no user impact, the module structure of the server got a revamp.
Allowing for custom images with unique functionalities and a smaller image size.
For more details, check out the project Readme

Change notes:

• fix: broken compaction due to used uuids by @jobulcke in #1373
• chore: restructuring to maintenance by @jobulcke in #1375
• feat: simplify maintenance by @jobulcke in #1374
• chore(deps): bump org.owasp:dependency-check-maven from 9.0.9 to 10.0.4 by @dependabot in #1385
• chore(deps): bump com.google.guava:guava from 33.0.0-jre to 33.3.1-jre by @dependabot in #1387
• chore(deps): bump org.springframework.data:spring-data-commons from 3.2.3 to 3.3.4 by @dependabot in #1388
• fix: flaky test by @jobulcke in #1381
• chore: db cleanup by @jobulcke in #1382
• feat: charset included in headers by @jobulcke in #1380
• fix: broken compaction rework by @jobulcke in #1383
• chore: fix view stats by @rorlic in #1396
• hotfix: dropped member reader perfo by @jobulcke in #1394
• feat: drop unnecessary db check on ingest by @jobulcke in #1399
• feat: modular restructure by @Yalz in #1400
• fix: missing page members by @Yalz in #1401
• chore: bump jena to next major version by @jobulcke in #1402
• feat: relax timebased fragmentation by @jobulcke in #1403
• docs: update project structure in docs by @Yalz in #1404
• Fix/broken image by @rorlic in #1407
• feat: configurable version delimiter by @jobulcke in #1405

Full Changelog: v3.4.0...v3.5.0

v3.4.0

What's Changed

📊 Fragmentation Optimalisation

Under the hood, the fragmentation processes (bucketisation + pagination) went through a big structural change to optimise database interactions.
Additionally, database views and tables have been optimised in function of this.

🚧 Known Issues

Timebased Linear Caching

Whilst preparing for this release, we noticed that the timebased linear caching feature was causing troubles in our current write process to the database. Therefore, it has been disabled for now, but we will try to enable it again as soon as possible

Change Notes

• feat: pagination rework by @Yalz in #1369
• feat: move all bucket db queries to ItemWriter by @jobulcke in #1370

Full Changelog: v3.3.0...v3.4.0

v3.3.0

What's Changed

Fragmentation revamp: Bucketisation & Pagination 📇

Although no breaking changes on the front end have been introduced, you might see in the logs that there is one fragmentation job that will run periodically.
This job will, for each view, first define the buckets in the bucketisation step. After that, the pagination step will assign the bucketised members to pages.
This was done in terms of performance to decouple the ingestion process from fragmentation.
Note that further releases will improve on performance of these processes.

Fragmentation will check every 30 seconds for unprocessed members.
If you wish to use a specific interval, update the ldes-server.fragmentation-cron spring value with a valid spring cron.

Fragmentation metrics 📈

In terms of observability, we also provide an update for all you Prometheus fans!
You will now have 3 metrics to follow up the internal processes of your LDES Server:

• ldes_server_ingested_members_count : Will keep track of the ingested members per collection.
• ldes_server_bucket_members_count : Will keep track of all bucketised members per view.
• ldes_server_pagination_members_count : Will keep track of all paginated members per view.

Skolemization ♻️

As per request of one of our onboarders, we introduced skolemization.
This feature allows the LDES Server to replace blank nodes with skolemized URIs.
For more details, check out the new documentation

Known Issues 🚧

Compaction vs Fragmentation

Whilst preparing for this release, we noticed how the newly created compaction URIs conflict with our pagination.
This issue is being tracked as part of #1365
We advice you to turn off compaction. This can be done by setting the compaction cron to -.

Change Notes

• chore: finalise 3.2.0 by @Yalz in #1332
• doc: update event source documentation by @jeflaenen in #1333
• Use latest cache actions by @rorlic in #1345
• chore: #1346: auto enable Spring Batch table creation + filter out ac… by @Yalz in #1349
• feat: skolemization by @jobulcke in #1351
• chore: improve ci pipeline by @Yalz in #1352
• feat: data model: retention + fragmentation by @Yalz in #1343
• fix: missing skolemization domain shacl uri validation by @jobulcke in #1354
• feat: include expanded metrics + general fragmentation fixes by @Yalz in #1356
• fix compaction by @Yalz in #1358
• feat: run fragmentation jobs per view in parallel by @Yalz in #1359
• fix: broken needs_pagination db view by @jobulcke in #1360
• chore: make fragmentation jobs parallel by @Yalz in #1361
• chore: removed incorrect unique constraint and added some indexes by @Yalz in #1362
• chore: finalise 3.3.0 release by @Yalz in #1363
• docs: skolemization updates by @Yalz in #1367

New Contributors

• @rorlic made their first contribution in #1345

Full Changelog: v3.0.0...v3.3.0

3.2.0-alpha

What's Changed

Resolving memory issue in Pagination

Due to an issue in pagination in version 3.1.0, the memory kept increasing in long runs.
This has now been resolved. Why therefor highly advise users to move to this release as soon as possible.

Under the hood: Database model changes

For those who might already have noticed, a new database model is showing itself between our releases.
This is with the goal of having a more performant solution.

Once all parts of the database are migrated, a full overview of the new database will be provided for those interested

Change Notes

• feat: new data model ingest by @pj-cegeka in #1316
• feat: less db calls for event stream children by @jobulcke in #1320
• chore: retired mongo references cleanup by @jobulcke in #1315
• doc: change of metrics URL by @jeflaenen in #1326
• fix: query optimisation + async events by @Yalz in #1324
• feat: async event handling by @Yalz in #1328
• fix: decrease load on bucketisation/pagination by @Yalz in #1329
• fix: memory impact by @Yalz in #1330
• Fix: restart with closed collection by @pj-cegeka in #1305

Full Changelog: v3.1.0-alpha...3.2.0-alpha

v3.1.0 alpha

What's Changed

Liquibase introduction

From the 3.1.0 release on, the LDES server contains a migration path between each release.
No extra configuration is needed, all needed tables and migrations will be done on the fly.

Fragmentation rework (batching)

The fragmentation got a big rework to focus towards a more performant LDES Server.
The 2 processes (bucketisation and pagination) are now split up:

• Bucketisation: Based on the configured fragmentation strategy and a given LDES member, the member can be assinged to a number of buckets. This will be reflected in the relations in the LDES fragments.
• Pagination: For each of the assigned buckets, the members will be paginated. This happens based on the configured memberCount per page.

These processes will now work with a time interval to process more members at the same time.

Geospatial fragmentation update

Previously, there was an issue in the geospatial fragmentation where only the tiles that contains the points of a shape were taken.
In the below example, A3 and D3 were taken as tiles of a geospatial fragmentation, but the others were ignored.
From now on, also tiles B3, C2 and C3 are taken.

Closing of an LDES

If, as a dataowner, you know no more data will be ingested in an Event Stream, there is now the ability to close the stream (turns the latest page immutable).
This gives the advantage that the LDES Client can stop following the LDES without needlessly polling for updates that will never come.
To close an LDES, please visit the swagger endpoint.

Change notes

• f:eat: split pagination and fragmentation by @pj-cegeka in #1294
• fix: add liquibase profile by @pj-cegeka in #1297
• feat: VSDSPUB-1283: Close LDES by @ferre-vaes in #1296
• fix: adapt ldes closure endpoint by @ferre-vaes in #1304
• feat: calculate intersecting tiles of geo feature by @ferre-vaes in #1309
• feat: fragmentation batching by @Yalz in #1311
• feat: new data model definition for event streams by @jobulcke in #1307
• fix: invalid documentation by @jobulcke in #1312
• fix: code smells by @jobulcke in #1317
• chore: finalise 3.1.0 by @Yalz in #1318

Full Changelog: v3.0.0-alpha...v3.1.0-alpha

v3.0.0

What's Changed

Postgres implementation

To deliver a more performant and transactional experience, the LDES Server moves from its trusted mongodb database
to a PostgreSQL database.
This however is only the first step in a peformance focus and more updates on the database structure might come.

For more info on how to upgrade, check out the configuration guide.

Change notes

• feat: add eventSource to admin api by @pj-cegeka in #1281
• feat: postgres implementation by @Yalz in #1289
• Fix: eventsource delete view postgres by @pj-cegeka in #1290
• chore: finalise 3.0.0 by @Yalz in #1292

Full Changelog: v2.15.0-alpha...v3.0.0-alpha

v2.15.0-alpha: Merge pull request #1282 from Informatievlaanderen/develop

What's Changed

• fix: attach link to Hulp Nodig button (#1277)
• Feat: fetch streaming LDES fragment (#1272)
• fix: invalid swagger ui in docs (#1273)
• feat: cleanup actuator hostname customizer (#1269)
• docs: reduce documentation padding (#1268)

Full Changelog: v2.14.0-alpha...v2.15.0-alpha

v2.14.0-alpha

What's Changed

• fix: cleanup test resource files in de ldes-server-admin module in #1257
• fix: slow retention insert query in #1263
• feat: member model extractor performance increase in #1264
• fix: mongock order failure in #1266

Full Changelog: v2.13.0-alpha...v2.14.0-alpha

v2.13.0-alpha

What's Changed

• ci: prepare release 2.13.0 by @jobulcke in #1224
• fix: change expected exceptions to warnings by @Tomvbe in #1222
• feat: add type check to timestamp and version of path on ingested member by @pj-cegeka in #1217
• Use replicasets in mongodb docker by @sandervd in #1226
• fix: unexpected empty shacl behavior by @jobulcke in #1220
• fix: non rdf content type by @Tomvbe in #1228
• docs: #1136 : generalise API examples by @Yalz in #1231
• fix: slow view operations by @jobulcke in #1232
• feat: return validation message in shacl report format by @pj-cegeka in #1227
• Feat: improve blank node validation by @pj-cegeka in #1237
• fix: unexpected error when compaction is run when no data is present by @jobulcke in #1238
• fix: treeValue should be a resource with by-reference fragmentation by @jobulcke in #1239
• fix: memberid extraction fails with bnode versionof by @jobulcke in #1242
• fix: members deleted twice by @jobulcke in #1234
• feat: use in memory sequence for ingest by @Tomvbe in #1244
• fix: actuator uses internal addresses by @jobulcke in #1246
• fix: docs/_site folder was not completely ignored by @jobulcke in #1249
• fix: Remove ModelParser and move static functions to the respectful fragmentation class that uses them by @fvaes-cegeka in #1247
• feat: add interactive swagger docs to github pages by @Tomvbe in #1248
• fix: sonarcloud improvements by @Yalz in #1250
• chore: update codeowners by @jobulcke in #1252
• fix: disable ActuatorHostNameCustomizer by @pj-cegeka in #1251
• ci: finalise 2.13.0 by @Yalz in #1256

New Contributors

• @sandervd made their first contribution in #1226
• @fvaes-cegeka made their first contribution in #1247

Full Changelog: v2.12.0-alpha...v2.13.0-alpha