Database Lab Engine v3.2.0

[NikolayS]

Database Lab Engine v3.2.0

Release v3.2.0 extends DLE UI with two new tabs, "Logs" and "Configuration", to improve the user experience when configuring or reconfiguring DLE for managed Postgres services such as AWS RDS and Google Cloud SQL. It also delivers a good number of new features and improvements.

Postgres 15 is already supported!

PostgreSQL 15 is already supported by DLE and included in the list of default images, with many extensions shipped with it: https://postgres.ai/docs/database-lab/supported-databases.

AWS Marketplace – the easiest way to get DLE for any Postgres database

DLE v3.2.0 has been already published in AWS Marketplace: https://aws.amazon.com/marketplace/pp/prodview-wlmm2satykuec, offering the Standard license, with support, embedded monitoring, simplified installation, and access to a wider variety of supported Postgres images (including extensions not available by default). An instance with DLE delivers "dev/test databases as a service" and levels up all dev&test processes. For example, on an r5.xlarge instance (4 vCPUs, 32 GiB RAM) with 200 GiB disk space, it is possible to run up to 30 database clones simultaneously (and even more, if shared_buffers is adjusted to lower values) spending as low as $360 per month – less than for a single regular RDS clone of the same size.

New contributor

🙇 The development team greets a new contributor: @Adrinlol.

What's new

New features

• New tab in UI, "Configuration" and corresponding API to allow editing of certain configuration parameters (only in logical mode; secret values are masked) and viewing the current config (in any mode): https://gitlab.com/postgres-ai/database-lab/-/merge_requests/561, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/575, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/594, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/601, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/612, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/613, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/614 (@yuridevx, @Adrinlol), https://gitlab.com/postgres-ai/database-lab/-/merge_requests/608, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/609, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/612 (@Adrinlol), https://gitlab.com/postgres-ai/database-lab/-/merge_requests/580, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/615
• New tab in UI, "Logs" to allow seeing DLE log messages to support troubleshooting (secret values are masked): https://gitlab.com/postgres-ai/database-lab/-/merge_requests/579 (@Adrinlol), https://gitlab.com/postgres-ai/database-lab/-/merge_requests/585, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/593
• Manage logs of PostgreSQL containers to simplify diagnostics (previously, if an error happens inside Postgres container and the container is destroyed, it was impossible to find the the details of the error): https://gitlab.com/postgres-ai/database-lab/-/merge_requests/558, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/570, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/598, https://gitlab.com/postgres-ai/docs/-/merge_requests/481 (@denis256)
• Support queryPreprocessing in logicalRestore and allow using inline SQL in queryPreprocessing: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/576
• Simplify Postgres image customization – for initialization in logical mode, perform comprehensive testing of the source database and compare a container created using the Postgres image provided in configuration with the source DB: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/573, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/578 (:bow: contribution by @Adrinlol), https://gitlab.com/postgres-ai/database-lab/-/merge_requests/596
• Show details of continuous synchronization of PGDATA (API+CLI; currently only for physical mode): https://gitlab.com/postgres-ai/database-lab/-/merge_requests/562, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/595 (@denis256)
• Show the progress of the full refresh process https://gitlab.com/postgres-ai/database-lab/-/merge_requests/574, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/581, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/584, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/587, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/588, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/591, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/610 (@Adrinlol)
• Support PostgreSQL 15: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/599, https://gitlab.com/postgres-ai/custom-images/-/merge_requests/42, https://gitlab.com/postgres-ai/custom-images/-/merge_requests/44, https://gitlab.com/postgres-ai/custom-images/-/merge_requests/45, https://gitlab.com/postgres-ai/docs/-/merge_requests/504

Improvements and fixes

• Redesign the Retrieval service: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/549
• Mounting of socket directory for non-guest virtualization roles: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/557 (@denis256)
• Add a link to the guide "PostgreSQL configuration in clones": https://gitlab.com/postgres-ai/database-lab/-/merge_requests/559
• Add session manager to CF template: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/560
• Add maintenance_work_mem for Logical mode: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/567
• Report the type of DLE subscription in UI: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/563
• Detect if DLE is running in a Docker container: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/571, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/603
• Apply the configuration and then start Postgres on the restore stage, to reduce the number of restarts needed: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/518, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/547 (@yuridevx), https://gitlab.com/postgres-ai/database-lab/-/merge_requests/558, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/572
• Fix the creation of a restricted user: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/586
• Build CE UI Docker image after merging into the master branch: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/590
• Adjust container readiness health checks: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/597, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/602
• Remove deprecated eslint analyzers: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/605
• Increase the number of parallel jobs in logical mode from 2/2 to 4/4 – usually, many more CPU cores are available: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/611

Diff showing all changes

Diff between versions v3.1.2 and v3.2.0: v3.1.2...v3.2.0.

Migration from DLE v3.1.x and configuration notes

Note: if you have an older version, check the previous release notes as well: https://gitlab.com/postgres-ai/database-lab/-/releases.

Configuration directory

The config directory, by default, is ~/.dblab/engine/configs and the corresponding mounted volume must be writable:

  ...
  --volume ~/.dblab/engine/configs:/home/dblab/configs
  ...

Logs directory

Make sure that ~/.dblab/engine/logs is mounted as a volume to the DLE container:

  ...
  --volume ~/.dblab/engine/configs:/home/dblab/configs \
  --volume ~/.dblab/engine/meta:/home/dblab/meta \
  --volume ~/.dblab/engine/logs:/home/dblab/logs \
  ...

Alternatively, you can mount the entire directory (including configs and meta – but make sure that these subdirectories exist, DLE doesn't create them yet):

  ...
  --volume ~/.dblab/engine:/home/dblab
  ...

Query preprocessing

The queryPreprocessing section supports inline SQL now. Examples can be found here: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/576#examples

Configuration UI

Because the configuration can be viewed and edited in the UI, make sure:

• secret values are masked (e.g., verificationToken, Platform accessToken, you can find all masking rules here: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/608#masking-rules)
• DLE configuration matches the reference configuration files in structure and naming; it is not recommended to have custom names and sections

Where to get help

To get help, reach out to the Postgres.ai team and the growing community of Database Lab users and contributors: https://postgres.ai/contact.

Provide feedback, contribute

Any feedback will be greatly appreciated:

• Twitter: @Database_Lab
• Database Lab Community Slack: https://slack.postgres.ai/

If you want to help the project grow, consider various ways of contributing:

• Star the GitHub repo: https://github.com/postgres-ai/database-lab-engine
• Spread the word using Twitter (please mention @Database_Lab!) or your social network of choice
• Consider translating the README.md to one or several languages you know
• Consider implementing one of the features that will make DLE better: see CONTRIBUTING.md and the list of "good first issues" on GitLab

---

This discussion was created from the release Database Lab Engine v3.2.0.