fix(deps): build psycopg3 instead of using pre-build binary

[gietschess]

Description

Since the update of the base python docker image from alpine:3.16 to 3.20 we've had segmentation faults while connecting to aws rds postgres databases. With the debug environment variable from the psycopg2 dependency we found connection issues while connecting to our database (see logs below).

Building the psycopg3 postgresql adapter instead of using the pre-build dependency as it's described in the psycopg3 installation documentation the connection could be established again as expected.

Also see the slack thread on this issue.

Test results

I've locally build and tested both the alpine and debian django +nginx dockerimages and tested them against the bitnami and aws rds postgres database.

Extra information

Additional logs:

Debug logs with the alpine image containing the psycopg2-binary 2.9.9 dependency:

[19] psyco_connect: dsn = 'dbname=dd client_encoding=UTF8 user=aws_admin password=bla host=defectdojo-rds-dev port=5432', async = 0
[19] connection_setup: init connection object at 0x7f18ab9af240, async 0, refcnt = [1] connection_setup: init connection object at 0x7f18ab9af240, async 0, refcnt = 11
[19] con_connect: connecting in SYNC mode
[19] conn_connect: new PG connection at 0x7f18aaf8d4b0
[19] conn_connect: PQconnectdb(dbname=dd client_encoding=UTF8 user=aws_admin password=bla host=defectdojo-rds-dev port=5432) returned BAD
[19] connection_init: FAILED
!!! uWSGI process 19 got Segmentation Fault !!!

Debug logs with the alpine image containing the self-build psycopg2 2.9.9 dependency:

[19] psyco_connect: dsn = 'dbname=dd client_encoding=UTF8 user=aws_admin password=bla host=defectdojo-rds-dev port=5432', async = 0
[19] connection_setup: init connection object at 0x7f87c53e3240, async 0, refcnt = 1
[19] con_connect: connecting in SYNC mode
[19] conn_connect: new PG connection at 0x7f87c49248a0
[19] conn_connect: server standard_conforming_strings parameter: on
[19] conn_connect: server requires E'' quotes: NO
[19] conn_connect: using protocol 3
[19] conn_connect: client encoding: UTF8
[19] clear_encoding_name: UTF8 -> UTF8
[19] conn_set_fast_codec: encoding=UTF8
[19] conn_set_fast_codec: PyUnicode_DecodeUTF8
[19] conn_connect: DateStyle ISO, MDY
[19] connection_setup: good connection object at 0x7f87c53e3240, refcnt = 1

[dryrunsecurity]

Hi there 👋, @DryRunSecurity here, below is a summary of our analysis and findings.

DryRun Security	Status	Findings
Configured Codepaths Analyzer	✅	0 findings
IDOR Analyzer	✅	0 findings
Sensitive Files Analyzer	❕	1 finding
Server-Side Request Forgery Analyzer	✅	0 findings
SQL Injection Analyzer	✅	0 findings
Authn/Authz Analyzer	✅	0 findings
Secrets Analyzer	✅	0 findings

Note

🟢 Risk threshold not exceeded.

Change Summary (click to expand)

The following is a summary of changes in this pull request made by me, your security buddy 🤖. Note that this summary is auto-generated and not meant to be a definitive list of security issues but rather a helpful summary from a security perspective.

Summary:

The provided code changes cover updates to the requirements.txt, Dockerfile.nginx-alpine, and Dockerfile.django-alpine files for the DefectDojo application. The changes primarily focus on updating dependencies, configuring the Docker build process, and improving the security and reliability of the application deployment.

The key changes include updating the PostgreSQL client version, adding new build dependencies, updating Node.js and Yarn versions, and implementing security-conscious practices such as user and permissions management, environment variable handling, and including additional fixtures and tests in the Docker image.

From an application security perspective, these changes are generally positive as they demonstrate the maintainers' commitment to keeping the application's dependencies up-to-date and addressing potential security vulnerabilities. The updates to the PostgreSQL client and the addition of new build dependencies suggest that the application is being actively maintained and improved.

Additionally, the security-conscious practices, such as user and permissions management, and the inclusion of additional fixtures and tests, indicate that the maintainers are taking steps to harden the application's deployment and ensure its overall security and reliability.

Files Changed:

1. requirements.txt:

   • The version of the psycopg library has been updated from psycopg[binary]==3.2.1 to psycopg[c]==3.2.1, which likely changes the database driver used by the application.

2. Dockerfile.nginx-alpine:

   • The PostgreSQL client version is updated from 14 to 16.
   • The python3-dev and libpq-dev packages are added to the list of installed packages.
   • The Node.js version is updated to 20.11.0, and the Yarn version is updated to 1.22.19.
   • Several environment variables related to NGINX configuration are included, which should be carefully reviewed for secure settings.

3. Dockerfile.django-alpine:

   • The PostgreSQL client version is updated from 14 to 16.
   • The python3-dev and libpq-dev packages are added to the build dependencies.
   • The Dockerfile sets up a dedicated user (appuser) with a specific UID and GID, and sets the appropriate permissions and ownership for the application files and directories.
   • Several environment variables related to the application configuration, such as admin user details, Celery worker settings, and UWSGI settings, are set.
   • Additional fixtures and unit tests are copied to the Docker image.

Powered by DryRun Security

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[mtesauro]

@gietschess

Note: There's a PR to move to psycopg3 that's closer to being merged as well #10348

[gietschess]

@gietschess

Note: There's a PR to move to psycopg3 that's closer to being merged as well #10348

with this pr we would still use the not recommended prebuild binary version of psycopg.

i don't really mind about the postgres clients version, just wanted to keep it up to date as alpine informs about using a old client version when installing version 14:

* Setting postgresql14 as the default version
*
* You are using 'postgresql14'. It's recommended to upgrade to the latest
* major version provided by package 'postgresql16'.
* Use command 'pg_versions' to switch between versions.
*

[kiblik]

Original tests have not been failing. Is there some way to improve tests to detect SegFault earlier?

[gietschess]

hi @kiblik ,
i couldn't reproduce the segfault with the bitnami postgres database from the helmchart and the alpine docker image but with a external postgres database.
my best guess would be that the segfault results due to some encryption configuration of the database but i'm not completly sure about that.

[mtesauro]

@gietschess

Note: There's a PR to move to psycopg3 that's closer to being merged as well #10348

with this pr we would still use the not recommended prebuild binary version of psycopg.

Yes, but if we're moving to psycopg3 then maybe do a PR for the non-binary version of psychopg3 instead of the non-binary version of psycopg2. At least that's what I was thinking.

At one point in time we switched to the binary version of psycopg for a specific reason but that was quite some time ago so I don't have a problem with using the non-binary/source version of that module.

I'd prefer not to use non-binary 2 over binary 3 of that module when non-binary 3 is a valid option.

i don't really mind about the postgres clients version, just wanted to keep it up to date as alpine informs about using a old client version when installing version 14:

The one place where this can cause problems is for iron installs - we've planning on deprecating that but it's not "official' yet. I just don't want to optimize for one distro and break others if we don't absolutely have to - from a quick look at building psychopg3, only libpq5. Just checked with the most "interesting" distro (RHEL 8) and pip has no issues building psychopg3 assuming you have the needed Postgres dev packages installed.

So I'll make my comments as resolved as far as the PG 16 is concerned.

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[gietschess]

alright, i've tested with the prebuild psycopg3 binary alpine image which also leads into segmentation faults.
with the self build ones both debian and alpine looked fine.

[mtesauro]

alright, i've tested with the prebuild psycopg3 binary alpine image which also leads into segmentation faults.
with the self build ones both debian and alpine looked fine.

OK. Perfect. Feel free to change this PR to use the source/you-build-it psycopg3 or close it and open a new one if that's easier for you.

I'm happy to take a PR to move to a built version of psycopg3 instead of the binary version that's currently in place after #10348 was merged.

[mtesauro]

Just realized you already changed to psycopg[c] from psycopg[binary]

Approved

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[hajali-amine]

Do we know when this will be released? We're facing the same issue with OVH managed PG databases 👀

[mtesauro]

Do we know when this will be released? We're facing the same issue with OVH managed PG databases 👀

@hajali-amine This went into the dev branch so will be part of the 2.37.0 (August) release. Minor version releases happen on the 1st Monday of the month or for August 5th.

If you need it sooner than that, you can always change requirements.txt like the PR does and build your own docker images using the dockerfiles in this repo.