Docker container fails to start with Python errors

[ablankenship10]

Describe the bug
Docker Compose fails to start service and constantly restarts

To Reproduce
Steps to reproduce the behavior:

1. create docker-compose.yml in project
2. run docker-compose up

Expected behavior
The service starts successfully and accepts connections

docker-compose.yml

version: '3.1'

services:
  local-data-api:
    image: koxudaxi/local-data-api
    restart: always
    environment:
      ENGINE: PostgresSQLJDBC
      POSTGRES_HOST: db
      POSTGRES_PORT: 5432
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: secret
      RESOURCE_ARN: 'arn:aws:rds:us-east-1:123456789012:cluster:dummy'
      SECRET_ARN: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy'
    ports:
      - '8080:80'
  db:
    image: mdillon/postgis
    restart: always
    environment:
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: dev
    ports:
      - '5432:5432'

Error Messages

local-data-api_1  | [2020-11-08 11:06:34 +0000] [1] [INFO] Starting gunicorn 20.0.4
local-data-api_1  | [2020-11-08 11:06:34 +0000] [1] [INFO] Listening at: http://0.0.0.0:80 (1)
local-data-api_1  | [2020-11-08 11:06:34 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
local-data-api_1  | [2020-11-08 11:06:34 +0000] [7] [INFO] Booting worker with pid: 7
local-data-api_1  | [2020-11-08 11:06:34 +0000] [7] [ERROR] Exception in worker process
local-data-api_1  | Traceback (most recent call last):
local-data-api_1  |   File "/usr/local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
local-data-api_1  |     worker.init_process()
local-data-api_1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/workers.py", line 57, in init_process
local-data-api_1  |     super(UvicornWorker, self).init_process()
local-data-api_1  |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 119, in init_process
local-data-api_1  |     self.load_wsgi()
local-data-api_1  |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
local-data-api_1  |     self.wsgi = self.app.wsgi()
local-data-api_1  |   File "/usr/local/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
local-data-api_1  |     self.callable = self.load()
local-data-api_1  |   File "/usr/local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
local-data-api_1  |     return self.load_wsgiapp()
local-data-api_1  |   File "/usr/local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
local-data-api_1  |     return util.import_app(self.app_uri)
local-data-api_1  |   File "/usr/local/lib/python3.8/site-packages/gunicorn/util.py", line 358, in import_app
local-data-api_1  |     mod = importlib.import_module(module)
local-data-api_1  |   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
local-data-api_1  |     return _bootstrap._gcd_import(name[level:], package, level)
local-data-api_1  |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
local-data-api_1  |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
local-data-api_1  |   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
local-data-api_1  |   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
local-data-api_1  |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
local-data-api_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
local-data-api_1  |   File "/app/local_data_api/main.py", line 28, in <module>
local-data-api_1  |     setup()
local-data-api_1  |   File "/app/local_data_api/settings.py", line 58, in setup
local-data-api_1  |     raise NotImplementedError("Engine not already implemented")
local-data-api_1  | NotImplementedError: Engine not already implemented
local-data-api_1  | [2020-11-08 11:06:34 +0000] [7] [INFO] Worker exiting (pid: 7)
local-data-api_1  | {"loglevel": "info", "workers": 1, "bind": "0.0.0.0:80", "graceful_timeout": 120, "timeout": 120, "keepalive": 5, "errorlog": "-", "accesslog": "-", "workers_per_core": 1.0, "use_max_workers": null, "host": "0.0.0.0", "port": "80"}
local-data-api_1  | [2020-11-08 11:06:34 +0000] [1] [INFO] Shutting down: Master
local-data-api_1  | [2020-11-08 11:06:34 +0000] [1] [INFO] Reason: Worker failed to boot.
local-data-api_1  | {"loglevel": "info", "workers": 1, "bind": "0.0.0.0:80", "graceful_timeout": 120, "timeout": 120, "keepalive": 5, "errorlog": "-", "accesslog": "-", "workers_per_core": 1.0, "use_max_workers": null, "host": "0.0.0.0", "port": "80"}
api_local-data-api_1 exited with code 3

Desktop (please complete the following information):

• OS: [e.g. iOS] Ubuntu 20.04
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[ablankenship10]

Dug around in the code and looks like the README info for Postgres should be updated to show
ENGINE: PostgresSQL

instead of

ENGINE: PostgreSQLJDBC

[ablankenship10]

Separate issue, or question rather, but I wont open a new one right now.. I used the dummy values for the ARNs and AWS-SDK is yelling at me that it doesn't belong to my AWS user. Do I need a real active cluster? I'm still in early development and have not spun one up yet. Can I use this fully locally?

[koxudaxi]

@ablankenship10
I'm sorry for the late reply.

Dug around in the code and looks like the README info for Postgres should be updated to show
ENGINE: PostgresSQL

instead of

ENGINE: PostgreSQLJDBC

Thank you for your report
I will fix a document soon.

Separate issue, or question rather, but I wont open a new one right now.. I used the dummy values for the ARNs and AWS-SDK is yelling at me that it doesn't belong to my AWS user. Do I need a real active cluster? I'm still in early development and have not spun one up yet. Can I use this fully locally?

Yes, I believe to able use this app on a local-machine.
And, I wouldn't need a real-aws environment.

I have tested your setting with this command in my local. The result is good.

$ aws --endpoint-url http://127.0.0.1:8080 rds-data execute-statement --resource-arn "arn:aws:rds:us-east-1:123456789012:cluster:dummy" --sql "SELECT datname FROM pg_database"  --secret-arn "arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy" --database dev
{
    "numberOfRecordsUpdated": 0,
    "records": [
        [
            {
                "stringValue": "postgres"
            }
        ],
        [
            {
                "stringValue": "dev"
            }
        ],
        [
            {
                "stringValue": "template1"
            }
        ],
        [
            {
                "stringValue": "template0"

Also, I ran the same api in python. It works fine.

I guess your AWS-SDK may validate dummy ARN.
What language did you use for AWS-SDK?
I tested aws-cli
aws-cli/2.0.56 Python/3.7.3 Linux/5.4.58-050458-generic exe/x86_64.ubuntu.18
And, AWS-SDK for python 1.10.28

[ablankenship10]

Thanks for your reply, I'm using Node.js (https://www.npmjs.com/package/aws-sdk)

I've run into such issues before with other packages using aws-sdk. If I remember right there is a way to avoid this, let me try and see if i can make it work correctly.

[koxudaxi]

In python, I use dummy Credentials.

import boto3; client = boto3.client('rds-data', endpoint_url='http://127.0.0.1:8080', aws_access_key_id='aaa',  aws_secret_access_key='bbb') 

Also, some users use this app for Node.js
they posted examples in this issue.
#41

You may need to cheat Credentials:thinking:

[ablankenship10]

Ah hah! Your example above lead me to setting an Endpoint URL in the typeORM ConnectionOptions serviceConfigOptions fields that are passed to the AWS-SDK like so:

{
  type: 'aurora-data-api-pg',
  database: 'devpg',
  secretArn: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:dummy',
  resourceArn: 'arn:aws:rds:us-east-1:123456789012:cluster:dummy',
  region: 'us-east-1',
  serviceConfigOptions: {
    endpoint: 'http://127.0.0.1:8080',
  }
}

Thanks a ton! (And thanks for responding to my email as well!)

[koxudaxi]

I'm happy 😄

I will improve the documents.

Thank you very much.

[koxudaxi]

@ablankenship10

Dug around in the code and looks like the README info for Postgres should be updated to show
ENGINE: PostgresSQL

instead of

ENGINE: PostgreSQLJDBC

I have checked your docker-compose.yml again.
The engine name is invalid.

PostgresSQLJDBC is wrong.
the correct is PostgreSQLJDBC.

local-data-api/docker-compose-postgres.yml

Line 8 in 3651f0c

ENGINE: PostgreSQLJDBC

version: '3.1'

services:
  local-data-api:
    image: koxudaxi/local-data-api
    restart: always
    environment:
      ENGINE: PostgresSQLJDBC

I recommend choice PostgreSQLJDBC for better behavior.
Because PostgreSQLJDBC uses a JDBC driver that is more similar to the real aurora.
For example, the engine supports result-metadata.

[ablankenship10]

Ah that seems to work too, so both PostgreSQLJDBC and PostgreSQL are supported but JDBC is better?

[koxudaxi]

Yes, JDBC is better:) Also, I guess AWS Aurora uses a JDBC driver

[ablankenship10]

Perfect, il use that then :)

Another question as you may know more about the internals of whats going on with the AWS SDK, whats going on with the executeStatement vs executeSql calls? It seems like the latter is deprecated and the former is what other libraries are using to make calls. Any idea if you can just pass a straight string with values into the executeStatement method rather than breaking the values off into the parameters field? I'm having issues with certain datatypes and not sure why I cant just pass a normal postres SQL string through the data-api proxy rather than doing some weird proprietary AWS format

[koxudaxi]

Yes, executeSql is deprecated. local-data-api doesn't support the API.

AWS team is developing DataAPI actively.
local-data-api doesn't support some features.
For example, I have never seen this option
They have updated API when I don't know.

But, The api usually treat parameters as specified type
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/data-api.html#data-api-parameter-mapping

What type did you give a parameter to local-data-api?

[ablankenship10]

Thanks for the link. I'm Using TypeORM (nodejs/typescript) and failing to use an array(string) field. Its supported by Postgres with SQL so unsure why they made a weird API instead of just letting you pass a normal SQL string directly. From what I can tell it is supposed to be supported now via the data api using arrayValues but for some reason the packages im using (data-api-client via TypeORM) are not supporting it yet and that even when they tried it returns Not Supported Error.