feat: support PostgreSQL as optional storage backend

.github/workflows/github-actions.yml

@@ -1,32 +1,45 @@
 name: github-actions
 on: [push, pull_request]
 jobs:
-    test:
-        runs-on: ubuntu-latest
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        db_url: ["http://localhost:8000", "postgres://postgres:postgres@localhost:5432/gene_normalizer_test"]
+    services:
+      postgres:
+        image: postgres:14
         env:
-            AWS_ACCESS_KEY_ID: ${{ secrets.DUMMY_AWS_ACCESS_KEY_ID }}
-            AWS_SECRET_ACCESS_KEY: ${{ secrets.DUMMY_AWS_SECRET_ACCESS_KEY }}
-            AWS_DEFAULT_REGION: us-east-2
-            AWS_DEFAULT_OUTPUT: text
-            GENE_NORM_DB_URL: http://localhost:8000
-            GENE_TEST: true
-        steps:
-            - uses: actions/checkout@v3
+          POSTGRES_USER: 'postgres'
+          POSTGRES_DB: 'gene_normalizer_test'
+          POSTGRES_PASSWORD: 'postgres'
+        ports:
+          - 5432:5432
+    env:
+      AWS_ACCESS_KEY_ID: ${{ secrets.DUMMY_AWS_ACCESS_KEY_ID }}
+      AWS_SECRET_ACCESS_KEY: ${{ secrets.DUMMY_AWS_SECRET_ACCESS_KEY }}
+      AWS_DEFAULT_REGION: us-east-2
+      AWS_DEFAULT_OUTPUT: text
+      GENE_NORM_DB_URL: ${{ matrix.db_url }}
+      GENE_TEST: true
+    steps:
+      - uses: actions/checkout@v3
 
-            - name: Setup Python
-              uses: actions/setup-python@v4
-              with:
-                  python-version: 3.8
+      - name: Setup Python
+        uses: actions/setup-python@v4
+        with:
+            python-version: 3.8
 
-            - name: Install dependencies
-              run: |
-                  python3 -m pip install pipenv
-                  pipenv install --dev
+      - name: Install dependencies
+        run: |
+          python3 -m pip install pipenv
+          pipenv install --dev
 
-            - name: Build local DynamoDB
-              run: |
-                chmod +x ./tests/unit/dynamodb_build.bash
-                ./tests/unit/dynamodb_build.bash
+      - name: Build local DynamoDB
+        if: ${{ env.GENE_NORM_DB_URL == 'http://localhost:8000' }}
+        run: |
+          chmod +x ./tests/unit/dynamodb_build.bash
+          ./tests/unit/dynamodb_build.bash
 
-            - run: pipenv run flake8
-            - run: pipenv run pytest tests/
+      - run: pipenv run flake8
+      - run: pipenv run pytest tests/

.gitignore

@@ -118,6 +118,7 @@ venv/
 ENV/
 env.bak/
 venv.bak/
+.python-version
 
 # Spyder project settings
 .spyderproject
@@ -160,4 +161,4 @@ dynamodb_local_latest/*
 
 Pipfile.lock
 *.toml
-*.zip
+*.zip

Pipfile

@@ -16,7 +16,7 @@ boto3 = "*"
 gene = {editable = true, path = "."}
 gffutils = "*"
 "biocommons.seqrepo" = "*"
-psycopg2-binary = "*"
+psycopg = {version = "*", extras=["binary"]}
 pytest = "*"
 pre-commit = "*"
 flake8 = "*"

README.md

@@ -1,31 +1,19 @@
 [![DOI](https://zenodo.org/badge/309797998.svg)](https://zenodo.org/badge/latestdoi/309797998)
 
-# Gene Normalization
+# Gene Normalizer
 Services and guidelines for normalizing gene terms
 
-Installing with pip:
+## Installation
+
+The Normalizer is available via PyPI:
 
 ```commandline
 pip install gene[dev]
 ```
 
 The `[dev]` argument tells pip to install packages to fulfill the dependencies of the `gene.etl` package.
 
-## Developer instructions
-Following are sections include instructions specifically for developers.
-
-### Installation
-For a development install, we recommend using Pipenv. See the
-[pipenv docs](https://pipenv-fork.readthedocs.io/en/latest/#install-pipenv-today)
-for direction on installing pipenv in your compute environment.
-
-Once installed, from the project root dir, just run:
-
-```commandline
-pipenv shell
-pipenv lock && pipenv sync
-pipenv install --dev
-```
+### External requirements
 
 Gene Normalization relies on [SeqRepo](https://github.com/biocommons/biocommons.seqrepo) data, which you must download yourself.
 
@@ -44,103 +32,122 @@ PermissionError: [Error 13] Permission denied: '/usr/local/share/seqrepo/2021-01
 
 You will want to do the following:\
 (*Might not be ._fkuefgd, so replace with your error message path*)
+
 ```console
 sudo mv /usr/local/share/seqrepo/2021-01-29._fkuefgd /usr/local/share/seqrepo/2021-01-29
-exit
 ```
 
-### Deploying DynamoDB Locally
+### Database Initialization
 
-We use Amazon DynamoDB for our database. To deploy locally, follow [these instructions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.DownloadingAndRunning.html).
+The Normalizer supports two data storage options:
 
-### Init coding style tests
+* [DynamoDB](https://aws.amazon.com/dynamodb), a NoSQL service provided by AWS. This is our preferred storage solution. In addition to cloud deployment, Amazon also provides a tool for local service, which can be installed [here](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.DownloadingAndRunning.html). Once downloaded, you can start service by running `java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb` in a terminal (add a `-port <VALUE>` option to use a different port)
+* [PostgreSQL](https://www.postgresql.org/), a well-known relational database technology. Once starting the Postgres server process, [ensure that a database is created](https://www.postgresql.org/docs/current/sql-createdatabase.html) (we typically name ours `gene_normalizer`).
 
-Code style is managed by [flake8](https://github.com/PyCQA/flake8) and checked prior to commit.
+By default, the Gene Normalizer expects to find a DynamoDB instance listening at `http://localhost:8000`. Alternative locations can be specified in two ways:
 
-We use [pre-commit](https://pre-commit.com/#usage) to run conformance tests.
+The first way is to set the `--db_url` option to the URL endpoint.
 
-This ensures:
+```commandline
+gene_update --update_all --db_url="http://localhost:8001"
+```
 
-* Check code style
-* Check for added large files
-* Detect AWS Credentials
-* Detect Private Key
+The second way is to set the `GENE_NORM_DB_URL` environment variable to the URL endpoint.
+```commandline
+export GENE_NORM_DB_URL="http://localhost:8001"
+```
 
-Before first commit run:
+To use a PostgreSQL instance instead of DynamoDB, provide a PostgreSQL connection URL instead, e.g.
 
 ```commandline
-pre-commit install
+export GENE_NORM_DB_URL="postgresql://postgres@localhost:5432/gene_normalizer"
 ```
 
+### Adding and refreshing data
 
-### Running unit tests
+Use the `gene_update` command in a shell to update the database.
 
-By default, tests will employ an existing DynamoDB database. For test environments where this is unavailable (e.g. in CI), the `GENE_TEST` environment variable can be set to initialize a local DynamoDB instance with miniature versions of input data files before tests are executed.
+#### Update source(s)
+
+The normalizer currently pulls data from [HGNC](https://www.genenames.org/), [Ensembl](https://useast.ensembl.org/index.html), and [NCBI](https://www.ncbi.nlm.nih.gov/gene/).
+
+To update one source, simply set `--normalizer` to the source you wish to update. The normalizer will check to see if local source data is up-to-date, acquire the most recent data if not, and use it to populate the database.
+
+For example, run the following to acquire the latest HGNC data if necessary, and update the HGNC gene records in the normalizer database:
 
 ```commandline
-export GENE_TEST=true
+gene_update --normalizer="hgnc"
 ```
 
-Running unit tests is as easy as pytest.
+To update multiple sources, you can use the `--normalizer` option with the source names separated by spaces.
+
+#### Update all sources
+
+To update all sources, use the `--update_all` flag:
 
 ```commandline
-pipenv run pytest
+gene_update --update_all
 ```
 
-### Updating the gene normalization database
+### Starting the gene normalization service
 
-Before you use the CLI to update the database, run the following in a separate terminal to start a local DynamoDB service on `port 8000`:
+Once the Gene Normalizer database has been loaded, from the project root, run the following:
 
-```
-java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb
+```commandline
+uvicorn gene.main:app --reload
 ```
 
-To change the port, simply add `-port value`.
+Next, view the OpenAPI docs on your local machine:
 
-#### Update source(s)
-The sources we currently use are: HGNC, Ensembl, and NCBI.
+http://127.0.0.1:8000/gene
+
+## Developer instructions
+The following sections include instructions specifically for developers.
 
-To update one source, simply set `--normalizer` to the source you wish to update.
+### Installation
+For a development install, we recommend using Pipenv. See the
+[pipenv docs](https://pipenv-fork.readthedocs.io/en/latest/#install-pipenv-today)
+for direction on installing pipenv in your compute environment.
 
-From the project root, run the following to update the HGNC source:
+Once installed, clone the repo and initialize the environment:
 
 ```commandline
-python3 -m gene.cli --normalizer="hgnc"
+git clone https://github.com/cancervariants/gene-normalization
+cd gene-normalization
+pipenv shell
+pipenv update
+pipenv install --dev
 ```
 
-To update multiple sources, you can use the `--normalizer` flag with the source names separated by spaces.
+### Init coding style tests
 
-#### Update all sources
+Code style is managed by [flake8](https://github.com/PyCQA/flake8) and checked prior to commit.
 
-To update all sources, use the `--update_all` flag.
+We use [pre-commit](https://pre-commit.com/#usage) to run conformance tests.
 
-From the project root, run the following to update all sources:
+This ensures:
 
-```commandline
-python3 -m gene.cli --update_all
-```
+* Check code style
+* Check for added large files
+* Detect AWS Credentials
+* Detect Private Key
 
-#### Specifying the database URL endpoint
-The default URL endpoint is `http://localhost:8000`.
-There are two different ways to specify the database URL endpoint.
+Before first commit run:
 
-The first way is to set the `--db_url` flag to the URL endpoint.
 ```commandline
-python3 -m gene.cli --update_all --db_url="http://localhost:8001"
+pre-commit install
 ```
 
-The second way is to set the `GENE_NORM_DB_URL` to the URL endpoint.
-```commandline
-export GENE_NORM_DB_URL="http://localhost:8001"
-python3 -m gene.cli --update_all
-```
+### Running unit tests
+
+By default, tests will employ an existing database. For test environments where this is unavailable (e.g. in CI), the `GENE_TEST` environment variable can be set to initialize a local DynamoDB instance with miniature versions of input data files before tests are executed.
 
-### Starting the gene normalization service
-From the project root, run the following:
 ```commandline
- uvicorn gene.main:app --reload
+export GENE_TEST=true
 ```
 
-Next, view the OpenAPI docs on your local machine:
+Running unit tests is as easy as pytest.
 
-http://127.0.0.1:8000/gene
+```commandline
+pipenv run pytest
+```

gene/__init__.py

@@ -34,8 +34,8 @@ def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
 
 
-from gene.schemas import SourceName, NamespacePrefix, SourceIDAfterNamespace, ItemTypes  # noqa: E402, E501
-ITEM_TYPES = {k.lower(): v.value for k, v in ItemTypes.__members__.items()}
+from gene.schemas import SourceName, NamespacePrefix, SourceIDAfterNamespace, RefType  # noqa: E402, E501
+ITEM_TYPES = {k.lower(): v.value for k, v in RefType.__members__.items()}
 
 # Sources we import directly (HGNC, Ensembl, NCBI)
 SOURCES = {source.value.lower(): source.value