governor-postgresql-transaction-id

Synopsis

senzing_governor.py contains a Governor class that can be used to govern the behavior of programs like:

1. stream-loader.py
2. redoer.py

when used with PostgreSQL database.

Overview

The senzing_governor.py in this repository monitors the age of the PostgreSQL Transaction IDs to determine when to pass back control to the caller.

Essentially it does the following SQL statement:

SELECT age(datfrozenxid) FROM pg_database WHERE datname = (%s);

If the age is greater than a high-watermark, SENZING_GOVERNOR_POSTGRESQL_HIGH_WATERMARK, then the Governor waits until the watermark recedes to a low-watermark, SENZING_GOVERNOR_POSTGRESQL_LOW_WATERMARK.

The lowering of the watermark must be done manually by using PostgreSQL VACUUM.

Contents

1. Preamble
   1. Legend
2. Expectations
3. Test using Command Line Interface
   1. Prerequisites for CLI
   2. Download
   3. Environment variables for CLI
   4. Run command
4. Configuration
5. References

Preamble

At Senzing, we strive to create GitHub documentation in a "don't make me think" style. For the most part, instructions are copy and paste. Whenever thinking is needed, it's marked with a "thinking" icon 🤔. Whenever customization is needed, it's marked with a "pencil" icon ✏️. If the instructions are not clear, please let us know by opening a new Documentation issue describing where we can improve. Now on with the show...

Legend

1. 🤔 - A "thinker" icon means that a little extra thinking may be required. Perhaps there are some choices to be made. Perhaps it's an optional step.
2. ✏️ - A "pencil" icon means that the instructions may need modification before performing.
3. ⚠️ - A "warning" icon means that something tricky is happening, so pay attention.

Expectations

• Space: This repository and demonstration require 10 KB free disk space.
• Time: Budget 20 minutes to get the demonstration up-and-running, depending on CPU and network speeds.
• Background knowledge: This repository assumes a working knowledge of:
  • PostgreSQL

Test using Command Line Interface

Prerequisites for CLI

🤔 The following tasks need to be complete before proceeding. These are "one-time tasks" which may already have been completed.

1. Install system dependencies:
   1. Use apt based installation for Debian, Ubuntu and others
      1. See apt-packages.txt for list
   2. Use yum based installation for Red Hat, CentOS, openSuse and others.
      1. See yum-packages.txt for list
2. Install Python dependencies:
   1. See requirements.txt for list
      1. Installation hints

Download

1. Get a local copy of senzing_governor.py and programs that test the governor.

   1. ✏️ Specify a director of where to download files. Example:

      export SENZING_GOVERNOR_PROJECT_DIR=~/test-governor

   2. Make project directory. Example:

      mkdir -p ${SENZING_GOVERNOR_PROJECT_DIR}

   3. Download files. Example:

      curl -X GET \
        --output ${SENZING_GOVERNOR_PROJECT_DIR}/senzing_governor.py \
        https://raw.githubusercontent.com/Senzing/governor-postgresql-transaction-id/main/senzing_governor.py
      
      curl -X GET \
        --output ${SENZING_GOVERNOR_PROJECT_DIR}/senzing_governor_tester.py \
        https://raw.githubusercontent.com/Senzing/governor-postgresql-transaction-id/main/senzing_governor_tester.py
      
      chmod +x ${SENZING_GOVERNOR_PROJECT_DIR}/senzing_governor_tester.py
      
      curl -X GET \
        --output ${SENZING_GOVERNOR_PROJECT_DIR}/senzing_governor_tester_context_manager.py \
        https://raw.githubusercontent.com/Senzing/governor-postgresql-transaction-id/main/senzing_governor_tester_context_manager.py
      
      chmod +x ${SENZING_GOVERNOR_PROJECT_DIR}/senzing_governor_tester_context_manager.py

2. 🤔 Alternative: The entire git repository can be downloaded by following instructions at Clone repository

Environment variables for CLI

1. Update PYTHONPATH. Example:

   export PYTHONPATH=${PYTHONPATH}:${SENZING_GOVERNOR_PROJECT_DIR}

2. ✏️ Identify PostgreSQL database. The format of the URL can be seen at SENZING_DATABASE_URL Example:

   export SENZING_GOVERNOR_DATABASE_URLS=postgresql://postgres:postgres@localhost:5432/G2

   1. 🤔 Optional: Multiple databases can be specified in a list. Example:

      export SENZING_GOVERNOR_DATABASE_URLS=postgresql://postgres:postgres@localhost:5432/G2,postgresql://postgres:postgres@localhost:5432/G2-RES

   2. 🤔 Optional: The comma is used as a list item separator. If a different separator is needed it can be specified. Example:

      export SENZING_GOVERNOR_LIST_SEPARATOR="+"
      export SENZING_GOVERNOR_DATABASE_URLS="postgresql://postgres:postgres@localhost:5432/G2+postgresql://postgres:postgres@localhost:5432/G2-RES"

Run command

1. Run the command. Example:

   ${SENZING_GOVERNOR_PROJECT_DIR}/senzing_governor_tester.py

2. For more examples of use, see Examples of CLI.

Configuration

Configuration values specified by environment variable or command line parameter.

• SENZING_GOVERNOR_DATABASE_URLS
• SENZING_GOVERNOR_INTERVAL
• SENZING_GOVERNOR_LIST_SEPARATOR
• SENZING_GOVERNOR_POSTGRESQL_HIGH_WATERMARK
• SENZING_GOVERNOR_POSTGRESQL_LOW_WATERMARK
• SENZING_GOVERNOR_PROJECT_DIR
• SENZING_GOVERNOR_WAIT

References

• Development
• Errors
• Examples