DataOps Observability Integration Agents are part of DataKitchen's Open Source Data Observability. They connect to various ETL, ELT, BI, data science, data visualization, data governance, and data analytic tools. They provide logs, messages, metrics, overall run-time start/stop, subtask status, and scheduling information to DataOps Observability.
Before you begin you will need to clone the repository.
# Mac with Homebrew
brew install git
# Ubuntu
sudo apt install git
git clone <URL>Create a development environment by running the following:
# this project requires python 3.11
cd /your/cloned/repo/
python3.11 -m venv venv
source venv/bin/activate
# Linux
pip install .[dev,release]
# Note: when running on MacOS with zsh shell, enclose the directive in single quotes to stop the shell
# interpreting it as a wild card.
pip install '.[dev,release]'Next, you should set up pre-commit.
pre-commit installTests will work by default from the terminal when run from root.
tests can be run from the root with the following command.
pytest .Some fixtures are specified as plugins from the root conftest.py. PyCharm by default sets the root to tests or the
test directory. You will need to manually set the root for tests that use the plugin.
Edit Configurations... --> Edit Configuration Templates --> Python Tests --> Pytest --> Working directory.
Change it to be the root of the repo.
A configuration has been saved in .run/Template Python tests.run.xml. Hopefully pycharm will pick that up
automatically.
A lot of tests require data that be shared. By default, all testlib/**/fixtures files are installed as pytest plugins. That means the fixture is accessible just by referencing it, and allows us to reference just a single set of data.
Please use it.
See <root>/conftest.py for details.
After setting up your environment and entering mypy can be from invoke
inv mypyThe rest of the tools can be run as a part of pre-commit, or through invoke
inv lintTIP: You can run these both together like so:
inv lint mypyTo test that the docker image builds, you can run
# --tag may be specified multiple times.
inv build-image [--tag tag]A release note must be created in order to merge into main. You can do so manually after installing the development
tools, or you can use the invoke mk-note command which will prompt you for the correct information.
You'll need your ticket number and release type.
Valid release note categories are:
- fixed
- added
- deprecated
- removed
- changed
- chore
these follow the Keep a Changlog format.
Run the following:
# e.g., towncrier create AG-17.fixed.md --edit
towncrier create your-ticket.type.md
# or; and follow the prompts
inv mk-noteThese notes will be consumed by leadership and tech-comms. Be thorough and describe what you did!
Note: chore types won't appear on changelog. They also do not require a ticket, just press enter when prompted for board name.
Releases should be done strictly through the build-system. Tagging should be done through the build-system.
On either a build of main or a manual build of main, you should see a series of manual jobs in the pipeline.
- Trigger Breaking Release
- Trigger Feature Release
- Trigger Bugfix Release
Choose one (and only one) of these jobs and hit the "play" button.
These will increment the versions and push them to the appropriate release channels.
We recommend you start by going through the Data Observability Overview Demo.
For support requests, join the Data Observability Slack and ask post on #support channel.
Talk and Learn with other data practitioners who are building with DataKitchen. Share knowledge, get help, and contribute to our open-source project.
Join our community here:
For details on contributing a new Agent or running the project for development, check out our contributing guide.
DataKitchen DataOps Observability Agents are Apache 2.0 licensed.