Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add devcontainer for examples #49

Merged
merged 9 commits into from
Feb 2, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,11 @@ Data Factory does not support unit testing out of the box. The only way to valid

## Getting started

1. Set up an empty Python project with your favorite testing library
2. Install the package using your preferred package manager:
* Pip: `pip install data-factory-testing-framework`
* Poetry: `poetry add data-factory-testing-framework`
3. Start writing tests
### Start writting tests
ydaponte marked this conversation as resolved.
Show resolved Hide resolved

To get started using the tests, refer to the following [README](./examples/README.md)

### Start contributing to the framework (PLACEHOLDER - TO BE UPDATED)
ydaponte marked this conversation as resolved.
Show resolved Hide resolved

## Features - Examples

Expand Down
38 changes: 38 additions & 0 deletions examples/.devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/python
{
"name": "Python 3",
// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
"image": "mcr.microsoft.com/vscode/devcontainers/python:0-3.11",
"features": {
"ghcr.io/devcontainers-contrib/features/bandit:2": {},
"ghcr.io/devcontainers-contrib/features/black:2": {},
"ghcr.io/devcontainers-contrib/features/curl-apt-get:1": {},
"ghcr.io/devcontainers-contrib/features/flake8:2": {},
"ghcr.io/devcontainers-contrib/features/pylint:2": {}
},
"customizations": {
"vscode": {
"extensions": [
"ms-python.python",
"ms-python.black-formatter",
"ms-python.flake8",
"eamodio.gitlens",
"VisualStudioExptTeam.vscodeintellicode",
"VisualStudioExptTeam.intellicode-api-usage-examples",
"DavidAnson.vscode-markdownlint"
]
}
},
// Features to add to the dev container. More info: https://containers.dev/features.
// "features": {},
// Use 'forwardPorts' to make a list of ports inside the container available locally.
// "forwardPorts": [],
// Use 'postCreateCommand' to run commands after the container is created.
"postCreateCommand": "bash scripts/postCreateCommand.sh"
// Configure tool-specific properties.
// "customizations": {},
// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
// "remoteUser": "root",
//"runArgs": ["--env-file","src/.env"]
}
104 changes: 104 additions & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@

# Data Factory - Using the Testing Framework

To use the Testing Framework, is enough to import the published package resulting from the developement in the current repo located in src/data_factory_testing_framework.
ydaponte marked this conversation as resolved.
Show resolved Hide resolved

## Getting started with writting tests
ydaponte marked this conversation as resolved.
Show resolved Hide resolved

### Running without a Dev Container
1. Set up an empty Python project with your favorite testing library
2. Install the package using your preferred package manager:
* Pip: `pip install data-factory-testing-framework`
* Poetry: `poetry add data-factory-testing-framework`
3. Start writing tests

### Running with a Dev Container

#### Pre-requirements using Dev Containers
To use a Dev Container, you need to have the following software in addition to the previous pre-requisites:

- Docker
- Visual Studio Code Remote Development Extension Pack

In order to open the project in a container follow the following steps:

- Open Visual Studio Code and clone the repository.
- Hit Control-Shift-P to open the command palette and type Dev Containers: Open Folder in Container ...
- When prompted, select the *examples* directory of the Project
- Wait for the container to build, check the logs for more information.

When the container successfully starts you can start writting your data factory pipeline tests.
ydaponte marked this conversation as resolved.
Show resolved Hide resolved


## Features - Examples
ydaponte marked this conversation as resolved.
Show resolved Hide resolved

The samples seen below is the _only_ code that you need to write! The framework will take care of the rest.

1. Evaluate activities (e.g. a WebActivity that calls Azure Batch API)

```python
# Arrange
activity: Activity = pipeline.get_activity_by_name("Trigger Azure Batch Job")
state = PipelineRunState(
parameters=[
RunParameter(RunParameterType.Global, "BaseUrl", "https://example.com"),
RunParameter(RunParameterType.Pipeline, "JobId", "123"),
],
variables=[
PipelineRunVariable("JobName", "Job-123"),
])
state.add_activity_result("Get version", DependencyCondition.SUCCEEDED, {"Version": "version1"})

# Act
activity.evaluate(state)

# Assert
assert "https://example.com/jobs" == activity.type_properties["url"].value
assert "POST" == activity.type_properties["method"].value
body = activity.type_properties["body"].get_json_value()
assert "123" == body["JobId"]
assert "Job-123" == body["JobName"]
assert "version1" == body["Version"]
```

2. Evaluate Pipelines and test the flow of activities given a specific input

```python
# Arrange
pipeline: PipelineResource = test_framework.repository.get_pipeline_by_name("batch_job")

# Runs the pipeline with the provided parameters
activities = test_framework.evaluate_pipeline(pipeline, [
RunParameter(RunParameterType.Pipeline, "JobId", "123"),
RunParameter(RunParameterType.Pipeline, "ContainerName", "test-container"),
RunParameter(RunParameterType.Global, "BaseUrl", "https://example.com"),
])

set_variable_activity: Activity = next(activities)
assert set_variable_activity is not None
assert "Set JobName" == set_variable_activity.name
assert "JobName" == activity.type_properties["variableName"]
assert "Job-123" == activity.type_properties["value"].value

get_version_activity = next(activities)
assert get_version_activity is not None
assert "Get version" == get_version_activity.name
assert "https://example.com/version" == get_version_activity.type_properties["url"].value
assert "GET" == get_version_activity.type_properties["method"]
get_version_activity.set_result(DependencyCondition.Succeeded,{"Version": "version1"})

create_batch_activity = next(activities)
assert create_batch_activity is not None
assert "Trigger Azure Batch Job" == create_batch_activity.name
assert "https://example.com/jobs" == create_batch_activity.type_properties["url"].value
assert "POST" == create_batch_activity.type_properties["method"]
body = create_batch_activity.type_properties["body"].get_json_value()
assert "123" == body["JobId"]
assert "Job-123" == body["JobName"]
assert "version1" == body["Version"]

with pytest.raises(StopIteration):
next(activities)
```

> See examples folder for more samples
4 changes: 4 additions & 0 deletions examples/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
python-dateutil
python-dotenv
pytest
data-factory-testing-framework
3 changes: 3 additions & 0 deletions examples/scripts/postCreateCommand.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/bin/bash
ydaponte marked this conversation as resolved.
Show resolved Hide resolved
python -m pip install --upgrade pip
pip install -r examples/requirements.txt