Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add devcontainer for examples #49

Merged
merged 9 commits into from
Feb 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 13 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,24 +56,30 @@ These tests are great to have, but miss the following benefits that unit tests,

## Getting started

1. Install the dotnet runtime from [here](https://dotnet.microsoft.com/en-us/download/dotnet/8.0).
### Start writing tests with a Dev Container

To get started using the tests, refer to the following [README](./examples/README.md)

### Start writing tests without a Dev Container

1. Set up an empty Python project with your favorite testing library
2. Install the dotnet runtime from [here](https://dotnet.microsoft.com/en-us/download/dotnet/8.0).
Using only the runtime and not the SDK should be sufficient.
This is required to run some expression functions on dotnet just like in Data Factory.
2. Set up an empty Python project with your favorite testing library

3. Set up an empty Python project with your favorite testing library
More information:
[docs_Setup](/docs/environment_setup/unit_test_setup.md)

3. Install the package using your preferred package manager:
4. Install the package using your preferred package manager:

Pip: `pip install data-factory-testing-framework`

4. Create a Folder in your project and copy the JSON Files with the pipeline definitions locally.
5. Create a Folder in your project and copy the JSON Files with the pipeline definitions locally.

More information:
[docs Json](/docs/environment_setup/json_pipeline_files.md)
[Json Guidance](/docs/environment_setup/json_pipeline_files.md)

5. Start writing tests
6. Start writing tests

## Features - Examples

Expand Down
39 changes: 39 additions & 0 deletions examples/.devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/python
{
"name": "Python 3",
// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
"image": "mcr.microsoft.com/vscode/devcontainers/python:0-3.11",
"features": {
"ghcr.io/devcontainers-contrib/features/bandit:2": {},
"ghcr.io/devcontainers-contrib/features/black:2": {},
"ghcr.io/devcontainers-contrib/features/curl-apt-get:1": {},
"ghcr.io/devcontainers-contrib/features/flake8:2": {},
"ghcr.io/devcontainers-contrib/features/pylint:2": {}
},
"customizations": {
"vscode": {
"extensions": [
"ms-python.python",
"ms-python.black-formatter",
"ms-python.flake8",
"eamodio.gitlens",
"VisualStudioExptTeam.vscodeintellicode",
"VisualStudioExptTeam.intellicode-api-usage-examples",
"DavidAnson.vscode-markdownlint",
"yzhang.markdown-all-in-one"
]
}
},
// Features to add to the dev container. More info: https://containers.dev/features.
// "features": {},
// Use 'forwardPorts' to make a list of ports inside the container available locally.
// "forwardPorts": [],
// Use 'postCreateCommand' to run commands after the container is created.
"postCreateCommand": "bash ./.devcontainer/postCreateCommand.sh"
// Configure tool-specific properties.
// "customizations": {},
// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
// "remoteUser": "root",
//"runArgs": ["--env-file","src/.env"]
}
3 changes: 3 additions & 0 deletions examples/.devcontainer/postCreateCommand.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/bin/bash
python -m pip install --upgrade pip
pip install -r ./.devcontainer/requirements.txt
4 changes: 4 additions & 0 deletions examples/.devcontainer/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
python-dateutil
python-dotenv
pytest
data-factory-testing-framework
108 changes: 108 additions & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@

# Data Factory - Using the Testing Framework

To use the Testing Framework, is enough to import the published package resulting from the development
in the current repo.
The package is located in src/data_factory_testing_framework.

## Getting started with writing tests

### Running without a Dev Container

1. Set up an empty Python project with your favorite testing library
2. To use the Testing Framework, install the data-factory-testing-framework package from PyPI via your preferred package
manager:
* Pip: `pip install data-factory-testing-framework`
* Poetry: `poetry add data-factory-testing-framework`
3. Start writting tests

### Running within a Dev Container

#### Pre-requirements using Dev Containers

To use a Dev Container, you need to have the following software in addition to the previous pre-requisites:

* Docker
* Visual Studio Code Remote Development Extension Pack

In order to open the project in a container follow the following steps:

* Open Visual Studio Code and clone the repository.
* Hit Control-Shift-P to open the command palette and type Dev Containers: Open Folder in Container ...
* When prompted, select the *examples* directory of the Project
* Wait for the container to build, check the logs for more information.

When the container successfully starts you can start writing your data factory pipeline tests.

## Features - Examples
ydaponte marked this conversation as resolved.
Show resolved Hide resolved

The samples seen below is the *only* code that you need to write! The framework will take care of the rest.

1. Evaluate activities (e.g. a WebActivity that calls Azure Batch API)

```python
# Arrange
activity: Activity = pipeline.get_activity_by_name("Trigger Azure Batch Job")
state = PipelineRunState(
parameters=[
RunParameter(RunParameterType.Global, "BaseUrl", "https://example.com"),
RunParameter(RunParameterType.Pipeline, "JobId", "123"),
],
variables=[
PipelineRunVariable("JobName", "Job-123"),
])
state.add_activity_result("Get version", DependencyCondition.SUCCEEDED, {"Version": "version1"})

# Act
activity.evaluate(state)

# Assert
assert "https://example.com/jobs" == activity.type_properties["url"].value
assert "POST" == activity.type_properties["method"].value
body = activity.type_properties["body"].get_json_value()
assert "123" == body["JobId"]
assert "Job-123" == body["JobName"]
assert "version1" == body["Version"]
```

2. Evaluate Pipelines and test the flow of activities given a specific input

```python
# Arrange
pipeline: PipelineResource = test_framework.repository.get_pipeline_by_name("batch_job")

# Runs the pipeline with the provided parameters
activities = test_framework.evaluate_pipeline(pipeline, [
RunParameter(RunParameterType.Pipeline, "JobId", "123"),
RunParameter(RunParameterType.Pipeline, "ContainerName", "test-container"),
RunParameter(RunParameterType.Global, "BaseUrl", "https://example.com"),
])

set_variable_activity: Activity = next(activities)
assert set_variable_activity is not None
assert "Set JobName" == set_variable_activity.name
assert "JobName" == activity.type_properties["variableName"]
assert "Job-123" == activity.type_properties["value"].value

get_version_activity = next(activities)
assert get_version_activity is not None
assert "Get version" == get_version_activity.name
assert "https://example.com/version" == get_version_activity.type_properties["url"].value
assert "GET" == get_version_activity.type_properties["method"]
get_version_activity.set_result(DependencyCondition.Succeeded,{"Version": "version1"})

create_batch_activity = next(activities)
assert create_batch_activity is not None
assert "Trigger Azure Batch Job" == create_batch_activity.name
assert "https://example.com/jobs" == create_batch_activity.type_properties["url"].value
assert "POST" == create_batch_activity.type_properties["method"]
body = create_batch_activity.type_properties["body"].get_json_value()
assert "123" == body["JobId"]
assert "Job-123" == body["JobName"]
assert "version1" == body["Version"]

with pytest.raises(StopIteration):
next(activities)
```

> See examples folder for more samples
Loading