docs: add basic and advanced documentation and refactor main readme

microsoft · Feb 6, 2024 · 116d5c7 · 116d5c7
1 parent 2d2b7dd
commit 116d5c7
Show file tree

Hide file tree

Showing 22 changed files with 100 additions and 153 deletions.
diff --git a/README.md b/README.md
@@ -52,26 +52,38 @@ The following pages go deeper into different topics and concepts of the framewor
 
 ### Basic
 
-1. [Installing and initializing the framework](docs/basic/installing-and-initializing-framework.md)
-2. [State](docs/basic/state.md)
-3. [Activity testing](docs/basic/activity-testing.md)
-4. [Pipeline testing](docs/basic/pipeline-testing.md)
-5. [Getting started detailed](docs/basic/getting-started-detailed.md)
+1. [Repository setup](docs/basic/repository_setup.md)
+2. [Installing and initializing the framework](docs/basic/installing_and_initializing_framework.md)
+3. [State](docs/basic/state.md)
+4. [Activity testing](docs/basic/activity_testing.md)
+5. [Pipeline testing](docs/basic/pipeline_testing.md)
 
 ### Advanced
 
 1. [Debugging your activities and pipelines](docs/advanced/debugging.md)
-2. [Development workflow](docs/basic/)
-3. [CI integration](docs/advanced/ci-integration.md)
-4. [Overriding expression functions](docs/advanced/overriding-expression-functions.md)
-5. [Framework internals](docs/advanced/framework-internals.md)
+2. [Development workflow](docs/advanced/development_workflow.md)
+3. [CI integration](docs/advanced/ci_integration.md)
+4. [Overriding expression functions](docs/advanced/overriding_expression_functions.md)
+5. [Framework internals](docs/advanced/framework_internals.md)
+
+### Getting started
+
+If you are new to Python or you want to quickly get started in an opinionated way, you can use the following guide: [Getting started](docs/getting_started.md)
 
 ## Examples
 
 More advanced examples demonstrating the capabilities of the framework:
 
+Fabric:
+
 1. [Batch job example](examples/fabric/batch_job/README.md)
-2. [Copy blobs example](examples/data_factory/copy_blobs/README.md)
+
+Azure Data Factory:
+
+1. [Copy blobs example](examples/data_factory/copy_blobs/README.md)
+2. [Batch job example](examples/data_factory/batch_job/README.md)
+
+## Roadmap
 
 ## Contributing
 

diff --git a/docs/advanced/ci-integration.md → docs/advanced/ci_integration.md b/docs/advanced/ci-integration.md → docs/advanced/ci_integration.md
diff --git a/docs/advanced/debugging.md b/docs/advanced/debugging.md
@@ -1,4 +1,3 @@
-# Tips
+# Debugging
 
-1. After parsing a data factory resource file, you can use the debugger to easily discover which classes are actually
-   initialized so that you can cast them to the correct type.
+As the framework is dynamically parsing and interpreting data factory resource files, it can be difficult to figure out with which objects you are working with. It is recommended to use the debugger during development of your tests to get a better idea of what activities are being returned and to see the structure of the activity and its properties.
diff --git a/docs/advanced/development-workflow.md → docs/advanced/development_workflow.md b/docs/advanced/development-workflow.md → docs/advanced/development_workflow.md
diff --git a/docs/advanced/framework-internals.md b/docs/advanced/framework-internals.md
diff --git a/docs/advanced/framework_internals.md b/docs/advanced/framework_internals.md
@@ -0,0 +1,3 @@
+# Framework internals
+
+This page will be used to describe the internals of the testing framework. It will be used to document the architecture, design decisions, and implementation details of the framework.
diff --git a/...vanced/overriding-expression-functions.md → ...vanced/overriding_expression_functions.md b/...vanced/overriding-expression-functions.md → ...vanced/overriding_expression_functions.md
diff --git a/docs/basic/activity-testing.md → docs/basic/activity_testing.md b/docs/basic/activity-testing.md → docs/basic/activity_testing.md
@@ -64,7 +64,7 @@ pipeline = test_framework.repository.get_pipeline_by_name("trigger_job_pipeline"
 activity = pipeline.get_activity_by_name("Trigger job")
 ```
 
-Create a `PipelineRunState` instance with the input used in the `url` expression, so a globalParameter `BaseUrl` and a variable `JobName`. See [State](docs/basic/state.md) for a detailed explanation of the `PipelineRunState` class.
+Create a `PipelineRunState` instance with the input used in the `url` expression, so a globalParameter `BaseUrl` and a variable `JobName`.
 
 ```python
 state = PipelineRunState(
@@ -76,9 +76,11 @@ state = PipelineRunState(
     ])
 ```
 
+> See [State](docs/basic/state.md) for a detailed explanation of the `PipelineRunState` class.
+
 ## Act
 
-Call the `evaluate` method on the activity with the `PipelineRunState` instance. This will evaluate all expressions in the activity using the provided state and store the result in a `value` property.
+Call the `evaluate` method on the `activity` with the `PipelineRunState` instance. This will evaluate all expressions in the activity using the provided state and store the result in a `value` property.
 
 ```python
 activity.evaluate(state)
@@ -126,4 +128,4 @@ activity: IfConditionActivity = pipeline.get_activity_by_name("If condition")
 child_activity = activity.if_true_activities[0]  # Or loop through them by name
 ```
 
-If you would like to evaluate the condition of a control activity in combination with the child activities, then write a pipeline test as described in [Pipeline testing](docs/basic/pipeline-testing.md).
+If you would like to evaluate the condition of a control activity in combination with the child activities, then write a pipeline test as described in [Pipeline testing](pipeline_testing.md).
diff --git a/docs/basic/getting-started-detailed.md b/docs/basic/getting-started-detailed.md
diff --git a/.../installing-and-initializing-framework.md → .../installing_and_initializing_framework.md b/.../installing-and-initializing-framework.md → .../installing_and_initializing_framework.md
@@ -31,5 +31,5 @@ activity = pipeline.get_activity_by_name("webactivity_name")
 
 See the following pages for more information on how to write tests for activities and pipelines:
 
-1. [Activity testing](activity-testing.md)
-2. [Pipeline testing](pipeline-testing.md)
+1. [Activity testing](activity_testing.md)
+2. [Pipeline testing](pipeline_testing.md)
diff --git a/docs/basic/pipeline-testing.md → docs/basic/pipeline_testing.md b/docs/basic/pipeline-testing.md → docs/basic/pipeline_testing.md
@@ -2,9 +2,9 @@
 
 The framework provides a way to evaluate a pipeline and validate the execution flow of each activity and ensure that the outcome of one activity is correctly passed to the next activity.
 
-Get acquainted with the [Activity testing](activity-testing.md) page before proceeding with pipeline testing, because similar concepts are used.
+Get acquainted with the [Activity testing](activity_testing.md) page before proceeding with pipeline testing, because similar concepts are used.
 
-Make sure to have initialized the framework before writing tests. See [Installing and initializing the framework](installing-and-initializing-framework.md) for more information.
+Make sure to have initialized the framework before writing tests. See [Installing and initializing the framework](installing_and_initializing_framework.md) for more information.
 
 Now that you have a `TestFramework` instance, you can start writing tests for your pipelines. A test always follows the Arrange-Act-Assert pattern:
 
@@ -116,7 +116,7 @@ The next(activities) method might throw an exception if the expression is invali
 3. Activity outputs being referenced are set through `activity.set_result` method
 4. Dependency conditions are met
 
-## No more activities
+### Asserting end of pipeline execution
 
 When the generator has no more activities to return, it will raise a `StopIteration` exception. This is a signal that all activities have been evaluated and the pipeline has been executed as expected.
 
@@ -127,11 +127,11 @@ with pytest.raises(StopIteration):
 
 ## Control activities
 
-The following activities are `ControlActivities` and will not be returned by the generator: `IfCondition`, `ForEach`, `Switch` and `Until`. The conditions of these activities are automatically evaluated and the child activities are executed based on the result of the condition.
+The following activities are `ControlActivities` and will not be returned by the generator: `IfCondition`, `ForEach`, `Switch` and `Until`. The conditions of these activities are automatically evaluated and the child activities are evaluated and returned based on the result of the condition.
 
 ## Execute child pipelines
 
-Child pipelines and their activities can be included in the evaluation of the parent pipeline by setting the `should_evaluate_child_pipelines` of the `TestFramework` constructor to True`. The framework will evaluate the parameters of the`ExecutePipelineActivity` and then evaluate the child pipeline with the provided parameters.
+Child pipelines and their activities can be included in the evaluation of the parent pipeline by setting the `should_evaluate_child_pipelines` of the `TestFramework` constructor to `True`. The framework will evaluate the parameters of the`ExecutePipelineActivity` and then evaluate the child pipeline with the provided parameters.
 
 ```python
 test_framework = TestFramework(should_evaluate_child_pipelines=True)

diff --git a/docs/basic/repository_setup.md b/docs/basic/repository_setup.md
@@ -0,0 +1,23 @@
+# Repository setup
+
+## Git integration
+
+To be able to write tests for data factory, you need to have the pipeline and activity definitions available. The recommended way to do this is to sync the Data Factory instance to a git repository, so that you can create a `tests` folder in the same repository and write tests for your data factory. The git integration process can be found here:
+
+1. [Fabric - Git integration process](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-integration-process)
+2. [Azure Data Factory - Git integration process](https://learn.microsoft.com/azure/data-factory/source-control)
+
+### Alternative for Azure Data Factory
+
+If you want to download a single JSON file for testing purposes, you can do so by following these steps:
+
+1. Open your Data Factory instance, and open the pipeline you want to test.
+2. Click on the action ellipses
+3. Click "Download support files"
+4. Extract the zip file containing the pipeline definition in a folder of your choice.
+
+> Remember the location of this folder, as you will need it to initialize the framework.
+
+Once your repository is set up, you can install and initialize the framework as described in the [installing and initializing the framework](installing_and_initializing_framework.md) page.
+
+![image](../images/download_support_files.png)
diff --git a/docs/environment_setup/images/env_variable.png b/docs/environment_setup/images/env_variable.png
diff --git a/docs/environment_setup/images/installing_poetry_power_shell.png b/docs/environment_setup/images/installing_poetry_power_shell.png
diff --git a/docs/environment_setup/images/new_project.png b/docs/environment_setup/images/new_project.png
diff --git a/docs/environment_setup/images/pipinstall_poetry.png b/docs/environment_setup/images/pipinstall_poetry.png
diff --git a/docs/environment_setup/images/poetry_framework.png b/docs/environment_setup/images/poetry_framework.png
diff --git a/docs/environment_setup/images/power_shell_invoke.png b/docs/environment_setup/images/power_shell_invoke.png
diff --git a/docs/environment_setup/json_pipeline_files.md b/docs/environment_setup/json_pipeline_files.md
diff --git a/docs/environment_setup/unit_test_setup.md b/docs/environment_setup/unit_test_setup.md
diff --git a/docs/getting_started.md b/docs/getting_started.md
@@ -0,0 +1,38 @@
+# Getting started
+
+This getting started focuses on novice users who are not familiar with Python or package management. It will guide you through the process of setting up a Python project, installing the framework and downloading the pipeline files so that you can start writing tests.
+
+## Download your Data Factory pipeline files
+
+The framework is designed to work with the JSON files that define your data factory pipelines and activities. You can download these files from your data factory environment.
+
+## Installing the framework in your project
+
+If you are using Visual Studio Code:
+
+1. Create a new folder for your project.
+2. Install the following framework from the terminal:
+
+   ```bash
+   pip install data-factory-testing-framework
+   ```
+
+3. Install pytest as testing library. All examples in this documentation are using pytest.
+
+   ```bash
+   pip install pytest
+   ```
+
+4. Download the pipeline files from your data factory environment and place them in the project folder as described in the [repository setup](basic/repository_setup.md) page.
+
+Additional resources:
+
+* [Get Started Tutorial for Python in Visual Studio Code](https://code.visualstudio.com/docs/python/python-tutorial)
+* [Integrated Terminal in Visual Studio Code](https://code.visualstudio.com/docs/terminal/basics)
+* [pytest: helps you write better programs — pytest documentation](https://docs.pytest.org/en/7.4.x/)
+
+Once setup, you can read the following pages to learn how to write tests for your data factory:
+
+1. [Initializing the framework](basic/installing_and_initializing_framework.md)
+2. [Activity testing](basic/activity_testing.md)
+3. [Pipeline testing](basic/pipeline_testing.md)
diff --git a/...t_setup/images/download_support_files.png → docs/images/download_support_files.png b/...t_setup/images/download_support_files.png → docs/images/download_support_files.png