style: add linting checks (#60)

* style: add linting checks

added markdown, yaml, general file formatting checks to improve readability, version control handling, and consistency.

.github/workflows/ci-cd.yml

@@ -1,42 +1,43 @@
+---
 name: CI-CD
 
-on: [push]
+on: [push]  # yamllint disable-line rule:truthy
 
 jobs:
   build:
     runs-on: ubuntu-latest
     permissions:
       contents: read
     steps:
-      #----------------------------------------------
+      # ----------------------------------------------
       #       check-out repo and set-up python
-      #----------------------------------------------
+      # ----------------------------------------------
       - name: Check out repository
         uses: actions/checkout@v3
       - name: Set up python
         id: setup-python
         uses: actions/setup-python@v4
         with:
           python-version: '3.9'
-      #----------------------------------------------
+      # ----------------------------------------------
       #  -----  install & configure poetry  -----
-      #----------------------------------------------
+      # ----------------------------------------------
       - name: Install Poetry
         uses: snok/install-poetry@v1
         with:
           virtualenvs-create: true
           virtualenvs-in-project: true
           installer-parallel: true
 
-      #----------------------------------------------
+      # ----------------------------------------------
       # install dependencies if cache does not exist
-      #----------------------------------------------
+      # ----------------------------------------------
       - name: Install dependencies
         if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
         run: poetry install --no-interaction --no-root
-      #----------------------------------------------
+      # ----------------------------------------------
       # install your root project, if required
-      #----------------------------------------------
+      # ----------------------------------------------
       - name: Install project
         run: poetry install --no-interaction
       #----------------------------------------------
@@ -51,9 +52,9 @@ jobs:
         run: |
           poetry version 0.0.1.alpha${{ github.run_number}}
           poetry build
-      #----------------------------------------------
+      # ----------------------------------------------
       # upload dist
-      #----------------------------------------------
+      # ----------------------------------------------
       - name: Upload dist
         uses: actions/upload-artifact@v3
         with:

.markdownlint.yaml

@@ -0,0 +1,3 @@
+---
+MD013:
+  line_length: 120

.markdownlintignore

@@ -0,0 +1,3 @@
+# ignore Microsoft-generated files
+SECURITY.md
+SUPPORT.md

.pre-commit-config.yaml

@@ -1,24 +1,40 @@
+---
 repos:
--   repo: https://github.com/python-poetry/poetry
+  - repo: https://github.com/python-poetry/poetry
     rev: 1.6.1
     hooks:
-    -   id: poetry-check
+      - id: poetry-check
         args: []
-    -   id: poetry-lock
+      - id: poetry-lock
         args: ["--no-update"]
-- repo: https://github.com/astral-sh/ruff-pre-commit
-  rev: v0.1.5
-  hooks:
-    - id: ruff
-      args: [ --fix ]
-    - id: ruff-format
--   repo: local
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.5.0
     hooks:
-    -   id: pytest
+      - id: end-of-file-fixer
+      - id: trailing-whitespace
+      - id: mixed-line-ending
+        args: [--fix=auto]
+  - repo: https://github.com/adrienverge/yamllint
+    rev: v1.33.0
+    hooks:
+      - id: yamllint
+        args: [--strict]
+  - repo: https://github.com/igorshubovych/markdownlint-cli
+    rev: v0.38.0
+    hooks:
+      - id: markdownlint-fix
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.1.5
+    hooks:
+      - id: ruff
+        args: [--fix]
+      - id: ruff-format
+  - repo: local
+    hooks:
+      - id: pytest
         name: pytest
         entry: poetry run pytest .
         language: system
         types: [python]
         require_serial: true
         pass_filenames: false
-

.vscode/settings.json

@@ -9,4 +9,4 @@
     "python.testing.cwd": "${workspaceFolder}",
     "python.testing.pytestEnabled": true,
     "python.testing.unittestEnabled": false,
-}
+}

.yamllint.yaml

@@ -0,0 +1,4 @@
+---
+rules:
+  new-lines:
+    type: platform

DEVELOPMENT.md

@@ -24,7 +24,8 @@ poetry run pre-commit run
 
 ### Run Linting
 
-We use Ruff to lint our code as it provides similar rule sets to well-established linters into one (e.g., black, flake8, isort, and pydocstyle).
+We use Ruff to lint our code as it provides similar rule sets to well-established linters into one
+(e.g., black, flake8, isort, and pydocstyle).
 
 To run linting, run the following command:
 

README.md

@@ -1,64 +1,80 @@
 # Data Factory - Testing Framework
 
-A test framework that allows you to write unit and functional tests for Data Factory pipelines against the git integrated json resource files.
+A test framework that allows you to write unit and functional tests for Data Factory
+pipelines against the git integrated json resource files.
 
 Supporting currently:
+
 * [Fabric Data Factory](https://learn.microsoft.com/en-us/fabric/data-factory/)
 * [Azure Data Factory v2](https://learn.microsoft.com/en-us/azure/data-factory/concepts-pipelines-activities?tabs=data-factory)
 
 Planned:
-* [Azure Synapse Analytics](https://learn.microsoft.com/en-us/azure/data-factory/concepts-pipelines-activities?context=%2Fazure%2Fsynapse-analytics%2Fcontext%2Fcontext&tabs=data-factory/)
 
+* [Azure Synapse Analytics](https://learn.microsoft.com/en-us/azure/data-factory/concepts-pipelines-activities?context=%2Fazure%2Fsynapse-analytics%2Fcontext%2Fcontext&tabs=data-factory/)
 
 ## Disclaimer
 
-This unit test framework is not officially supported. It is currently in an experimental state and has not been tested with every single data factory resource. It should support all activities out-of-the-box but has not been thoroughly tested, please report any issues in the issues section and include an example of the pipeline that is not working as expected.
+This unit test framework is not officially supported.
+It is currently in an experimental state and has not been tested with every single data factory resource.
+It should support all activities out-of-the-box but has not been thoroughly tested,
+please report any issues in the issues section and include an example of the pipeline that is not working as expected.
 
-If there's a lot of interest in this framework, then we will continue to improve it and move it to a production-ready state. 
+If there's a lot of interest in this framework, then we will continue to improve it and move it to a production-ready state.
 
 ## Features
 
 Goal: Validate that the evaluated pipeline configuration with its expressions is behaving as expected on runtime.
 
 1. Evaluate expressions with their functions and arguments instantly by using the framework's internal expression parser.
-2. Test a pipeline or activity against any state to assert the expected outcome. A state can be configured with pipeline parameters, global parameters, variables and activity outputs.
+2. Test a pipeline or activity against any state to assert the expected outcome.
+   A state can be configured with pipeline parameters, global parameters, variables and activity outputs.
 3. Simulate a pipeline run and evaluate the execution flow and outcome of each activity.
 4. Dynamically supports all activity types with all their attributes.
 
-> Pipelines and activities are not executed on any Data Factory environment, but the evaluation of the pipeline configuration is validated locally. This is different from the "validation" functionality present in the UI, which only validates the syntax of the pipeline configuration.
-
+> Pipelines and activities are not executed on any Data Factory environment,
+> but the evaluation of the pipeline configuration is validated locally.
+> This is different from the "validation" functionality present in the UI,
+> which only validates the syntax of the pipeline configuration.
 
 ## Why
 
-Data Factory does not support unit testing out of the box. The only way to validate your changes is through manual testing or running e2e tests against a deployed data factory. These tests are great to have, but miss the following benefits that unit tests, like using this unit test framework, provide:
+Data Factory does not support unit testing out of the box.
+The only way to validate your changes is through manual testing or running e2e tests against a deployed data factory.
+These tests are great to have, but miss the following benefits that unit tests, like using this unit test framework, provide:
 
-* Shift left with immediate feedback on changes - Evaluate any individual data factory resource (pipelines, activities, triggers, datasets, linked services etc..), including (complex) expressions
+* Shift left with immediate feedback on changes - Evaluate any individual data factory resource
+  (pipelines, activities, triggers, datasets, linked services etc..), including (complex) expressions
 * Allows testing individual resources (e.g. activity) for many different input values to cover more scenarios.
-* Less issues in production - due to the fast nature of writing and running unit tests, you will write more tests in less time and therefore have a higher test coverage. This means more confidence in new changes, fewer risks in breaking existing features (regression tests), and thus far fewer issues in production.
+* Less issues in production - due to the fast nature of writing and running unit tests,
+  you will write more tests in less time and therefore have a higher test coverage.
+  This means more confidence in new changes, fewer risks in breaking existing features (regression tests),
+  and thus far fewer issues in production.
 
-> Even though Data Factory is UI-driven writing unit tests, and might not be in the nature of it. How can you be confident that your changes will work as expected, and that existing pipelines will not break, without writing unit tests?
+> Even though Data Factory is UI-driven writing unit tests, and might not be in the nature of it.
+> How can you be confident that your changes will work as expected,
+> and that existing pipelines will not break, without writing unit tests?
 
 ## Getting started
 
 1. Set up an empty Python project with your favorite testing library
-   
-   More information: 
+
+   More information:
    [docs_Setup](/docs/environment_setup/unit_test_setup.md)
-   
+
 2. Install the package using your preferred package manager:
-   
+
     Pip: `pip install data-factory-testing-framework`
-   
+
 3. Create a Folder in your project and copy the JSON Files with the pipeline definitions locally.
-   
-   More information: 
+
+   More information:
    [docs Json](/docs/environment_setup/json_pipeline_files.md)
-  
+
 4. Start writing tests
 
 ## Features - Examples
 
-The samples seen below are the _only_ code that you need to write! The framework will take care of the rest. 
+The samples seen below are the _only_ code that you need to write! The framework will take care of the rest.
 
 1. Evaluate activities (e.g. a WebActivity that calls Azure Batch API)
 
@@ -86,53 +102,53 @@ The samples seen below are the _only_ code that you need to write! The framework
     assert "Job-123" == body["JobName"]
     assert "version1" == body["Version"]
    ```
-
-   2. Evaluate Pipelines and test the flow of activities given a specific input
-
-       ```python
-       # Arrange
-       pipeline: PipelineResource = test_framework.repository.get_pipeline_by_name("batch_job")
-
-       # Runs the pipeline with the provided parameters
-       activities = test_framework.evaluate_pipeline(pipeline, [
-           RunParameter(RunParameterType.Pipeline, "JobId", "123"),
-           RunParameter(RunParameterType.Pipeline, "ContainerName", "test-container"),
-           RunParameter(RunParameterType.Global, "BaseUrl", "https://example.com"),
-       ])
-
-       set_variable_activity: Activity = next(activities)
-       assert set_variable_activity is not None
-       assert "Set JobName" == set_variable_activity.name
-       assert "JobName" == activity.type_properties["variableName"]
-       assert "Job-123" == activity.type_properties["value"].value
-
-       get_version_activity = next(activities)
-       assert get_version_activity is not None
-       assert "Get version" == get_version_activity.name
-       assert "https://example.com/version" == get_version_activity.type_properties["url"].value
-       assert "GET" == get_version_activity.type_properties["method"]
-       get_version_activity.set_result(DependencyCondition.Succeeded,{"Version": "version1"})
-
-       create_batch_activity = next(activities)
-       assert create_batch_activity is not None
-       assert "Trigger Azure Batch Job" == create_batch_activity.name
-       assert "https://example.com/jobs" == create_batch_activity.type_properties["url"].value
-       assert "POST" == create_batch_activity.type_properties["method"]
-       body = create_batch_activity.type_properties["body"].get_json_value()
-       assert "123" == body["JobId"]
-       assert "Job-123" == body["JobName"]
-       assert "version1" == body["Version"]
-
-       with pytest.raises(StopIteration):
-           next(activities)
-       ```
-
-> See the [Examples](/examples) folder for more samples 
 
+2. Evaluate Pipelines and test the flow of activities given a specific input
+
+    ```python
+    # Arrange
+    pipeline: PipelineResource = test_framework.repository.get_pipeline_by_name("batch_job")
+
+    # Runs the pipeline with the provided parameters
+    activities = test_framework.evaluate_pipeline(pipeline, [
+        RunParameter(RunParameterType.Pipeline, "JobId", "123"),
+        RunParameter(RunParameterType.Pipeline, "ContainerName", "test-container"),
+        RunParameter(RunParameterType.Global, "BaseUrl", "https://example.com"),
+    ])
+
+    set_variable_activity: Activity = next(activities)
+    assert set_variable_activity is not None
+    assert "Set JobName" == set_variable_activity.name
+    assert "JobName" == activity.type_properties["variableName"]
+    assert "Job-123" == activity.type_properties["value"].value
+
+    get_version_activity = next(activities)
+    assert get_version_activity is not None
+    assert "Get version" == get_version_activity.name
+    assert "https://example.com/version" == get_version_activity.type_properties["url"].value
+    assert "GET" == get_version_activity.type_properties["method"]
+    get_version_activity.set_result(DependencyCondition.Succeeded,{"Version": "version1"})
+
+    create_batch_activity = next(activities)
+    assert create_batch_activity is not None
+    assert "Trigger Azure Batch Job" == create_batch_activity.name
+    assert "https://example.com/jobs" == create_batch_activity.type_properties["url"].value
+    assert "POST" == create_batch_activity.type_properties["method"]
+    body = create_batch_activity.type_properties["body"].get_json_value()
+    assert "123" == body["JobId"]
+    assert "Job-123" == body["JobName"]
+    assert "version1" == body["Version"]
+
+    with pytest.raises(StopIteration):
+        next(activities)
+    ```
+
+> See the [Examples](/examples) folder for more samples
 
 ## Registering missing expression functions
 
-As the framework is interpreting expressions containing functions, these functions are implemented in the framework, but there may be bugs in some of them. You can override their implementation through:
+As the framework is interpreting expressions containing functions, these functions are implemented in the framework,
+but there may be bugs in some of them. You can override their implementation through:
 
 ```python
    FunctionsRepository.register("concat", lambda arguments: "".join(arguments))
@@ -141,7 +157,8 @@ As the framework is interpreting expressions containing functions, these functio
 
 ## Tips
 
-1. After parsing a data factory resource file, you can use the debugger to easily discover which classes are actually initialized so that you can cast them to the correct type.
+1. After parsing a data factory resource file, you can use the debugger to easily discover which classes are actually
+   initialized so that you can cast them to the correct type.
 
 ## Recommended development workflow for Azure Data Factory v2
 
@@ -160,7 +177,7 @@ As the framework is interpreting expressions containing functions, these functio
 
 This project welcomes contributions and suggestions.  Most contributions require you to agree to a
 Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
-the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
+the rights to use your contribution. For details, visit <https://cla.opensource.microsoft.com>.
 
 When you submit a pull request, a CLA bot will automatically determine whether you need to provide
 a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
@@ -177,7 +194,3 @@ trademarks or logos is subject to and must follow
 [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
 Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
 Any use of third-party trademarks or logos are subject to those third-party's policies.
-
-
-
-

SECURITY.md

@@ -14,7 +14,7 @@ Instead, please report them to the Microsoft Security Response Center (MSRC) at
 
 If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com).  If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey).
 
-You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc). 
+You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc).
 
 Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:
 

SUPPORT.md

@@ -1,13 +1,13 @@
 # Support
 
-## How to file issues and get help  
+## How to file issues and get help
 
-This project uses GitHub Issues to track bugs and feature requests. Please search the existing 
-issues before filing new issues to avoid duplicates.  For new issues, file your bug or 
+This project uses GitHub Issues to track bugs and feature requests. Please search the existing
+issues before filing new issues to avoid duplicates.  For new issues, file your bug or
 feature request as a new Issue.
 
 For help and questions about using this project, feel free ask your question via GitHub issues. In the future we will think about other channels.
 
-## Microsoft Support Policy  
+## Microsoft Support Policy
 
 Support for this **PROJECT or PRODUCT** is limited to the resources listed above.