Merge pull request #551 from nautobot/develop

Merge 1.4.2 into main

development/Dockerfile

@@ -26,6 +26,7 @@ ENV NAUTOBOT_ROOT ${NAUTOBOT_ROOT}
 # and CI and local development may have a newer version of Poetry
 # Since this is only used for development and we don't ship this container, pinning Poetry back is not expressly necessary
 # We also don't need virtual environments in container
+ARG POETRY_VERSION=1.5.1
 RUN curl -sSL https://install.python-poetry.org -o /tmp/install-poetry.py && \
     python /tmp/install-poetry.py && \
     rm -f /tmp/install-poetry.py && \

docs/admin/admin_install.md

@@ -28,7 +28,7 @@ echo nautobot-golden-config >> local_requirements.txt
 
 Once installed, the plugin needs to be enabled in your Nautobot configuration. The following block of code below shows the additional configuration required to be added to your `nautobot_config.py` file:
 
-- Append `"nautobot_golden_config"` to the `PLUGINS` list, and `"nautobot_plugin_nornir"` if it was not already there (more info [here](https://github.com/nautobot/nautobot-plugin-nornir)).
+- Append `"nautobot_golden_config"` to the `PLUGINS` list, and `"nautobot_plugin_nornir"` if it was not already there (more info [here](https://docs.nautobot.com/projects/plugin-nornir/en/latest/)).
 - Append the `"nautobot_golden_config"` dictionary to the `PLUGINS_CONFIG` dictionary, and `"nautobot_plugin_nornir"` if it was not already there.
 
 ```python
@@ -64,11 +64,6 @@ PLUGINS_CONFIG = {
 }
 ```
 
-The following block of code below shows the additional configuration required to be added to your `nautobot_config.py` file:
-
-- append `"nautobot_golden_config"` to the `PLUGINS` list, and `"nautobot_plugin_nornir"` if it was not already there (more info [here](https://github.com/nautobot/nautobot-plugin-nornir)).
-- append the `"nautobot_golden_config"` dictionary to the `PLUGINS_CONFIG` dictionary, and `"nautobot_plugin_nornir"` if it was not already there.
-
 Once the Nautobot configuration is updated, run the Post Upgrade command (`nautobot-server post_upgrade`) to run migrations and clear any cache.
 
 ```shell
@@ -101,7 +96,7 @@ The plugin behavior can be controlled with the following list of settings.
 | enable_postprocessing     | True                          | False    | A boolean to represent whether or not to generate intended configurations to push, with extra processing such as secrets rendering.                                        |
 | postprocessing_callables  | ['mypackage.myfunction']      | []      | A list of function paths, in dotted format, that are appended to the available methods for post-processing the intended configuration, for instance, the `render_secrets`. |
 | postprocessing_subscribed | ['mypackage.myfunction']      | []      | A list of function paths, that should exist as postprocessing_callables, that defines the order of application of during the post-processing process.                      |
-| platform_slug_map         | {"cisco_wlc": "cisco_aireos"} | None    | A dictionary in which the key is the platform slug and the value is what netutils uses in any "network_os" parameter.                                                      |
+| platform_slug_map         | {"cisco_wlc": "cisco_aireos"} | None    | A dictionary in which the key is the platform slug and the value is what netutils uses in any "network_os" parameter within `netutils.config.compliance.parser_map`.                                                      |
 | sot_agg_transposer        | "mypkg.transposer"            | None    | A string representation of a function that can post-process the graphQL data.                                                                                              |
 | per_feature_bar_width     | 0.15                          | 0.15    | The width of the table bar within the overview report                                                                                                                      |
 | per_feature_width         | 13                            | 13      | The width in inches that the overview table can be.                                                                                                                        |
@@ -111,4 +106,4 @@ The plugin behavior can be controlled with the following list of settings.
     Over time the compliance report will become more dynamic, but for now allow users to configure the `per_*` configs in a way that fits best for them.
 
 !!! note
-    Review [`nautobot_plugin_nornir`](https://pypi.org/project/nautobot-plugin-nornir/) for Nornir and dispatcher configuration options.
+    Review [`nautobot_plugin_nornir`](https://docs.nautobot.com/projects/plugin-nornir/en/latest/user/app_feature_dispatcher/) for Nornir and dispatcher configuration options.

docs/admin/admin_upgrade.md

@@ -6,8 +6,8 @@ When a new release comes out it may be necessary to run a migration of the datab
 
 ## v1.0.0
 
-`v1.0.0` Provides a breaking change for the users running pre `1.0.0` code sourced from `develop` branch of the plugin. Only users of the `Backup Repository Matching Rule` and `Intended Repository Matching Rule` features are affected by following behaviour: migration script will only migrate the first repository from the list into the new default settings.
+`v1.0.0` Provides a breaking change for the users running pre `1.0.0` code sourced from `develop` branch of the plugin. Only users of the `Backup Repository Matching Rule` and `Intended Repository Matching Rule` features are affected by following behavior: migration script will only migrate the first repository from the list into the new default settings.
 
-Because of this specific behaviour, please review your configuration and capture it before attempting to upgrade if using above features.
+Because of this specific behavior, please review your configuration and capture it before attempting to upgrade if using above features.
 
-Users running the released packages are not affected by this behaviour.
+Users running the released packages are not affected by this behavior.

docs/admin/release_notes/version_1.4.md

@@ -2,6 +2,23 @@
 
 - Change min version of Nautobot from 1.4.0 to 1.5.3 which is required for the use of NautobotUIViewset, Notes mixins etc.
 
+## v1.4.2 - 2023-08
+
+### Changed
+
+- [519](https://github.com/nautobot/nautobot-plugin-golden-config/pull/519) - docs-only: large fixes and template troubleshooting section.
+
+### Fixed
+
+- [492](https://github.com/nautobot/nautobot-plugin-golden-config/pull/492) - Fix count of in scope devices on settings detail view.
+- [498](https://github.com/nautobot/nautobot-plugin-golden-config/pull/498) - Fix deepdiff dependency.
+- [501](https://github.com/nautobot/nautobot-plugin-golden-config/pull/501) - Update docs for adding CustomField data with datasources.
+- [503](https://github.com/nautobot/nautobot-plugin-golden-config/pull/503) - Switch from deprecated FilterSet to new FilterSetMixin.
+- [504](https://github.com/nautobot/nautobot-plugin-golden-config/pull/504) - Fix extend queryfilter to export.
+- [511](https://github.com/nautobot/nautobot-plugin-golden-config/pull/511) - Fix `log_failure` function missing argument.
+- [523](https://github.com/nautobot/nautobot-plugin-golden-config/pull/523) - Fix docs site by pinning dev dependencies.
+- [530](https://github.com/nautobot/nautobot-plugin-golden-config/pull/530) - Fix, removing ConfigCompliance model import from 0005 migration.
+
 ## v1.4.1 - 2023-05
 
 ### Fixed

docs/dev/dev_adr.md

@@ -134,7 +134,7 @@ The API view, under the path `config-postprocessing`, uses custom permissions, n
 
 ### Renders Secrets
 
-It was decided to restrict the usage of Jinja filters to only the ones related to getting Nautobot secrets values (defined here), plus the `encrypt_type5` and `encrypt_type7` filters from Netutils. Remember that this function is not defined to replace the regular Jinja rendering done for creating the Intended configuration, only to add secrets information on the fly. This avoids undesired behavior on this synchronous operation.
+It was decided to restrict the usage of Jinja filters to only the ones related to getting Nautobot secrets values (defined here), plus the `encrypt_<vendor>_type5` and `encrypt__<vendor>_type7` filters from [Netutils](https://netutils.readthedocs.io/en/latest/dev/code_reference/password/#netutils.password). Remember that this function is not defined to replace the regular Jinja rendering done for creating the Intended configuration, only to add secrets information on the fly. This avoids undesired behavior on this synchronous operation.
 
 This function performs an additional permission validation, to check if the requesting user has permissions to view the `SecretsGroup` requested.
 

docs/dev/dev_environment.md

@@ -4,7 +4,7 @@
 
 The development environment can be used in two ways:
 
-1. `Recommended` All services are spun up using Docker and a local mount so you can develop locally, but Nautobot is spun up within the Docker container.
+1. `Recommended` All services are spun up using Docker and a local volume mount attached so you can develop locally, but Nautobot is spun up within the Docker container.
 2. With a local poetry environment if you wish to develop outside of Docker with the caveat of using external services provided by Docker for PostgresQL and Redis.
 
 This is a quick reference guide if you're already familiar with the development environment provided, which you can read more about later in this document.
@@ -321,6 +321,9 @@ When trying to debug an issue, one helpful thing you can look at are the logs wi
 !!! note
 	The `-f` tag will keep the logs open, and output them in realtime as they are generated.
 
+!!! info
+    Want to limit the log output even further? Use the `--tail <#>` command line argument in conjunction with `-f`.
+
 So for example, our plugin is named `nautobot-golden-config`, the command would most likely be `docker logs nautobot_golden_config_nautobot_1 -f`. You can find the name of all running containers via `docker ps`.
 
 If you want to view the logs specific to the worker container, simply use the name of that container instead.

docs/requirements.txt

@@ -1,5 +1,6 @@
+griffe==0.30.1
 mkdocs==1.3.1
 mkdocs-material==8.4.2
 mkdocs-version-annotations==1.0.0
-mkdocstrings==0.19
-mkdocstrings-python==0.7.1
+mkdocstrings==0.22.0
+mkdocstrings-python==1.1.2

docs/user/app_faq.md

@@ -43,19 +43,26 @@ Understanding that there will never be consensus on what should go into a featur
 The current supported platform and the associated *default* platform slug names are the following for:
 
 * arista_eos
+* aruba_aoscx
 * bigip_f5
 * cisco_aireos
 * cisco_asa
 * cisco_ios
+* cisco_ios_xr
+* citrix_netscaler
 * cisco_nxos
+* extreme_netiron
 * fortinet_fortios
 * juniper_junos
 * linux
+* mikrotik_routeros
+* mrv_optiswitch
 * nokia_sros
+* paloalto_panos
 
 The expected "network_os" parameter must be as defined by netutils and golden config uses the platform slug to map from the device to the appropriate "network_os" that netutils expects. However, there an ability to map the actual platform slug for compliance and parsing tasks via the plugin settings in your "nautobot_config.py", and documented on the primary Readme.
 
-To provide a concrete example of this, note the following example that demonstrates how you can transpose any platform slug name to the expected one, as well as map multiple keys to a single netutils expected key.
+To provide a concrete example of this, note the following example that demonstrates how you can transpose any platform slug name to the expected one, as well as map multiple keys to a single netutils expected key. The `platform_slug_map` is only used for configuration compliance job. The json key is the Nautobot platform slug, and the json value is the "network_os" parameter defined in `netutils.config.compliance.parser_map`.
 ```json
 {
     "platform_slug_map":  {
@@ -73,9 +80,13 @@ The current supported platform and the associated *default* platform slug names
 * arista_eos
 * cisco_asa
 * cisco_ios
-* cisco_xr
+* cisco_ios_xr
 * cisco_nxos
 * juniper_junos
+* mikrotik_routeros
+* mikrotik_routeros_api
+* ruckus_fastiron
+* ruckus_smartzone_api
 
 In many use cases, this can be extended with a custom dispatcher for nornir tasks, which is controlled in the [nornir-nautobot](https://github.com/nautobot/nornir-nautobot) repository. Additionally the [`nautobot_plugin_nornir`](https://pypi.org/project/nautobot-plugin-nornir/) provide the ability to leverage the `dispatcher_mapping` configuration parameter to map and/or extend for your environment. Please see the instructions there for further details.
 
@@ -131,4 +142,4 @@ These errors have been accurate so far, that is not to say that there is no way
 
 * Incorrectly configured Secrets
 * Filtering to nothing when presumption is the filter works a certain way
-* Referencing an OS that is not recognized
+* Referencing an OS that is not recognized

docs/user/app_feature_backup.md

@@ -46,9 +46,17 @@ The `backup_path_template` can be set in the UI.  For navigation details [see](.
 
 ### Device Login Credentials
 
-The credentials/secrets management is further described within the [nautbot-plugin-nornir](https://github.com/nautobot/nautobot-plugin-nornir)
-repository. For the simplest use case you can set environment variables for `NAPALM_USERNAME`, `NAPALM_PASSWORD`, and `DEVICE_SECRET`. For more
-complicated use cases, please refer to the plugin documentation linked above.
+The credentials/secrets management occurs within the [nautobot-plugin-nornir](https://github.com/nautobot/nautobot-plugin-nornir) library and is described in the [Navigating Credentials](https://docs.nautobot.com/projects/plugin-nornir/en/latest/user/app_feature_credentials/) documentation. For the simplest use case you can set environment variables for `NAPALM_USERNAME`, `NAPALM_PASSWORD`, and `DEVICE_SECRET` in conjunction with the `credentials` string shown below in your configuration for `nautobot-plugin-nornir`.
+
+```python
+PLUGINS_CONFIG = {
+    "nautobot_plugin_nornir": {
+        "nornir_settings": {
+           "credentials": "nautobot_plugin_nornir.plugins.credentials.env_vars.CredentialsEnvVars"
+        },
+    }
+}
+```
 
 ## Starting a Backup Job
 

docs/user/app_feature_compliance.md

@@ -58,7 +58,7 @@ The "Configs to Match" section represents the configuration root elements. This
 what a line starts with only. Meaning, there is an implicit greediness to the matching. All matches must start form the beginning of the line.
 
 !!! note
-    If accidentally the data is "corrupted" with a bad tested match, simply delete the devices an re-run the compliance process.
+    If the data is accidentally "corrupted" with a bad tested match, simply delete the devices an re-run the compliance process.
 
 !!! note
     The mapping of "network_os" as defined by netutils is provided via the plugin settings in your nautobot_config.py, and documented on the primary Readme.
@@ -85,7 +85,7 @@ Please note the following about the compliance details page.
 
 ## Supported Platforms
 
-Platforms support technically come from the options provided by [nornir-nautobot](https://github.com/nautobot/nornir-nautobot) for nornir dispatcher tasks and
+Platforms support technically come from the options provided by [nornir-nautobot](https://github.com/nautobot/nornir-nautobot) for Nornir dispatcher tasks and
 [netutils](https://github.com/networktocode/netutils) for configuration compliance and parsing. However, for reference, the valid slug's of the platforms are
 provided in the [FAQ](./app_faq).
 

docs/user/app_feature_config_postprocessing.md

@@ -9,7 +9,7 @@ Challenging use cases when using the running configuration as intended:
 
 - Because the intended configuration is stored in the database, and in an external Git repository, it should **not** contain any secret.
 - The format of the running configuration is not always the same as the configuration to push, examples include:
-    - Pushing snmpv3 configurations, which do not show up in the running config
+    - Pushing SNMPv3 configurations, which do not show up in the running config
     - VTP configurations where the configurations is not in the running config at all
     - Implicit configurations like a "no shutdown" on an interface
 - The configurations used to get the configuration to the intended state may require to be ordered to not cause an outage.
@@ -41,7 +41,7 @@ The `render_secrets` function performs an extra Jinja rendering on top of an int
 - `get_secret_by_secret_group_slug`: as the name suggests, it returns the secret_group value, for a secret type, from its `slug`.
 
 !!! note
-    Other default Django or Netutils filters are not available in this Jinja environment. Only `encrypt_type5` and `encrypt_type7` can be used together with the `get_secret` filters.
+    Other default Django or Netutils filters are not available in this Jinja environment. Only `encrypt_<vendor>_type5` and `encrypt_<vendor>_type7` can be used together with the `get_secret` filters.
 
 Because this rendering is separated from the standard generation of the intended configuration, you must use the `{% raw %}` Jinja syntax to avoid being processed by the initial generation stage.