Add support API endpoints as NotificationSources

[chadell]

Introduction

The main goal of this plugin is to be able to handle automatically circuit maintenance notifications, including:

• notification data collection/fetching
• notification data normalisation (delegated to the circuit-maintenance-parser library)
• create/update circuit maintenance and relationships

From the beginning, we envisioned that notification data could come via different channels, for instance, receiving emails (traditional method) or via API (via active pulling or reacting to webhooks). Initially, the focused only in the first use case (email) because it is still the most common use-case.

Goal: Add support API endpoints as NotificationSources

In this discussion we want to provide a HLD about the different options we have to implement this feature, considering the pros and cons.

Notification data collection/fetching

The current design of NotificationSource model it's quite open in the database, and depends on a more specific implementation as Source that take the initalisation data directly from the plugin configuration.

This loose implementation helps to define any kind of integration schema, accepting any attribute you may need, and deciding which type of integration based on the url (and other specific details in each case). Today, the plugin supports 3 types of Sources: IMAP, GmailAPIOauth and GmailAPIOauth, but all these classes inherit from a [EmailSource](https://github.com/nautobot/nautobot-plugin-circuit-maintenance/blob/develop/nautobot_circuit_maintenance/handle_notifications/sources.py#L185) class.

In order to evolve this design we will need to create a new Source class ready to interact with external API endpoints, adding a field in NotificationSource and Source about the type, that could help to be more specific during the initialisation of the Source, instead of deducing it from the url as commented above.

Also, with the recently added support in Nautobot for Secrets, we could consider adding support to handle NotificationSource directly from Nautobot. Today, the key configuration from a NotificationSource can't be updated in Nautobot (UI/API) because we decided to not store secrets in the database. With the new backends available in the Secrets Provider Plugin we could revisit it and support both ways.

One way or another, with the new API source, we will be able to get the information from API endpoints, most likely JSON data. But, this data will have a different schema from one provider to another, so we have to move to the next discussion point: data normalisation.

Notification data normalisation

One of the first design decisions of this plugin was to externalise the normalisation data to another library, so the global community could benefit from the parser library even they are not using the Nautobot plugin, so increase the odd of biggest impact.

This parser library serves 2 purposes:

• Define what a standard maintenance notification looks like, following some recommendations.
• Define a simple entry point to parse a circuit maintenance notification and convert it to the previous object

Today, because this library is mostly driven by Networktocode, the parser library is only focused on parsing notifications from email, but it could be extended to support other data normalisation such as a custom JSON to the reference maintenance object.

The key question here is to decide, from the Circuit Maintenance plugin perspective, where it makes more sense to implement the data normalisation required for processing the new data gathered from API sources, that even doesn't require the same level of parsing than an email, still needs to convert between every provider schema to the expect object.

Option 1: Implement the API data normalisation in the Plugin

• It could help to quicker development of API integration due being in the same project

Option 2: Implement the API data normalisation in the Parser

• Data normalisation happens only in one place, it is not split in between the plugin (for API data) and the parser (for email data), less complexity and inconsistencies
• Data normalisation for API is still a problem for the whole community, not only Nautobot one, so we benefit from others implementations

[chadell]

Please @pke11y @glennmatthews @dgarros share your points about it

[pke11y]

@chadell I believe developing an API integration for the circuit providers would be a huge benefit to the plugin. Getting accurate email data and the various iterations and formatting styles can be a barrier to extending the support for existing and new providers. I think it would make it easier for others to contribute too.

I like the idea of keeping the normalisation per provider in the parser library (option 2), architecturally it makes sense. However, I'm not sure the use case for parsing emails outside of the plugin is that strong. I'd be interested to hear if someone is actually using it in that way. Perhaps API parsing might be more useful.

I'm not sure how many providers have API services. It would be great if there was some way to leverage public/private cloud API's to get the data too.

[chadell]

Thanks @pke11y for your feedback.
Currently, I have no notice about the library being used by other projects other than Nautobot, mostly because this would not be public repositories. I imagine people using it on internal scripts/apps that work closely with its custom logic and not making them public. However, the repository has 27 stars from people outside of NTC and we receive some issue on the past, so I would like to think that someone is actually reusing it. Another idea could be to launch a poll on the public slack to get people interested in.

For providers offering an API, I would say that all the Cloud Service Provider, for instance, AWS, that we could take as an example for the implementation.

[scetron]

I think that in this case it makes more sense to implement the functionality as part of the parser in option 2.

Organizationally, it keeps all of the ability to parse something in one place whether from email, api, or other source. It makes the parser totally usable in some other project and gives more value to the parsers in addition to the reasons that you listed.

A point against putting it in the plugin is that there seems to be limited use for this data/capability outside of doing circuit maintenance, at least that I am aware of, so in a sense, something that is involved in parsing shouldn't be split out into two separate places, unless the utility of putting it in the plugin is for some specific purpose other than more quickly developing it.