feat(platform): Support manually setting up webhooks #8750

ntindle · 2024-11-24T09:41:06Z

Resolves Add Custom Webhook block #8748

The webhooks system as is works really well for full blown enterprise webhooks managed via a UI. It does not work for more "chill guy" webhook tools that just send notifications sometimes.

Changes 🏗️

Checklist 📋

For code changes:

I have clearly listed my changes in the PR description
I have made a test plan
I have tested my changes according to the test plan:
- ...

Example test plan

Create from scratch and execute an agent with at least 3 blocks
Import an agent from file upload, and confirm it executes correctly
Upload agent to marketplace
Import an agent from marketplace and confirm it executes correctly
Edit an agent from monitor, and confirm it executes correctly

For configuration changes:

.env.example is updated or already compatible with my changes
docker-compose.yml is updated or already compatible with my changes
I have included a list of my configuration changes in the PR description (under Changes)

Examples of configuration changes

Changing ports
Adding new services that need to communicate with each other
Secrets or environment variable changes
New or infrastructure changes such as databases

… hooks

…e hooks + other infra

…inier/open-1961-implement-github-on-pull-request-block

…h/Node | Graph/Node -> GraphModel/NodeModel

…quest-block

…inier/open-1961-implement-github-on-pull-request-block

…quest-block

Resolves #8357 - Add user flow to explicitly confirm deleting credentials linked to in-use webhooks - Add logic to deregister, remove, and unlink webhooks before deleting parent credentials - backend: Add `NeedConfirmation` exception - frontend: Add `AlertDialog` UI component

Resolves #8738 - Disable webhook blocks if `PLATFORM_BASE_URL` is not set - Throw `MissingConfigError` when trying to set up a node-webhook-link if `PLATFORM_BASE_URL` is not set - Add `MissingConfigError` - Add field validator to `Config.platform_base_url` and `Config.frontend_base_url`

ntindle · 2024-11-24T09:42:07Z

autogpt_platform/backend/backend/blocks/compass/triggers.py

+        credentials: CredentialsMetaInput[Literal["compass"], Literal["api_key"]] = (
+            CredentialsField(
+                provider="compass",
+                supported_credential_types={"api_key"},
+            )
+        )


I don't have any real concept of credentials that I need here. If we generate a secret on the platform side, that would be useful so we can ensure its a trusted source who sent the hook. Maybe we should store that in the credential?

For that we already have the Webhook.secret. This mechanism doesn't need credentials, so we should just omit the credentials field.

We'll have to amend this check in on_node_activate to only enforce credentials if there is an actual credentials field on block.input_schema:

AutoGPT/autogpt_platform/backend/backend/integrations/webhooks/graph_lifecycle_hooks.py

Lines 114 to 128 in 68f4aea

has_everything_for_webhook = (

resource is not None

and CREDENTIALS_FIELD_NAME in node.input_default

and event_filter_input_name in node.input_default

and any(is_on for is_on in node.input_default[event_filter_input_name].values())

)

if has_everything_for_webhook and resource:

logger.debug(f"Node #{node} has everything for a webhook!")

if not credentials:

credentials_meta = node.input_default[CREDENTIALS_FIELD_NAME]

raise ValueError(

f"Cannot set up webhook for node #{node.id}: "

f"credentials #{credentials_meta['id']} not available"

)

This is also one of the two places where the presence of an event filter is enforced.

Basically the only thing it should do for your case is create a Webhook and store it in the DB. The easy way is to just skip parts of the process and then display the webhook ingress URL and webhook secret on the block in the UI. The neat way would be to have a picker for the webhooks for a given provider, on blocks that can't auto-create their webhook (like GitHub).

ntindle · 2024-11-24T09:42:31Z

autogpt_platform/backend/backend/integrations/webhooks/__init__.py

@@ -1,5 +1,8 @@
 from typing import TYPE_CHECKING

+from backend.integrations.webhooks.simple_webhook_manager import CompassWebhookManager, SimpleWebhooksManager
+from pyparsing import C


auto imports strike again

what's next, the rest of the alphabet? :+)

ntindle · 2024-11-24T09:43:16Z

autogpt_platform/backend/backend/integrations/webhooks/simple_webhook_manager.py

+class SimpleWebhooksManager(BaseWebhooksManager):
+    PROVIDER_NAME = "simple_hook_manager"
+
+    DEFAULT_HEADERS = {"Accept": "application/json"}
+
+    async def _register_webhook(
+        self,
+        credentials: Credentials,
+        webhook_type: CompassWebhookType,
+        resource: str,
+        events: list[str],
+        ingress_url: str,
+        secret: str,
+    ) -> tuple[str, dict]:
+
+        webhook_id = uuid.uuid4()
+        config = {}
+
+        return str(webhook_id), config
+
+    async def _deregister_webhook(
+        self, webhook: integrations.Webhook, credentials: Credentials
+    ) -> None:
+        if webhook.credentials_id != credentials.id:
+            raise ValueError(
+                f"Webhook #{webhook.id} does not belong to credentials {credentials.id}"
+            )
+        # If we reach here, the webhook was successfully deleted or didn't exist
+


all i need is a url and data shape to make my example work. Not sure what is needed for that but this feels overkill, and somehow underkill

ntindle · 2024-11-24T09:44:36Z

Example of hte hooks I'm getting (ignore the get requests, that's me) https://play.svix.com/view/e_UHwNM5156Yqkb3v5LnoJTE8I0EM/2pHjEII1gqLCbg40krh0XlKvb1W

Pwuts · 2024-11-25T12:24:26Z

autogpt_platform/backend/backend/data/block.py

@@ -53,6 +53,7 @@ class BlockCategory(Enum):
    COMMUNICATION = "Block that interacts with communication platforms."
    DEVELOPER_TOOLS = "Developer tools such as GitHub blocks."
    DATA = "Block that interacts with structured data."
+    HARDWARE = "Block that interacts with hardware."


ROBOTICS?

Hardware is very very broad.

Pwuts · 2024-11-25T12:26:04Z

autogpt_platform/backend/backend/integrations/webhooks/simple_webhook_manager.py

+        webhook_id = uuid.uuid4()
+        config = {}
+
+        return str(webhook_id), config


We should avoid generating and storing fake data like this. If we can't programmatically register a webhook, _register_webhook either shouldn't be called or should return empty output. The Webhook.provider_webhook_id will simply be empty, because we don't and can't know it.

github-actions · 2024-11-25T17:44:57Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

github-actions · 2024-11-25T18:19:13Z

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

netlify · 2024-11-25T18:20:17Z

✅ Deploy Preview for auto-gpt-docs ready!

Name	Link
🔨 Latest commit	`23ab904`
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs/deploys/6744bf954a6dbc00086d022c
😎 Deploy Preview	https://deploy-preview-8750--auto-gpt-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

ntindle · 2024-11-26T04:59:58Z

This does not resolve #8748. I was narrowly targeting a time window of free time I have near Thanksgiving. That ship sailed unfortunately so closing PR

Pwuts and others added 30 commits October 17, 2024 00:11

Add Graph.on_update, Block.on_node_update, Block.on_node_delete…

857ae69

… hooks

Add credentials injection to Graph.on_update etc hooks

2f405c3

Add webhooks and their links to nodes to DB schema

921f614

update webhooks db schema

946b35e

Introduce PLATFORM_BASE_URL config attribute

dfe36c1

Add WebhooksManager base and GitHub implementation + graph lifecycl…

321ef9d

…e hooks + other infra

Add webhook ingress endpoint

f4ce0f4

fix(blocks): Allow having an input and output pin with the same name

40e846e

fix bootstrapping issues

6dc9e0c

hide input pins on webhook blocks

ab5b336

dedup generateInputHandles

41ccae7

refactor(backend): Make RedisEventQueue generic

a82acc7

apply event type filter in webhook ingress endpoint

36b87eb

add webhook ping endpoint

13e6a75

hide payload input

f117d3f

fix graph creation

c604d27

fix enum usage

34d82bf

smol cleanup

98aed6d

fix webhook resource string formatting

9f37671

add debug stuff for webhooks

e5c95f6

fix webhook ingress URL

5e995f9

fix github events

aa12a5a

Merge branch 'dev' of github.com:Significant-Gravitas/AutoGPT into re…

707df04

…inier/open-1961-implement-github-on-pull-request-block

[Reduce code change size] Revert CreatableGraph/CreatableNode -> Grap…

5158b87

…h/Node | Graph/Node -> GraphModel/NodeModel

[Reduce code change size] Take Redis generic refactor out of the PR

5e70973

Merge branch 'dev' into reinier/open-1961-implement-github-on-pull-re…

6885fa8

…quest-block

Merge branch 'dev' of github.com:Significant-Gravitas/AutoGPT into re…

2734d83

…inier/open-1961-implement-github-on-pull-request-block

Fix frontend_base_url / platform_base_url clash

19b38fb

Set default platform_base_url & fix agent output block bug

1a23591

Fix CORS issue, make ValueError 400, toast saveAgent HTTP error

0343402

Pwuts and others added 8 commits November 21, 2024 19:00

Rename sender output to triggered_by_user

4c53eb3

Update description of triggered_by_user

7c0b336

Fix description of payload output to be generic

0f7fa4b

Add pull_request_url output

2dcbcfe

Merge branch 'dev' into reinier/open-1961-implement-github-on-pull-re…

397ae0b

…quest-block

feat: wip eod commit

0ac0ac9

github-actions bot added platform/backend AutoGPT Platform - Back end platform/blocks size/l labels Nov 24, 2024

ntindle commented Nov 24, 2024

View reviewed changes

ntindle changed the title ~~feat: wip eod commit~~ ref: webhooks simplification Nov 24, 2024

Update triggers.py

463c8a6

ntindle mentioned this pull request Nov 24, 2024

feat(platform, blocks): Webhook-triggered blocks #8358

Merged

14 tasks

Pwuts reviewed Nov 25, 2024

View reviewed changes

Base automatically changed from reinier/open-1961-implement-github-on-pull-request-block to dev November 25, 2024 17:42

github-actions bot added the conflicts Automatically applied to PRs with merge conflicts label Nov 25, 2024

Merge branch 'dev' into exploration/other-types-of-hooks

23ab904

github-actions bot removed the conflicts Automatically applied to PRs with merge conflicts label Nov 25, 2024

Pwuts changed the title ~~ref: webhooks simplification~~ feat(platform): Support manually setting up webhooks Nov 25, 2024

ntindle closed this Nov 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(platform): Support manually setting up webhooks #8750

feat(platform): Support manually setting up webhooks #8750

ntindle commented Nov 24, 2024 •

edited by Pwuts

Loading

ntindle Nov 24, 2024

Pwuts Nov 25, 2024

Pwuts Nov 25, 2024 •

edited

Loading

Pwuts Nov 25, 2024 •

edited

Loading

ntindle Nov 24, 2024

Pwuts Nov 25, 2024

ntindle Nov 24, 2024

ntindle commented Nov 24, 2024

Pwuts Nov 25, 2024

Pwuts Nov 25, 2024 •

edited

Loading

github-actions bot commented Nov 25, 2024

github-actions bot commented Nov 25, 2024

netlify bot commented Nov 25, 2024

ntindle commented Nov 26, 2024

	has_everything_for_webhook = (
	resource is not None
	and CREDENTIALS_FIELD_NAME in node.input_default
	and event_filter_input_name in node.input_default
	and any(is_on for is_on in node.input_default[event_filter_input_name].values())
	)

	if has_everything_for_webhook and resource:
	logger.debug(f"Node #{node} has everything for a webhook!")
	if not credentials:
	credentials_meta = node.input_default[CREDENTIALS_FIELD_NAME]
	raise ValueError(
	f"Cannot set up webhook for node #{node.id}: "
	f"credentials #{credentials_meta['id']} not available"
	)

feat(platform): Support manually setting up webhooks #8750

feat(platform): Support manually setting up webhooks #8750

Conversation

ntindle commented Nov 24, 2024 • edited by Pwuts Loading

Changes 🏗️

Checklist 📋

For code changes:

For configuration changes:

ntindle Nov 24, 2024

Choose a reason for hiding this comment

Pwuts Nov 25, 2024

Choose a reason for hiding this comment

Pwuts Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

Pwuts Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

ntindle Nov 24, 2024

Choose a reason for hiding this comment

Pwuts Nov 25, 2024

Choose a reason for hiding this comment

ntindle Nov 24, 2024

Choose a reason for hiding this comment

ntindle commented Nov 24, 2024

Pwuts Nov 25, 2024

Choose a reason for hiding this comment

Pwuts Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Nov 25, 2024

github-actions bot commented Nov 25, 2024

netlify bot commented Nov 25, 2024

✅ Deploy Preview for auto-gpt-docs ready!

ntindle commented Nov 26, 2024

ntindle commented Nov 24, 2024 •

edited by Pwuts

Loading

Pwuts Nov 25, 2024 •

edited

Loading

Pwuts Nov 25, 2024 •

edited

Loading

Pwuts Nov 25, 2024 •

edited

Loading