Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(platform): Support manually setting up webhooks #8750

Closed
wants to merge 87 commits into from
Closed
Show file tree
Hide file tree
Changes from 85 commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
857ae69
Add `Graph.on_update`, `Block.on_node_update`, `Block.on_node_delete`…
Pwuts Oct 16, 2024
2f405c3
Add credentials injection to `Graph.on_update` etc hooks
Pwuts Oct 17, 2024
921f614
Add webhooks and their links to nodes to DB schema
Pwuts Oct 17, 2024
946b35e
update webhooks db schema
Pwuts Oct 17, 2024
dfe36c1
Introduce `PLATFORM_BASE_URL` config attribute
Pwuts Oct 17, 2024
321ef9d
Add `WebhooksManager` base and GitHub implementation + graph lifecycl…
Pwuts Oct 17, 2024
f4ce0f4
Add webhook ingress endpoint
Pwuts Oct 17, 2024
40e846e
fix(blocks): Allow having an input and output pin with the same name
Pwuts Oct 19, 2024
6dc9e0c
fix bootstrapping issues
Pwuts Oct 19, 2024
ab5b336
hide input pins on webhook blocks
Pwuts Oct 20, 2024
41ccae7
dedup `generateInputHandles`
Pwuts Oct 20, 2024
a82acc7
refactor(backend): Make `RedisEventQueue` generic
Pwuts Oct 20, 2024
36b87eb
apply event type filter in webhook ingress endpoint
Pwuts Oct 21, 2024
13e6a75
add webhook ping endpoint
Pwuts Oct 21, 2024
f117d3f
hide `payload` input
Pwuts Oct 21, 2024
c604d27
fix graph creation
Pwuts Oct 21, 2024
34d82bf
fix enum usage
Pwuts Oct 21, 2024
98aed6d
smol cleanup
Pwuts Oct 21, 2024
9f37671
fix webhook resource string formatting
Pwuts Oct 21, 2024
e5c95f6
add debug stuff for webhooks
Pwuts Oct 21, 2024
5e995f9
fix webhook ingress URL
Pwuts Oct 21, 2024
aa12a5a
fix github events
Pwuts Oct 21, 2024
707df04
Merge branch 'dev' of github.com:Significant-Gravitas/AutoGPT into re…
majdyz Oct 24, 2024
5158b87
[Reduce code change size] Revert CreatableGraph/CreatableNode -> Grap…
majdyz Oct 24, 2024
5e70973
[Reduce code change size] Take Redis generic refactor out of the PR
majdyz Oct 24, 2024
6885fa8
Merge branch 'dev' into reinier/open-1961-implement-github-on-pull-re…
majdyz Oct 24, 2024
2734d83
Merge branch 'dev' of github.com:Significant-Gravitas/AutoGPT into re…
majdyz Oct 25, 2024
19b38fb
Fix frontend_base_url / platform_base_url clash
majdyz Oct 25, 2024
1a23591
Set default platform_base_url & fix agent output block bug
majdyz Oct 25, 2024
0343402
Fix CORS issue, make ValueError 400, toast saveAgent HTTP error
majdyz Oct 25, 2024
90d1fc7
Propagate clean error message from github error to user
majdyz Oct 26, 2024
dfeecf4
Propagate clean error message from github error to user
majdyz Oct 26, 2024
089b236
Merge branch 'dev' into reinier/open-1961-implement-github-on-pull-re…
majdyz Oct 27, 2024
1ebadb7
Merge branch 'dev' of github.com:Significant-Gravitas/AutoGPT into re…
majdyz Oct 27, 2024
35f044d
Lint
majdyz Oct 27, 2024
3e14f45
Merge remote-tracking branch 'origin/reinier/open-1961-implement-gith…
majdyz Oct 27, 2024
94d7567
Refactor to use new RedisEventBus
majdyz Oct 27, 2024
e9d1b1b
Skip webhook activation on no events
majdyz Oct 28, 2024
4b8cd25
Fix webhook changes breakage
majdyz Oct 28, 2024
2b45c28
Remove already done note
majdyz Oct 28, 2024
843d1cb
Unify channel name format
majdyz Oct 28, 2024
c357ea7
undo removing subgraph stuff
Pwuts Oct 28, 2024
57b532d
fix Node <-> Webhook reference and NodeModel usages
Pwuts Oct 29, 2024
9e0695f
smol clean
Pwuts Oct 29, 2024
02e952c
remove user auth middleware from webhook ingress endpoint
Pwuts Oct 29, 2024
d818af6
fix `add_execution` webhook mechanism
Pwuts Oct 29, 2024
4c68a1c
fix GitHub PR trigger block propagating output from base implementation
Pwuts Oct 29, 2024
e7dc870
move event output from base to PR trigger block
Pwuts Oct 30, 2024
5c82edc
Merge branch 'dev' into reinier/open-1961-implement-github-on-pull-re…
Pwuts Nov 4, 2024
dd446a7
change event filter input to a MultiSelect
Pwuts Nov 5, 2024
a525ce6
Merge branch 'dev' into reinier/open-1961-implement-github-on-pull-re…
Pwuts Nov 5, 2024
ca942cc
fix registration of pyro (de)serializers on `DatabaseManager`
Pwuts Nov 9, 2024
00882c0
format
Pwuts Nov 9, 2024
a573848
Merge branch 'dev' into reinier/open-1961-implement-github-on-pull-re…
Pwuts Nov 9, 2024
1de18a8
format frontend
Pwuts Nov 9, 2024
b1fac14
fix `NodeMultiSelectInput` click behavior
Pwuts Nov 9, 2024
d4c5e63
fix `NodeMultiSelectInput` scroll behavior
Pwuts Nov 9, 2024
ce56136
add docs for webhook-triggered blocks
Pwuts Nov 9, 2024
2a32b3b
fix collapsers?
Pwuts Nov 9, 2024
90418e0
docs: wording & formatting
Pwuts Nov 10, 2024
f40aef8
formatting
Pwuts Nov 11, 2024
8f708a2
fix code block titles
Pwuts Nov 11, 2024
35ffa09
add instruction to add event filter
Pwuts Nov 11, 2024
4048cc8
formatting
Pwuts Nov 11, 2024
060b1bc
Merge branch 'dev' into reinier/open-1961-implement-github-on-pull-re…
Pwuts Nov 12, 2024
b7c7028
fix pre-commit isort
Pwuts Nov 12, 2024
9e4f657
fix CI
Pwuts Nov 12, 2024
bdc3590
Merge branch 'dev' into reinier/open-1961-implement-github-on-pull-re…
Pwuts Nov 12, 2024
63a41f7
make `GithubPullRequestTriggerBlock` test an actual test
Pwuts Nov 13, 2024
865ac4b
address feedback
Pwuts Nov 13, 2024
fd0a3c2
add instructions for `PLATFORM_BASE_URL` to .env.example
Pwuts Nov 13, 2024
3ae2128
move github example payload
Pwuts Nov 15, 2024
5bb9e91
fix(backend): Add migrations to fix credentials inputs
Pwuts Nov 15, 2024
5ec5834
Merge branch 'dev' into reinier/open-1961-implement-github-on-pull-re…
Pwuts Nov 18, 2024
9e16e74
Merge branch 'dev' into reinier/open-1961-implement-github-on-pull-re…
Pwuts Nov 19, 2024
bc79d3b
Remove `ProviderName` enum
Pwuts Nov 19, 2024
4a5b783
Improve input descriptions and fix tooltips on GitHub Pull Request Tr…
Pwuts Nov 21, 2024
4c53eb3
Rename `sender` output to `triggered_by_user`
Pwuts Nov 21, 2024
7c0b336
Update description of `triggered_by_user`
Pwuts Nov 21, 2024
0f7fa4b
Fix description of `payload` output to be generic
Pwuts Nov 21, 2024
2dcbcfe
Add `pull_request_url` output
Pwuts Nov 21, 2024
397ae0b
Merge branch 'dev' into reinier/open-1961-implement-github-on-pull-re…
Pwuts Nov 21, 2024
8761c03
Remove webhooks before deleting credentials
Pwuts Nov 21, 2024
6eb643d
Prevent use of webhook system if `PLATFORM_BASE_URL` is not set
Pwuts Nov 21, 2024
0ac0ac9
feat: wip eod commit
ntindle Nov 24, 2024
463c8a6
Update triggers.py
ntindle Nov 24, 2024
23ab904
Merge branch 'dev' into exploration/other-types-of-hooks
ntindle Nov 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions autogpt_platform/backend/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,15 @@ SUPABASE_URL=http://localhost:8000
SUPABASE_SERVICE_ROLE_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyAgCiAgICAicm9sZSI6ICJzZXJ2aWNlX3JvbGUiLAogICAgImlzcyI6ICJzdXBhYmFzZS1kZW1vIiwKICAgICJpYXQiOiAxNjQxNzY5MjAwLAogICAgImV4cCI6IDE3OTk1MzU2MDAKfQ.DaYlNEoUrrEn2Ig7tqibS-PHK5vgusbcbo7X36XVt4Q
SUPABASE_JWT_SECRET=your-super-secret-jwt-token-with-at-least-32-characters-long

# For local development, you may need to set FRONTEND_BASE_URL for the OAuth flow for integrations to work.
FRONTEND_BASE_URL=http://localhost:3000
## For local development, you may need to set FRONTEND_BASE_URL for the OAuth flow
## for integrations to work. Defaults to the value of PLATFORM_BASE_URL if not set.
# FRONTEND_BASE_URL=http://localhost:3000

## PLATFORM_BASE_URL must be set to a *publicly accessible* URL pointing to your backend
## to use the platform's webhook-related functionality.
## If you are developing locally, you can use something like ngrok to get a publc URL
## and tunnel it to your locally running backend.
PLATFORM_BASE_URL=https://your-public-url-here

## == INTEGRATION CREDENTIALS == ##
# Each set of server side credentials is required for the corresponding 3rd party
Expand Down
7 changes: 0 additions & 7 deletions autogpt_platform/backend/backend/blocks/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,13 +60,6 @@ def all_subclasses(cls: Type[T]) -> list[Type[T]]:
input_schema = block.input_schema.model_fields
output_schema = block.output_schema.model_fields

# Prevent duplicate field name in input_schema and output_schema
duplicate_field_names = set(input_schema.keys()) & set(output_schema.keys())
if duplicate_field_names:
raise ValueError(
f"{block.name} has duplicate field names in input_schema and output_schema: {duplicate_field_names}"
)

# Make sure `error` field is a string in the output schema
if "error" in output_schema and output_schema["error"].annotation is not str:
raise ValueError(
Expand Down
2 changes: 1 addition & 1 deletion autogpt_platform/backend/backend/blocks/agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ def get_executor_manager_client():

@thread_cached
def get_event_bus():
from backend.data.queue import RedisExecutionEventBus
from backend.data.execution import RedisExecutionEventBus

return RedisExecutionEventBus()

Expand Down
75 changes: 75 additions & 0 deletions autogpt_platform/backend/backend/blocks/compass/triggers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
from typing import Literal
from backend.data.block import (
Block,
BlockCategory,
BlockOutput,
BlockSchema,
BlockWebhookConfig,
)
from backend.data.model import CredentialsField, CredentialsMetaInput, SchemaField
from backend.integrations.webhooks.simple_webhook_manager import CompassWebhookType
from pydantic import BaseModel


class Transcription(BaseModel):
text: str
speaker: str
end: float
start: float
duration: float


class TranscriptionDataModel(BaseModel):
date: str
transcription: str
transcriptions: list[Transcription]


class CompassAITriggerBlock(Block):
class Input(BlockSchema):
class EventsFilter(BaseModel):
all: bool = True

payload: TranscriptionDataModel = SchemaField(hidden=True)
events: EventsFilter = SchemaField(
description="Filter the events to be triggered on."
)

credentials: CredentialsMetaInput[Literal["compass"], Literal["api_key"]] = (
CredentialsField(
provider="compass",
supported_credential_types={"api_key"},
)
)
Comment on lines +38 to +43
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have any real concept of credentials that I need here. If we generate a secret on the platform side, that would be useful so we can ensure its a trusted source who sent the hook. Maybe we should store that in the credential?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For that we already have the Webhook.secret. This mechanism doesn't need credentials, so we should just omit the credentials field.

Copy link
Member

@Pwuts Pwuts Nov 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll have to amend this check in on_node_activate to only enforce credentials if there is an actual credentials field on block.input_schema:

has_everything_for_webhook = (
resource is not None
and CREDENTIALS_FIELD_NAME in node.input_default
and event_filter_input_name in node.input_default
and any(is_on for is_on in node.input_default[event_filter_input_name].values())
)
if has_everything_for_webhook and resource:
logger.debug(f"Node #{node} has everything for a webhook!")
if not credentials:
credentials_meta = node.input_default[CREDENTIALS_FIELD_NAME]
raise ValueError(
f"Cannot set up webhook for node #{node.id}: "
f"credentials #{credentials_meta['id']} not available"
)

This is also one of the two places where the presence of an event filter is enforced.

Copy link
Member

@Pwuts Pwuts Nov 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically the only thing it should do for your case is create a Webhook and store it in the DB. The easy way is to just skip parts of the process and then display the webhook ingress URL and webhook secret on the block in the UI. The neat way would be to have a picker for the webhooks for a given provider, on blocks that can't auto-create their webhook (like GitHub).


class Output(BlockSchema):
transcription: str = SchemaField(
description="The transcription of the compass transcription."
)

def __init__(self):
super().__init__(
id="9464a020-ed1d-49e1-990f-7f2ac924a2b7",
description="This block forwards an input value as output, allowing reuse without change.",
categories={BlockCategory.HARDWARE},
input_schema=CompassAITriggerBlock.Input,
output_schema=CompassAITriggerBlock.Output,
webhook_config=BlockWebhookConfig(
provider="simple_hook_manager",
webhook_type=CompassWebhookType.TRANSCRIPTION,
resource_format="{transcription_id}",
event_filter_input="events",
event_format="transcription.{event}",
),
# test_input=[
# {"input": "Hello, World!"},
# {"input": "Hello, World!", "data": "Existing Data"},
# ],
# test_output=[
# ("output", "Hello, World!"), # No data provided, so trigger is returned
# ("output", "Existing Data"), # Data is provided, so data is returned.
# ],
)

def run(self, input_data: Input, **kwargs) -> BlockOutput:
yield "transcription", input_data.payload.transcription
Loading
Loading