add /verifyrun cog #36
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
msaroufim
reviewed
Nov 24, 2024
There was a problem hiding this comment.
There is some time we could save in having a single command verify all of modal, nvidia and amd one after the other and letting us know how many of those 3 succeeded. Also as far as the UX goes, why not have verify actually trigger the bot 3 times instead of having a human manually verify in a thread
| This cog provides functionality to verify that either a GitHub Actions or | ||
| Modal run completed successfully by checking for specific message patterns | ||
| in a Discord thread. It supports verification of two types of runs: | ||
| 1. GitHub Actions runs - Identified by "GitHub Action triggered!" message |
There was a problem hiding this comment.
So this seems to test a trigger but we've had issues where this wouldn't have helped like for example when we had timeout issues with both the NVIDIA and AMD runner
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds a
/verifyruncommand that does the same work as the smoke test but is easier to use.Checklist
Before submitting this PR, ensure the following steps have been completed:
- Run the cluster bot on your server:
- Start a training run by with the slash command
- Wait for the training run to complete.
- Copy the URL for the thread started by the cluster bot in response to
- Click on the 3 dots (
- Select Copy Message Link.
- Using the copied URL, run the smoke test:
- Verify that the smoke test script responds with:
For more information on running a cluster bot on your own server, see/run.You may need to exercise some judgement about the script and GPU type.
your
/runmessage ("Cluster Bot started a thread: ..."):...) to the cluster bot's message.README.md.