Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cursor): Add CursorReplayStrategy with red dot painting and self-correction #823

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

onyedikachi-david
Copy link


Fixes #760
/claim #760

What kind of change does this PR introduce?

Feature

Summary

This PR introduces the CursorReplayStrategy feature, which allows the AI model to suggest coordinates for a red dot on a screenshot representing the intended action target location. This new strategy provides the ability to visualize and self-correct initial suggestions, improving replay accuracy. It also includes an update to the screenshot capturing functionality to support capturing all monitors.

Checklist

  • My code follows the style guidelines of OpenAdapt
  • I have performed a self-review of my code
  • If applicable, I have added tests to prove my fix is functional/effective
  • I have linted my code locally prior to submission
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
  • New and existing unit tests pass locally with my changes

How can your code be run and tested?

  1. Test the CursorReplayStrategy by running the AI model and verifying the red dot is correctly suggested and visualized on the screenshot.
  2. Use the self-correction mechanism to adjust the red dot position and verify the corrections are applied accurately.

Other information

No additional context needed.


@abrichr abrichr mentioned this pull request Jul 5, 2024
7 tasks
@abrichr
Copy link
Member

abrichr commented Jul 5, 2024

Thank you @onyedikachi-david !

Can you please update your PR description with images containing the screenshot with the red dot? Ideally multiple over time as the model self corrects.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement model cursor for visual feedback
2 participants