Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Generate a software accuracy report #171

Open
Udayraj123 opened this issue Mar 2, 2024 · 1 comment
Open

[Feature] Generate a software accuracy report #171

Udayraj123 opened this issue Mar 2, 2024 · 1 comment
Labels

Comments

@Udayraj123
Copy link
Owner

Requirements

Software accuracy report

  • We can assume that a ground-truth CSV output is provided, which can be matched with OMRChecker's output and get an accuracy score/report.
  • Need to list the erroneous changes and show the marked images for them.
  • We can get an accuracy number based on review portal's output (ground truth)

Ideas to start with

Further ideas

This implementation can be extended to create a test benchmark for any existing OMR checking software. We need to blackbox the logic and standardise the inputs and outputs in this case.

@Udayraj123 Udayraj123 converted this from a draft issue Mar 2, 2024
@Udayraj123 Udayraj123 changed the title Generate a software accuracy report [Feature] Generate a software accuracy report Mar 2, 2024
@Udayraj123
Copy link
Owner Author

Crosslinking discord thread: https://discord.com/channels/590134763784896514/1270815064256938183/1270816309940326572

Types of accuracy metrics
    - csv-based accuracy benchmark (using moderation portal)
    - Needs to fit in the per-directory approach
    - bubble detection/reading level accuracy
        - confidence metrics
    - preProcessor level accuracies
        - confidence metrics for the matchers
    - aggregate metric
    - speed graphs to identify

- Since we have 4-5 confidence metrics and 2-3 accuracy metrics, it'd be great to find a color spectrum that caters to these (color strips of 3 at a time if needed). Where all-white means most accurate and all-black means least accurate, and the weighted magnitude of the color gives the numeric representation.
- Can also make a Radar chart of this for explanation purposes as it's across different dimensions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: No status
Development

No branches or pull requests

1 participant