[WIP] Add frontend for text based needles #5165

clanig · 2023-05-30T07:56:13Z

This PR is for the frontend integration of the OCR feature.
It depends on os-autoinst/os-autoinst#2276 to be merged.

github-actions · 2023-05-30T07:56:42Z

Great PR! Please pay attention to the following items before merging:

Files matching docs/*.asciidoc:

Consider generating documentation locally to verify it is rendered correctly using tools/generate-htmldoc

This is an automatically generated QA checklist based on modified files.

Martchus · 2023-05-30T09:05:09Z

assets/javascripts/needlediff.js

-          this.ctx.drawImage(this.needleImg, orig.xpos, orig.ypos, usedWith, a.height, x, a.ypos, usedWith, a.height);
+          if (this.needleImg) {
+            this.ctx.drawImage(this.needleImg, orig.xpos, orig.ypos, usedWith, a.height, x, a.ypos, usedWith, a.height);
+          } else {


This whole else-block should better be extracted into a separate function (to avoid this one from becoming to big).

Martchus · 2023-05-30T09:05:59Z

assets/javascripts/needlediff.js

+            }
+
+            // Write OCR lines to Canvas context line by line
+            var ocrTxtXPos = Math.ceil(a.xpos + a.width / 2);


It would generally be nice to use const/let in new code, e.g.

Suggested change

var ocrTxtXPos = Math.ceil(a.xpos + a.width / 2);

const ocrTxtXPos = Math.ceil(a.xpos + a.width / 2);

This counts also for all the other occurences of var … (in new code).

Martchus · 2023-05-30T09:07:14Z

assets/javascripts/needlediff.js

+          } else {
+            // Calculate font size with 1px margin
+            var ocrLinesFontSize = 14;
+            var ocrLines = a['ocr_str'].split('\n');


Suggested change

var ocrLines = a['ocr_str'].split('\n');

var ocrLines = a.ocr_str.split('\n');

Martchus · 2023-05-30T09:09:03Z

assets/javascripts/needleeditor.js

+      var changeMarginButton = document.getElementById('change-margin');
+      // Only OCR areas can be assigned a refstr
+      var refstrTxtArea = document.getElementById('txtarea-refstr');
+      if (a.type == "match") {


Suggested change

if (a.type == "match") {

if (a.type === "match") {

Martchus · 2023-05-30T09:09:21Z

assets/javascripts/needleeditor.js

+        refstrTxtArea.value = '';
+        refstrTxtArea.classList.add('disabled');
+        refstrTxtArea.disabled = "disabled";
+      } else if (a.type == "ocr") {


Suggested change

} else if (a.type == "ocr") {

} else if (a.type === "ocr") {

Martchus · 2023-05-30T09:22:44Z

lib/OpenQA/Task/Needle/Save.pm

-        my $error = "Image $imagename could not be found!";
-        $app->log->error("Failed to save needle: $error");
-        return $minion_job->fail({error => "<strong>Failed to save $needlename.</strong><br>$error"});
+    unless ($no_match_areas) {


The double-negative is not very nice and we won't need to extra variable here.

Suggested change

unless ($no_match_areas) {

if (@match_areas) {

The same counts for the other unless ($no_match_areas) occurrences.

Martchus · 2023-05-30T09:24:06Z

lib/OpenQA/WebAPI/Controller/Step.pm

@@ -154,7 +154,7 @@ sub edit ($self) {
                    width => int $match->{w},
                    ypos => int $match->{y},
                    height => int $match->{h},
-                    type => 'match',
+		    type => 'match',


This change looks unintended. Likely tidy will fix it for you (which you'll have to run anyways).

Martchus · 2023-05-30T09:25:21Z

lib/OpenQA/WebAPI/Controller/Step.pm

@@ -449,6 +450,8 @@ sub calc_matches ($needle, $areas) {
            type => $area->{result},
            similarity => int($area->{similarity} + 0.5),
        );
+        $match{ocr_str} = $area->{ocr_str} if defined $area->{ocr_str};
+	print($area->{ocr_str}) if defined $area->{ocr_str};


This is likely for local debugging and should be removed before merging.

Martchus · 2023-05-30T09:27:16Z

docs/Contributing.asciidoc

@@ -134,8 +134,7 @@ Installation Guide.
 In the case of os-autoinst, only a few http://www.cpan.org/[CPAN] modules are
 required. Basically `Carp::Always`, `Data::Dump`. `JSON` and `YAML`. On the other
 hand, several external tools are needed including
-http://wiki.qemu.org/Main_Page[QEMU],
-https://code.google.com/p/tesseract-ocr/[Tesseract] and
+http://wiki.qemu.org/Main_Page[QEMU] and


Maybe we should mention GOCR then instead?

Martchus · 2023-05-30T09:28:46Z

assets/javascripts/needlediff.js

+            while ((ocrLines.length * ocrLinesFontSize + (ocrLines.length - 1)) > (a.height - 2)) {
+              ocrLinesFontSize--;
+            }


Does this really need to be computed via a loop?

The addition of the OCR feature revealed character encoding bugs that might have been invisible previously. This commit fixes the character encoding in multiple places to ensure the correct displaying of special characters in the web frontend.

mergify · 2023-09-14T15:01:04Z

This pull request is now in conflicts. Could you fix it? 🙏

clanig · 2023-09-14T15:03:34Z

I have updated the code. There were multiple encoding bugs as well.
This code is based on an old codebase. So I am going to rebase now.

mergify · 2024-02-06T13:25:06Z

This pull request is now in conflicts. Could you fix it? 🙏

kalikiana marked this pull request as draft May 30, 2023 08:07

Martchus requested changes May 30, 2023

View reviewed changes

clanig added 4 commits September 12, 2023 09:00

Display OCR result of test on step screenshot

d20ed4e

Implement OCR in needle editor

882d8b1

Update docs for OCR implementation

c8103e1

Fix character encoding for OCR

3a4e3e1

The addition of the OCR feature revealed character encoding bugs that might have been invisible previously. This commit fixes the character encoding in multiple places to ensure the correct displaying of special characters in the web frontend.

clanig force-pushed the add_frontend_for_text_based_needles branch from 0f737df to 3a4e3e1 Compare September 14, 2023 15:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add frontend for text based needles #5165

[WIP] Add frontend for text based needles #5165

clanig commented May 30, 2023

github-actions bot commented May 30, 2023

Martchus May 30, 2023

Martchus May 30, 2023

Martchus May 30, 2023

Martchus May 30, 2023

Martchus May 30, 2023

Martchus May 30, 2023

Martchus May 30, 2023

Martchus May 30, 2023

Martchus May 30, 2023

Martchus May 30, 2023

mergify bot commented Sep 14, 2023

clanig commented Sep 14, 2023

mergify bot commented Feb 6, 2024

	var ocrTxtXPos = Math.ceil(a.xpos + a.width / 2);
	const ocrTxtXPos = Math.ceil(a.xpos + a.width / 2);

	var ocrLines = a['ocr_str'].split('\n');
	var ocrLines = a.ocr_str.split('\n');

[WIP] Add frontend for text based needles #5165

Are you sure you want to change the base?

[WIP] Add frontend for text based needles #5165

Conversation

clanig commented May 30, 2023

github-actions bot commented May 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Sep 14, 2023

clanig commented Sep 14, 2023

mergify bot commented Feb 6, 2024