Find screen text/image and apply condition - see example using image / ocr

[TomAsl50]

Hi Ken,

I want to find screen for specific text/image and apply the condition but I am stuck and not able to think of any function to use it.

Example the Pseudocode below:

If screentext equals "MY" or thisimage.png
click proceed.png
else
click exitimage.png

Kindly advise.

[kensoh]

Hi Tom you can use something like below. present() will check if some image is on the screen now.

If you use exist() instead, TagUI will keep checking until the default timeout of 10 seconds before returning false

if present('thisimage.png')
    click proceed.png
else
    click exitimage.png

[kensoh]

Adding on, below is an example of checking the condition using text instead through OCR instead of an image -

if present('some text using ocr')
    click proceed.png
else
    click exitimage.png

[TomAsl50]

Hi Ken,

when I apply the above code format. I got below error:

ERROR - [LINE 36] cannot understand step If present('images/my.png')

Do I miss any configuration?

Kindly advise.

Thanks.

[kensoh]

Hi Tom, can you paste a minimal script that you have that can trigger this error? Let me test that from my side.

Can you also check that you have v6.14 of TagUI? - https://tagui.readthedocs.io/en/latest/setup.html

[kensoh]

Also adding on, sorry I have a typo (I copied straight from your original example), the if should be lowercase.

[TomAsl50]

Hi Ken,

My bad. I should use if instead If. Its work fine now.

Also how about if statement with multiple condition OR, AND? Can this be done?

Thank you.

[kensoh]

Yes you can try if condition1 or condition2 etc, if condition1 and condition2 etc

[TomAsl50]

Hi Ken,

I try the code as below, my screen does not have the image 1 or 2 in present() but it still treat it as valid condition

if present('images/1.png') or present('images/2.png') 
    click images/3.png
    wait 5
    click images/Save.png
else
    click images/Back.png

[kensoh]

Hi Tom, can you add a live step just before above block of instructions?

Then try below to see what is the output.

echo `present('images/1.png')`

echo `present('images/2.png')`

Also, can you attach a .js file that is generated while it is still in the live mode to this issue? I want to see if for some reason the logic understanding is broken. Also, if you can, get the latest copy of TagUI to see if it is related to any recent bug-fix since Dec 2020 - see this link on how to update #1001 (comment)

[kensoh]

Adding on, attach the logs in tagui\src\tagui.sikuli folder so I can have a look at the response from the computer vision engine.

[TomAsl50]

Hi Ken,

I install the latest TagUI this month, version 6.14 if not mistaken. Please find the attached screenshot and .js file.

result.zip

tagui_sikuli.zip

[kensoh]

Thanks Tom! From you logs, I confirm that

1. the conversion from human language conditions to JavaScript code to do comparison is correct
2. the visual automation part returning response is behaving as expected from TagUI's perspective

Can you attach here what is your 1.png and 2.png?

For example, on my laptop, when I run below flow, the response is correct depending on whether let my desktop be visible during the automation or if I hide my desktop so that the image of the icon is not visible.

a.tag

if present('icon.png')
    echo YES
else
    echo NO

icon.png

tagui\src\tagui.sikuli\tagui.log for the case of icon being hidden from view

[tagui] START  - listening for inputs

[tagui] INPUT  - [1] present /Users/kensoh/Desktop/icon.png
[tagui] ACTION - present /Users/kensoh/Desktop/icon.png
[tagui] OUTPUT - [1] ERROR

[tagui] FINISH - stopped listening

[TomAsl50]

Hi Ken,

Below is the image file. Those image text actually not appear in my screen. My screen is showing US MB52 in text (in a grid column)

[kensoh]

Oh I see.. It sounds like the confidence level for the computer vision takes this as a match.

Is your application a web application? If yes, can you try writing the condition using a web identifier instead?

If it is a desktop app, see if you can use keyboard or visual automation to navigate to the field to copy out the text, before retrieving it using value_of_field = clipboard() and the checking this value as a condition.

[TomAsl50]

Hi Ken,

Below is new code. I manage to copy the text to clipboard but when I add in if comparison, the TagUi return error aborted.

keyboard [shift][end]
keyboard [ctrl]c
result = clipboard()
echo result

if result equals to "MB52"
click images/View.png
else
click images/Back.png

[kensoh]

Oh great! I think you have a typo? There is a requirement for indentation so that TagUI knows the steps are part of if or else conditions -

failed with error

result = 'MB52'
if result equals to "MB52"
echo a
else
echo b

runs and prints a

result = 'MB52'
if result equals to "MB52"
    echo a
else
    echo b

[TomAsl50]

Hi Ken,

Thanks for highlight. The if statement now is working, additionally when I add in for loop as below. It show an errors.

for n from 1 to 2
{
echo n
if result equals to "MB52"
click images/View.png
else
click images/Back.png
break
}

ERROR - missing indentation for {
ERROR - automation aborted due to above

Even I try to remove {} also appear the same error.

Kindly advise. Thanks.

[kensoh]

Hi Tom, can you attach you script here so that I can see your actual indentation?

Above looks wrong because after if and else statements, there should be indentation.

Also, try getting the latest version from here - #1001 (comment)

The TagUI v6.14 was from Dec 2020, and there has been a of bug-fix on indentation since then.

[TomAsl50]

Hi Ken,

The statement when I paste in this comment. It does not show indentation.

Below is the snapshot from my script file.

[kensoh]

Oh I see above is incorrect.

After echo n the next line should not have extra indentation because they belong to the same block.

Can you try remove the curly brackets and also align the lines something like below. Let me know if that works!

for n from 1 to 2
    echo n
    if result equals to "MB52"
        click images/View.png
    else
        click images/Back.png
        break

[TomAsl50]

Hi Ken,

Its all working now.

Thank you for your great help. :-)