Visual automation for 2 monitors - use r.vision('Screen(1).click("/path/image.png")')

[ck81]

Hi Ken,

Understand that you have mentioned in #119 (dated Apr 4, 2020)

Are you using multiple monitors (laptop monitor and a separate monitor)? It might not work as the tool is designed to work with a single screen. If yes, try disconnecting so that you are only using your laptop monitor

And in #143 (dated May 14, 2020)

use only laptop screen and detach any external monitors

It has been more than one year now. Was wondering if the sikuli library has been improved such that RPA-Python now supports image recognition in monitor 2.

Or do you have any tweaks or hacks for this?

In today's working environment, a lot of people have 2 monitors. Most RPA tools can run with 2 monitors. It would be really good if TagUI or RPA-Python can also run with 2 monitors.

[kensoh]

Hi CK, actually SikuliX already supports multiple monitors.

However, it requires users to explicitly pick which monitor to search, for every step. And it is too much code change for upstream TagUI project now to cater to this use case of multiple monitors (the risk is breaking functionality for primary use case of 1 monitor).

From above SikuliX link, there is a SCREEN variable which supposedly is a constant. Can you try if the following is able to enable automation on a 2nd monitor? I tried assigning to Screen(0) which didn't crash SikuliX, but I can't test Screen(1) because I don't have an external monitor.

By right, a constant can't be changed at all. But if this SCREEN value can be changed with below, then easy solution!

r.vision('SCREEN = Screen(1)')

[ck81]

r.vision('SCREEN = Screen(1)')

Ken, you're amazing! Yes, it worked!

I used

r.vision('Screen(1).click("d:/tagui/img1.png")')

And it clicked the image in my second monitor!

Note: For those who want to use this, make sure that you use the fullpath to the image you want to click. At first I thought it's not working. Then I remembered that for vision statement, you need to use fullpath to image.

Ken, this is really cool! You should really document this somewhere in the documentation so that people are aware that TagUI-RPA supports 2 monitors!

[kensoh]

Thanks CK for your confirmation! CC @ruthtxh

Quite a long time ago I saw this but I assume in SikuliX documentation that SCREEN is a constant means it can't be changed. Normally a constant if you try to change it will crash a program. Does the following work by redefining SCREEN constant?

r.vision('SCREEN = Screen(1)')
r.click('d:/tagui/img1.png')

If it doesn't work, that means the SCREEN constant is still not something that can be changed. And the fallback is doing something like below, which you can do directly (without your first line of assigning Screen(1) to SCREEN).

r.vision('Screen(1).click("d:/tagui/img1.png")')

If above is the only way, then maybe a helper function can be explored to redirect steps like r.click() to 2nd monitor.

[ck81]

r.vision('SCREEN = Screen(1)')
r.click('d:/tagui/img1.png')

Just tried. This doesn't work.

[kensoh]

Thanks CK, I see.. Then I guess will be limited to using r.vision() workaround for now.

In your Python script, you can also do some helper functions something like below to make it easier.

However, note that it will be very limited because it is 'hard-coding' to interact by image instead of (x,y) coordinates, using OCR etc, and below keyboard2() doesn't take care of modifier keys. How I come up with below is referencing tagui/src/tagui.sikuli/tagui.py file. What to fill up in these helper functions can also be found from SikuliX doc for SikuliX syntax.

def click2(image_identifier)
    return r.vision('Screen(1).click("' + image_identifier + '")')

def type2(image_identifier, text_to_type)
    return r.vision('Screen(1).type("' + image_identifier + '","' + text_to_type + '")')

def keyboard2(text_to_type)
    return r.vision('Screen(1).type("' + text_to_type + '")')