Looping through list of URLs in datatables csv to and visit link - merged to #941

[ruthtxh]

Posting a query from a user who reached out to me on LinkedIn below:

I really enjoyed the using the tool, it really simplifies automation 100x

my main gripe on tagui is that i can't (or don't know how to) run the script on multiple urls stored in a csv (or stored as rows in a text file)

eg.

list_of_urls.csv:
sites
www.site1.com
www.site2.com

myflow.tag:
for sites in list_of_urls.csv:
visit site1
do something
...

///

so I had to resort to use the py rpa package which had some nasty gotchas eg:

1. cannot have spaces in folders to C:/ , so had to start project in C:/tagui/flow/your_script.py
2. urls need to start with 'http'... www will throw an error
3. if you're on vsc, the built in terminal wont work, need to execute your_script.py in powershell or cmd

overall painful but happy with the simplicity!

[ruthtxh]

Here's a workaround to read the url from a datatable.csv list and visit the url.

dump `url` to url.txt
load url.txt to url
https://`url.split('//')[1]`

Link to other issues:
#936
#953

[ruthtxh]

For the Python RPA package, it is @kensoh's personal project and issues are separately tracked here https://github.com/tebelorg/RPA-Python/issues.

Regarding the points:

1. cannot have spaces in folders to C:/ , so had to start project in C:/tagui/flow/your_script.py
   reply: yes this is a known issue for TagUI as well and we are looking into it.
2. urls need to start with 'http'... www will throw an error
   reply: TagUI searches the syntax beginning with 'http' or 'https' to identify it is a web page

[kensoh]

Thanks @ruthtxh for your replies! Maybe this is an opportunity to add some 'macro' functionality to TagUI.

Ie, user can define some text replacement shortcuts so that certain text will be replaced before parsing.

Eg a line starting with 'visit ' will be replaced and be interpreted as

https://`parameter.split('//')[1]`

This is done for the 'wait for ' which maps to 'hover '. But I haven't further implemented this. The difference between this way and adding new step is new step involves more work and need to regenerate dictionaries for the 20 over languages. This macro idea will just work for the syntax user defines and not other languages.

[kensoh]

Assigning to myself for now. Can explore user macros for next release

[kensoh]

Adding feedback from Kevin that visit step is easier

[kensoh]

Related to #941

[kensoh]

Issue #949 should be part of user macros. Tracking it here and closing #949

---

Met a power user and he suggested that an authenticate step would be helpful.

For eg, when user is required to enter password and 2FA manually.

Thinking through, under the hood, it would be setting timeout to 300 seconds, doing a hover step on the target element, and then setting timeout back to the previous value (default is 10 seconds). This way, for 5 minutes, users can slowly enter password and 2FA code without breaking the automation. Once logged in and the element appears, automation continues.

[kensoh]

Hi @ruthtxh, actually doing below directly works for the datatable. Not sure if recent change fixes this or we missed it -

`url`

[kensoh]

Regardless, the idea of user macros is still important

And there is still no ability to access a URL from variable directly (above is from data table).

[kensoh]

Closing this issue as the use case here is already working ok, but the topic of user macros should still be explored.

Merging to #941