Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

--wait-for / wait_for mechanism for waiting until a JavaScript expression returns true #72

Closed
psychemedia opened this issue May 10, 2022 · 5 comments
Closed

--wait-for / wait_for mechanism for waiting until a JavaScript expression returns true #72

psychemedia opened this issue May 10, 2022 · 5 comments
Labels
enhancement New feature or request

Comments

@psychemedia
Copy link

psychemedia commented May 10, 2022 •
edited by simonw
Loading

The Playwright .wait_for_function() allows you to wait for a certain condition to met before progressing to the next step.

It would be useful to support this sort of conditional delay when there is uncertainty about what an appropriate explicit --wait delay might be.
@psychemedia psychemedia changed the title Using waitForFunction instead of explicit --wait durations Support waitForFunction as well as explicit --wait durations May 10, 2022
@simonw
Copy link
Owner

simonw commented May 19, 2022 •
edited
Loading

Interesting idea. It might be possible to support this as:

shot-scraper datasette.io --wait-for-function "document.querySelector('iframe')"

Since it looks like page.wait_for_function(...) accepts a string.

There's a default timeout of 30s. Not sure if it's worth having an extra --wait-for-function-timeout option which lets you change that default. I'm inclined to leave that off at first.

@simonw simonw added the enhancement New feature or request label May 19, 2022
@simonw
Copy link
Owner

simonw commented May 19, 2022

I like --wait-for "..." better. It's shorter.

@simonw
Copy link
Owner

simonw commented May 19, 2022

Built a simple test page:

<html>
<body><h1>Here it comes...</h1>
<script>
setTimeout(() => {
  var div = document.createElement('div');
  div.innerHTML = 'DIV after 2 seconds';
  document.body.appendChild(div);
}, 2000);
</script>
</body>
</html>

Used this for a prototype:

shot-scraper div-after-2-seconds.html -o no-wait.png -w 300 -h 200

no-wait

shot-scraper div-after-2-seconds.html -o wait-for.png -w 300 -h 200 \
  --wait-for "document.querySelector('div')"

wait-for

Here's the prototype:

diff --git a/shot_scraper/cli.py b/shot_scraper/cli.py
index f2c96c6..3d238ae 100644
--- a/shot_scraper/cli.py
+++ b/shot_scraper/cli.py
@@ -117,6 +117,7 @@ def cli():
 @click.option(
     "--wait", type=int, help="Wait this many milliseconds before taking the screenshot"
 )
+@click.option("--wait-for", help="Wait until this JS expression returns true")
 @click.option(
     "--timeout",
     type=int,
@@ -151,6 +152,7 @@ def shot(
     retina,
     quality,
     wait,
+    wait_for,
     timeout,
     interactive,
     devtools,
@@ -198,6 +200,7 @@ def shot(
         "height": height,
         "quality": quality,
         "wait": wait,
+        "wait_for": wait_for,
         "timeout": timeout,
         "padding": padding,
         "retina": retina,
@@ -637,6 +640,7 @@ def take_shot(
         output = filename_for_url(url, ext="png", file_exists=os.path.exists)
     quality = shot.get("quality")
     wait = shot.get("wait")
+    wait_for = shot.get("wait_for")
     padding = shot.get("padding") or 0
 
     selectors = shot.get("selectors") or []
@@ -673,10 +677,14 @@ def take_shot(
 
     if wait:
         time.sleep(wait / 1000)
+
     javascript = shot.get("javascript")
     if javascript:
         _evaluate_js(page, javascript)
 
+    if wait_for:
+        page.wait_for_function(wait_for)
+
     screenshot_args = {}
     if quality:
         screenshot_args.update({"quality": quality, "type": "jpeg"})

@simonw
Copy link
Owner

simonw commented May 19, 2022

I decided to evaluate any JavaScript from --javascript before the --wait-for code is executed.

@simonw simonw changed the title Support waitForFunction as well as explicit --wait durations --wait-for / wait_for mechanism for waiting until a JavaScript expression returns true May 19, 2022
@simonw simonw closed this as completed in 84d30ed May 19, 2022
@simonw
Copy link
Owner

simonw commented May 19, 2022

Documentation is now live here: https://shot-scraper.datasette.io/en/latest/screenshots.html#waiting-until-a-specific-condition

simonw added a commit that referenced this issue May 19, 2022
@simonw
Fixed error in example for #72
1494ae7
simonw added a commit that referenced this issue May 19, 2022
@simonw
Release 0.14
048f064 
Refs #74, #73, #72
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@simonw @psychemedia