Ability to run against other browsers

[simonw]

Playwright can do this, so it's not too hard to get shot-scraper to do this too.

[simonw]

@palewire made a start on this here:

• Add an option to install chrome or firefox #52

[simonw]

shot 'https://www.whatismybrowser.com/detect/what-is-my-user-agent/' -o /tmp/default.png -h 400 -w 800

shot-scraper shot 'https://www.whatismybrowser.com/detect/what-is-my-user-agent/' -o /tmp/firefox.png -b firefox -h 400 -w 800

[simonw]

% shot-scraper javascript https://simonwillison.net/ 'navigator.userAgent'
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/100.0.4863.0 Safari/537.36"
% shot-scraper javascript https://simonwillison.net/ 'navigator.userAgent' -b firefox
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0"

[simonw]

Got this error in CI: https://github.com/simonw/shot-scraper/runs/5728682320?check_suite_focus=true

AttributeError: 'BrowserType' object has no attribute 'chromium'

It was from this:

shot-scraper pdf https://datasette.io/tools

[simonw]

Here's what happens if you attempt to use a browser that isn't yet installed. I think this is OK:

(shot-scraper) shot-scraper % shot-scraper javascript https://simonwillison.net/ 'navigator.userAgent' -b chrome-beta
Traceback (most recent call last):
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/bin/shot-scraper", line 33, in <module>
    sys.exit(load_entry_point('shot-scraper', 'console_scripts', 'shot-scraper')())
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/simon/Dropbox/Development/shot-scraper/shot_scraper/cli.py", line 349, in javascript
    context, browser_obj = _browser_context(p, auth, browser=browser)
  File "/Users/simon/Dropbox/Development/shot-scraper/shot_scraper/cli.py", line 210, in _browser_context
    browser_obj = p.chromium.launch(**browser_kwargs)
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 11417, in launch
    self._sync(
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 111, in _sync
    return task.result()
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/playwright/_impl/_browser_type.py", line 90, in launch
    Browser, from_channel(await self._channel.send("launch", params))
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 39, in send
    return await self.inner_send(method, params, False)
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 63, in inner_send
    result = next(iter(done)).result()
playwright._impl._api_types.Error: Chromium distribution 'chrome-beta' is not found at /Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta
Run "playwright install chrome-beta"
(shot-scraper) shot-scraper % shot-scraper install chrome-beta
Usage: shot-scraper install [OPTIONS]
Try 'shot-scraper install -h' for help.

Error: Got unexpected extra argument (chrome-beta)
(shot-scraper) shot-scraper % shot-scraper install -bchrome-beta
Switching to root user to install dependencies...
Password:
+ rm -rf '/Applications/Google Chrome Beta.app'
+ cd /tmp
+ curl -o ./googlechromebeta.dmg -k https://dl.google.com/chrome/mac/universal/beta/googlechromebeta.dmg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  182M  100  182M    0     0  36.0M      0  0:00:05  0:00:05 --:--:-- 39.7M
+ hdiutil attach -nobrowse -quiet -noautofsck -noautoopen -mountpoint /Volumes/googlechromebeta.dmg ./googlechromebeta.dmg

+ cp -rf '/Volumes/googlechromebeta.dmg/Google Chrome Beta.app' /Applications
+ hdiutil detach /Volumes/googlechromebeta.dmg
"disk2" ejected.
+ rm -rf /tmp/googlechromebeta.dmg
+ '/Applications/Google Chrome Beta.app/Contents/MacOS/Google Chrome Beta' --version
Google Chrome 100.0.4896.60 beta
(shot-scraper) shot-scraper % 
(shot-scraper) shot-scraper % shot-scraper javascript https://simonwillison.net/ 'navigator.userAgent' -b chrome-beta
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/100.0.4896.60 Safari/537.36"
(shot-scraper) shot-scraper % 

[simonw]

I'm going to add Firefox to the CI suite, update the documentation and call this done.