--skip and --fail options for handling HTTP 4xx and 4xx errors

[simonw]

The script that updates https://simonw.github.io/datasette-screenshots/ has a problem: occasionally it takes a screenshot that looks like this:

I don't want to manually check the shots every time I the GitHub Actions runs, so I'd like to have an option for shot-scraper multi that can spot if a page comes back with a non-200 error and logs a warning but keeps going.

That way any failures in that repo at least won't over-write a good screenshot.

[simonw]

This should probably be accompanied by an option that causes the entire script to quit with an error status code (such that CI will fail) if a 4xx or 5xx is encountered.

[simonw]

Asked ChatGPT:

What are some good names for an option for a CLI script which means "fail with a non-zero exit code if an HTTP error occurs"

It came up with some bad options, so I asked:

Are there any commands like "curl" that have an option like this? List the commands and their options

Got back:

Yes, there are several commands like curl that have an option to fail with a non-zero exit code when an HTTP error occurs. Some examples include:

• curl: -f, --fail
• wget: --no-clobber
• httpie: --fail
• http (from httpie): --fail

These commands also have --silent,--show-error, -S options which make them not output anything to stdout, these options also make them fail with a non-zero exit code if an HTTP error occurs.

I'm going to go with --fail.

[simonw]

I'm going to add --skip and --fail to the shot-scraper shot and shot-scraper multi commands.

Should I add them to pdf and html and accessibility and javascript as well?

I'm leaning towards "yes", because each of those commands might be used like so:

shot-scraper pdf https://unreliable-website.com/ -o latest.pdf

And people may want to overwrite latest.pdf only if the web page is up.

[simonw]

Demo:

% echo '- url: https://latest.datasette.io/404' | shot-scraper multi - --fail
Error: 404 error for https://latest.datasette.io/404
% echo $?
1
% echo '- url: https://latest.datasette.io/404' | shot-scraper multi - --skip
404 error for https://latest.datasette.io/404, skipping
% echo $?                                                                    
0

[simonw]

More demos:

% shot-scraper html https://simonwillison.net/404 --skip
404 error for https://simonwillison.net/404, skipping
% echo $?
0
% shot-scraper html https://simonwillison.net/404 --fail
Error: 404 error for https://simonwillison.net/404
% echo $?
1

% shot-scraper javascript https://simonwillison.net/404 'document.title' --skip
404 error for https://simonwillison.net/404, skipping
% echo $?
0
% shot-scraper javascript https://simonwillison.net/404 'document.title' --fail
Error: 404 error for https://simonwillison.net/404
% echo $?
1

% shot-scraper pdf https://simonwillison.net/404 --skip
404 error for https://simonwillison.net/404, skipping
% echo $?
0
% shot-scraper pdf https://simonwillison.net/404 --fail
Error: 404 error for https://simonwillison.net/404
% echo $?                                              
1

% shot-scraper accessibility https://simonwillison.net/404 --skip
404 error for https://simonwillison.net/404, skipping
% echo $?
0
% shot-scraper accessibility https://simonwillison.net/404 --fail
Error: 404 error for https://simonwillison.net/404
% echo $?                                                        
1