browse: (1) apply url validation also to scrape_links(), (2) add unit-tests for scrape_links()

[coditamar]

Background

• Validating URLs is important and we wish to apply it to all browse functions and not only scrape_text()
• Continuing to add unit-test to enable us to develop with confidence, this time adding tests to scrape_links()

Changes

1. extracting the URL validation from scrape_text() into a separate function get_validated_response()
2. applied get_validated_response() also to scrape_links()
3. added tests for scrape_links() in tests/test_browse_scrape_links.py

Documentation

Functions include comments.
test_browse_scrape_links.py includes explanations.

Test Plan

Tests where added

PR Quality Checklist

• [v] My pull request is atomic and focuses on a single change.
• [v] I have thoroughly tested my changes with multiple different prompts.
• [v] I have considered potential risks and mitigations for my changes.
• [v] I have documented my changes clearly and comprehensively.
• [v] I have not snuck in any "extra" small tweaks changes

[nponeccop]

These lines are non-PEP8 compliant whitespace. Remove them (but ensure that there is a final CRLF

[nponeccop]

Now you overdone it. You should leave the final line separator. The exact rules are pretty hard, you can just run the flake8 and see if it stops complaining at the EOL :-\

But this is lesser evil, I approve

[nponeccop]

@coditamar There are conflicts now

[coditamar]

I will make the needed changes or open a clean PR

[coditamar]

Overall changed:
browse:
(1) apply url validation also to scrape_links(),
(2) add unit-tests for scrape_links(),
(3) fix url validation (cleaned conflict),
(4) moved related unit-test under the right folder

To be frank, I think that the changes that made the conflicts were a bit messy -- it wasn't "dry" for example

I hope I made the code cleaner

@nponeccop @richbeales

[richbeales]

Conflicting files
scripts/browse.py
tests/test_browse_scrape_text.py