Skip to content

Latest commit

 

History

History
69 lines (50 loc) · 1.72 KB

README.md

File metadata and controls

69 lines (50 loc) · 1.72 KB
Raw

Archive

A simple python script to generate a sitemap of a given website and archive all the pages not already stored in the wayback machine. This now available to use an API as well!

Check it out the documentation here

Setup

$ git clone https://github.com/apurvmishra99/archiver.git
$ cd archiver
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -r requirements.txt

Usage

Usage: archive.py [OPTIONS] URL

Options:
  -m, --max_urls INTEGER  The max number of urls to collect. The default value
                          is 50.Use 0 to set it as infinite.
  -d, --days INTEGER      The time difference(in days) of the current copy of
                          the page if it exists and we want to archive it
                          again. The default value is 7 days. Use 0 to archive
                          all pages again.
  --help                  Show this message and exit.

Example

$ python archive.py --days=7 --max_urls=50 https://apurvmishra.xyz

Alternative use

If you just want to scrape all the internal links on the website and write it to a txt file you can scrape_all_internal_links.py

Usage

Usage: scrape_all_internal_links.py [OPTIONS] URL

Options:
  --max_urls INTEGER  The max number of urls to collect. Use 0 to set it as
                      infinite.
  --help              Show this message and exit.

Example

$ python scrape_all_internal_links.py --max_urls=50 https://apurvmishra.xyz

TODO

  • Package the script
  • Convert to async
  • Add command line option to just generate the sitemap

Tested On

Pop!_OS 20.04 LTS
Python v3.7.6