Skip to content

Latest commit

 

History

History
32 lines (20 loc) · 793 Bytes

README.md

File metadata and controls

32 lines (20 loc) · 793 Bytes
Raw

url collector (Current release: 0.11.1)

a small cli utility that collects all urls from one (hierarchical) sitemap and offers a random selection of all urls. You can use JSON or YAML as output format

Installation

  1. Make sure, you have the rust and cargo tooling installed
  2. Run cargo install --git https://github.com/stmh/url-collector.git

Or

  • Download the binary from the releases

Usage

Get help for all arguments

url-collector --help

Typical usage

# Will collect 10 random urls from the sitemap
url-collector --url https://spiegel.de --num-urls 10   
# Will get 100000 urls from a basic auth site and print yaml from it
url-collector --url https://myprivatewebsite.org --authentication user:seCret --num-urls 100000 --output yaml

Have fun