galer

             __
   __    _ _(_ )   __  _ __ 
 /'_ '\/'_' )| | /'__'( '__)
( (_) ( (_| || |(  ___| |
'\__  '\__,_(___'\____(_)
( )_) |
 \___/'  @dwisiswant0

A fast tool to fetch URLs from HTML attributes by crawl-in. Inspired by the @omespino Tweet, which is possible to extract src, href, url and action values by evaluating JavaScript through Chrome DevTools Protocol.

---

Resources

• Installation
  • from Binary
  • from Source
  • from GitHub
• Usage
  • Basic Usage
  • Flags
  • Examples
    • Single URL
    • URLs from list
    • from Stdin
  • Library
• TODOs
• Help & Bugs
• License
• Version
• Acknowledgement

Installation

from Binary

The installation is easy. You can download a prebuilt binary from releases page, unpack and run! or with

▶ (sudo) curl -sSfL https://git.io/galer | sh -s -- -b /usr/local/bin

from Source

If you have go1.22+ compiler installed and configured:

▶ go install -v github.com/dwisiswant0/galer@latest

from GitHub

▶ git clone https://github.com/dwisiswant0/galer
▶ cd galer
▶ go build .
▶ (sudo) install galer /usr/local/bin

Usage

Basic Usage

Simply, galer can be run with:

▶ galer -u "http://domain.tld"

Flags

▶ galer -h

This will display help for the tool. Here are all the switches it supports.

Flag	Description
-u, --url	Target to fetches (single target URL or list)
-e, --extension	Show only certain extensions (comma-separated, e.g. js,php)
-c, --concurrency	Concurrency level (default: 50)
--in-scope	Show in-scope URLs/same host only
-o, --output	Save fetched URLs output into file
-t, --timeout	Maximum time (seconds) allowed for connection (default: 60)
-s, --silent	Silent mode (suppress an errors)
-v, --verbose	Verbose mode show error details unless you weren't use silent
-h, --help	Display its helps

Examples

Single URL

▶ galer -u "http://domain.tld"

URLs from list

▶ galer -u /path/to/urls.txt

from Stdin

▶ cat urls.txt | galer

In case you want to chained with other tools:

▶ subfinder -d domain.tld -silent | httpx -silent | galer

Library

You can use galer as library.

▶ go get github.com/dwisiswant0/galer/pkg/galer@latest

For example:

package main

import (
	"fmt"

	"github.com/dwisiswant0/galer/pkg/galer"
)

func main() {
	cfg := &galer.Config{
		Timeout: 60,
	}
	cfg = galer.New(cfg)

	run, err := cfg.Crawl("https://twitter.com")
	if err != nil {
		panic(err)
	}

	for _, url := range run {
		fmt.Println(url)
	}
}

TODOs

• Enable to set extra HTTP headers
• Provide randomly User-Agent
• Bypass headless browser
• Add exception for specific extensions

Help & Bugs

If you are still confused or found a bug, please open the issue. All bug reports are appreciated, some features have not been tested yet due to lack of free time.

Status

Caution

galer has NOT reached 1.0 yet. Therefore, this library is currently not supported and does not offer a stable API; use at your own risk.

There are no guarantees of stability for the APIs in this library, and while they are not expected to change dramatically. API tweaks and bug fixes may occur.

Pronunciation

id_ID • /gäˈlər/ — kalau galer jangan dicium baunya, langsung cuci tangan, bego!

Acknowledgement

• Omar Espino for the idea, that's why this tool was made!

License

sebel is released by @dwisiswant0 under the MIT license. See LICENSE.