Skip to content

Latest commit

 

History

History
16 lines (10 loc) · 619 Bytes

README.md

File metadata and controls

16 lines (10 loc) · 619 Bytes
Raw

Simple Web Scraping Colly App with Fiber

This is a basic Go application using the Fiber framework to create scraping tasks in colly.

How to Run

  1. Clone the repository.
  2. Navigate to the project directory.
  3. Run docker compose up --build.
  4. Visit http://127.0.0.1:3000/api/healthchecker in a web browser or use a tool like curl to test it.
  5. Send GET request to http://127.0.0.1:3000/scrape/coursera to start scraping Coursera courses. And http://127.0.0.1:3000/scrape/quotes to scrape quotes.toscrape.com.

What It Does

  • Scrapes data from websites and stores in PostgreSQL database.