Unable to resolve "./scraperDrivers/allDrivers.js"

[rosevinnur]

Unable to resolve "./scraperDrivers/allDrivers.js" from "utils/api.js"

[agilgur5]

Yes, I haven't completed documenting this as there's still a lot of WIP (less than a week old), but per the .gitignore located in the utils/ directory and the last commit in that directory, there is intentionally no site-specific scraping code committed.

You can either find some of this code elsewhere or write some scraping code for your site of choice on your own. I was planning to eventually split out the scraping API/spec as a separate library/spec that folks can implement scraper plugins against (i.e. just return data formatted a certain way and it will work). A site-specific scraper currently does not take very much code to implement -- the one I'm using is ~30 LoC, see an example below.

I'll document this at a later point, but if you want to use this immediately, here's some reference code below. Note that the metadata per item and the "api"/layout of different sites can vary widely, so the spec is also a WIP.

For now, allDrivers.js is just a wrapper file that looks like:

import * as site1Driver from './site1Driver.js'
import * as site2Driver from './site2Driver.js'

const drivers = {
  site1Driver,
  site2Driver
}

export default drivers

and a site-specific scraper driver might look like:

export const latestURL = 'http://sitelinkhere/?page='

export function getLatest ($) {
  return $('.classname')
    .map((index, el) => ({
      title: $(el).find...,
      cover: $(el).find('img').attr('src'),
      link: $(el).find('a').attr('href'),
      release: $(el).find...
    }))
    .get()
}

export function getChapters ($) {
  const title = $('selector1').text().trim()
  const chapters = $('selector2')
    .map((index, el) => ({
      link: $(el).attr('href').replace('//', 'http://'),
      title: $(el).text().match(/[0-9]+/)[0] || '0'
    }))
    .get()
  const tags = $('selector3')
    .map((index, el) => $(el).text().trim())
    .get()
  const summary = $('.some-classname').text().trim()
  return { title, chapters, tags, summary }
}

export function getPages ($) {
  return $('selector1')
    .map((index, el) =>
      $(el).attr('value').replace('//', 'http://')
    )
    .get()
}

export function getImage ($) {
  return $('imgselector').attr('src')
}

This repo in its WIP state currently only works as a plain latest reader client of a single site with no bells or whistles (i.e. no storage/caching besides that of the network layer, no search, etc)