Skip to content
NikoKrause edited this page Nov 18, 2018 · 14 revisions

Steven Selph's Scraper

This scraper downloads images and metadata primarily from online databases like thegamesdb for use with EmulationStation It uses hashes of roms to provide the most accurate results.

scraped

Supported Systems:

A complete list is HERE.

Platforms Supported:

Anything you can compile Go on. I provide precompiled binaries for:

  • Windows
  • Mac
  • Linux
  • Raspberry Pi

Note that if you are using RetroPie, the scraper can easily be used and installed through the RetroPie Setup Script - you can see the documentation for it HERE

Installing:

You can install compiled binaries from the releases page HERE

you can also compile from source as described in the README

Usage

$ cd <rom directory>
$ scraper

ROMs will be scanned and a gamelist.xml file will be created. All images will be placed inside the images folder.

All Systems

$ scraper -scrape_all -thumb_only

MAME/FBA

The scraper now supports MAME/FBA but using file names instead of hashing. Since it uses a different DB and lookup method, several of the command line flags no longer apply. When the -mame flag is used it disables all other databases and the mamedb only has the one size of image so the flags about thumbnails, gdb, ovgdb, etc don't do anything.

$ scraper -mame

You can choose your preference of image type with the mame_img flag. If you prefer marquees but want to fallback to titles then snapshots you can do:

$ scraper -mame -mame_img "m,t,s"

Command Line Flags

There are several command flags you can pass. To see a full list use -help

$ scraper -help

Full List:

Usage of scraper:
  -add_not_found
    	If true, add roms that are not found as an empty gamelist entry.
  -append
    	If the gamelist file already exists, skip files that are already listed and only append new files.
  -console_img string
    	Comma-separated order to prefer images, s=snapshot, b=boxart, f=fanart, a=banner, l=logo, 3b=3D boxart, cart=cartridge, clabel=cartridge label, mix3=Standard 3 mix, mix4=Standard 4 mix. (default "b")
  -console_src string
    	Comma-separated order to prefer console sources, ss=screenscraper, ovgdb=OpenVGDB, gdb=theGamesDB (default "gdb")
  -convert_videos
    	If true, convert videos for the Raspberry Pi (e.g. 320x240@30fps) NOTE: This needs ffmpeg installed
  -download_images
    	If false, don't download any images, instead see if the expected file is stored locally already. (default true)
  -download_marquees
    	If true, download marquees.
  -download_videos
    	If true, download videos.
  -extra_ext string
    	Comma-separated list of extensions to also include in the scraper.
  -hash_file file
    	The file containing hash information.
  -image_dir directory
    	The directory to place downloaded images to locally. (default "images")
  -image_path path
    	The path to use for images in gamelist.xml. If scrape_all is used, only image_dir is used. (default "images")
  -image_suffix suffix
    	The suffix added after rom name when creating image files. (default "-image")
  -img_format jpg or png
    	jpg or png, the format to write the images. (default "jpg")
  -img_workers N
    	Use N worker threads to process images. If 0, then use the same value as workers.
  -lang string
    	The order to choose for language if there is more than one for a value. (en, fr, es, de, pt) (default "en")
  -mame
    	If true we want to run in MAME mode.
  -mame_img string
    	Comma-separated order to prefer images, s=snap, t=title, m=marquee, c=cabinet, b=boxart, 3b=3D-boxart, fly=flyer. (default "t,m,s,c")
  -mame_src string
    	Comma-separated order to prefer mame sources, ss=screenscraper, adb=arcadeitalia, mamedb=mamedb-mirror, gdb=theGamesDB-neogeo (default "adb,gdb")
  -marquee_dir directory
    	The directory to place downloaded marquees to locally. (default "images")
  -marquee_format jpg or png
    	jpg or png, the format to write the marquees. (default "png")
  -marquee_path path
    	The path to use for marquees in gamelist.xml. If scrape_all is used, only marquee_dir is used. (default "images")
  -marquee_suffix suffix
    	The suffix added after rom name when creating marquee files. (default "-marquee")
  -max_height height
    	The max height of images. Larger images will be resized.
  -max_width width
    	The max width of images. Larger images will be resized. (default 400)
  -missing file
    	The file where information about ROMs that weren't scraped is added.
  -nested_img_dir
    	Use a nested img directory structure that matches rom structure.
  -no_thumb
    	Don't add thumbnails to the gamelist.
  -output_file file
    	The XML file to output to. If scrape_all is used, this is ignored and the gamelist in the system path. (default "gamelist.xml")
  -overview_len N
    	If set it will truncate the overview of roms to N characters + ellipsis.
  -refresh
    	Information will be attempted to be downloaded again but won't remove roms that are not scraped.
  -region string
    	The order to choose for region if there is more than one for a value. xx is a special region that will choose any region. (default "us,wor,eu,jp,fr,xx")
  -retries N
    	Retry a rom N times on an error. (default 2)
  -rom_dir directory
    	The directory containing the roms file to process. (default ".")
  -rom_path path
    	The path to use for roms in gamelist.xml. (default ".")
  -scrape_all
    	If true, scrape all systems listed in es_systems.cfg. All dir/path flags will be ignored.
  -skip_check
    	Skip the check if thegamesdb.net is up.
  -ss_password password
    	The password for registered ScreenScraper users.
  -ss_user username
    	The username for registered ScreenScraper users.
  -strip_unicode
    	If true, remove all non-ascii characters.
  -thumb_only
    	Download the thumbnail for both the image and thumb (faster).
  -thumb_suffix suffix
    	The suffix added after rom name when creating thumb files. (default "-thumb")
  -update_cache
    	If false, don't check for updates on locally cached files. (default true)
  -use_filename
    	If true, use the filename minus the extension as the game title in xml.
  -use_nointro_name
    	Use the name in the No-Intro DB instead of the one in the GDB. (default true)
  -version
    	Print the release version and exit.
  -video_dir directory
    	The directory to place downloaded videos to locally. (default "images")
  -video_path path
    	The path to use for videos in gamelist.xml. If scrape_all is used, only video_dir is used. (default "images")
  -video_suffix suffix
    	The suffix added after rom name when creating video files. (default "-video")
  -workers N
    	Use N worker threads to process roms. (default 1)