Spider/Crawler written to debug sites and TRACK LINKS.
It's annoying when a spider says there was an error fetching a page but doesn't say what page that bad link was on!
It's written in C# C-Sharp
To get started you probably want:
branches/new-dev/installer/NIE_Spider.msi
TODO:
- I think trunk vs. new-dev branch should be swapped
- Not sure of doc