New Feature Requests and Possible Enhancements

Ideas to include/enhance the application

• Parse Options from the commandline and add non-UI mode.

• Maybe add 500px, instagram, flickr, pinterest or other tumblr similar pages?

• Redesign the UI layout and the controls.

• Implement a filesystem watcher that reads .txt files from a user-definable folder with new instructions. E.g. download blog X with tags Y now.

• Theme support.

• Move the blog index (i.e. .tumblr) files to a separate, user definable folder?

• Linux (G)UI

Code Enhancements

Possible things to improve.

• Code refactoring.
• Unit tests.
• Documentation (code, user manuals, web page).
• Redesign and clean up the database structures?
  • We could implement a FolderBrowser that allows to browse through the filesystem from the main user interface, thus working like a FTP client. It loads and updates the blog manager with the databases found in the corresponding directory and you can submit them to the queue.
• Use more events driven code, for example in the AbstractDownloader.cs
• In the AbstractDownloader.cs and Crawler classes, use TPL Dataflow/Actor model instead of creating a huge list of Tasks using a blockingcollection/getconsumingenumerable.
• Use HttpClient instead of HttpWebRequests for downloading.
• Better handling of the TimeoutExceptions that were introduced to terminate dead connections in wonky internet connections. I.e. notify the user that a connection dropped and act accordingly.
• ..

How to add new website crawler

It should be quite straight forward now to add new sites like 500px, instagram, or twitter. The downloader and most of the UI should be able to take different websites.

You can check this commit for the addition of the Twitter downloader as the most recent example. This commit also contains code refactorings to make the naming of the classes and methods more general. You can also check this commit for the addition of the Tumblr tag search downloader as the previous example. Older examples include the Tumblr liked-by downloader, the Tumblr downloader for private blogs and the Tumblr search downloader.

In essence, you have to do:

• Implement the ICrawler interface and override the Crawl method to start the crawler and a DownloadBlogAsync task from the IDownloader interface. See here for an example implementation in the TumblrTagSearchCrawler.
• The url validator needs adjustments to detect proper urls.
• Add your BlogType.
• Add your Crawler to the CrawlerFactory.
• Add your BlogType to the BlogFactory.
• You might want to add a new DetailsView.cs if you want different checkboxes or statistics.