New Feature Requests and Possible Enhancements

Ideas to include/enhance the application

• Parse Options from the commandline and add non-UI mode.

• Maybe add 500px, instagram, flickr, pinterest or other tumblr similar pages?

• Redesign the UI layout and the controls.

• Re-download of a blog.

• File rename function. Multiple possible implementation variants:

  • Freely defined pattern based on user input.
  • After posts name.
  • Add a leading number indicating the blogs post order.

• Implement a filesystem watcher that reads .txt files from a user-definable folder with new instructions. E.g. download blog X with tags Y now.

• Theme support.

• Move the blog index (i.e. .tumblr) files to a separate, user definable folder?

• Linux (G)UI

Code Enhancements

Possible things to improve.

• Code refactoring.
• Unit tests.
• Documentation (code, user manuals, web page).
• Redesign and clean up the database structures?
  • We could implement a FolderBrowser that allows to browse through the filesystem from the main user interface, thus working like a FTP client. It loads and updates the blog manager with the databases found in the corresponding directory and you can submit them to the queue.
• Use more events driven code, for example in the AbstractDownloader.cs
• In the AbstractDownloader.cs and Crawler classes, use TPL Dataflow/Actor model instead of creating a huge list of Tasks using a blockingcollection/getconsumingenumerable.
• Use HttpClient instead of HttpWebRequests for downloading.
• Better handling of the TimeoutExceptions that were introduced to terminate dead connections in wonky internet connections. I.e. notify the user that a connection dropped and act accordingly.
• ..

How to add new website crawler

It should be quite straight forward now to add new sites like 500px, instagram, or twitter. The downloader and most of the UI should be able to take different websites.

You can check this commit for my addition of the Tumblr tag search downloader for the most recent example. Older examples include the Tumblr liked-by downloader, the Tumblr downloader for private blogs and the Tumblr search downloader.

In essence, you have to do:

• Implement the ICrawler interface and override the Crawl method to start the crawler and a DownloadBlogAsync task from the IDownloader interface. See here for an example implementation in the TumblrTagSearchCrawler.
• The url validator needs adjustments to detect proper urls.
• Add your BlogType.
• Add your Crawler to the CrawlerFactory.
• Add your BlogType to the BlogFactory.
• You might want to add a new DetailsView.cs if you want different checkboxes or statistics.