Refactor DataLayer into DataSource and DataProcessing layers

[sergeyk]

Right now, DataLayer does image-specific processing of LevelDB data.
It should be separated into LevelDBDataSourceLayer, and ImageProcessingLayer.
That way, other data sources can more easily be added: HDF5, image directories, CSV, etc.

[kloudkl]

Would you like to create the first DataSourceLayer for the HDF5DataLayer in #147?

[kloudkl]

But for backward compatibility and not breaking all the existing user codes, we probably need to have both versions for a while and deprecate the old version after the end of an announced maintenance period. Shall we set a version release schedule from now on?

[shelhamer]

@kloudkl we're still in a "break it now so it sets right" stage of development, so we aren't ready for versioned releases, but we'll do our best to make the transition comfortable.

[sergeyk]

@kloudkl no, I think #147 should be merged before this refactoring

[kloudkl]

Let me make it more clear. I did not mean that #147 should be refactored before being merged.

Switching easily among different data sources using the same processing layer will greatly reduce code redundancy and is a feature that many users would embrace wholeheartedly. What I asked for is a concrete example of the design that @sergeyk was thinking about.

[kloudkl]

Not everything has to be a separate layer considering the significant success of #128. The DataProcessing stuffs fit more naturally in a processing pipeline which should stay in the refactored data layer to avoid redundant memory copy.

The data source should be a plain field of data layer similarly to avoid creating too many kinds of layers. The data source will be defined in the proto and instantiated by a data source factory. The major methods of the DataSource base class that I can think of are has_next_batch, next_batch, load and save. Any better ideas?

[futurely]

A series of recent collaborative efforts to refactor the data layers have completed this task.