Use OpenWormReader to generate connectome

[travs]

This is closed when we are using PyOpenWorm's data via OpenWormReader (#46) to get the connectome we use in c302.

@ahrasheed

[pgleeson]

This really depends on openworm/owmeta#177 getting completed.

What's still lacking in the use of these multiple connectome data sources is a rigorous analysis, comparison and documentation of the different data sets, which will become more important as the new connectome is used. Unless there's a well informed paper trail showing that the project knows about the history of data produced in this area (and why we have chosen to use what we do), it will be difficult to convince c elegans physiologists we know what we're talking about...

Maybe there's a need for a separate https://github.com/openworm/ConnectomeDatasets project to document and make scripts to analyse/compare the different data sets out there and can be used as a basis/reference for what's used in PyOpenWorm?

[pgleeson]

Time to revisit this since there has been progress with openworm/owmeta#177.

Would it make sense to do this already though, before the latest changes in PyOpenWorm are merged to master? Anyone installing from master would get a different connectome than those installing from dev (or pip it seems)...

[slarson]

OK; so now we've got a release to master :)

So folks who have been waiting on this one maybe now we can get back to it. @ahrasheed are you up for it? I will also comment over on #27 which has more on this.

[pgleeson]

OpenWormReader.py has been now updated and is being successfully tested with the latest PyOpenWorm from master.

However, the OpenWormReader takes about 3 minutes to extract the connectivity data using PyOpenWorm. Below is the log from running on Travis-CI

$ python OpenWormReader.py
2016-01-29 14:54:45,436: OpenWormReader - INFO: Initialising OpenWormReader
2016-01-29 14:54:45,469: root - INFO: Connected to database
2016-01-29 14:54:55,024: OpenWormReader - INFO: Finished initialising OpenWormReader
2016-01-29 14:58:15,009: OpenWormReader - INFO: Total cells read 451
2016-01-29 14:58:15,009: OpenWormReader - INFO: Total connections read 7319

This needs to be sped up significantly if it's to be used for generating connections elsewhere. Also, there should be 2 methods (as in SpreadsheetDataReader.py) one for neurons->neurons and one for motorneurons->muscles

[slarson]

Awesome! Should this issue be closed then?

The performance issue is regrettable and has been known for a while: openworm/owmeta#90

It would be good to turn attention back to it now.

[pgleeson]

Technically this is done now, the next step of using PyOpenWorm/OpenWormReader in c302 is really covered here: #27. but that is unlikey to happen until performance is improved in some way...