Spike2IO cannot read smrx recordings

[whbdupree]

I am unable to read recordings in the .smrx format

[whbdupree]

In [1]: from neo.io import Spike2IO
r
In [2]: r=Spike2IO(filename='test.smrx')

In [3]: x=r.read(cascade=True,lazy=False)

KeyError Traceback (most recent call last)
in ()
----> 1 x=r.read(cascade=True,lazy=False)

/usr/lib/python3.6/site-packages/neo/io/baseio.py in read(self, lazy, cascade, **kargs)
119 if not cascade:
120 return bl
--> 121 seg = self.read_segment(lazy=lazy, cascade=cascade, **kargs)
122 bl.segments.append(seg)
123 bl.create_many_to_one_relationship()

/usr/lib/python3.6/site-packages/neo/io/spike2io.py in read_segment(self, take_ideal_sampling_rate, lazy, cascade)
90 """
91
---> 92 header = self.read_header(filename=self.filename)
93
94 # ~ print header

/usr/lib/python3.6/site-packages/neo/io/spike2io.py in read_header(self, filename)
195 channelHeader += HeaderReader(fid, np.dtype(dt))
196
--> 197 channelHeader.type = dict_kind[channelHeader.kind]
198 #~ print i, channelHeader
199 channelHeaders.append(channelHeader)

KeyError: 215

[samuelgarcia]

Yes. Only smr are supported.
I don't known how much smr and smrx are different. If it is only some conversion 32bit to 64bit this should be easy to adapt. But if smrx is a completely new file format then it is a piece of work.

Do you have small files you could give and specification of the format ?

[whbdupree]

I can't share any data from my current project for reasons; best not to use these at all to be safe. I will see if I can get a short snippet from an unrelated project.

…

On Oct 27, 2017 3:15 AM, "Garcia Samuel" ***@***.***> wrote: Yes. Only smr are supported. I don't known how much smr and smrx are different. If it is only some conversion 32bit to 64bit this should be easy to adapt. But if smrx is a completely new file format then it is a piece of work. Do you have small files you could give and specification of the format ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#424 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIfZbk3nXjdsYcfXOT-cPtF7Ksc6AgH9ks5swYL4gaJpZM4QIIcF> .

[samuelgarcia]

If not too difficult, this snippet should be as small as possible and should contains all possible objects (signals, spikes, event, epoch..). Thanks.

[whbdupree]

Well I am totally unable to work with .smrx files at the moment. Do you have a sample of such a .smr file that you could upload or send to me? I could then have a colleague convert it to .smrx.

…

On Oct 27, 2017 5:49 AM, "Garcia Samuel" ***@***.***> wrote: If not too difficult, this snippet should be as small as possible and should contains all possible objects (signals, spikes, event, epoch..). Thanks. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#424 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIfZbpP9xHvOmnv9AYlWb_DOAzJ1ZGwyks5swacwgaJpZM4QIIcF> .

[sLopezdeDiego]

Does any of you have a specification of the smrx file format? I would like to add support for this, but I have not been able to find any documentation for smrx.

[samuelgarcia]

I still do not have theses specifications.
maybe a plexon custumers could ask for them diretcly ?

[neuralcircuitsTV]

It would be very useful to be able to read .smrx files, as these are now standard for Spike2 data acquisition. I can provide a file if helpful. This is taken from the Spike2 v9 help files:

The 64-bit filing system (*.smrx)

The new 64-bit filing system was designed to be logically compatible with the 32-bit system; by this we mean that it can hold the same types of data as the original without information loss, though some of these types are extended. It is also likely that we will add further data types, as required, in the future. Features include:

1.Times are stored as 64-bit integers. At a time resolution of 1 ns (nanosecond, 10-9 seconds), the maximum duration is around 256 years.

2.The file size is limited only by the capabilities of the operating system and by the size of a file that you can manage conveniently for archival. As I write this disk drives have a maximum size of a few TB. The file format is designed to make recovery of damaged files relatively straightforward.

3.The files have a built-in data lookup system designed to minimise the number of disk reads required to locate data on any channel.

4.The overhead for severely fragmented waveform data has been reduced to a few bytes per fragment.

We have removed many of the limits on things like the number of channels in a file and the length of channels comments and units. Before Spike2 version 8.03 the program enforced the original limits on the length of strings (but treated as Unicode characters). From version 8.03 onwards, a 64-bit smrx file in Spike2 can have 20 Unicode characters of channel title, 10 of channel units and comments can be up to 100 characters. The underlying file format does not impose a fixed limit, but practical considerations make it useful to define them.

[samuelgarcia]

Dear Tim,
I am the original contributor of Spike2IO

I agree. This smrx format have to be incorporated in neo.

To develop a new format a developer need:

• full specficiation of the format
• small file for the test chain
• time
• help
• personnal need of the format

At the moment, I don't have any of theses.
But maybe, if you have time we could collaborate on this format.

Cheers

Samuel

[neuralcircuitsTV]

Dear Samuel
Thanks, this would be great, and I know researchers who would benefit from being able to directly import smrx files. If there is some time in the future, I put a short example data file in a repo, lfp-juxta. Happy to collaborate. Best, Tim

[bendichter]

@samuelgarcia

We'd like to offer conversion from smrx to NWB, and I think the best way to offer this would be:
SMRX -> NEO -> SpikeExtractors -> NWB

I was able to find some code to do this in MATLAB as part of the Brainstorm package here in case that is useful.

I found a link for Python code here but the form threw an error. It also says this code only works on Windows, which I know won't work for many NWB users, but at least it's a start.

edit: The link is now fixed, and leads to a .exe which I cannot open on OSX Catalina using wine. I'm guessing it's 32-bit

[samuelgarcia]

Hi ben.

2 independent answers:

• For pure file conversion, in my mind the chain "SMRX -> NEO -> SpikeExtractors -> NWB" is not optimal. I think the
  "SMRX -> NEO -> NWB" would be better, the neo API is really more sophisticated than the spikeextractors layer.
  Neo handle : multi segment -aka pauses in recording- , multiple sampling rate, annotation, array annotaion, epch, event, ....
  The spikeinterface layer for IO for design with a unique goal : spikesorting. The neo layer is more universal to handle datasets
  The central core/strength of neo is "handle datset with relationship between object". This is crutial to cover all dataset conversion
  details.

• For the srmx, as I said in April I need :

  • full specficiation of the format >>> maybe the code you mention can help for this
  • small file for the test chain >>> if you work on this conversion maybe you can provide a small file here
  • time >>> I am back from vaccation, I think it is OK now.
  • help >>> could you help me on this ?

Cheers

[bendichter]

@samuelgarcia

I agree, fewer steps in the conversion is better. I haven't been able to try out #796 yet, but if that works for this use-case I agree that would be better, particularly since it supports more flexibility and types of data than SpikeInterface.

That being said, we also want to use SpikeInterface spike sorting pipelines on SMRX data, so we'd need some path that connects SMRX to SpikeExtractor regardless of if it is our preferred conversion strategy.

I've been working on the requirements you mentioned:

• full specification of the format - I have not been able to find this and I think it is unlikely that we will be able to get it. We have been able to get a pre-compiled Python API with some example python code for usage of that API, but even the API itself is not open. How does this map onto the scope of NEO? Do you support reading of closed formats with provided API libraries? I'm not crazy about it myself.
• For example data, I found this file, here but I can't verify its quality
• For time and help, we are working on a contract that would allow us to direct some person-hours towards this problem. Will keep you updated as this develops

[bendichter]

@samuelgarcia CED just released a pip-installable API: https://pypi.org/project/sonpy/

[bendichter]

@eduardacenteno is interested in working on this. See https://github.com/SpikeInterface/spikeextractors/blob/2881a5973f1648de665049868c8c800c1166c621/spikeextractors/extractors/cedextractors/utils.py#L54

[samuelgarcia]

Close by #987