TTLs not faithfully recorded on linux version

[fmarbach]

I am saving frame TTLs from 4 cameras to the digital input of the open ephys board.
Summary of problem: A single camera works, 2 cameras sort of, 3-4 cameras tend to show frequent gaps across all digital inputs (see plot titled "Timestamps of all channels"). A function generator records very faithfully alone (100hz square wave), but when acquired together with 2 cameras shows gaps as well. In the GUI, TTLs display fine, without gaps. The only thing that helped was switching to a windows machine.
Details:

• Ubuntu 20.04.4 LTS
• Open ephys gui v0.5.5.3
• Data is saved in binary, loaded using the open-ephys/open-ephys-python-tools
• This did not help: Updating ubuntu, open ephys gui, switching computer, recording to analog input
  
  

[jsiegle]

Thanks for the detailed report...we will try to replicate this on our end and will get back to you.

[jsiegle]

Hi @fmarbach, I ran the following test on my Ubuntu machine:

• Digital inputs 1-4 connected to output channels 1-4 of a Pulse Pal
• Pulse Pal configured to output 100 Hz, 83 Hz, 66 Hz, and 60 Hz pulses, with 1 ms pulse widths
• 128 channels of simultaneous headstage data

I had the same observation as you -- there were no gaps in the events displayed in the LFP Viewer, but when I looked at the event data that was written, there were gaps of up to 3 seconds. This was true for both the Binary Format and the Open Ephys Format, so it's not a format-specific problem.

To find the source of the issue, I programmed the File Reader to generate artificial events. Once I reached around 1500 events per buffer (regardless of whether they came from a single channel or multiple channels), the data file no longer contained all of the events.

I was able to fix the problem by increasing the size of the event queue (which passes events from the Record Node to the write thread). This was originally set at 512, which should be able to handle a theoretical event rate of 22 kHz for a 23 ms buffer size. However, it appears that on Linux the event queue is not emptied smoothly, which can lead to overflows, and causing the behavior we both observed.

After changing the event queue size to 10,000 (theoretical event rate of 434 kHz), I performed the same test. This time, there were no dropped events at all in an 8-minute recording.

We will put out a patch release ASAP with this fix. We really appreciate your help with this!

[fmarbach]

Hi Josh, Thanks a lot for the swift fix! I am happy you checked the different formats, I had neglected that and was going to check myself next week. Will have a look at the patch and hopefully happily return to linux! Fred

…

On Mar 10, 2022, at 6:00 PM, Josh Siegle ***@***.***> wrote: Hi @fmarbach <https://github.com/fmarbach>, I ran the following test on my Ubuntu machine: Digital inputs 1-4 connected to output channels 1-4 of a Pulse Pal Pulse Pal configured to output 100 Hz, 83 Hz, 66 Hz, and 60 Hz pulses, with 1 ms pulse widths 128 channels of simultaneous headstage data I had the same observation as you -- there were no gaps in the events displayed in the LFP Viewer, but when I looked at the event data that was written, there were gaps of up to 3 seconds. This was true for both the Binary Format and the Open Ephys Format, so it's not a format-specific problem. To find the source of the issue, I programmed the File Reader to generate artificial events. Once I reached around 1500 events per buffer (regardless of whether they came from a single channel or multiple channels), the data file no longer contained all of the events. I was able to fix the problem by increasing the size of the event queue (which passes events from the Record Node to the write thread). This was originally set at 512, which should be able to handle a theoretical event rate of 22 kHz for a 23 ms buffer size. However, it appears that on Linux the event queue is not emptied smoothly, which can lead to overflows, and causing the behavior we both observed. After changing the event queue size to 10,000 (theoretical event rate of 434 kHz), I performed the same test. This time, there were no dropped events at all in an 8-minute recording. We will put out a patch release ASAP with this fix. We really appreciate your help with this! — Reply to this email directly, view it on GitHub <open-ephys/plugin-GUI#506>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZIGX7P7Q36KY26YFJDCYLU7I2DRANCNFSM5QGYVYUQ>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.

[jsiegle]

Here are the test binaries for the patch release: https://openephysgui.jfrog.io/artifactory/Test/linux/open-ephys-v0.5.5.4-linux-beta.zip

Can you try out this version and let us know if it solves the problem?

[fmarbach]

Hi Josh, Thanks, this is great. i’ll be away for a few days, so will get back to you with testing wednesday next week. Fred

…

Sent from my iPhone

On Mar 11, 2022, at 2:34 AM, Josh Siegle ***@***.***> wrote:  Here are the test binaries for the patch release: https://openephysgui.jfrog.io/artifactory/Test/linux/open-ephys-v0.5.5.4-linux-beta.zip Can you try out this version and let us know if it solves the problem? — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

[fmarbach]

Hi Josh, just fyi i am delayed by about a week from now to test this, sick at home… will let you know once tested, Fred

…

On Mar 11, 2022, at 8:01 AM, Fred Marbach ***@***.***> wrote: Hi Josh, Thanks, this is great. i’ll be away for a few days, so will get back to you with testing wednesday next week. Fred Sent from my iPhone > On Mar 11, 2022, at 2:34 AM, Josh Siegle ***@***.***> wrote: > >  > > Here are the test binaries for the patch release: https://openephysgui.jfrog.io/artifactory/Test/linux/open-ephys-v0.5.5.4-linux-beta.zip <https://openephysgui.jfrog.io/artifactory/Test/linux/open-ephys-v0.5.5.4-linux-beta.zip> > Can you try out this version and let us know if it solves the problem? > > — > Reply to this email directly, view it on GitHub <open-ephys/plugin-GUI#506>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZIGXZUWOUHN7DJEIWDUFDU7KWLJANCNFSM5QGYVYUQ>. > Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. > You are receiving this because you were mentioned. >

[fmarbach]

Hi Josh, sorry for the delay with the test — i can confirm with this version i don’t see the dropped TTL issue. I will let you know if anything turns up still, but i assume for now this bug is fixed! Many thanks, Fred

…

On Mar 16, 2022, at 12:38 PM, Fred Marbach ***@***.***> wrote: Hi Josh, just fyi i am delayed by about a week from now to test this, sick at home… will let you know once tested, Fred > On Mar 11, 2022, at 8:01 AM, Fred Marbach ***@***.*** ***@***.***>> wrote: > > Hi Josh, > > Thanks, this is great. > i’ll be away for a few days, so will get back to you with testing wednesday next week. > > Fred > > Sent from my iPhone > >> On Mar 11, 2022, at 2:34 AM, Josh Siegle ***@***.*** ***@***.***>> wrote: >> >>  >> >> Here are the test binaries for the patch release: https://openephysgui.jfrog.io/artifactory/Test/linux/open-ephys-v0.5.5.4-linux-beta.zip <https://openephysgui.jfrog.io/artifactory/Test/linux/open-ephys-v0.5.5.4-linux-beta.zip> >> Can you try out this version and let us know if it solves the problem? >> >> — >> Reply to this email directly, view it on GitHub <open-ephys/plugin-GUI#506>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZIGXZUWOUHN7DJEIWDUFDU7KWLJANCNFSM5QGYVYUQ>. >> Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. >> You are receiving this because you were mentioned. >>

[jsiegle]

That's great, thanks for testing!

[anjaldoshi]

Closing as it is fixed in v0.5.5.4.