.h5 file not found after preprocessing (motion correction) step

[rl72]

Hello,

I have been trying to preprocess a video, and in particular to apply motion correction to it (because I have done the other steps such as downsampling and spatially filtering it in the Inscopix Data Processing Software before exporting it as a .tiff file for motion correction and cell extraction) but I do not find the output .h5 file once the program runs turboreg (I have tried to save the motion corrected and the df/f movies) hence I can't further run cell extraction algorithms. Errors such as these appear :

Error using h5create (line 71)
The value of 'Dataset' is invalid. Expected DATASET to be nonempty.

Error in writeHDF5Data (line 74)
h5create(saveDir,options.datasetname,dataDims,'Datatype',inputClass,'ChunkSize',dataDimsChunkCopy,'Deflate',options.deflateLevel);

Error in calciumImagingAnalysis/modelPreprocessMovieFunction (line 645)
movieSaved = writeHDF5Data(thisMovie,savePathStr,'datasetname',options.outputDatasetName,'addInfo',{options.turboreg,analysisOptionStruct,optionsCopy},'addInfoName',{'/movie/processingSettings','/movie/analysisOperations','/movie/modelPreprocessMovieFunctionOptions'},'deflateLevel',options.deflateLevel)

Error in calciumImagingAnalysis/modelPreprocessMovie (line 37)
obj.modelPreprocessMovieFunction('folderListPath',folderListInfo,'fileFilterRegexp',options.fileFilterRegexp,'datasetName',options.datasetName,'frameList',[]);

Error in calciumImagingAnalysis/runPipeline (line 1582)
obj.(thisFxn{1});

Error in calciumImagingAnalysis/display (line 882)
obj.runPipeline;
@@@@@@@

I am new to this and not sure of what is causing this issue, what could I try to fix this?

Thanks a lot for your support and guidance,
Best wishes

[bahanonu]

Hi, hope all is well. To help with this, could you:

• go into the processing_info sub-folder located in the folder where your movie is and send over the output of the .*_preprocess.log file located there?
• run disp(ciapkg.version); in the MATLAB command window and let me know what the output is? Since your errors say calciumImagingAnalysis class whereas most recent version should be ciatah.

As a heads up:

• It is preferred if you perform spatial filtering after the motion correction, as the motion correction methods/algorithm uses a different spatial filter to enhance features in the movie that can be lost by doing another set of filtering beforehand, thus reducing accuracy of alignment.
• CIAtah can downsample Inscopix ISXD files and convert directly to HDF5 in batch (preferred), see https://bahanonu.github.io/ciatah/pipeline_detailed/#converting-inscopix-isxd-files-to-hdf5.

[rl72]

Hi, thanks a lot for your answer and for your advice regarding spatial filtering, I will try that out next time!

Regarding the version of the package, here is the output :
verStr =

'v3.26.2
 20210325200706'

v3.26.2

In fact, at first I used Ciatah but then I tried the CalciumAnalysisImaging version to see if maybe there was an incompatibility issue between Ciatah and my Matlab version which was older (2017), but the errors are the same in both cases. Here is a link to the log file for a video I tried preprocessing with Ciatah: https://file.re/2021/04/09/20210409094621preprocess/

Thanks again and best wishes

[rl72]

Hi, thanks a lot for your answer and for your advice regarding spatial filtering, I will try that out next time!

Regarding the version of the package, here is the output :
verStr =

'v3.26.2
20210325200706'
v3.26.2

In fact, at first I used Ciatah but then I tried the CalciumAnalysisImaging version to see if maybe there was an incompatibility issue between Ciatah and my Matlab version which was older (2017), but the errors are the same in both cases. Here is the log file :
20210409094621preprocess.txt

Thanks and best wishes

[bahanonu]

@rl72, it looks like both input and output HDF5 dataset names are blank, which is causing the HDF5 save function to fail, e.g. see below in the log file you sent:

                   datasetName: [1×0 char]
             outputDatasetName: [1×0 char]

After creating the CIAtah object (e.g. obj = ciatah;) either run obj.setMovieInfo and ensure the 3rd option is /1 or run the below commands:

obj.outputDatasetName = '/1';
obj.inputDatasetName = '/1';

This should allow you to run modelPreprocessMovieFunction and get movies output fine. Let me know if that works on your end.

[rl72]

Hi Biafra,

Thanks a lot for your input, everything worked out in the end nicely! I ran CNMFe and EXTRACT smoothly, but I noticed that when running EXTRACT through the CIAtah pipeline, it was possible to modify only a few parameters (the essential ones, such as choosing between GPU/CPU, setting the SNR for cell identification, the number of partitions etc) so I was wondering if there was a way to set more advanced parameters to run EXTRACT in CIAtah?

Thanks and best wishes

[bahanonu]

@rl72 Great! That's good to hear.

re: EXTRACT, yep, I was planning to push an update soon, similar to the CNMF-e sub-module, to allow access to more (all) advanced parameters.

[rl72]

Oh that would be amazing indeed, can't wait to try it out when it is updated then !

On the same topic, I tried running the CIAtah cell sorter on the EXTRACT output (which I ran separately in advance). I used the code provided in the EXTRACT repository but I kept receiving the following error messages in a loop for a very long time until it runs out of memory :

@@@@@@@
Index exceeds matrix dimensions.

Error in compareSignalToMovie (line 101)
nPoints = options.inputMovieDims(3);

Error in signalSorter>viewMontage (line 2463)
croppedPeakImages = compareSignalToMovie(inputMovie, inputImage, thisTrace,'getOnlyPeakImages',1,'waitbarOn',0,'extendedCrosshairs',2,'crossHairVal',maxValMovie*options.crossHairPercent,'outlines',1,'signalPeakArray',signalPeakArray,'cropSize',cropSizeLength,'addPadding',1,'outlineVal',NaN,'xCoords',xCoords,'yCoords',yCoords,'inputDatasetName',options.inputDatasetName,'inputMovieDims',options.inputMovieDims,'hdf5Fid',options.hdf5Fid,'keepFileOpen',options.keepFileOpen);

Error in signalSorter>chooseSignals (line 981)
[objCutImagesCollection{i}, ~] = viewMontage(options.inputMovie,thisImage,options,thisTrace,[signalPeakIdx{i}],minValMovie,maxValMovie,options.cropSizeLength,i,1);

Error in signalSorter (line 608)
[choices, safeExit] = chooseSignals(options,1:nSignals, inputImages,inputSignals,objMap, valid, inputStr,tmpDir,sessionID,signalPeakIdx,signalSnr,inputImagesThres,inputImageSizes,peakOutputStat,ROItraces,imgStats,inputSignalSignal,inputSignalNoise,inputImagesBoundaryIndices,signalPeakIdxOriginal,signalPeaksOriginal,instructionStr);
@@@@@@@
@@@@@@@
Index exceeds matrix dimensions.

Error in signalSorter>chooseSignals (line 1039)
title(['imageCorr = ' num2str(round(peakOutputStat.outputMeanImageCorrs(i)*sigDig)/sigDig) 10 'Press Y to go to last sorted signal.'],'FontSize',options.fontSize,'Interpreter','tex');

Error in signalSorter (line 608)
[choices, safeExit] = chooseSignals(options,1:nSignals, inputImages,inputSignals,objMap, valid, inputStr,tmpDir,sessionID,signalPeakIdx,signalSnr,inputImagesThres,inputImageSizes,peakOutputStat,ROItraces,imgStats,inputSignalSignal,inputSignalNoise,inputImagesBoundaryIndices,signalPeakIdxOriginal,signalPeaksOriginal,instructionStr);
@@@@@@@
0 0
cellmap type: max
@@@@@@@
Index exceeds matrix dimensions.

Error in signalSorter>chooseSignals (line 1271)
' | imageCorr = ' num2str(round(peakOutputStat.outputMeanImageCorrs(i)*sigDig)/sigDig) ',' num2str(peakOutputStat.outputMeanImageCorrs2(i))...

Error in signalSorter (line 608)
[choices, safeExit] = chooseSignals(options,1:nSignals, inputImages,inputSignals,objMap, valid, inputStr,tmpDir,sessionID,signalPeakIdx,signalSnr,inputImagesThres,inputImageSizes,peakOutputStat,ROItraces,imgStats,inputSignalSignal,inputSignalNoise,inputImagesBoundaryIndices,signalPeakIdxOriginal,signalPeaksOriginal,instructionStr);
@@@@@@@
All warnings have the state 'off'.
calculating movie min/max...
finding centroids...
Finding cell neighbors: 2|100
Done assigning neighbor IDs...

ans =

'Frame 1/22725'

Would you know why is this happening?

Thanks a lot!

[bahanonu]

@rl72 Did you run signalSorter directly from the command line? If so, can you provide the command(s) you used to run it? That'll help me debug.

[rl72]

yes, I ran the following :

iopts.inputMovie = M;
iopts.valid = 'neutralStart';
iopts.cropSizeLength = 20;
iopts.cropSize = 20;
iopts.medianFilterTrace = 0;
iopts.subtractMean = 0;
iopts.backgroundGood = [208,229,180]/255;
iopts.backgroundBad = [244,166,166]/255;
iopts.backgroundNeutral = repmat(230,[1 3])/255;

[inputImagesSorted, inputSignalsSorted, choices] = signalSorter(output.spatial_weights, output.temporal_weights', 'options',iopts);

[bahanonu]

Thanks. Can you send the following info:

• Matlab version.
• Size of M, e.g. disp(size(M));
• Output on the command window before the errors begin.

[rl72]

Yes of course :

• Matlab R2018b
• disp(size(M));
  1 11
• and as for the output before error starts, please find the screenshots attached
  
  

[bahanonu]

Great, thanks. Looks like M is not in the correct format (x y nFrames). Is that a path to a movie? If so, give the full path rather than just the filename. Else if it is the movie itself, the dimensions do not match the output from EXTRACT.

[rl72]

If the full path of my file is "D:\lumi_data\data_day4\trial.h5", should I run M as = ('D:\lumi_data\data_day4\trial.h5:/1') ? If yes, it is not working either.

[bahanonu]

Try below and let me know if that works for you. If you run into issues, run h5disp('D:\lumi_data\data_day4\trial.h5') to make sure dataset name is /1.

iopts.inputMovie = 'D:\lumi_data\data_day4\trial.h5';
iopts.readMovieChunks = 1; % Binary: 1 = read movie from HDD, 0 = load entire movie into RAM
iopts.inputDatasetName = '/1';

CIAtah does not accept filename and HDF5 dataset name (as with EXTRACT currently) as a single string but has separate variables for each (to avoid certain issues that can arise).

[rl72]

Oh it did work now, thank you so much! There are a few minor errors that appeared (please see below) but the cell sorting interface seems to be otherwise working fine. Thanks again for taking the time to help with this!

[bahanonu]

Great! Glad to hear it is working. Let me know if any other questions come up.

Those warnings should not affect function, but I'll double check in my copy of 2018b (normally testing in Matlab 2019+ these days) to see if they can be removed.