error with job_kwargs in ironclust

[paul-aparicio]

Hello! I am trying to run ironclust and I get the following error:

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[29], line 2
      1 # run sorter
----> 2 sorting_IC = si.run_sorter_by_property(sorter_name='ironclust', recording=recording_cache, grouping_property='group', 
      3                                        folder=pth+sess+'processed/results_ic', verbose=False, **params)
      5 print(f'Ironclust found {len(sorting_IC.get_unit_ids())} units')

File /media/paul/storage/pdev/spikeinterface/src/spikeinterface/sorters/launcher.py:308, in run_sorter_by_property(sorter_name, recording, grouping_property, folder, mode_if_folder_exists, engine, engine_kwargs, verbose, docker_image, singularity_image, working_folder, **sorter_params)
    297     job = dict(
    298         sorter_name=sorter_name,
    299         recording=rec,
   (...)
    304         **sorter_params,
    305     )
    306     job_list.append(job)
--> 308 sorting_list = run_sorter_jobs(job_list, engine=engine, engine_kwargs=engine_kwargs, return_output=True)
    310 unit_groups = []
    311 for sorting, group in zip(sorting_list, recording_dict.keys()):

File /media/paul/storage/pdev/spikeinterface/src/spikeinterface/sorters/launcher.py:106, in run_sorter_jobs(job_list, engine, engine_kwargs, return_output)
    103 if engine == "loop":
    104     # simple loop in main process
    105     for kwargs in job_list:
--> 106         sorting = run_sorter(**kwargs)
    107         if return_output:
    108             out.append(sorting)

File /media/paul/storage/pdev/spikeinterface/src/spikeinterface/sorters/runsorter.py:199, in run_sorter(sorter_name, recording, folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, output_folder, **sorter_params)
    188             raise RuntimeError(
    189                 "The python `spython` package must be installed to "
    190                 "run singularity. Install with `pip install spython`"
    191             )
    193     return run_sorter_container(
    194         container_image=container_image,
    195         mode=mode,
    196         **common_kwargs,
    197     )
--> 199 return run_sorter_local(**common_kwargs)

File /media/paul/storage/pdev/spikeinterface/src/spikeinterface/sorters/runsorter.py:259, in run_sorter_local(sorter_name, recording, folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, output_folder, **sorter_params)
    257 SorterClass.set_params_to_folder(recording, folder, sorter_params, verbose)
    258 # This writes parameters and recording to binary and could ideally happen in the host
--> 259 SorterClass.setup_recording(recording, folder, verbose=verbose)
    260 # This NEEDS to happen in the docker because of dependencies
    261 SorterClass.run_from_folder(folder, raise_error, verbose)

File /media/paul/storage/pdev/spikeinterface/src/spikeinterface/sorters/basesorter.py:234, in BaseSorter.setup_recording(cls, recording, output_folder, verbose)
    232     all_params = json.load(f)
    233     sorter_params = all_params["sorter_params"]
--> 234 cls._setup_recording(recording, sorter_output_folder, sorter_params, verbose)

File /media/paul/storage/pdev/spikeinterface/src/spikeinterface/sorters/external/ironclust.py:173, in IronClustSorter._setup_recording(cls, recording, sorter_output_folder, params, verbose)
    171 dataset_dir = sorter_output_folder / "ironclust_dataset"
    172 # Generate three files in the dataset directory: raw.mda, geom.csv, params.json
--> 173 MdaRecordingExtractor.write_recording(
    174     recording=recording, save_path=str(dataset_dir), verbose=False, **get_job_kwargs(params, verbose)
    175 )
    177 samplerate = recording.get_sampling_frequency()
    178 num_channels = recording.get_num_channels()

File /media/paul/storage/pdev/spikeinterface/src/spikeinterface/extractors/mdaextractors.py:103, in MdaRecordingExtractor.write_recording(recording, save_path, params, raw_fname, params_fname, geom_fname, dtype, **job_kwargs)
     66 @staticmethod
     67 def write_recording(
     68     recording,
   (...)
     75     **job_kwargs,
     76 ):
     77     """Write a recording to file in MDA format.
     78 
     79     Parameters
   (...)
    101             * progress_bar
    102     """
--> 103     job_kwargs = fix_job_kwargs(job_kwargs)
    104     assert recording.get_num_segments() == 1, (
    105         "MdaRecording.write_recording() can only write a single segment " "recording"
    106     )
    107     save_path = Path(save_path)

File /media/paul/storage/pdev/spikeinterface/src/spikeinterface/core/job_tools.py:67, in fix_job_kwargs(runtime_job_kwargs)
     64 job_kwargs = get_global_job_kwargs()
     66 for k in runtime_job_kwargs:
---> 67     assert k in job_keys, (
     68         f"{k} is not a valid job keyword argument. " f"Available keyword arguments are: {list(job_keys)}"
     69     )
     71 # remove mutually exclusive from global job kwargs
     72 for k, v in runtime_job_kwargs.items():

AssertionError: verbose is not a valid job keyword argument. Available keyword arguments are: ['n_jobs', 'total_memory', 'chunk_size', 'chunk_memory', 'chunk_duration', 'progress_bar', 'mp_context', 'max_threads_per_process']

I have tried explicitly setting the verbose option in run_sorter, and using set_global_job_kwargs, but I get the same error. Thanks for any help!

[zm711]

This seems like a bug to me. We recently removed verbose as a job_kwarg in general, but maybe we missed a spot. Just to confirm what version of spikeinterface are you running?

[chrishalcrow]

Hey @paul-aparicio , I think I found the bug and have proposed a fix. As @zm711 said, it's to do with the fact that verbose used to be classed as a job_kwarg, but no more! Would you be able to checkout PR #3535 and see if it fixes the problem?

[paul-aparicio]

Hi, thanks for your responses! When I run si.version is returns v0.102.0, though I downloaded the latest version of main on 11/11. Sorry. I am new to working with git, so i may take me some time to figure out how to download the patch and get it to work, but I will try! Thanks again.

[chrishalcrow]

Hi, thanks for your responses! When I run si.version is returns v0.102.0, though I downloaded the latest version of main on 11/11. Sorry. I am new to working with git, so i may take me some time to figure out how to download the patch and get it to work, but I will try! Thanks again.

No worries. To get the most up to date release you can download the github repository by going to a place you'd like to store github repos and running
git clone https://github.com/SpikeInterface/spikeinterface.git
This basically downloads the spikeinterface github repo: you should now see a folder called spikeinterface. Go into this directory and install this version of spikeinterace using e.g.
pip install -e .
Now you want to try out the PR I made. There's a few ways to do this. My preference is to get the github cli: https://cli.github.com/ and then running gh pr checkout 3535 while in the spikeinterface folder. Some more details here: https://spikeinterface.readthedocs.io/en/latest/get_started/installation.html#from-source

If you have any problems, you can ping me here, or try and find me somewhere else (slack, e-mail)!

[paul-aparicio]

Thanks again for your help. I installed spikeinterface and the cli.github tool as you suggested and then tried running gh pr checkout 3535 in the spikeinterface folder but I get the error:

could not determine base repo: fatal: not a git repository (or any of the parent directories): .git
/snap/gh/502/usr/bin/git: exit status 128

I checked to make sure there was a .git folder. I also tried running git initand made sure I was logged in.

[chrishalcrow]

Thanks for trying to get this to work (the first time is always painful!). When I go to my spikeinterface git directory from the terminal (by running cd /whatever/the/path/is/to/spikeinterface) and run git status, the following is printed:

On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

Does the same happen for you?

[paul-aparicio]

Hi, once I ran git status, I got the same output and now everything worked. I ran the code and got the following error about cpython not being installed:

/media/paul/storage/pdev/spikeinterface/src/spikeinterface/extractors/mdaextractors.py:130: DeprecationWarning: auto_cast_uint is deprecated and will be removed in 0.103. Use the `unsigned_to_signed` function instead.
  write_binary_recording(
/media/paul/storage/pdev/spikeinterface/src/spikeinterface/core/recording_tools.py:151: ResourceWarning: unclosed file <_io.TextIOWrapper name='/media/paul/Test/TY20210401_640_signalCheck_morning/processed/results_ic/0/sorter_output/ironclust_dataset/raw.mda' mode='r+' encoding='UTF-8'>
  executor.run()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
---------------------------------------------------------------------------
SpikeSortingError                         Traceback (most recent call last)
Cell In[13], line 2
      1 # run sorter
----> 2 sorting_IC = si.run_sorter_by_property(sorter_name='ironclust', recording=recording_cache, grouping_property='group', 
      3                                        folder=pth+sess+'processed/results_ic', verbose=False, **params)
      5 print(f'Ironclust found {len(sorting_IC.get_unit_ids())} units')

File /media/paul/storage/pdev/spikeinterface/src/spikeinterface/sorters/launcher.py:308, in run_sorter_by_property(sorter_name, recording, grouping_property, folder, mode_if_folder_exists, engine, engine_kwargs, verbose, docker_image, singularity_image, working_folder, **sorter_params)
    297     job = dict(
    298         sorter_name=sorter_name,
    299         recording=rec,
   (...)
    304         **sorter_params,
    305     )
    306     job_list.append(job)
--> 308 sorting_list = run_sorter_jobs(job_list, engine=engine, engine_kwargs=engine_kwargs, return_output=True)
    310 unit_groups = []
    311 for sorting, group in zip(sorting_list, recording_dict.keys()):

File /media/paul/storage/pdev/spikeinterface/src/spikeinterface/sorters/launcher.py:106, in run_sorter_jobs(job_list, engine, engine_kwargs, return_output)
    103 if engine == "loop":
    104     # simple loop in main process
    105     for kwargs in job_list:
--> 106         sorting = run_sorter(**kwargs)
    107         if return_output:
    108             out.append(sorting)

File /media/paul/storage/pdev/spikeinterface/src/spikeinterface/sorters/runsorter.py:199, in run_sorter(sorter_name, recording, folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, output_folder, **sorter_params)
    188             raise RuntimeError(
    189                 "The python `spython` package must be installed to "
    190                 "run singularity. Install with `pip install spython`"
    191             )
    193     return run_sorter_container(
    194         container_image=container_image,
    195         mode=mode,
    196         **common_kwargs,
    197     )
--> 199 return run_sorter_local(**common_kwargs)

File /media/paul/storage/pdev/spikeinterface/src/spikeinterface/sorters/runsorter.py:261, in run_sorter_local(sorter_name, recording, folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, output_folder, **sorter_params)
    259 SorterClass.setup_recording(recording, folder, verbose=verbose)
    260 # This NEEDS to happen in the docker because of dependencies
--> 261 SorterClass.run_from_folder(folder, raise_error, verbose)
    262 if with_output:
    263     sorting = SorterClass.get_result_from_folder(folder, register_recording=True, sorting_info=True)

File /media/paul/storage/pdev/spikeinterface/src/spikeinterface/sorters/basesorter.py:302, in BaseSorter.run_from_folder(cls, output_folder, raise_error, verbose)
    299         print(f"{sorter_name} run time {run_time:0.2f}s")
    301 if has_error and raise_error:
--> 302     raise SpikeSortingError(
    303         f"Spike sorting error trace:\n{error_log_to_display}\n"
    304         f"Spike sorting failed. You can inspect the runtime trace in {output_folder}/spikeinterface_log.json."
    305     )
    307 return run_time

SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
  File "/media/paul/storage/pdev/spikeinterface/src/spikeinterface/sorters/basesorter.py", line 262, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File "/media/paul/storage/pdev/spikeinterface/src/spikeinterface/sorters/external/ironclust.py", line 268, in _run_from_folder
    raise Exception("ironclust returned a non-zero exit code")
Exception: ironclust returned a non-zero exit code

Spike sorting failed. You can inspect the runtime trace in /media/paul/Test/TY20210401_640_signalCheck_morning/processed/results_ic/0/spikeinterface_log.json.

The log was:

                            < M A T L A B (R) >
                  Copyright 1984-2021 The MathWorks, Inc.
             R2021b Update 2 (9.11.0.1837725) 64-bit (glnxa64)
                             December 14, 2021


                            < M A T L A B (R) >
                  Copyright 1984-2021 The MathWorks, Inc.
             R2021b Update 2 (9.11.0.1837725) 64-bit (glnxa64)
                             December 14, 2021

 
 
To get started, type doc.
To get started, type doc.
For product information, visit www.mathworks.com.
For product information, visit www.mathworks.com.
 
 
===================================================
===================================================
IronClust Version: 2
IronClust Version: 2
===================================================
===================================================
--------------------------------------------------------------------------------Python commands require a supported version of CPython. See Getting Started with Python.Python commands require a supported version of CPython. See Getting Started with Python.

I tried installing cpython into the same env using conda install cpython, but it gave the same error?

[chrishalcrow]

Hello, now I have no idea. But at least spikeinterface is now getting ironclust started! I don't see any cpython error in your trace? Where can you see that?

Maybe @magland can help.

[magland]

Maybe @magland can help.

Sorry, unfortunately I'm not familiar with how ironclust works.

[paul-aparicio]

It's at the end of the log output from ironclust. It's printed for off to the right. Thanks for working through this with me!

[paul-aparicio]

I think it has something to do with spikeinterface. The same installed version of ironclust works fine with an older version of spikeinterface (v0.13).

[alejoe91]

Seems it's this issue: flatironinstitute/ironclust#62