Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

21.0.1 stuck changing data types #2725

Closed
sitek opened this issue Feb 28, 2022 · 8 comments
Closed

21.0.1 stuck changing data types #2725

sitek opened this issue Feb 28, 2022 · 8 comments
Labels
bug

Comments

@sitek
Copy link
Contributor

sitek commented Feb 28, 2022

What happened?

Similar to #2652, processing stuck for multiple days:

220228-11:18:01,591 nipype.workflow INFO:
	 [Node] Running "save_midthickness" ("nipype.interfaces.io.DataSink")
220228-11:18:02,10 nipype.workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_FLT01_wf.anat_preproc_wf.surface_recon_wf.gifti_surface_wf.save_midthickness".
220228-11:18:49,404 nipype.interface WARNING:
	 Changing /{filepath}/derivatives/fmriprep/sub-FLT01/anat/sub-FLT01_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz dtype from float64 to uint8
220228-11:18:52,346 nipype.interface WARNING:
	 Changing /{filepath}/derivatives/fmriprep/sub-FLT01/anat/sub-FLT01_space-MNI152NLin2009cAsym_dseg.nii.gz dtype from float64 to int16

What command did you use?

singularity run --cleanenv -B /bgfs:/bgfs $sing_img \
  $data_dir $out_dir participant \
  --participant-label $sub \
  --fs-license-file $fs_license \
  --skip_bids_validation \
  --debug all \
  --nprocs $nprocs \
  --output-spaces T1w func fsnative MNI152NLin2009cAsym

What version of fMRIPrep are you running?

21.0.1

How are you running fMRIPrep?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

Work directory

Please copy and paste any relevant log output.

No response

Additional information / screenshots

No response
@sitek sitek added the bug label Feb 28, 2022
@effigies
Copy link
Member

effigies commented Mar 1, 2022

Are you able to look at the running processes (e.g., with htop) to see what is hanging? Also can you run with -vv to get more nipype debug info? Also, --debug is not intended for end user use. It is almost certainly not doing something you want.

@mgxd
Copy link
Collaborator

mgxd commented Mar 11, 2022

Are you reusing a FreeSurfer directory? If so, you might be rerunning the entire surface reconstruction - the output layout has changed in the 21.0.x series, including the FreeSurfer subjects dir (now $out_dir/sourcedata/freesurfer). You can use the --fs-subjects-dir flag to ensure fmriprep is using the correct directory

@sitek
Copy link
Contributor Author

sitek commented Mar 14, 2022

Thanks @effigies and @mgxd – delayed response as I've slowly been rerunning things (kept hitting SLURM time limits; also, I thought it could be a SLURM memory issue [similar to #2623] so I used --mem to restrict the amount of memory fmriprep is aware of, but it's in MB, not GB, so it hung on topup for a while -_- ).
Re: FreeSurfer: This is a new dataset so I'm rerunning everything fresh, so I expect it to take a little bit. I'm starting it again with correct memory allocation and clean work/out directories, so I'll update you on how things go.

@sitek
Copy link
Contributor Author

sitek commented Mar 18, 2022

Update: stuck on topup for the last four days (running for 93 hours per htop).

FreeSurfer appears to be finished – here's the latest fmriprep update (as of 03-18 at 10:40am):

220316-04:14:22,798 nipype.workflow INFO:
[MultiProc] Running 1 tasks, and 0 jobs ready. Free memory (GB): 99.80/100.00, Free processors: 7/8.
Currently running:
* fmriprep_wf.single_subject_FLT01_wf.fmap_preproc_wf.wf_auto_00000.topup

currently running command:

topup --config=/opt/conda/lib/python3.8/site-packages/sdcflows/data/flirtsch/b02b0.cnf --datain=/{workdir}/fmriprep_2022.03.14/fmriprep_wf/single_subject_FLT01_wf/fmap_preproc_wf/wf_auto_00000/topup/sub-FLT01_dir-AP_magnitude_run-1_epi_idx-000_merged_encfile.txt --imain=/{workdir}/fmriprep_2022.03.14/fmriprep_wf/single_subject_FLT01_wf/fmap_preproc_wf/wf_auto_00000/concat_blips/sub-FLT01_dir-AP_magnitude_run-1_epi_idx-000_merged.nii.gz --out=sub-FLT01_dir-AP_magnitude_run-1_epi_idx-000_merged_base --iout=sub-FLT01_dir-AP_magnitude_run-1_epi_idx-000_merged_corrected.nii.gz --fout=sub-FLT01_dir-AP_magnitude_run-1_epi_idx-000_merged_field.nii.gz --jacout=jac --logout=sub-FLT01_dir-AP_magnitude_run-1_epi_idx-000_merged_topup.log --rbmout=xfm --dfout=warpfield

I have 6 BOLD runs, each with an EPI field map pair (assigned to the *_bold.nii.gz and *_sbref.nii.gz for each run in the IntendedFor json field).topup appears to still be working on the first pair.

@effigies
Copy link
Member

Very likely to be too many volumes in fmriprep_wf/single_subject_FLT01_wf/fmap_preproc_wf/wf_auto_00000/concat_blips/sub-FLT01_dir-AP_magnitude_run-1_epi_idx-000_merged.nii.gz. What's your full file listing and IntendedFor/B0Field* metadata?

@sitek
Copy link
Contributor Author

sitek commented Mar 18, 2022

fmap directory:

$ ls -lh fmap
total 67M
-rw-r--r-- 1 krs228 bchandrasekaran 3.3K Feb 18 17:19 sub-FLT01_dir-AP_run-1_part-mag_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 2.2M Feb 17 19:05 sub-FLT01_dir-AP_run-1_part-mag_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.1K Feb 17 19:05 sub-FLT01_dir-AP_run-1_part-phase_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 3.9M Feb 17 19:05 sub-FLT01_dir-AP_run-1_part-phase_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.3K Feb 18 17:20 sub-FLT01_dir-AP_run-2_part-mag_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 2.2M Feb 17 19:05 sub-FLT01_dir-AP_run-2_part-mag_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.1K Feb 17 19:05 sub-FLT01_dir-AP_run-2_part-phase_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 3.9M Feb 17 19:05 sub-FLT01_dir-AP_run-2_part-phase_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.3K Feb 18 17:21 sub-FLT01_dir-AP_run-3_part-mag_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 2.2M Feb 17 19:05 sub-FLT01_dir-AP_run-3_part-mag_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.1K Feb 17 19:05 sub-FLT01_dir-AP_run-3_part-phase_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 3.9M Feb 17 19:05 sub-FLT01_dir-AP_run-3_part-phase_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.3K Feb 18 17:22 sub-FLT01_dir-AP_run-4_part-mag_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 2.2M Feb 17 19:05 sub-FLT01_dir-AP_run-4_part-mag_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.1K Feb 17 19:05 sub-FLT01_dir-AP_run-4_part-phase_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 3.9M Feb 17 19:05 sub-FLT01_dir-AP_run-4_part-phase_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.3K Feb 18 17:23 sub-FLT01_dir-AP_run-5_part-mag_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 2.1M Feb 17 19:05 sub-FLT01_dir-AP_run-5_part-mag_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.1K Feb 17 19:05 sub-FLT01_dir-AP_run-5_part-phase_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 3.9M Feb 17 19:05 sub-FLT01_dir-AP_run-5_part-phase_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.3K Feb 18 17:24 sub-FLT01_dir-AP_run-6_part-mag_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 2.2M Feb 17 19:05 sub-FLT01_dir-AP_run-6_part-mag_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.1K Feb 17 19:05 sub-FLT01_dir-AP_run-6_part-phase_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 3.9M Feb 17 19:05 sub-FLT01_dir-AP_run-6_part-phase_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.3K Feb 18 17:19 sub-FLT01_dir-PA_run-1_part-mag_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 2.2M Feb 17 19:05 sub-FLT01_dir-PA_run-1_part-mag_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.1K Feb 17 19:05 sub-FLT01_dir-PA_run-1_part-phase_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 3.9M Feb 17 19:05 sub-FLT01_dir-PA_run-1_part-phase_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.3K Feb 18 17:20 sub-FLT01_dir-PA_run-2_part-mag_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 2.2M Feb 17 19:05 sub-FLT01_dir-PA_run-2_part-mag_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.1K Feb 17 19:05 sub-FLT01_dir-PA_run-2_part-phase_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 3.9M Feb 17 19:05 sub-FLT01_dir-PA_run-2_part-phase_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.3K Feb 18 17:21 sub-FLT01_dir-PA_run-3_part-mag_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 2.2M Feb 17 19:05 sub-FLT01_dir-PA_run-3_part-mag_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.1K Feb 17 19:05 sub-FLT01_dir-PA_run-3_part-phase_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 3.9M Feb 17 19:05 sub-FLT01_dir-PA_run-3_part-phase_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.3K Feb 18 17:22 sub-FLT01_dir-PA_run-4_part-mag_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 2.2M Feb 17 19:05 sub-FLT01_dir-PA_run-4_part-mag_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.1K Feb 17 19:05 sub-FLT01_dir-PA_run-4_part-phase_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 3.9M Feb 17 19:05 sub-FLT01_dir-PA_run-4_part-phase_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.3K Feb 18 17:23 sub-FLT01_dir-PA_run-5_part-mag_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 2.2M Feb 17 19:05 sub-FLT01_dir-PA_run-5_part-mag_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.1K Feb 17 19:05 sub-FLT01_dir-PA_run-5_part-phase_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 3.9M Feb 17 19:05 sub-FLT01_dir-PA_run-5_part-phase_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.3K Feb 18 17:24 sub-FLT01_dir-PA_run-6_part-mag_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 2.2M Feb 17 19:05 sub-FLT01_dir-PA_run-6_part-mag_epi.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran 3.1K Feb 17 19:05 sub-FLT01_dir-PA_run-6_part-phase_epi.json
-rw-r--r-- 1 krs228 bchandrasekaran 3.9M Feb 17 19:05 sub-FLT01_dir-PA_run-6_part-phase_epi.nii.gz

func directory:

$ ls -lh func
total 1.1G
-r--r--r-- 1 krs228 bchandrasekaran  15K Feb 17 19:15 sub-FLT01_task-stgrid_run-01_bold.json
-r--r--r-- 1 krs228 bchandrasekaran 180M Feb 17 19:15 sub-FLT01_task-stgrid_run-01_bold.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran  116 Feb 17 19:17 sub-FLT01_task-stgrid_run-01_events.tsv
-r--r--r-- 1 krs228 bchandrasekaran 5.7K Feb 17 19:17 sub-FLT01_task-stgrid_run-01_sbref.json
-r--r--r-- 1 krs228 bchandrasekaran 2.0M Feb 17 19:17 sub-FLT01_task-stgrid_run-01_sbref.nii.gz
-r--r--r-- 1 krs228 bchandrasekaran  14K Feb 17 19:17 sub-FLT01_task-stgrid_run-02_bold.json
-r--r--r-- 1 krs228 bchandrasekaran 176M Feb 17 19:16 sub-FLT01_task-stgrid_run-02_bold.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran  116 Feb 17 19:17 sub-FLT01_task-stgrid_run-02_events.tsv
-r--r--r-- 1 krs228 bchandrasekaran 5.7K Feb 17 19:17 sub-FLT01_task-stgrid_run-02_sbref.json
-r--r--r-- 1 krs228 bchandrasekaran 2.0M Feb 17 19:17 sub-FLT01_task-stgrid_run-02_sbref.nii.gz
-r--r--r-- 1 krs228 bchandrasekaran  14K Feb 17 19:07 sub-FLT01_task-tonecat_run-02_bold.json
-r--r--r-- 1 krs228 bchandrasekaran 176M Feb 17 19:07 sub-FLT01_task-tonecat_run-02_bold.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran  116 Feb 17 19:17 sub-FLT01_task-tonecat_run-02_events.tsv
-r--r--r-- 1 krs228 bchandrasekaran 5.7K Feb 17 19:13 sub-FLT01_task-tonecat_run-02_sbref.json
-r--r--r-- 1 krs228 bchandrasekaran 2.0M Feb 17 19:13 sub-FLT01_task-tonecat_run-02_sbref.nii.gz
-r--r--r-- 1 krs228 bchandrasekaran  16K Feb 17 19:09 sub-FLT01_task-tonecat_run-03_bold.json
-r--r--r-- 1 krs228 bchandrasekaran 208M Feb 17 19:09 sub-FLT01_task-tonecat_run-03_bold.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran  116 Feb 17 19:17 sub-FLT01_task-tonecat_run-03_events.tsv
-r--r--r-- 1 krs228 bchandrasekaran 5.7K Feb 17 19:13 sub-FLT01_task-tonecat_run-03_sbref.json
-r--r--r-- 1 krs228 bchandrasekaran 2.0M Feb 17 19:13 sub-FLT01_task-tonecat_run-03_sbref.nii.gz
-r--r--r-- 1 krs228 bchandrasekaran  14K Feb 17 19:11 sub-FLT01_task-tonecat_run-04_bold.json
-r--r--r-- 1 krs228 bchandrasekaran 176M Feb 17 19:10 sub-FLT01_task-tonecat_run-04_bold.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran  116 Feb 17 19:17 sub-FLT01_task-tonecat_run-04_events.tsv
-r--r--r-- 1 krs228 bchandrasekaran 5.7K Feb 17 19:13 sub-FLT01_task-tonecat_run-04_sbref.json
-r--r--r-- 1 krs228 bchandrasekaran 2.0M Feb 17 19:13 sub-FLT01_task-tonecat_run-04_sbref.nii.gz
-r--r--r-- 1 krs228 bchandrasekaran  16K Feb 17 19:13 sub-FLT01_task-tonecat_run-05_bold.json
-r--r--r-- 1 krs228 bchandrasekaran 208M Feb 17 19:12 sub-FLT01_task-tonecat_run-05_bold.nii.gz
-rw-r--r-- 1 krs228 bchandrasekaran  116 Feb 17 19:17 sub-FLT01_task-tonecat_run-05_events.tsv
-r--r--r-- 1 krs228 bchandrasekaran 5.7K Feb 17 19:13 sub-FLT01_task-tonecat_run-05_sbref.json
-r--r--r-- 1 krs228 bchandrasekaran 2.0M Feb 17 19:13 sub-FLT01_task-tonecat_run-05_sbref.nii.gz

And the IntendedFor data:

$ tail -n 5 fmap/*part-mag_epi.json
==> fmap/sub-FLT01_dir-AP_run-1_part-mag_epi.json <==
  "IntendedFor": [
    "func/sub-FLT01_task-tonecat_run-02_bold.nii.gz",
    "func/sub-FLT01_task-tonecat_run-02_sbref.nii.gz"
  ]
}

==> fmap/sub-FLT01_dir-AP_run-2_part-mag_epi.json <==
  "IntendedFor": [
    "func/sub-FLT01_task-tonecat_run-03_bold.nii.gz",
    "func/sub-FLT01_task-tonecat_run-03_sbref.nii.gz"
  ]
}

==> fmap/sub-FLT01_dir-AP_run-3_part-mag_epi.json <==
  "IntendedFor": [
    "func/sub-FLT01_task-tonecat_run-04_bold.nii.gz",
    "func/sub-FLT01_task-tonecat_run-04_sbref.nii.gz"
  ]
}

==> fmap/sub-FLT01_dir-AP_run-4_part-mag_epi.json <==
  "IntendedFor": [
    "func/sub-FLT01_task-tonecat_run-05_bold.nii.gz",
    "func/sub-FLT01_task-tonecat_run-05_sbref.nii.gz"
  ]
}

==> fmap/sub-FLT01_dir-AP_run-5_part-mag_epi.json <==
  "IntendedFor": [
    "func/sub-FLT01_task-stgrid_run-01_bold.nii.gz",
    "func/sub-FLT01_task-stgrid_run-01_sbref.nii.gz"
  ]
}

==> fmap/sub-FLT01_dir-AP_run-6_part-mag_epi.json <==
  "IntendedFor": [
    "func/sub-FLT01_task-stgrid_run-02_bold.nii.gz",
    "func/sub-FLT01_task-stgrid_run-02_sbref.nii.gz"
  ]
}

==> fmap/sub-FLT01_dir-PA_run-1_part-mag_epi.json <==
  "IntendedFor": [
    "func/sub-FLT01_task-tonecat_run-02_bold.nii.gz",
    "func/sub-FLT01_task-tonecat_run-02_sbref.nii.gz"
  ]
}

==> fmap/sub-FLT01_dir-PA_run-2_part-mag_epi.json <==
  "IntendedFor": [
    "func/sub-FLT01_task-tonecat_run-03_bold.nii.gz",
    "func/sub-FLT01_task-tonecat_run-03_sbref.nii.gz"
  ]
}

==> fmap/sub-FLT01_dir-PA_run-3_part-mag_epi.json <==
  "IntendedFor": [
    "func/sub-FLT01_task-tonecat_run-04_bold.nii.gz",
    "func/sub-FLT01_task-tonecat_run-04_sbref.nii.gz"
  ]
}

==> fmap/sub-FLT01_dir-PA_run-4_part-mag_epi.json <==
  "IntendedFor": [
    "func/sub-FLT01_task-tonecat_run-05_bold.nii.gz",
    "func/sub-FLT01_task-tonecat_run-05_sbref.nii.gz"
  ]
}

==> fmap/sub-FLT01_dir-PA_run-5_part-mag_epi.json <==
  "IntendedFor": [
    "func/sub-FLT01_task-stgrid_run-01_bold.nii.gz",
    "func/sub-FLT01_task-stgrid_run-01_sbref.nii.gz"
  ]
}

==> fmap/sub-FLT01_dir-PA_run-6_part-mag_epi.json <==
  "IntendedFor": [
    "func/sub-FLT01_task-stgrid_run-02_bold.nii.gz",
    "func/sub-FLT01_task-stgrid_run-02_sbref.nii.gz"
  ]
}

@sitek
Copy link
Contributor Author

sitek commented Mar 23, 2022 •
edited
Loading

Very likely to be too many volumes in fmriprep_wf/single_subject_FLT01_wf/fmap_preproc_wf/wf_auto_00000/concat_blips/sub-FLT01_dir-AP_magnitude_run-1_epi_idx-000_merged.nii.gz.

Yep, you were right – I had unused field map files stored in a subdirectory that I had included in my .bidsignore file, assuming fmriprep would ignore them (which it doesn't). After removing those files from the bids directory tree, fmrirpep (sans freesurfer) with 6 pairs of PE EPI images took 8 hours to run. Thanks for your feedback!

@sitek sitek closed this as completed Mar 23, 2022
@effigies
Copy link
Member

Nope. .bidsignore is a validator-only concept. Apologies for the confusion.

@tsalo tsalo mentioned this issue Jun 30, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@effigies @sitek @mgxd