RF: Update primary bold workflow to incorporate single shot resampling #3114

effigies · 2023-10-13T17:03:28Z

Changes proposed in this pull request

This is a pretty big one, so I'm pausing now. Things that this PR is doing:

Adding single-shot reampling interfaces for fieldmaps and BOLD volumes/series.
Gets rid of the old single_subject_wf and replaces it with what had been single_subject_fit_wf.
Creates a new workflow in workflows.bold.base to wrap around fit and further stages.
Adds an init_bold_native_wf to workflow.bold.fit, which handles slice timing correction and multi-echo, and always returns both a native-resampled file as well as something that can be resampled and the transforms/fieldmaps it should use.
- For single-echo, the resample-able thing is the STC'd file, and you get motion transforms and fieldmap ID back.
- For multi-echo, the resample-able thing is the optimal combination, and motion transforms and fieldmap are dropped.

codecov · 2023-10-13T18:07:57Z

Codecov Report

Attention: 298 lines in your changes are missing coverage. Please review.

Comparison is base (51319bb) 47.24% compared to head (a8630f9) 46.89%.
Report is 6 commits behind head on next.

❗ Current head a8630f9 differs from pull request most recent head 8187256. Consider uploading reports for the commit 8187256 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             next    #3114      +/-   ##
==========================================
- Coverage   47.24%   46.89%   -0.35%     
==========================================
  Files          49       53       +4     
  Lines        4009     4224     +215     
==========================================
+ Hits         1894     1981      +87     
- Misses       2115     2243     +128

Files	Coverage Δ
fmriprep/workflows/bold/resampling.py	`11.01% <ø> (ø)`
fmriprep/workflows/tests/test_base.py	`100.00% <ø> (ø)`
fmriprep/workflows/bold/t2s.py	`28.94% <0.00%> (ø)`
fmriprep/workflows/tests/__init__.py	`32.35% <50.00%> (ø)`
fmriprep/utils/asynctools.py	`57.14% <57.14%> (ø)`
fmriprep/workflows/bold/apply.py	`72.72% <72.72%> (ø)`
fmriprep/utils/misc.py	`27.77% <20.00%> (-9.73%)`	⬇️
fmriprep/workflows/bold/outputs.py	`20.68% <4.34%> (-0.45%)`	⬇️
fmriprep/workflows/base.py	`16.86% <30.76%> (+9.39%)`	⬆️
fmriprep/utils/transforms.py	`22.50% <22.50%> (ø)`
... and 3 more

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

effigies · 2023-10-13T21:12:52Z

It works on my machine:

231013-16:58:31,771 nipype.workflow INFO:
         [Job 506] Completed (fmriprep_23_2_wf.sub_04_wf.anat_fit_wf.ds_msmsulc_wf.ds_sphere_reg_msm).
231013-16:58:33,657 nipype.workflow IMPORTANT:
         fMRIPrep finished successfully!
231013-16:58:33,657 nipype.workflow IMPORTANT:
         Works derived from this fMRIPrep execution should include the boilerplate text found in /data/testing/fmriprep-ng-3046/logs/CITATION.md.

effigies · 2023-10-16T13:22:46Z

@mgxd Could you review the structure? This builds on your initial two commits, but ends up moving most of that into fit.py, as we will need this for things like calculating the goodvoxels mask and other things like that. I'm not committed to that decision, so it could be moved back.

@oesteban @feilong, if you have time for one thing, looking over fmriprep.interfaces.resampling would be greatly appreciated. I've currently broken out fieldmap reconstruction and single-shot resampling into two separate interfaces, but they could potentially be recombined as they were in the sdcflows interface.

This partially reverts commit 617b2a1.

mgxd

I'm fine with the structure, left a few comments as I was going through it

fmriprep/interfaces/resampling.py

fmriprep/utils/transforms.py

mgxd · 2023-10-17T14:30:07Z

fmriprep/utils/transforms.py

+    return nt.TransformChain([warp_transform, nt.Affine(affine)])
+
+
+def parse_combined_hdf5(h5_fn, to_ras=True):


i'm confused by the assertions in this - are we just comparing with the hardcoded ants parameters? if so, are there difference if run sloppy?

I have not noticed a difference in sloppy. The assertions are to make sure that we are not applying these modifications to files that they don't apply to.

@feilong could probably answer better than I could.

I wasn't sure if my understanding of the h5 file content format is correct. It was mainly based on my reverse engineering.

I tested the function extensively for T1 -> MNI transforms, and I am very sure it works properly in this case. I haven't tested it in other conditions.

fmriprep/workflows/bold/apply.py

Co-authored-by: Mathias Goncalves <goncalves.mathias@gmail.com>

effigies · 2023-10-17T14:55:16Z

Going to merge and keep things moving. Working on T1 resampling and CIFTI next.

feilong · 2023-10-24T17:45:25Z

I eyeballed fmriprep.interfaces.resampling and everything looks good to me.
I'm looking forward to testing it on my datasets soon.

23.2.0 (January 10, 2024) New feature release in the 23.2.x series. This release wraps up a significant refactor of fMRIPrep. The main new features can be used with the ``--level`` and ``--derivatives`` flags. The ``--level`` flag can take the arguments ``minimal``, ``resampling`` or ``full``. The default is ``full``, which should produce nearly the same results as previous versions. ``minimal`` will produce only the minimum necessary to deterministically generate the remaining derivatives. ``resampling`` will produce some additional derivatives, intended to simplify resampling with other tools. The ``--derivatives`` flag takes arguments of the form ``name=/path/to/dir``, for example ``--derivatives anat=$SMRIPREP_DIR``. If provided, fMRIPrep will read the specified directories for pre-computed derivatives. If a derivative is found, it will be used instead of computing it from scratch. If a derivative is not found, fMRIPrep will compute it and proceed as usual. Taken together, these features can allow a dataset provider to run a minimal fMRIPrep run, targeting many output spaces, while a user can then run a ``--derivatives`` run to generate additional derivatives in only the output spaces they need. Another use case is to provide an precomputed derivative to override the default fMRIPrep behavior, enabling easier workarounds for bugs or experimentation with alternatives. Additionally, this release includes a number of bug fixes and improvements. This release adds support for MSM-Sulc, improving the alignment of subject surfaces to the fsLR template. This process is enabled by default, but may be disabled with the ``--no-msm`` flag. This release resolves a number of issues with fieldmaps inducing distortions during correction. Phase difference and direct fieldmaps are now masked correctly, preventing the overestimation of distortions outside the brain. Additionally, we now implement Jacobian weighting during unwarping, which corrects for compression and expansion effects on signal intensity. To disable Jacobian weighting, use ``--ignore fmap-jacobian``. Finally, a new resampling method has been added, to better account for susceptibility distortion and motion in a single shot resampling to a volumetric target space. We anticipate extending this to surface targets in the future. * FIX: Restore --ignore sbref functionality (#3180) * FIX: Retrieve atlas ROIs at requested density (#3179) * FIX: Keep minctracc executable in FreeSurfer installation (#3175) * FIX: Exclude echo entity from optimally combined derivatives (#3166) * FIX: Disable boldref-space outputs unless requested (#3159) * FIX: Tag memory estimates in resamplers (#3150) * FIX: Final revisions for next branch (#3134) * FIX: Minor fixes to work with MSMSulc-enabled smriprep-next (#3098) * FIX: Connect EPI-to-fieldmap transform (#3099) * FIX: Use Py2-compatible version file template for fmriprep-docker (#3101) * FIX: Update connections to unwarp_wf, convert ITK transforms to text (#3077) * ENH: Allow --ignore fmap-jacobian to disable Jacobian determinant modulation during fieldmap correction (#3186) * ENH: Exclude non-steady-state volumes from confound correlation plot (#3171) * ENH: Pass FLAIR images to anatomical workflow builder to include in boilerplate (#3146) * ENH: Restore carpetplot and other final adjustments (#3131) * ENH: Restore CIFTI-2 generation (#3129) * ENH: Restore resampling to surface GIFTIs (#3126) * ENH: Restore confound generation (#3120) * ENH: Restore resampling BOLD to volumetric templates (#3121) * ENH: Restore resampling to T1w target (#3116) * ENH: Add MSMSulc (#3085) * ENH: Add reporting workflow for BOLD fit (#3082) * ENH: Generate anatomical derivatives useful for resampling (#3081) * RF: Load reportlets interfaces from nireports rather than niworkflows (#3176, #3184) * RF: Separate goodvoxels mask creation from fsLR resampling (#3170) * RF: Write out anatomical template derivatives (#3136) * RF: Update primary bold workflow to incorporate single shot resampling (#3114) * RF: Update derivative cache spec, calculate per-BOLD, reuse boldref2fmap (#3078) * RF: Split fMRIPrep into fit and derivatives workflows (#2913) * RPT: Rename CSF/WM confounds in fMRIPlot (#3172) * TST: Add smoke tests for full workflow and most branching flags (#3155) * TST: Add smoke-tests for bold_fit_wf (#3152) * DOC: Fix documentation and description for init_bold_grayords_wf (#3051) * DOC: Minor updates in outputs.rst (#3148) * STY: Apply a couple refurb suggestions (#3151) * STY: Fix flake8 warnings (#3044) * STY: Apply pyupgrade suggestions (#3043) * MNT: Restore mritotal subcommands to Dockerfile (#3149) * MNT: Update smriprep to 0.13.1 (#3153) * MNT: optimise size of PNG files (#3145) * MNT: update vendored docs script ``github_link.py`` (#3144) * MNT: Update tedana pin, test on Python 3.12 (#3141) * MNT: Bump environment (#3132) * MNT: Bump version requirements (#3107) * MNT: http:// → https:// (#3097) * MNT: Remove mritotal and dependencies from FreeSurfer ignore file (#3090) * MNT: Update environment (#3073) * MNT: Depend on newer sphinx (#3067) * MNT: Install ANTs from conda-forge (#3061) * MNT: Drop Python 3.8 and numpy 1.21 support (NEP29) (#3052) * MNT: update update_zenodo.py script (#3042) * MNT: Fix welcome message formatting and instructions (#3039) * MNT: Python 3.11 should be supported (#3038) * CI: Bump actions/setup-python from 4 to 5 (#3181) * CI: Stop testing legacy layout (#3079) * CI: Improve tag detection for docker builds (#3066) * CI: Clean up pre-release builds (#3040)

23.2.0 (January 10, 2024) New feature release in the 23.2.x series. This release wraps up a significant refactor of fMRIPrep. The main new features can be used with the ``--level`` and ``--derivatives`` flags. The ``--level`` flag can take the arguments ``minimal``, ``resampling`` or ``full``. The default is ``full``, which should produce nearly the same results as previous versions. ``minimal`` will produce only the minimum necessary to deterministically generate the remaining derivatives. ``resampling`` will produce some additional derivatives, intended to simplify resampling with other tools. The ``--derivatives`` flag takes arguments of the form ``name=/path/to/dir``, for example ``--derivatives anat=$SMRIPREP_DIR``. If provided, fMRIPrep will read the specified directories for pre-computed derivatives. If a derivative is found, it will be used instead of computing it from scratch. If a derivative is not found, fMRIPrep will compute it and proceed as usual. Taken together, these features can allow a dataset provider to run a minimal fMRIPrep run, targeting many output spaces, while a user can then run a ``--derivatives`` run to generate additional derivatives in only the output spaces they need. Another use case is to provide an precomputed derivative to override the default fMRIPrep behavior, enabling easier workarounds for bugs or experimentation with alternatives. Additionally, this release includes a number of bug fixes and improvements. This release adds support for MSM-Sulc, improving the alignment of subject surfaces to the fsLR template. This process is enabled by default, but may be disabled with the ``--no-msm`` flag. This release resolves a number of issues with fieldmaps inducing distortions during correction. Phase difference and direct fieldmaps are now masked correctly, preventing the overestimation of distortions outside the brain. Additionally, we now implement Jacobian weighting during unwarping, which corrects for compression and expansion effects on signal intensity. To disable Jacobian weighting, use ``--ignore fmap-jacobian``. Finally, a new resampling method has been added, to better account for susceptibility distortion and motion in a single shot resampling to a volumetric target space. We anticipate extending this to surface targets in the future. * FIX: Restore --ignore sbref functionality (nipreps#3180) * FIX: Retrieve atlas ROIs at requested density (nipreps#3179) * FIX: Keep minctracc executable in FreeSurfer installation (nipreps#3175) * FIX: Exclude echo entity from optimally combined derivatives (nipreps#3166) * FIX: Disable boldref-space outputs unless requested (nipreps#3159) * FIX: Tag memory estimates in resamplers (nipreps#3150) * FIX: Final revisions for next branch (nipreps#3134) * FIX: Minor fixes to work with MSMSulc-enabled smriprep-next (nipreps#3098) * FIX: Connect EPI-to-fieldmap transform (nipreps#3099) * FIX: Use Py2-compatible version file template for fmriprep-docker (nipreps#3101) * FIX: Update connections to unwarp_wf, convert ITK transforms to text (nipreps#3077) * ENH: Allow --ignore fmap-jacobian to disable Jacobian determinant modulation during fieldmap correction (nipreps#3186) * ENH: Exclude non-steady-state volumes from confound correlation plot (nipreps#3171) * ENH: Pass FLAIR images to anatomical workflow builder to include in boilerplate (nipreps#3146) * ENH: Restore carpetplot and other final adjustments (nipreps#3131) * ENH: Restore CIFTI-2 generation (nipreps#3129) * ENH: Restore resampling to surface GIFTIs (nipreps#3126) * ENH: Restore confound generation (nipreps#3120) * ENH: Restore resampling BOLD to volumetric templates (nipreps#3121) * ENH: Restore resampling to T1w target (nipreps#3116) * ENH: Add MSMSulc (nipreps#3085) * ENH: Add reporting workflow for BOLD fit (nipreps#3082) * ENH: Generate anatomical derivatives useful for resampling (nipreps#3081) * RF: Load reportlets interfaces from nireports rather than niworkflows (nipreps#3176, nipreps#3184) * RF: Separate goodvoxels mask creation from fsLR resampling (nipreps#3170) * RF: Write out anatomical template derivatives (nipreps#3136) * RF: Update primary bold workflow to incorporate single shot resampling (nipreps#3114) * RF: Update derivative cache spec, calculate per-BOLD, reuse boldref2fmap (nipreps#3078) * RF: Split fMRIPrep into fit and derivatives workflows (nipreps#2913) * RPT: Rename CSF/WM confounds in fMRIPlot (nipreps#3172) * TST: Add smoke tests for full workflow and most branching flags (nipreps#3155) * TST: Add smoke-tests for bold_fit_wf (nipreps#3152) * DOC: Fix documentation and description for init_bold_grayords_wf (nipreps#3051) * DOC: Minor updates in outputs.rst (nipreps#3148) * STY: Apply a couple refurb suggestions (nipreps#3151) * STY: Fix flake8 warnings (nipreps#3044) * STY: Apply pyupgrade suggestions (nipreps#3043) * MNT: Restore mritotal subcommands to Dockerfile (nipreps#3149) * MNT: Update smriprep to 0.13.1 (nipreps#3153) * MNT: optimise size of PNG files (nipreps#3145) * MNT: update vendored docs script ``github_link.py`` (nipreps#3144) * MNT: Update tedana pin, test on Python 3.12 (nipreps#3141) * MNT: Bump environment (nipreps#3132) * MNT: Bump version requirements (nipreps#3107) * MNT: http:// → https:// (nipreps#3097) * MNT: Remove mritotal and dependencies from FreeSurfer ignore file (nipreps#3090) * MNT: Update environment (nipreps#3073) * MNT: Depend on newer sphinx (nipreps#3067) * MNT: Install ANTs from conda-forge (nipreps#3061) * MNT: Drop Python 3.8 and numpy 1.21 support (NEP29) (nipreps#3052) * MNT: update update_zenodo.py script (nipreps#3042) * MNT: Fix welcome message formatting and instructions (nipreps#3039) * MNT: Python 3.11 should be supported (nipreps#3038) * CI: Bump actions/setup-python from 4 to 5 (nipreps#3181) * CI: Stop testing legacy layout (nipreps#3079) * CI: Improve tag detection for docker builds (nipreps#3066) * CI: Clean up pre-release builds (nipreps#3040)

mgxd and others added 23 commits October 12, 2023 13:04

RF: Move function to utils.misc module

3181f53

WIP: Start on apply workflow

6575633

ENH: Add resampling interfaces

294bc93

RF: Construct pe_info from ResampleSeries inputs

b508c7a

ENH: Add relaxed version of sdcflows GetReadoutTime interface

f8dc35a

DOC: Add comments convincing us that we are projecting correctly

81a395e

RF: Focus apply workflow on preparing further resampling

700bb40

RF: Create parent bold workflow as namespace

5931389

RF: Replace single_subject_wf() finally

a6789d2

RF: Move get_estimator

d94c3f8

DOC: Restore single_subject_wf docstring

328c4a9

RF: bold_file -> bold_series; wf_key -> bold_file

cf90bac

RF: Add bold native datasink workflow

61ab81a

FIX: Consistently sort bold series to ensure common lookups

96efe6f

RF: Use continues rather than multiple loops

1f23d76

RF: Move growing bold-specific workflow back into bold.base

d3bf1f7

ENH: Restore t2star reporting

6cf81fc

RF: init_bold_native_wf has been moved

5dc518e

FIX: missing import

66ccfec

FIX: Some bugs/style issues picked up by flake8

b0e3445

FIX: Import init_bold_wf

db2a963

TEST: Move tests to match get_estimator

b34bff0

FIX: Bad imports

a4ca684

effigies added 6 commits October 13, 2023 14:16

FIX: Clean up missing connections, bad imports

13ffca0

FIX: SimpleInterface uses _run_interface()

5d7a2e4

RF: Retrieve boldref from workflow, not outputdir

aacd7bc

FIX: Handle affine detection better

89ae68d

FIX: Correctly set defalt values for traits

15ab396

FIX: Handle (ijk)- pe-dirs and missing/solo HMC transforms

e14d360

effigies added 4 commits October 13, 2023 16:40

FIX: fmap_wf may not be defined

b062b48

FIX: Missing kwarg

47f305b

FIX: Sort on echo time

c3dde8a

FIX: dunder

e3b4feb

effigies added 4 commits October 13, 2023 22:10

DOC: Small updates to fix doc builds

3308813

FIX: Bad import, connection

fbee379

FIX: Return workflow and bad connection

d2fcd38

FIX: Ensure connections and names make sense

4564048

effigies force-pushed the enh/resampler branch 2 times, most recently from 55e275e to 170bc16 Compare October 15, 2023 12:48

FIX: Connect bold_mask to bold_t2s_wf

d4a84ad

effigies force-pushed the enh/resampler branch from 170bc16 to d4a84ad Compare October 15, 2023 22:45

FIX: Datasink config

79a7497

effigies and others added 4 commits October 16, 2023 09:45

DOC: Add docstrings for bold_wf and bold_fit_wf

00fe8f5

DOC/RF: Improve graph intelligibility

617b2a1

DOC: Document init_bold_native_wf

fa8c85c

Partially revert "DOC/RF: Improve graph intelligibility"

a8630f9

This partially reverts commit 617b2a1.

mgxd reviewed Oct 17, 2023

View reviewed changes

effigies and others added 2 commits October 17, 2023 10:49

FIX: Check for existence of input files

d14df3d

Co-authored-by: Mathias Goncalves <goncalves.mathias@gmail.com>

Apply suggestions from code review

8187256

Co-authored-by: Mathias Goncalves <goncalves.mathias@gmail.com>

effigies merged commit 978ae51 into nipreps:next Oct 17, 2023
2 of 3 checks passed

effigies deleted the enh/resampler branch October 17, 2023 14:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RF: Update primary bold workflow to incorporate single shot resampling #3114

RF: Update primary bold workflow to incorporate single shot resampling #3114

effigies commented Oct 13, 2023

codecov bot commented Oct 13, 2023 •

edited

Loading

effigies commented Oct 13, 2023

effigies commented Oct 16, 2023

mgxd left a comment

mgxd Oct 17, 2023

effigies Oct 17, 2023

effigies Oct 17, 2023 •

edited

Loading

feilong Oct 24, 2023

effigies commented Oct 17, 2023

feilong commented Oct 24, 2023

		return nt.TransformChain([warp_transform, nt.Affine(affine)])


		def parse_combined_hdf5(h5_fn, to_ras=True):

RF: Update primary bold workflow to incorporate single shot resampling #3114

RF: Update primary bold workflow to incorporate single shot resampling #3114

Conversation

effigies commented Oct 13, 2023

Changes proposed in this pull request

codecov bot commented Oct 13, 2023 • edited Loading

Codecov Report

effigies commented Oct 13, 2023

effigies commented Oct 16, 2023

mgxd left a comment

Choose a reason for hiding this comment

mgxd Oct 17, 2023

Choose a reason for hiding this comment

effigies Oct 17, 2023

Choose a reason for hiding this comment

effigies Oct 17, 2023 • edited Loading

Choose a reason for hiding this comment

feilong Oct 24, 2023

Choose a reason for hiding this comment

effigies commented Oct 17, 2023

feilong commented Oct 24, 2023

codecov bot commented Oct 13, 2023 •

edited

Loading

effigies Oct 17, 2023 •

edited

Loading