RF: Split fMRIPrep into fit and derivatives workflows #2913
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
effigies
force-pushed
the
rf/split_workflow
branch
2 times, most recently
from
9133f5c
to
ca4e44d
Compare
effigies
force-pushed
the
rf/split_workflow
branch
from
460fcda
to
66cb0bb
Compare
effigies
force-pushed
the
rf/split_workflow
branch
from
42ccd1b
to
28805d6
Compare
effigies
added a commit
that referenced
this pull request
Feb 2, 2023
Back when I commenced some work on #2207 (from which I'll need to familiarise myself with #2913), I started by looking at the CLI and thinking about what the right interface would be for such a capability (see eg. #2805, #2806). I immediately found some command-line options that really felt as though they were in the wrong place. So here I've separated out the set of CLI changes that I'd generated in the course of that little experiment. The git diff is quite unclean, so I've produced the set of options that would be moved and their corresponding old and new option groups. | Option | Old option group | New option group | |-----------------------------|-------------------------------|------------------------------------------------------| | `--anat-only` | Options to handle performance | Options for performing only a subset of the workflow | | `--boilerplate-only` | Options to handle performance | Options for performing only a subset of the workflow | | `--md-only-boilerplate` | Options to handle performance | Options for modulating outputs | | `--cifti-output` | Surface preprocessing options | Workflow configuration | | `--error-on-aroma-warnings` | Options to handle performance | Specific options for running ICA_AROMA | | `--verbose` | Options to handle performance | Other options | | `--me-output-echos` | Workflow configuration | Options for modulating outputs | | `--medial-surface-nan` | Workflow configuration | Options for modulating outputs | | `--no-submm-recon` | Surface preprocessing options | Specific options for FreeSurfer processing | | `--output-layout` | Other options | Options for modulating outputs | | `--sloppy` | Other options | Options to handle performance | | `--track-carbon` | Other options | Options for carbon tracking | | `--country-code` | Other options | Options for carbon tracking | New option groups: - "Options for performing only a subset of the workflow" - "Options for modulating outputs" - "Options for carbon tracking" Deleted option groups: - "Surface preprocessing options" In particular, "Options for performing only a subset of the workflow" is what I was looking for in the context of #2207, where I had initially added command-line options "`--fmri_withhold`" and "`--fmri_regenerate`" (the naming of such options should perhaps be discussed in #2913). Open to modification of proposed changes, or indeed addition of other changes (since this would be a good opportunity to review placement of the complete set of command-line options). For instance I'm currently looking at `[ --output-spaces, --cifti-output, --me-output-echos ]` as a candidate option group.
…kflow' and 'mgxd/rf/fit-io' into rf/split_workflow
This reverts commit f683374.
…to rf/split_workflow
Factor reference workflows into workflows.bold.reference Add HMC output workflow
|
Current state: Adding fieldmap processing back this morning. |
|
@mgxd @oesteban @feilong Here's the current fit-only output. Anything obvious missing? Should we get started on a resampler from BOLD to MNI using just derivatives? I think in the short term a standalone program would be better than trying to integrate it into the fMRIPrep workflow (though we will eventually do that). If I can find time between calls today, I'll update the workflow to work with the recently merged sdcflows changes. Also, how do people feel about merging this into |
|
Is the HTML report still produced? Should the goodvoxels mask be produced too? Otherwise, looks reasonable to me. Also, I'm good with pushing to |
|
HTML is created, but I'm not generating any fMRI reports at this point. Generating reports from the derivatives only is also on my todo list. For goodvoxels mask: graph LR;
fsnative --> desc-ribbon_mask.nii.gz --> desc-goodvoxels_mask.nii.gz;
bold.nii.gz --> desc-stc_bold.nii.gz --> desc-preproc_bold.nii.gz;
desc-hmc_xfm.txt & fieldmap.nii.gz & desc-fmapreg_xfm.txt --> desc-preproc_bold.nii.gz;
desc-preproc_bold.nii.gz --> desc-goodvoxels_mask.nii.gz;
All of these steps should be deterministic, so should not be part of the absolute minimal mode, IMO. It would definitely be good to approach these "second derivatives" in the same way we approach our minimal derivatives, and provide the option to spend a bit more time on the front end to save a lot of time for the ultimate derivatives. |
|
Perhaps also include the surface |
effigies
added a commit
that referenced
this pull request
Jan 10, 2024
23.2.0 (January 10, 2024) New feature release in the 23.2.x series. This release wraps up a significant refactor of fMRIPrep. The main new features can be used with the ``--level`` and ``--derivatives`` flags. The ``--level`` flag can take the arguments ``minimal``, ``resampling`` or ``full``. The default is ``full``, which should produce nearly the same results as previous versions. ``minimal`` will produce only the minimum necessary to deterministically generate the remaining derivatives. ``resampling`` will produce some additional derivatives, intended to simplify resampling with other tools. The ``--derivatives`` flag takes arguments of the form ``name=/path/to/dir``, for example ``--derivatives anat=$SMRIPREP_DIR``. If provided, fMRIPrep will read the specified directories for pre-computed derivatives. If a derivative is found, it will be used instead of computing it from scratch. If a derivative is not found, fMRIPrep will compute it and proceed as usual. Taken together, these features can allow a dataset provider to run a minimal fMRIPrep run, targeting many output spaces, while a user can then run a ``--derivatives`` run to generate additional derivatives in only the output spaces they need. Another use case is to provide an precomputed derivative to override the default fMRIPrep behavior, enabling easier workarounds for bugs or experimentation with alternatives. Additionally, this release includes a number of bug fixes and improvements. This release adds support for MSM-Sulc, improving the alignment of subject surfaces to the fsLR template. This process is enabled by default, but may be disabled with the ``--no-msm`` flag. This release resolves a number of issues with fieldmaps inducing distortions during correction. Phase difference and direct fieldmaps are now masked correctly, preventing the overestimation of distortions outside the brain. Additionally, we now implement Jacobian weighting during unwarping, which corrects for compression and expansion effects on signal intensity. To disable Jacobian weighting, use ``--ignore fmap-jacobian``. Finally, a new resampling method has been added, to better account for susceptibility distortion and motion in a single shot resampling to a volumetric target space. We anticipate extending this to surface targets in the future. * FIX: Restore --ignore sbref functionality (#3180) * FIX: Retrieve atlas ROIs at requested density (#3179) * FIX: Keep minctracc executable in FreeSurfer installation (#3175) * FIX: Exclude echo entity from optimally combined derivatives (#3166) * FIX: Disable boldref-space outputs unless requested (#3159) * FIX: Tag memory estimates in resamplers (#3150) * FIX: Final revisions for next branch (#3134) * FIX: Minor fixes to work with MSMSulc-enabled smriprep-next (#3098) * FIX: Connect EPI-to-fieldmap transform (#3099) * FIX: Use Py2-compatible version file template for fmriprep-docker (#3101) * FIX: Update connections to unwarp_wf, convert ITK transforms to text (#3077) * ENH: Allow --ignore fmap-jacobian to disable Jacobian determinant modulation during fieldmap correction (#3186) * ENH: Exclude non-steady-state volumes from confound correlation plot (#3171) * ENH: Pass FLAIR images to anatomical workflow builder to include in boilerplate (#3146) * ENH: Restore carpetplot and other final adjustments (#3131) * ENH: Restore CIFTI-2 generation (#3129) * ENH: Restore resampling to surface GIFTIs (#3126) * ENH: Restore confound generation (#3120) * ENH: Restore resampling BOLD to volumetric templates (#3121) * ENH: Restore resampling to T1w target (#3116) * ENH: Add MSMSulc (#3085) * ENH: Add reporting workflow for BOLD fit (#3082) * ENH: Generate anatomical derivatives useful for resampling (#3081) * RF: Load reportlets interfaces from nireports rather than niworkflows (#3176, #3184) * RF: Separate goodvoxels mask creation from fsLR resampling (#3170) * RF: Write out anatomical template derivatives (#3136) * RF: Update primary bold workflow to incorporate single shot resampling (#3114) * RF: Update derivative cache spec, calculate per-BOLD, reuse boldref2fmap (#3078) * RF: Split fMRIPrep into fit and derivatives workflows (#2913) * RPT: Rename CSF/WM confounds in fMRIPlot (#3172) * TST: Add smoke tests for full workflow and most branching flags (#3155) * TST: Add smoke-tests for bold_fit_wf (#3152) * DOC: Fix documentation and description for init_bold_grayords_wf (#3051) * DOC: Minor updates in outputs.rst (#3148) * STY: Apply a couple refurb suggestions (#3151) * STY: Fix flake8 warnings (#3044) * STY: Apply pyupgrade suggestions (#3043) * MNT: Restore mritotal subcommands to Dockerfile (#3149) * MNT: Update smriprep to 0.13.1 (#3153) * MNT: optimise size of PNG files (#3145) * MNT: update vendored docs script ``github_link.py`` (#3144) * MNT: Update tedana pin, test on Python 3.12 (#3141) * MNT: Bump environment (#3132) * MNT: Bump version requirements (#3107) * MNT: http:// → https:// (#3097) * MNT: Remove mritotal and dependencies from FreeSurfer ignore file (#3090) * MNT: Update environment (#3073) * MNT: Depend on newer sphinx (#3067) * MNT: Install ANTs from conda-forge (#3061) * MNT: Drop Python 3.8 and numpy 1.21 support (NEP29) (#3052) * MNT: update update_zenodo.py script (#3042) * MNT: Fix welcome message formatting and instructions (#3039) * MNT: Python 3.11 should be supported (#3038) * CI: Bump actions/setup-python from 4 to 5 (#3181) * CI: Stop testing legacy layout (#3079) * CI: Improve tag detection for docker builds (#3066) * CI: Clean up pre-release builds (#3040)
NingAnMe
added a commit
to NingAnMe/fmriprep
that referenced
this pull request
Jan 11, 2024
23.2.0 (January 10, 2024) New feature release in the 23.2.x series. This release wraps up a significant refactor of fMRIPrep. The main new features can be used with the ``--level`` and ``--derivatives`` flags. The ``--level`` flag can take the arguments ``minimal``, ``resampling`` or ``full``. The default is ``full``, which should produce nearly the same results as previous versions. ``minimal`` will produce only the minimum necessary to deterministically generate the remaining derivatives. ``resampling`` will produce some additional derivatives, intended to simplify resampling with other tools. The ``--derivatives`` flag takes arguments of the form ``name=/path/to/dir``, for example ``--derivatives anat=$SMRIPREP_DIR``. If provided, fMRIPrep will read the specified directories for pre-computed derivatives. If a derivative is found, it will be used instead of computing it from scratch. If a derivative is not found, fMRIPrep will compute it and proceed as usual. Taken together, these features can allow a dataset provider to run a minimal fMRIPrep run, targeting many output spaces, while a user can then run a ``--derivatives`` run to generate additional derivatives in only the output spaces they need. Another use case is to provide an precomputed derivative to override the default fMRIPrep behavior, enabling easier workarounds for bugs or experimentation with alternatives. Additionally, this release includes a number of bug fixes and improvements. This release adds support for MSM-Sulc, improving the alignment of subject surfaces to the fsLR template. This process is enabled by default, but may be disabled with the ``--no-msm`` flag. This release resolves a number of issues with fieldmaps inducing distortions during correction. Phase difference and direct fieldmaps are now masked correctly, preventing the overestimation of distortions outside the brain. Additionally, we now implement Jacobian weighting during unwarping, which corrects for compression and expansion effects on signal intensity. To disable Jacobian weighting, use ``--ignore fmap-jacobian``. Finally, a new resampling method has been added, to better account for susceptibility distortion and motion in a single shot resampling to a volumetric target space. We anticipate extending this to surface targets in the future. * FIX: Restore --ignore sbref functionality (nipreps#3180) * FIX: Retrieve atlas ROIs at requested density (nipreps#3179) * FIX: Keep minctracc executable in FreeSurfer installation (nipreps#3175) * FIX: Exclude echo entity from optimally combined derivatives (nipreps#3166) * FIX: Disable boldref-space outputs unless requested (nipreps#3159) * FIX: Tag memory estimates in resamplers (nipreps#3150) * FIX: Final revisions for next branch (nipreps#3134) * FIX: Minor fixes to work with MSMSulc-enabled smriprep-next (nipreps#3098) * FIX: Connect EPI-to-fieldmap transform (nipreps#3099) * FIX: Use Py2-compatible version file template for fmriprep-docker (nipreps#3101) * FIX: Update connections to unwarp_wf, convert ITK transforms to text (nipreps#3077) * ENH: Allow --ignore fmap-jacobian to disable Jacobian determinant modulation during fieldmap correction (nipreps#3186) * ENH: Exclude non-steady-state volumes from confound correlation plot (nipreps#3171) * ENH: Pass FLAIR images to anatomical workflow builder to include in boilerplate (nipreps#3146) * ENH: Restore carpetplot and other final adjustments (nipreps#3131) * ENH: Restore CIFTI-2 generation (nipreps#3129) * ENH: Restore resampling to surface GIFTIs (nipreps#3126) * ENH: Restore confound generation (nipreps#3120) * ENH: Restore resampling BOLD to volumetric templates (nipreps#3121) * ENH: Restore resampling to T1w target (nipreps#3116) * ENH: Add MSMSulc (nipreps#3085) * ENH: Add reporting workflow for BOLD fit (nipreps#3082) * ENH: Generate anatomical derivatives useful for resampling (nipreps#3081) * RF: Load reportlets interfaces from nireports rather than niworkflows (nipreps#3176, nipreps#3184) * RF: Separate goodvoxels mask creation from fsLR resampling (nipreps#3170) * RF: Write out anatomical template derivatives (nipreps#3136) * RF: Update primary bold workflow to incorporate single shot resampling (nipreps#3114) * RF: Update derivative cache spec, calculate per-BOLD, reuse boldref2fmap (nipreps#3078) * RF: Split fMRIPrep into fit and derivatives workflows (nipreps#2913) * RPT: Rename CSF/WM confounds in fMRIPlot (nipreps#3172) * TST: Add smoke tests for full workflow and most branching flags (nipreps#3155) * TST: Add smoke-tests for bold_fit_wf (nipreps#3152) * DOC: Fix documentation and description for init_bold_grayords_wf (nipreps#3051) * DOC: Minor updates in outputs.rst (nipreps#3148) * STY: Apply a couple refurb suggestions (nipreps#3151) * STY: Fix flake8 warnings (nipreps#3044) * STY: Apply pyupgrade suggestions (nipreps#3043) * MNT: Restore mritotal subcommands to Dockerfile (nipreps#3149) * MNT: Update smriprep to 0.13.1 (nipreps#3153) * MNT: optimise size of PNG files (nipreps#3145) * MNT: update vendored docs script ``github_link.py`` (nipreps#3144) * MNT: Update tedana pin, test on Python 3.12 (nipreps#3141) * MNT: Bump environment (nipreps#3132) * MNT: Bump version requirements (nipreps#3107) * MNT: http:// → https:// (nipreps#3097) * MNT: Remove mritotal and dependencies from FreeSurfer ignore file (nipreps#3090) * MNT: Update environment (nipreps#3073) * MNT: Depend on newer sphinx (nipreps#3067) * MNT: Install ANTs from conda-forge (nipreps#3061) * MNT: Drop Python 3.8 and numpy 1.21 support (NEP29) (nipreps#3052) * MNT: update update_zenodo.py script (nipreps#3042) * MNT: Fix welcome message formatting and instructions (nipreps#3039) * MNT: Python 3.11 should be supported (nipreps#3038) * CI: Bump actions/setup-python from 4 to 5 (nipreps#3181) * CI: Stop testing legacy layout (nipreps#3079) * CI: Improve tag detection for docker builds (nipreps#3066) * CI: Clean up pre-release builds (nipreps#3040)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This begins on #2207.
Todo:
--minimalderivatives modeinput -> proc -> derivatives -> bufferpattern--minimalDocumentation that should be reviewed
There will be some.