FIX: Tag memory estimates in resamplers #3150
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #3150 +/- ##
=======================================
Coverage 50.00% 50.00%
=======================================
Files 53 53
Lines 3914 3914
=======================================
Hits 1957 1957
Misses 1957 1957 ☔ View full report in Codecov by Sentry. |
effigies
force-pushed
the
fix/tag-resampling-memory
branch
from
fe3ada5
to
eb71d3b
Compare
tsalo
added a commit
to tsalo/aslprep
that referenced
this pull request
Nov 30, 2023
tsalo
added a commit
to PennLINC/aslprep
that referenced
this pull request
Dec 6, 2023
* Try this. * keep working. * Update plotting.py * Update base.py * Update base.py * Update. * Update plotting.py * Update plotting.py * fix. * Update plotting.py * Update plotting.py * Update plotting.py * Update. * Update cbf.py * Update cbf.py * Update cbf.py * Update cbf.py * Set Infs to 0. The Philips M0 scan has values of zero for some in-mask voxels, which, when used to calculate CBF, results in Infs. * Try just using standard ASL preproc pipeline. * Try that. * Update tests.py * Update outputs.py * Update base.py * Update base.py * More work. * More work. * More work. * Update utility.py * More work. * Run black and isort. * Update. * Update. * More work. * Update. * Fix. * Update gecbf.py * Update. * Update misc.py * Update fit.py * Update. * Update. * Fix? * Update fit.py * Update fit.py * Update fit.py * Update reference.py * Update reference.py * Update fit.py * Update fit.py * Whoops! * Update fit.py * Update fit.py * Update fit.py * Update fit.py * Update fit.py * Lots of changes. * Update config.py * Update base.py * Update base.py * Update base.py * Update base.py * Update. * Update base.py * Update. * Update. * Update base.py * Update confounds.py * Update. * Update base.py * Update. * Update confounds.py * Update. * Update. * Update. * Update workflows.rst * Update fit.py * Use new fMRIPrep! * Update. * Update. * update. * Update reference.py * Update. * Fix some things. * Fix many things. * Try 3.10. * Update. * Bump build Docker image version. * Update. * Update lint.yml * Update. * Update run.py * Update test_cli.py * Update test_cli.py * Update parser.py * Update parser.py * Update parser.py * Update parser.py * Update. * Update base.py * Update base.py * Update fit.py * Update fit.py * Update base.py * Update qc.py * Update apply.py * Update base.py * Update reference.py * Update base.py * Update base.py * Fix connections. * Make ApplyTransforms calls verbose. * Fix ApplyTransforms calls. * Try that. * Update. * Try fixing m0scan_native. * Try that then. * Try that then. * Update cbf.py * Update test_cli.py * Update aslprep_bids_config.json * Update test_cli.py * Write out native-space derivatives. * Update outputs.py * Update expected outputs. * Update base.py * Update expected outputs. * Update. * Parcellate data no matter what. * More cleanup. * Try adding numpy to build requirements. * Run isort. * Update __init__.py * Deal with style issues. * Add precomputed (will be empty). * Ignore unused parameter. * Try fixing install. * precomputed is a dictionary. * Outsource more workflows to fmriprep. * Try to implement functional cache. * Update BASIL output names. * Update base.py * Update hmc.py * Update bids.py * Remove GE workflows. * Try to fix GE HMC. * Whoops! * Run isort. * Update hmc.py * Update hmc.py * Update expected outputs. * Write out QC file. * Update utility.py * Add outputnode back in. * Try to debug motion params more. * Support single-volume motion parameters. * Add source to docstring. * Keep cleaning up GE processing. * Set confounds min volumes to 2. * Update confounds.py * Make QC calculations flexible. * Remove unused import. * Remove unused data files. * Write out QC file correctly. * Rename QC json filename. * Add confounds to GE outputs. * Drop unused functions and classes. * Try fixing tests. * Use GenericLabel instead of MultiLabel. * Use fMRIPrep's resampling interfaces. * Remove unnecessary comment. * Outsource more workflows to fMRIPrep. * Update expected_outputs_test_002.txt * Update expected outputs. * Update expected_outputs_test_001.txt * Don't use FixHeadersApplyTransforms. * Fix some connections in fit. * Add checks. * Fix logger name. * Retain niftis for now. * Override DDS for some workflow calls. * Disable aslref-space outputs unless requested Copies nipreps/fmriprep#3159. * Pass FLAIR images to anatomical workflow builder Copied from nipreps/fmriprep#3146. * Tag memory estimates in resamplers Copied from nipreps/fmriprep#3150. * Use M0 scan for reference for GE data. * Try to fix transform name. * Add reference image smoothing step. * Give smoothing output a BIDS-ish name. * Update fit.py * Remove unused modules. * Pin sMRIPrep version. * Reduce mask. * Squeeze masks before merging. * Fix module name. * Drop niftis from test outputs. * Update workflows.rst * Improve documentation. * Update api.rst * Update workflows.rst * Update citations. * More documentation improvements. * Remove dropped coverage files from expected outputs. * Update license and output docs. * Fix issues with CIFTI connections. * Update base Docker image. The new version has MSM. * Add level testing jobs. * Update expected outputs. * Try fixing more issues. * Draft workflow to produce CIFTI CBF maps. * Keep working on CIFTI CBF workflow. * Keep working on CIFTI CBF workflow. * Add reference images. * Update outputs.py * Address multiple bugs with GIFTI outputs. * Try outputting fsaverage. * Add docstrings. * Ignore unused argument. * Update tests. * Fix! * Update test_002 cache (it was already v3). * Update. * Update test_cli.py * Update config.yml * Update config.yml * Fix things. * Update test_cli.py * Update config.yml * Combine QC and confounds modules. * Remove unused functions/interfaces. * Remove another unused function. * Remove unused interface function. * Bump anatomical test data version. * Fix import. * Update expected outputs. * Create expected_outputs_qtab.txt * Grab MNI152NLin2009cAsym xfm for resampling. * Update expected outputs. * Add topup to boilerplate BibTeX file. * Revert some changes. * Update expected outputs. * Move asl_anat_wf from resampling to full. * Try outputting CIFTI for one test. * Add updated fsLR workflow from fMRIPrep. * Update resampling.py * Speed up test_003. * Use internal version of init_bold_surf_wf. * Fix style issues. * Remove unused script. * Fix typos. * Drop CIFTI test and sort expected outputs.
effigies
added a commit
that referenced
this pull request
Jan 10, 2024
23.2.0 (January 10, 2024) New feature release in the 23.2.x series. This release wraps up a significant refactor of fMRIPrep. The main new features can be used with the ``--level`` and ``--derivatives`` flags. The ``--level`` flag can take the arguments ``minimal``, ``resampling`` or ``full``. The default is ``full``, which should produce nearly the same results as previous versions. ``minimal`` will produce only the minimum necessary to deterministically generate the remaining derivatives. ``resampling`` will produce some additional derivatives, intended to simplify resampling with other tools. The ``--derivatives`` flag takes arguments of the form ``name=/path/to/dir``, for example ``--derivatives anat=$SMRIPREP_DIR``. If provided, fMRIPrep will read the specified directories for pre-computed derivatives. If a derivative is found, it will be used instead of computing it from scratch. If a derivative is not found, fMRIPrep will compute it and proceed as usual. Taken together, these features can allow a dataset provider to run a minimal fMRIPrep run, targeting many output spaces, while a user can then run a ``--derivatives`` run to generate additional derivatives in only the output spaces they need. Another use case is to provide an precomputed derivative to override the default fMRIPrep behavior, enabling easier workarounds for bugs or experimentation with alternatives. Additionally, this release includes a number of bug fixes and improvements. This release adds support for MSM-Sulc, improving the alignment of subject surfaces to the fsLR template. This process is enabled by default, but may be disabled with the ``--no-msm`` flag. This release resolves a number of issues with fieldmaps inducing distortions during correction. Phase difference and direct fieldmaps are now masked correctly, preventing the overestimation of distortions outside the brain. Additionally, we now implement Jacobian weighting during unwarping, which corrects for compression and expansion effects on signal intensity. To disable Jacobian weighting, use ``--ignore fmap-jacobian``. Finally, a new resampling method has been added, to better account for susceptibility distortion and motion in a single shot resampling to a volumetric target space. We anticipate extending this to surface targets in the future. * FIX: Restore --ignore sbref functionality (#3180) * FIX: Retrieve atlas ROIs at requested density (#3179) * FIX: Keep minctracc executable in FreeSurfer installation (#3175) * FIX: Exclude echo entity from optimally combined derivatives (#3166) * FIX: Disable boldref-space outputs unless requested (#3159) * FIX: Tag memory estimates in resamplers (#3150) * FIX: Final revisions for next branch (#3134) * FIX: Minor fixes to work with MSMSulc-enabled smriprep-next (#3098) * FIX: Connect EPI-to-fieldmap transform (#3099) * FIX: Use Py2-compatible version file template for fmriprep-docker (#3101) * FIX: Update connections to unwarp_wf, convert ITK transforms to text (#3077) * ENH: Allow --ignore fmap-jacobian to disable Jacobian determinant modulation during fieldmap correction (#3186) * ENH: Exclude non-steady-state volumes from confound correlation plot (#3171) * ENH: Pass FLAIR images to anatomical workflow builder to include in boilerplate (#3146) * ENH: Restore carpetplot and other final adjustments (#3131) * ENH: Restore CIFTI-2 generation (#3129) * ENH: Restore resampling to surface GIFTIs (#3126) * ENH: Restore confound generation (#3120) * ENH: Restore resampling BOLD to volumetric templates (#3121) * ENH: Restore resampling to T1w target (#3116) * ENH: Add MSMSulc (#3085) * ENH: Add reporting workflow for BOLD fit (#3082) * ENH: Generate anatomical derivatives useful for resampling (#3081) * RF: Load reportlets interfaces from nireports rather than niworkflows (#3176, #3184) * RF: Separate goodvoxels mask creation from fsLR resampling (#3170) * RF: Write out anatomical template derivatives (#3136) * RF: Update primary bold workflow to incorporate single shot resampling (#3114) * RF: Update derivative cache spec, calculate per-BOLD, reuse boldref2fmap (#3078) * RF: Split fMRIPrep into fit and derivatives workflows (#2913) * RPT: Rename CSF/WM confounds in fMRIPlot (#3172) * TST: Add smoke tests for full workflow and most branching flags (#3155) * TST: Add smoke-tests for bold_fit_wf (#3152) * DOC: Fix documentation and description for init_bold_grayords_wf (#3051) * DOC: Minor updates in outputs.rst (#3148) * STY: Apply a couple refurb suggestions (#3151) * STY: Fix flake8 warnings (#3044) * STY: Apply pyupgrade suggestions (#3043) * MNT: Restore mritotal subcommands to Dockerfile (#3149) * MNT: Update smriprep to 0.13.1 (#3153) * MNT: optimise size of PNG files (#3145) * MNT: update vendored docs script ``github_link.py`` (#3144) * MNT: Update tedana pin, test on Python 3.12 (#3141) * MNT: Bump environment (#3132) * MNT: Bump version requirements (#3107) * MNT: http:// → https:// (#3097) * MNT: Remove mritotal and dependencies from FreeSurfer ignore file (#3090) * MNT: Update environment (#3073) * MNT: Depend on newer sphinx (#3067) * MNT: Install ANTs from conda-forge (#3061) * MNT: Drop Python 3.8 and numpy 1.21 support (NEP29) (#3052) * MNT: update update_zenodo.py script (#3042) * MNT: Fix welcome message formatting and instructions (#3039) * MNT: Python 3.11 should be supported (#3038) * CI: Bump actions/setup-python from 4 to 5 (#3181) * CI: Stop testing legacy layout (#3079) * CI: Improve tag detection for docker builds (#3066) * CI: Clean up pre-release builds (#3040)
NingAnMe
added a commit
to NingAnMe/fmriprep
that referenced
this pull request
Jan 11, 2024
23.2.0 (January 10, 2024) New feature release in the 23.2.x series. This release wraps up a significant refactor of fMRIPrep. The main new features can be used with the ``--level`` and ``--derivatives`` flags. The ``--level`` flag can take the arguments ``minimal``, ``resampling`` or ``full``. The default is ``full``, which should produce nearly the same results as previous versions. ``minimal`` will produce only the minimum necessary to deterministically generate the remaining derivatives. ``resampling`` will produce some additional derivatives, intended to simplify resampling with other tools. The ``--derivatives`` flag takes arguments of the form ``name=/path/to/dir``, for example ``--derivatives anat=$SMRIPREP_DIR``. If provided, fMRIPrep will read the specified directories for pre-computed derivatives. If a derivative is found, it will be used instead of computing it from scratch. If a derivative is not found, fMRIPrep will compute it and proceed as usual. Taken together, these features can allow a dataset provider to run a minimal fMRIPrep run, targeting many output spaces, while a user can then run a ``--derivatives`` run to generate additional derivatives in only the output spaces they need. Another use case is to provide an precomputed derivative to override the default fMRIPrep behavior, enabling easier workarounds for bugs or experimentation with alternatives. Additionally, this release includes a number of bug fixes and improvements. This release adds support for MSM-Sulc, improving the alignment of subject surfaces to the fsLR template. This process is enabled by default, but may be disabled with the ``--no-msm`` flag. This release resolves a number of issues with fieldmaps inducing distortions during correction. Phase difference and direct fieldmaps are now masked correctly, preventing the overestimation of distortions outside the brain. Additionally, we now implement Jacobian weighting during unwarping, which corrects for compression and expansion effects on signal intensity. To disable Jacobian weighting, use ``--ignore fmap-jacobian``. Finally, a new resampling method has been added, to better account for susceptibility distortion and motion in a single shot resampling to a volumetric target space. We anticipate extending this to surface targets in the future. * FIX: Restore --ignore sbref functionality (nipreps#3180) * FIX: Retrieve atlas ROIs at requested density (nipreps#3179) * FIX: Keep minctracc executable in FreeSurfer installation (nipreps#3175) * FIX: Exclude echo entity from optimally combined derivatives (nipreps#3166) * FIX: Disable boldref-space outputs unless requested (nipreps#3159) * FIX: Tag memory estimates in resamplers (nipreps#3150) * FIX: Final revisions for next branch (nipreps#3134) * FIX: Minor fixes to work with MSMSulc-enabled smriprep-next (nipreps#3098) * FIX: Connect EPI-to-fieldmap transform (nipreps#3099) * FIX: Use Py2-compatible version file template for fmriprep-docker (nipreps#3101) * FIX: Update connections to unwarp_wf, convert ITK transforms to text (nipreps#3077) * ENH: Allow --ignore fmap-jacobian to disable Jacobian determinant modulation during fieldmap correction (nipreps#3186) * ENH: Exclude non-steady-state volumes from confound correlation plot (nipreps#3171) * ENH: Pass FLAIR images to anatomical workflow builder to include in boilerplate (nipreps#3146) * ENH: Restore carpetplot and other final adjustments (nipreps#3131) * ENH: Restore CIFTI-2 generation (nipreps#3129) * ENH: Restore resampling to surface GIFTIs (nipreps#3126) * ENH: Restore confound generation (nipreps#3120) * ENH: Restore resampling BOLD to volumetric templates (nipreps#3121) * ENH: Restore resampling to T1w target (nipreps#3116) * ENH: Add MSMSulc (nipreps#3085) * ENH: Add reporting workflow for BOLD fit (nipreps#3082) * ENH: Generate anatomical derivatives useful for resampling (nipreps#3081) * RF: Load reportlets interfaces from nireports rather than niworkflows (nipreps#3176, nipreps#3184) * RF: Separate goodvoxels mask creation from fsLR resampling (nipreps#3170) * RF: Write out anatomical template derivatives (nipreps#3136) * RF: Update primary bold workflow to incorporate single shot resampling (nipreps#3114) * RF: Update derivative cache spec, calculate per-BOLD, reuse boldref2fmap (nipreps#3078) * RF: Split fMRIPrep into fit and derivatives workflows (nipreps#2913) * RPT: Rename CSF/WM confounds in fMRIPlot (nipreps#3172) * TST: Add smoke tests for full workflow and most branching flags (nipreps#3155) * TST: Add smoke-tests for bold_fit_wf (nipreps#3152) * DOC: Fix documentation and description for init_bold_grayords_wf (nipreps#3051) * DOC: Minor updates in outputs.rst (nipreps#3148) * STY: Apply a couple refurb suggestions (nipreps#3151) * STY: Fix flake8 warnings (nipreps#3044) * STY: Apply pyupgrade suggestions (nipreps#3043) * MNT: Restore mritotal subcommands to Dockerfile (nipreps#3149) * MNT: Update smriprep to 0.13.1 (nipreps#3153) * MNT: optimise size of PNG files (nipreps#3145) * MNT: update vendored docs script ``github_link.py`` (nipreps#3144) * MNT: Update tedana pin, test on Python 3.12 (nipreps#3141) * MNT: Bump environment (nipreps#3132) * MNT: Bump version requirements (nipreps#3107) * MNT: http:// → https:// (nipreps#3097) * MNT: Remove mritotal and dependencies from FreeSurfer ignore file (nipreps#3090) * MNT: Update environment (nipreps#3073) * MNT: Depend on newer sphinx (nipreps#3067) * MNT: Install ANTs from conda-forge (nipreps#3061) * MNT: Drop Python 3.8 and numpy 1.21 support (NEP29) (nipreps#3052) * MNT: update update_zenodo.py script (nipreps#3042) * MNT: Fix welcome message formatting and instructions (nipreps#3039) * MNT: Python 3.11 should be supported (nipreps#3038) * CI: Bump actions/setup-python from 4 to 5 (nipreps#3181) * CI: Stop testing legacy layout (nipreps#3079) * CI: Improve tag detection for docker builds (nipreps#3066) * CI: Clean up pre-release builds (nipreps#3040)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We'll start with the baseline assumption that our resampler uses about 4*the original BOLD series size, which was generally a good estimate for antsApplyTransform.
Also tagging a few things with
run_without_submittingand some tasks that are surely using more than the default amount with 1GB, which is at least better. STC I assume uses 2x the total amount.All of this could stand profiling, and I'm curious to try out memray on the Python stuff at least.