Generate metrics from external regressors using F stats

[handwerkerd]

Closes #1009. This is an alternative approach to #1008 and #1021.

If a user provides external regressors, this will calculate a fit to those regressors to include as metrics in the component table and for use in decision trees.

Changes proposed in this pull request:

• Create a new module, metrics.external, for calculating metrics from external regressors
  • An F statistic is used with a Legendre polynomial detrending baseline and p, F, and R2 values are saved as metrics in the component table
  • These stats are saved for the Full F model using all regressors as well as partial F models. This makes it possible to both examine when the full model fits, but to also log if it's the motion vs ROI-based vs cardiac/respiration-based regressors that fit to a specific component time series.
  • Regressor labels need to be specified, but can include regular expresses to use all regressors in a file
  • Multiple models can be included so that it is possible to create one model with nuisance regressors to reject and another with task design regressors to potentiall retain. This was a suggestion from @dowdlelt so that it would be possible to identify and conservatively retain components that fit to the overall task design.
• Add a new parameter to the tedana CLI, --external, to pass in a TSV with external regressors.
• Demo decision trees that incorporate external regressor correlation metrics. There are called demo_ because they demonstrate how to use the new functionality. They might be similar to trees we'll want to recommend, but validation on actual data (and tweaking) will be easier when this is merged and more people are using it.
  • In the decision tree there is a new field external_regressor_config which is a list of dictionaries with the following keys:
    • info A description of what the config does that's saved
    • report A description to add to report.txt
    • statistic Currently the only option is "F" but this leaves open possibilities for additional options.
    • detrend: Will automatically calculate the number of polynomial detrending regressors if true, but can also be a number for the users to specify. If this is false, then will be set to 0 (mean removal) and log a warning.
    • regressors: A list of the regressor names to include
    • partial_models [optional] A list of dictionaries with titles and names for partial models, each model name is its own key and contains either a list of column labels or a regular express wildcard (i.e. "Motion": ["^mot_.*$"] means the Motion partial model will include any external regressor column label that begins with mot_ and "Motion": ["mot_x", "mot_y", "mot_z"] specifies 3 specific column label names
  • demo_external_regressors_motion_task_models.json uses all the above options, uses the partial models to add a classification_tag, but not change results and retains components that correlate to the task (R2>0.5), have kappa>elbow regardless of what rho is.
  • demo_external_regressors_single_model.json uses the minimum number of options to run with external regressors.
• Includes tests that should cover all new code.
  • In adding testing, I started to use the matching parameter to confirm the correct errors are triggered and caplog.text to confirm expected warnings or info statements are logged.
  • Also starting to add tests that check for specific numerical outputs.
• Documentation is updated to explain all this new functionality and outputs. This is primarily in building_decision_trees.rst and included_decision_trees.rst

[handwerkerd]

I think I've now addressed all feedback regarding code and I've started to add documentation to building_decision_trees.rst I still need to make flow charts for the new trees and add them to included_decision_trees.rst but otherwise this PR is (again) ready for review & for people to try it out.

[handwerkerd]

I think/hope this is fully done and ready for review. It's important to have people look at the code, but I'd also like to have people read the documentation (tedana --help https://tedana--1064.org.readthedocs.build/en/1064/building_decision_trees.html#external-regressor-configuration & https://tedana--1064.org.readthedocs.build/en/1064/included_decision_trees.html#demo-external-regressors-single-model) and try to run this to see if it both makes sense and runs as expected. (@tsalo @eurunuela @n-reddy @goodalse2019 @dowdlelt )

[tsalo]

I just noticed that nuisance was misspelled as nuissance in a few places.

[tsalo]

I really don't want to hold this PR up any more. I say we merge as-is, make a release-candidate release, and deal with any bugs or typos before the next actual release.

[handwerkerd]

I really don't want to hold this PR up any more. I say we merge as-is, make a release-candidate release, and deal with any bugs or typos before the next actual release.

I'd still like to have @n-reddy and/or @goodalse2019 at least look over the documentation and run before merging. It would be good to see this understandable and executable by someone who hasn't yet look deep into the code. If they have time in the next week or so, I'm happy to wait before merging.

[tsalo]

In that case, can we commit to merging on a specific date (e.g., next Friday- 08/09) if they don't provide feedback by then?

[goodalse2019]

@handwerkerd and @tsalo I will have finished looked over the documentation and run by tomorrow evening. I have tested it on one subject very basic to make sure it ran and all looked like it went well! I am going to try on a couple others expanding my regressors tsv file a bit more. For testing purposes - is there a lower end of # of subjects you prefer this to be run over (e.g., at least 10?) or will just a couple do? Also - is there a good way to provide a overview of results for people to look over just to see the tests? Like uploading pdf or ppt in mattermost with some of the output figures?

[handwerkerd]

I'm at a conference 8/6-9 so 8/12 would be a good day to merge if we don't get feedback requiring changes. (Earlier if we get feedback saying everything is fine.

@goodalse2019 At this point, we're not claiming the results will be better or worse, so I'd just like to see this functioning as expected on typical data. If you have more than one dataset sitting around, then trying this on 2-3 subjects with different acquisition parameters and making sure it runs as expected would be fine.
Seeing the results will be useful for the next step of trying to figure out what's actually beneficial. If it's not too large, I think you can upload a pdf as a comment on this PR, which will mean we'd be able to find it when continuing work on these methods. Thank you.

[goodalse2019]

Sounds good, we have two - I can test on a couple subject from both and will upload some results asap

[goodalse2019]

I've run the single model demo tree and attached are some output summaries after running on n = 2 in two me datasets:

Dataset 1A - single model
Dataset1_singlemodel.pdf
Dataset 2 - single model
Dataset2_singlemodel.pdf

Still working on the task_motion_models demo - dataset 1 has a task and I am trying to get that to work but first few attempts were no good on my end. I can run over a few more subs in each if you think helpful and update these documents. I will be out of town in a wedding 8/2-8/4 so won't be until 8/5 that I can pick up again.

[tsalo]

Thanks @goodalse2019. I think the results look reasonable, and the fact that it ran through is good news. @handwerkerd are you comfortable merging given her results?

[n-reddy]

I reviewed the documentation and it makes sense to me! I just made a few small edits for clarity. Still planning to test the code if I get a chance in the next couple days.

[handwerkerd]

Thank you all. I'm comfortable merging, but I'm currently traveling through Tuesday and wouldn't mind giving it one more quick check myself. If others are comfortable merging, then give approving reviews.

[goodalse2019]

Looks good to me, single model and documentation ran successfully and was straight forward IMO.

[n-reddy]

Just an update that both the single model and the task_motion_models worked well on two datasets I tested with different parameters! A small comment perhaps for future tweaks of the demo .json is that the classification tags on the demo_external_regressors_motion_task_models can be a little confusing. For example, see in the pdfs linked below that the highlighted component was tagged as 'Likely BOLD, External Regressors, Unlikely BOLD' in the motion and task model.

Minimal model: minimal.pdf

External regressors - Motion parameters only (single model): external_regressors_motion.pdf

External regressors - Motion parameters and task regressors: external_regressors_motionTask.pdf