[TST] Tests and better debugging #41

rmarkello · 2018-05-11T18:32:16Z

I'm working on adding some unit tests for tedana.utils! Unfortunately a large portion of the functions require data files to operate on (or, at least, would be much easier to test with data files).

Would it be reasonable to downsample some functional data from OpenNeuro and ship them with tedana in a tests/data directory? It'd be nice to be able to run some tests locally, especially considering tedana doesn't have any external software dependencies anymore!

Updated circle.yml to trigger new tests. More tests needed for all the functions that operate on data files?

emdupre · 2018-05-11T18:36:33Z

Certainly with io and utils, I think we could use down sampled data ! I worry that T2* map and similar functions might not be as resilient with a heavily down sampled data set, but it's something we could test !

Added a bunch of logging statements through the codebase, options to augment (--debug) or stifle (--quiet) them, and some comments throughout. Also updated `setup.py` so that once installed, `tedana` is callable from the command line (it had erroneous pointers to meica). Hoping I didn't change any of the results in the process of doing this...

rmarkello · 2018-05-11T21:43:18Z

This PR is working to address both #15 and #36.

I think #42 should be merged first and I'll deal with the conflicts that will arise from renaming / working on the same stuff. I'll hold off on working on this too much more until that's settled.

codecov · 2018-05-13T22:57:43Z

Codecov Report

❗ No coverage uploaded for pull request base (master@d780754). Click here to learn what that means.
The diff coverage is 54.7%.

@@            Coverage Diff            @@
##             master      #41   +/-   ##
=========================================
  Coverage          ?   23.02%           
=========================================
  Files             ?       23           
  Lines             ?     1355           
  Branches          ?        0           
=========================================
  Hits              ?      312           
  Misses            ?     1043           
  Partials          ?        0

Impacted Files	Coverage Δ
tedana/selection/_utils.py	`0% <ø> (ø)`
tedana/selection/select_comps.py	`0% <0%> (ø)`
tedana/tests/test_tedana.py	`0% <0%> (ø)`
tedana/workflows/t2smap.py	`0% <0%> (ø)`
tedana/cli/run_tedana.py	`0% <0%> (ø)`
tedana/decomposition/eigendecomp.py	`0% <0%> (ø)`
tedana/decomposition/_utils.py	`0% <0%> (ø)`
tedana/model/monoexponential.py	`5.97% <0%> (ø)`
tedana/workflows/tedana.py	`0% <0%> (ø)`
tedana/info.py	`100% <100%> (ø)`
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d780754...eea263b. Read the comment docs.

rmarkello · 2018-05-14T02:53:39Z

Alright, I think this might be ready to go. Travis is now responsible for all testing except the primary integration test (test_tedana.py), which Circle handles. I added a few tests for tedana.utils.utils, but there's still a lot more to be covered. Nonetheless, at this point I think it might be best to integrate this into master and build from there in more targeted PRs.

This closes #36 and closes #45. It's a partial response to #15, but certainly not enough to warrant closing that issue. This may also close #13, with all the work that @tsalo has been doing merged in.

Let me know if you'd like to see anything else added to this, or if you have any comments!

emdupre · 2018-05-14T03:06:54Z

tedana/info.py

+        'sphinx>=1.5.3',
+        'sphinx_rtd_theme',
+        'sphinx-argparse'
+    ],


Could we add numpydoc here ? Comes from #43

Self-assigning this !

tsalo · 2018-05-14T11:25:00Z

@rmarkello Would you be willing to add a coverage badge to the README?

emdupre · 2018-05-14T11:53:19Z

I was thinking about waiting until we got closer to 80% at our 0.1.0 milestone. Do you think it's valuable to have beforehand ?

…

On Mon, May 14, 2018, 07:25 Taylor Salo, ***@***.***> wrote: @rmarkello <https://github.com/rmarkello> Would you be willing to add a coverage badge to the README? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#41 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOUk550lyPVaGUxDg_k3Lxv9v4yZ2320ks5tyWmMgaJpZM4T74ZI> .

tsalo · 2018-05-14T12:03:53Z

It will be a little depressing at first, but at least for me it reminds me that there's something to aim for. Totally up to you though.

KirstieJane · 2018-05-14T12:23:36Z

I think tests are a very concrete way to have new contributors add to the project so a badge might be nice (although I do hear you @emdupre that it will feel sad to start off with!!)

If there are any issues that I can tweet/promote to new contributors just let me know. Remember that many of them won’t know about brain imaging, so the work that you three (and anyone else who knows tedana well) can do to make this easier is to open well defined issues with as much context included as you can! Then just tag me and I’ll share it!

emdupre · 2018-05-14T12:32:47Z

Thanks so much @KirstieJane !! Yes, I think we could add detail in #15 which might be nice to point new contributors in the right direction. And we can open more re-factoring issues, though it seems like having tests in place help for re-factoring, so I'm not sure what's the right order to promote those !

…

On Mon, May 14, 2018, 08:23 Kirstie Whitaker, ***@***.***> wrote: I think tests are a very concrete way to have new contributors add to the project so a badge might be nice (although I do hear you @emdupre <https://github.com/emdupre> that it will feel sad to start off with!!) If there are any issues that I can tweet/promote to new contributors just let me know. Remember that many of them won’t know about brain imaging, so the work that you three (and anyone else who knows *tedana* well) can do to make this easier is to open well defined issues with as much context included as you can! Then just tag me and I’ll share it! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#41 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOUk506jy1QAlzqcQvt13MfNvrzo7UPvks5tyXdIgaJpZM4T74ZI> .

rmarkello · 2018-05-14T14:36:54Z

Alright, I added the codecov badge (in all its 17% coverage glory) and made some minor edits to the README.md! Next up will be making some concrete issues that can be tackled wrt adding more tests!

@emdupre assuming this passes (which it should since I only changed the documentation!) it should be ready for your final review 😄

emdupre · 2018-05-14T18:48:49Z

README.md


-We ask that all contributions to tedana respect our [code of conduct](https://github.com/ME-ICA/tedana/blob/master/Code_of_Conduct.md).
-
-### :earth_americas: Mozilla Global Sprint (10-11 May, 2018) :earth_africa:


#mozsprint is over ! so sad 😢

emdupre · 2018-05-14T19:05:31Z

tedana/selection/select_comps.py

        cond2 = db.labels_.max() < len(nc) / 6
+        # not totally sure if 0 is a special label for DBSCAN


self-assign: add a #TODO here

emdupre · 2018-05-14T19:55:45Z

LGTM ! Merging as soon as tests pass 😄 Thanks !!!

* Some contributor updates * Added doc to Marco

* Decision tree refactor with minimal and kundu * Fix commented-out tedana workflow * Appease the style checker * All tremble before the mighty linter * Actually fix incorrect style checker issue * Unfix another style checker error * Attempt to make Black happy, even though it does not actually say what's wrong * ran black * Added elbows to reports * fixing kundu tree and added calc_median * kundu.json added comment * kundu kappa_elbow is GTE not GT * kundu dtm matches main and minimal updated * flake8 style fixes * fixed linting * fixed report elbow warning * removed unneeded second d_table calc function * Links building decision trees to index * Adds ComponentSelector to API docs * Set language to English * Fix dead nilearn link * Add load_config and ComponentSelector to API docs * Fix mixing matrix over-save bug * Separately modularized kappa & rho elbow calcs and created liberal rho elbow (#15) * kundu tree provisionalreject to unclassified * calc_rho_elbow progress * calc_rho_elbow done * Removed calc_varex_upper_p * Removed kappa_rho_elbow tests * both decision trees running * linting fixes * Enable tedana_reclassify as console script * No errors if no xcomp but also no decide_comps (#16) * Update tedana/io.py Co-authored-by: Taylor Salo <tsalo90@gmail.com> * Appease style checker * Appease the style checker? * Force to use up to date setuptools; installation bug otherwise * Remove out of date make entry * Create functional reclassify CLI * Replace blanks with n/a * Maybe appease black * Fix typo Co-authored-by: Eneko Uruñuela <e.urunuela@bcbl.eu> * BIDSify some outputs * Appease black * Heavily revise ComponentSelector module docs * Fixing mid kappa A inconsistency (#17) * Output codes in kundu.json * fixed kappa ratio * Update tedana/selection/selection_nodes.py Co-authored-by: Joshua Teves <jbtevespro@gmail.com> * minimal tree keep kappa>2rho Co-authored-by: Joshua Teves <jbtevespro@gmail.com> * Drops 3.6 support * Remove 3.6 support from CircleCI tests * Reformat comment * Reduce line length * Update lint in Makefile * Correctly collect API submodule doc * Fix errors * Fix more sphinx * working on selector init documentation * Breaking up outputs.rst * partially updated output_file_descriptions.rst * changed n_bold_comps to n_accepted_comps * n_bold_comps to n_accepted_comps * ComponentSelector.py API docs cleaned up * selection_nodes decision_docs updated * selection_nodes docstrings cleaned up * Fixed a test for selection_nodes * Updated faq for tedana_reclassify and tree options * docstrings in tedica and other small updates * Updated docstrings in selection_utils.py * Update docs/output_file_descriptions.rst * Working on improving selector documentation (#18) * working on selector init documentation * Breaking up outputs.rst * partially updated output_file_descriptions.rst * changed n_bold_comps to n_accepted_comps * n_bold_comps to n_accepted_comps * ComponentSelector.py API docs cleaned up * selection_nodes decision_docs updated * selection_nodes docstrings cleaned up * Fixed a test for selection_nodes * Updated faq for tedana_reclassify and tree options * docstrings in tedica and other small updates * Updated docstrings in selection_utils.py * Update docs/output_file_descriptions.rst Co-authored-by: Joshua Teves <jbtevespro@gmail.com> * Remove manual selection * Force user to pick a tree * Fix CLI test * Revert "Force user to pick a tree" This reverts commit 4fc656f. * Revert "Fix CLI test" This reverts commit 4038336. * Make kundu default tree * Attempt to fix error * Adds input data to registry * Revert "Adds input data to registry" This reverts commit c7349bd. * Adds input registration * Appease linter * Add class template start * Add previous workflow registry into new one * Fix failure to update tags and classifications in manual * Fix missing less likely BOOLD tag * Adds more useful reporting for unused metrics * Create generated metrics * Update line terminator * Force black to run before flake8 * Updates percentile call * more doc updates * fixed meica to v2.5 in docstrings * docs building again * more updates to building decision trees * improved docs (#19) * working on selector init documentation * Breaking up outputs.rst * partially updated output_file_descriptions.rst * changed n_bold_comps to n_accepted_comps * n_bold_comps to n_accepted_comps * ComponentSelector.py API docs cleaned up * selection_nodes decision_docs updated * selection_nodes docstrings cleaned up * Fixed a test for selection_nodes * Updated faq for tedana_reclassify and tree options * docstrings in tedica and other small updates * Updated docstrings in selection_utils.py * Update docs/output_file_descriptions.rst * more doc updates * fixed meica to v2.5 in docstrings * docs building again * more updates to building decision trees Co-authored-by: Joshua Teves <jbtevespro@gmail.com> * Get rid of optional method keyword * Revert "Get rid of optional method keyword" This reverts commit e5fdec1. * Revert "Updates percentile call" This reverts commit 9d6a487. * Revert "Update line terminator" This reverts commit 8cf697c. * Autodocument ComponentSelector methods/attributes (#20) * Rename ComponentSelector module. * Document the ComponentSelector directly. * fixed rename of component_selector * Fixed remaining transition to component_selector (#21) * working on selector init documentation * Breaking up outputs.rst * partially updated output_file_descriptions.rst * changed n_bold_comps to n_accepted_comps * n_bold_comps to n_accepted_comps * ComponentSelector.py API docs cleaned up * selection_nodes decision_docs updated * selection_nodes docstrings cleaned up * Fixed a test for selection_nodes * Updated faq for tedana_reclassify and tree options * docstrings in tedica and other small updates * Updated docstrings in selection_utils.py * Update docs/output_file_descriptions.rst * more doc updates * fixed meica to v2.5 in docstrings * docs building again * more updates to building decision trees * fixed rename of component_selector Co-authored-by: Joshua Teves <jbtevespro@gmail.com> * more doc updates * mostly classification_output_descriptions * Fixed io API and selector API warnings * message message * key parts of docs all updated * output_file_descriptions fully updated * filled testing gaps for component_selector * Updates integration test fnames * Try a numpy fix * Try again * Remove dead code * full selector coverage (#23) * Add tedana_reclassify tests * Actually add test to circle workflow * Maybe actually add it * Change o to outdir * Fix noreports maybe * Fix tedort * CircleCI are you okay? * Circle if you keep this up I will switch to Actions * Revert "Circle if you keep this up I will switch to Actions" This reverts commit ad29c0d. * Maybe silence duecredit and re-trigger Circle * Try something else * Guess that wasn't legal * Switch main to _main * Add to pyproject.toml * Force it to be editable * Add references to resources package * Dispose of sanity check * Add more reclassify tests * Adaptive mask is not a bool * Add label for setup.cfg * Revert "Adaptive mask is not a bool" This reverts commit f7db360. * Add resource files * Clarify variables * Update date and weep * Fixed NoLikelyBOLDBug (#24) * Fixed NoLikelyBOLDBug * Updated docs for Likely BOLD * Added note for when ICA will rerun * updated message * New verbose tag for more detailed logging. * at_least_num_exist to classification_doesnt_exist * Cleaned up selector logging output * fixed debug logging * Temporarily turn on force overwrite for redo ICA * Fixed I007 divergence * calc_varex_thresh now has num_highest_var_comps * fixed linting errors * Update integration test data * Adds csv and text file reading for manual acc/rej * Add tests for CustomEncoder * Adds bibtex warning check test * Appease linter * Fix unused metrics warning * Add reclassify tests and patches to test failures * Make stylistic changes. * Remove trailing whitespace. * Spacing in io. * More minor changes. * Add custom napoleon section "Generated Files" * Replace numTrue/numFalse with n_true/n_false. * Replace ifTrue/ifFalse with if_true/if_false. * Use fill_doc. * Style fixes. * more int32 * more int32 fun * Appease linter * Fixed style issues * Add RICA to Approach section of docs * Fixed CI style check failure * DTM documentation review (#30) * Standardization of usage descriptions * Minor grammar edits * Minor grammar/spelling edits * Update docs/faq.rst --------- * Rename reclassify force (#32) * changed tedana_reclassify and force * Added default messages to CLI workflows * clean up CLI default messages * added t2smap to function from CLI * style fix * Add defaults to --help output (#31) * added ica_reclassify to setup.cfg * Using a more persistent cache for the testing data (#33) * Cleans up how testing datasets are downloaded within test_integration.py. In Main & the current JT_DTM each dataset is downloaded in a slightly different way and the five-echo data are downloaded twice. * Added `data_for_testing_info` which gives the file hash location and local directory name for each of the four files we download. All tests are updated to use this function. * The local copy of testing data will now go into the `.testing_data_cache` subdirectory * The downloaded testing data will be in separate directories from the outputs so the downloaded directories can be completely static * When `download_test_data` is called, it will first download the metadata json to see if the last updated copy on osf.io is newer than the downloaded version and will only download if osf has a newer file. Downloading the metadata will happen frequently, but it will hopefully be fast. * The logger is now used to give a warning if osf.io cannot be accessed, but it will still run using cached data * Change to TestLGR.info * Fixing high variance classification mess (#34) * Added dec_reclassify_high_var_comps plus * clarified diff btwn rho_kundu and _liberal thresh * Clarified docs for minimal tree * Replace versioneer with hatch (#35) * Update gitignore. * Delete _version.py * Adopt new packaging. * Ignore the _version.py file. * Fix CI (#36) * Base the cache on pyproject.toml, not setup.cfg. * Also drop use of setup.py in publishing action. * Add flake8-pyproject as a requirement. (#37) * Try fixing coverage. (#38) * Improving ica_reclassify (#39) * ica_reclassify docs now rendering in usage.html * moves file parsing to ica_reclassify_workflow * added error checks and tests * Ica reclassify registry fixes (#42) * add pandas version check >= 1.5.2 and mod behavior (#938) * add version check and mod behavior if pandas >= 1.5.2 to prevent error in writing csv * formatting * adding P. Molfese --------- Co-authored-by: Molfese <molfesepj@MH02217917MDI.local> * readded InputHarvester and expanduser * fixed handler base_dir path * mixing matrix file always in registry --------- Co-authored-by: Peter J. Molfese <pmolfese@gmail.com> Co-authored-by: Molfese <molfesepj@MH02217917MDI.local> * Drop Python 3.6 and 3.7 support (#40) * Drop Python 3.6 and 3.7 support. * line_terminator --> lineterminator * added mixm to 4echo test (#43) * Updating Contributor Information (#41) * Some contributor updates * Added doc to Marco * Added flow charts and some text (#44) * Added flow charts and some text * Finished flow charts and text. Co-authored-by: marco7877 <marco7877@users.noreply.github.com> --------- Co-authored-by: marco7877 <marco7877@users.noreply.github.com> * RTDfix (#45) * Update documentation (#46) * Update docs. * Update docs/building_decision_trees.rst Co-authored-by: Dan Handwerker <7406227+handwerkerd@users.noreply.github.com> --------- Co-authored-by: Dan Handwerker <7406227+handwerkerd@users.noreply.github.com> * Output docs on one page (#47) * Output docs on one page * added new multi-echo lectures --------- Co-authored-by: Joshua Teves <joshua.teves@nih.gov> Co-authored-by: handwerkerd <7406227+handwerkerd@users.noreply.github.com> Co-authored-by: Taylor Salo <tsalo90@gmail.com> Co-authored-by: Eneko Uruñuela <e.urunuela@bcbl.eu> Co-authored-by: handwerkerd <dan.handwerker@gmail.com> Co-authored-by: Taylor Salo <tsalo006@fiu.edu> Co-authored-by: Eneko Uruñuela <e.urunuela@icloud.com> Co-authored-by: Neha Reddy <58482773+n-reddy@users.noreply.github.com> Co-authored-by: Peter J. Molfese <pmolfese@gmail.com> Co-authored-by: Molfese <molfesepj@MH02217917MDI.local> Co-authored-by: marco7877 <marco7877@users.noreply.github.com> Co-authored-by: Taylor Salo <salot@pennmedicine.upenn.edu>

[TST] Added some tests for utils.py

28b2622

Updated circle.yml to trigger new tests. More tests needed for all the functions that operate on data files?

rmarkello changed the title ~~[WIP, TST] Adding tests for tedana.utils~~ [WIP, TST] Tests and better debugging May 11, 2018

rmarkello added 2 commits May 11, 2018 17:47

[RF] Removed unnecessary dependencies, updated Dockerfile

0e21720

[FIX] Added scipy as dependency

6fcf2d3

emdupre added this to the 0.1.0 milestone May 13, 2018

Merge remote-tracking branch 'upstream/master' into utiltests

d12ded9

tsalo mentioned this pull request May 13, 2018

Coveralls #45

Closed

[TST] Update circle and travis

73cca70

rmarkello added 3 commits May 13, 2018 19:06

[RF] Logging and doc-strings

1a1ff2a

[TST] Testing data and more utils tests

35d5568

Merge remote-tracking branch 'upstream/master' into utiltests

ee81c75

rmarkello changed the title ~~[WIP, TST] Tests and better debugging~~ [TST] Tests and better debugging May 14, 2018

emdupre reviewed May 14, 2018

View reviewed changes

[DOC] Add numpydoc requirement to setup.py

533a070

[DOC] Add codecov badge to README.md

603389d

emdupre reviewed May 14, 2018

View reviewed changes

emdupre approved these changes May 14, 2018

View reviewed changes

Convert code comment to TODO

eea263b

emdupre merged commit 4c2a474 into ME-ICA:master May 14, 2018

emdupre mentioned this pull request May 14, 2018

[DOC, FIX, RF] Fix typos in documentation and refactor fit_decay/fit_decay_ts. #49

Merged

rmarkello mentioned this pull request May 15, 2018

FutureWarning in np.linalg.lstsq #36

Closed

rmarkello deleted the utiltests branch September 15, 2018 14:39

handwerkerd added a commit to handwerkerd/tedana that referenced this pull request May 1, 2023

Updating Contributor Information (ME-ICA#41)

3e48d9f

* Some contributor updates * Added doc to Marco

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TST] Tests and better debugging #41

[TST] Tests and better debugging #41

rmarkello commented May 11, 2018

emdupre commented May 11, 2018

rmarkello commented May 11, 2018

codecov bot commented May 13, 2018 •

edited

Loading

rmarkello commented May 14, 2018

emdupre May 14, 2018

emdupre May 14, 2018

tsalo commented May 14, 2018

emdupre commented May 14, 2018 via email

tsalo commented May 14, 2018

KirstieJane commented May 14, 2018

emdupre commented May 14, 2018 via email

rmarkello commented May 14, 2018

emdupre May 14, 2018

emdupre May 14, 2018

emdupre commented May 14, 2018


		We ask that all contributions to tedana respect our [code of conduct](https://github.com/ME-ICA/tedana/blob/master/Code_of_Conduct.md).

		### :earth_americas: Mozilla Global Sprint (10-11 May, 2018) :earth_africa:

		cond2 = db.labels_.max() < len(nc) / 6
		# not totally sure if 0 is a special label for DBSCAN

[TST] Tests and better debugging #41

[TST] Tests and better debugging #41

Conversation

rmarkello commented May 11, 2018

emdupre commented May 11, 2018

rmarkello commented May 11, 2018

codecov bot commented May 13, 2018 • edited Loading

Codecov Report

rmarkello commented May 14, 2018

emdupre May 14, 2018

Choose a reason for hiding this comment

emdupre May 14, 2018

Choose a reason for hiding this comment

tsalo commented May 14, 2018

emdupre commented May 14, 2018 via email

tsalo commented May 14, 2018

KirstieJane commented May 14, 2018

emdupre commented May 14, 2018 via email

rmarkello commented May 14, 2018

emdupre May 14, 2018

Choose a reason for hiding this comment

emdupre May 14, 2018

Choose a reason for hiding this comment

emdupre commented May 14, 2018

codecov bot commented May 13, 2018 •

edited

Loading