Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use HyperSpy's map() to reduce time and memory use in some EBSD processing methods #527

Merged
merged 27 commits into from
May 25, 2022

Conversation

hakonanes
Copy link
Member

Description of the change

This PR introduces many internal changes and some external ones.

The improved map() method in HyperSpy v1.7 is used in some EBSD processing methods, reducing memory use and the processing time! HyperSpy's minimal version is set to 1.7. The map() method does not print a progressbar even though we ask for one (see hyperspy/hyperspy#2947), so for now we add this manually. This leads to two bars for some reason, but two are better than none.

The above changes made the following kikuchipy.pattern.chunk functions unnecessary, and have hence been removed from the API (in v0.6): remove_static_background(), remove_dynamic_background(), get_image_quality().

The EBSD.remove_static_background() parameter relative, to maintain relative intensities between patterns during background removal, has been removed. It can still be passed, but a deprecation warning is raised if it is. Passing it after v0.7 is released will raise an error. This was done because the (bad) implementation required that all patterns had to be queried for the minimum and maximum intensity value before background removal, effectively touching every pattern chunk twice. Dask could for some reason handle this well previously, but not later versions (don't know exactly which versions), and lead to large memory use for large datasets. I deem keeping relative intensities between patterns an unimportant feature in kikuchipy, and hence have dropped the functionality entirely in one go.

I've added formatting of notebooks using black's Jupyter notebook extension (thanks to @harripj for mentioning it in orix!). This is also added to the CI workflow code style check.

FFTs are computed using functions from scipy.fft instead of numpy.fft.

Progress of the PR

Minimal example of the bug fix or new feature

These operations are faster since they use hyperspy.signals.BaseSignal.map():

>>> import kikuchipy as kp
>>> s = kp.data.nickel_ebsd_large()
>>> s.remove_static_background()
Removing the static background:
[########################################] | 100% Completed |  0.1s
[########################################] | 100% Completed |  0.1s
>>> s.remove_dynamic_background()
Removing the dynamic background:
[########################################] | 100% Completed |  0.1s
[########################################] | 100% Completed |  0.6s
>>> iq = s.get_image_quality()
Calculating the image quality:
[########################################] | 100% Completed |  0.1s
[########################################] | 100% Completed |  0.3s

For reviewers

  • The PR title is short, concise, and will make sense 1 year later.
  • New functions are imported in corresponding __init__.py.
  • New features, API changes, and deprecations are mentioned in the unreleased
    section in CHANGELOG.rst.
  • New contributors are added to .all-contributorsrc and the table is regenerated.

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
@hakonanes hakonanes added enhancement New feature or request documentation This relates to the documentation maintenance This relates to package maintenance 📋 tests labels May 25, 2022
@hakonanes hakonanes added this to the v0.6.0 milestone May 25, 2022
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation This relates to the documentation enhancement New feature or request maintenance This relates to package maintenance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant