Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add lambda filter for earthkit.data.Field objects #16

Merged
merged 6 commits into from
Dec 18, 2024

Conversation

frazane
Copy link
Contributor

@frazane frazane commented Dec 2, 2024

Purpose

Add a first prototype implementation of a "lambda filter", i.e. a filter wrapping arbitrary user-defined functions.

Code changes

Added a new EarthkitFieldLambdaFilter class. Far from being a complete and truly generic feature but can be improved later.

  • For now I implemented a filter for functions acting on earthkit.data.Field objects, later we might add similar classes for xarray-based and numpy-based functions
  • It only supports the case of single or multiple fields as inputs and a single output field, but could be extended to be more generic (maybe by extending SimpleFilter or creating a new class).
  • Replaces/transforms input fields (not intended to add new fields).

Example config:

- earthkitfieldlambda:
    fn: "path.to.function"
    param: ["u","v"]
    args: [10]
    kwargs: {some_option: True}

Extras

Adapted tests to make a single mars request and use numpy.testing module.

@FussyDuck
Copy link

FussyDuck commented Dec 2, 2024

CLA assistant check
All committers have signed the CLA.

Copy link
Member

@HCookie HCookie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution.

The filter looks good, and is well documentated.
I have left a few comments to be addressed,

  • Primarily, we use pytest as our test harness, so some of the if __name__ == "__main__" should be removed.

src/anemoi/transform/filters/lambda_filters.py Outdated Show resolved Hide resolved
src/anemoi/transform/filters/lambda_filters.py Outdated Show resolved Hide resolved
tests/test_filters.py Show resolved Hide resolved
tests/test_filters.py Show resolved Hide resolved
@b8raoult b8raoult merged commit 9e44cab into ecmwf:develop Dec 18, 2024
5 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants