Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up initial benchmark for worldcereal #4

Closed
JeroenVerstraelen opened this issue Jun 17, 2024 · 6 comments
Closed

Set up initial benchmark for worldcereal #4

JeroenVerstraelen opened this issue Jun 17, 2024 · 6 comments
Assignees

Comments

@JeroenVerstraelen
Copy link
Contributor

JeroenVerstraelen commented Jun 17, 2024

A simple initial benchmark workflow using the worldcereal scenario.
No regression test, no metrics

Github action, doesn't have to be recurrent yet.

@JeroenVerstraelen JeroenVerstraelen changed the title Set up initial benchmark for worldcereal: Set up initial benchmark for worldcereal Jun 17, 2024
soxofaan added a commit that referenced this issue Jun 19, 2024
@soxofaan
Copy link
Contributor

initial setup is done

  • added package with reusable test/benchmark utilities
  • added initial unit tests (also triggered from github actions)
  • added initial benchmark
    • githbub workflow just runs one benchmark at random now
    • benchmark still fails because there is no auth setup yet

@soxofaan
Copy link
Contributor

First benchmark batch job is now running, fully triggered from GitHub workflow:

j-24062104655b4a8d91b124f13a3b625f from workflow https://github.com/ESA-APEx/apex_algorithms/actions/runs/9616807287

@soxofaan
Copy link
Contributor

So the basics are in place now.

Quite some further finetuning and documentation to be covered (see TODO comments in the code)

soxofaan added a commit that referenced this issue Jun 21, 2024
soxofaan added a commit that referenced this issue Jun 25, 2024
to have a simpler use case than worldcereal_inference
soxofaan added a commit that referenced this issue Jun 25, 2024
@soxofaan
Copy link
Contributor

because the worldcereal use case is still failing, I now added a very simple max NDVI UDP and benchmark scenario which now successfully runs in the benchmark workflow

@soxofaan
Copy link
Contributor

Initial benchmark setup, with a daily run can be checked at https://github.com/ESA-APEx/apex_algorithms/actions/workflows/benchmarks.yaml

note that it sometimes works, and sometimes fails, but that is indication that the benchmark system works: one benchmark scenario (max_ndvi) finishes, while the worldcereal one does not work yet

@soxofaan
Copy link
Contributor

soxofaan commented Jul 1, 2024

@soxofaan soxofaan closed this as completed Jul 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants