This repository has been archived by the owner on Dec 20, 2024. It is now read-only.
Release 0.3.2 #187
Merged
Release 0.3.2 #187
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Update changelog --------- Co-authored-by: Harrison Cook <Harrison.cook@ecmwf.int> Co-authored-by: HCookie <48088699+HCookie@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* full shuffle of the dataset * added changelog entry --------- Co-authored-by: Ana Prieto Nemesio <91897203+anaprietonem@users.noreply.github.com>
…ion (#136) * Implementation of aw_rescaling * Pre-commit * Updated implementation based on feedback * Small fixes - training now worked for all cases * Docstrings GraphNodeAttributes, minor fixes * Update changelog * Removed obsolete config options * Docstrings * Unit testing * Updated documentation * area_weights uses AreaWeights from anemoi-graphs * pre-commit * if test to check for scaled_attribute
* fix: remove resolution check * feat: first implementation of Callback to check variable order in pre-training * feat: add variable order checks for pre-training and current training * tests: implement tests for variable order * docs: changelog * tests: make variable for number of fixed callbacks * refactor: remove nested if as per review * fix: remove resolution from config * Fix linting issues --------- Co-authored-by: Harrison Cook <Harrison.cook@ecmwf.int>
* [pre-commit.ci] pre-commit autoupdate updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.2 → v0.8.1](astral-sh/ruff-pre-commit@v0.7.2...v0.8.1) - [github.com/jshwi/docsig: v0.64.0 → v0.65.0](jshwi/docsig@v0.64.0...v0.65.0) * fix: pre-commit docsig * fix: qa --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Gert Mertes <gert.mertes@ecmwf.int>
* check env vars safely * changelog
* Introduced resume flag and checkpoint loading for transfer learning, removed metadata saving in checkpoints due to corruption error on big models, fixed logging to work in the transfer leanring setting * Added len of dataset computed dynamically * debugging validation * Small changes * Removed prints * Not working * small changes * Imputer changes * Added sanification of checkpoint, effective batch size, git pre commit * gpc * gpc * New implementation: do not store modified checkpoint, load it directly after changing it * Added logging * Transfer learning working: implemented checkpoint cleaning with large models * Reverted some changes concerning imputer issues * Reverted some changes concerning imputer issues * Cleaned code for final review * Changed changelog and assigned TODO correctly * Changed changelog and assigned TODO correctly * Addressed review: copy checkpoint before removing metadata file * gpc passed * Removed logger in debugging mode * removed dataset lenght due to checkpointing issues * Reintroduced correct config on graphtansformer * gpc passed * Removed patched for issue #57, code expects patched checkpoint already * Removed new path name for patched checkpoint (ignoring fully issue #57) + removed fix for missing config * Adapted changelog * Switched logging to info from debug
* remove saving of unused metadata for training ckpt, fixing #57
* Lower bound delta lat in power spectrum plot and align input color map for precip plots
Co-authored-by: Harrison Cook <Harrison.cook@ecmwf.int>
* wip: possibility to save checkpoint at the end of fitting loop * cleaning * fix for tzinfo type * remove changes in checkpoint file * add shield for info regarding docs building
…d models using scalar functionality (KNMI) (#199) * Initial commit w-MSE splitting functionality * New graph name to fix anemoi compatibility issues with existing graph files * Code fixes for limitedarea loss * code quality improvements * expand scalar to prevent index out of bound error * Updated callbacks for CERRA config * Change naming of stretched grid metrics * Add all four new metrics to CERRA config * Code quality improvements * Reduced model complexity of CERRA config to run on a single GPU * Temporary solution to be able to log the overall MSE inside/outside with variable scaling * Removed CERRA config that was used for testing * Update limited area mask scalar based on review comments Co-authored-by: Harrison Cook <Harrison.cook@ecmwf.int> * Added change log entry * Update test script after adding 'all' variable --------- Co-authored-by: Harrison Cook <Harrison.cook@ecmwf.int>
Fix remove `metadata.dataset.specific.forward.forward.attrs.variables_metadata.` from logger params
* Store numpy arrays in checkpoints * changelog * fix failing test * feat: save remove unconnected mask in supporting_arrays * feat: add output mask to supporting arrays * fix: keep get_node_weights as staticmethod --------- Co-authored-by: Mario Santa Cruz <48736305+JPXKQX@users.noreply.github.com> Co-authored-by: Mario Santa Cruz <mariosanta_cruz@hotmail.com>
* Identify stretched grid from graph * Code quality improvements * Added change log entry
* Identify stretched grid from graph * Code quality improvements * Added change log entry
* Allow metrics in normalised space * Update ScaleTensor - Generalise `without` and `subset` - Allow removal by dim * Subset within the loss function * Use internal model mapping
* wip: possibility to save checkpoint at the end of fitting loop * cleaning * update changelog and rename function * wip * wip * add docstrings and comments for better readability * remove change
* fix: log warning when no trainable tensors * feat: propose default configs * fix: update changelog * fix: update defaults * fix: update loss_scaling function * fix: update to latest config * fix: remove hidden node attributes (stretched) * fix: update plot of trainable params * fix: style
JPXKQX
requested review from
theissenhelen,
JesperDramsch,
gmertes and
b8raoult
as code owners
JPXKQX
requested review from
floriankrb,
anaprietonem,
HCookie and
mchantry
as code owners
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR brings the latest developments:
📚 Documentation preview 📚: https://anemoi-training--187.org.readthedocs.build/en/187/