Code

Dawn HPC improvements to pipeline#43

Code

Merged

JimCircadian merged 15 commits intoicenet-ai:v0.3.0_devicenet-ai/icenet-pipeline:v0.3.0_devfrom JimCircadian:38_dawn_hpcJimCircadian/icenet-pipeline:38_dawn_hpcCopy head branch name to clipboardAug 21, 2024

+351-21

Commits

Commits on Jun 8, 2024

• Dev icenet-ai#38: adding support for incremental HPC environment installation from cloned tensorflow-gpu

  JimCircadian

  committed
  0e6bd7b

• Dev icenet-ai#38: sticking in some stubs for dawn use

  JimCircadian

  committed
  7cdd3e2

• Basic resnet working in scripts

  JimCircadian

  committed
  7cb21c2

• Ensuring tensorboard is not included

  JimCircadian

  committed
  cf08987

• Allowing a test file to be included, committing shared test script for comparison

  JimCircadian

  committed
  6665d7c

• Number of steps equal to dataset size

  JimCircadian

  committed
  f4adba6

• Added horovod test

  JimCircadian

  committed
  86bc2da

• Demonstration training working with loss/metric hack

  JimCircadian

  committed
  23a3db5

• Successful implementation of horovod distributed node training

  JimCircadian

  committed
  dc150b9

• Updating for a more significant run before optimising HP, transfer of code and ensembling

  JimCircadian

  committed
  ed59fa7

• Updating current scripts and reordering data processing to prioritise time consuming ERA5

  JimCircadian

  committed
  a133b3e

• New script for sanitising tfrecorddatasets automatically

  JimCircadian

  committed
  158fbd5

Commits on Aug 20, 2024

• Dev icenet-ai#38: Ensuring dawn specifics aren't polluting the variables passed through

  JimCircadian

  committed
  5f2289b

• Update run_predict_ensemble.sh

  Show description for fcd47ab
  

  JimCircadian

  and

  bnubald

  authored
  fcd47ab

• Fixing missing nodes specification in getopt

  JimCircadian

  committed
  aadfc50