【Hackathon 6th No.37】GraphCastNet 代码迁移至 PaddleScience #897

MayYouBeProsperous · 2024-05-14T09:13:41Z

PR types

Others

PR changes

APIs, Docs

Describe

复现运行结果

paddle-bot · 2024-05-14T09:13:45Z

Thanks for your contribution!

MayYouBeProsperous · 2024-06-06T13:12:38Z

麻烦制作数据集下载链接：https://aistudio.baidu.com/datasetdetail/252766

lijialin03 · 2024-06-07T04:29:18Z

麻烦制作数据集下载链接：https://aistudio.baidu.com/datasetdetail/252766

数据集下载链接分别为：
dataset.zip
dataset-step12.zip
params.zip
template_graph.zip
stats.zip
graphcast-jax2paddle.csv
jax_graphcast_small_output.npy

lijialin03 · 2024-06-13T04:12:12Z

我看标题中的WIP去掉了，代码现在是可以review的状态嘛

MayYouBeProsperous · 2024-06-13T08:56:53Z

是的，可以 review 了。
CI 报错好像是注释的括号匹配问题，应该怎么解决呢

lijialin03 · 2024-06-13T12:03:46Z

是的，可以 review 了。 CI 报错好像是注释的括号匹配问题，应该怎么解决呢

好嘞，这个先不管吧，等review完再说

lijialin03 · 2024-06-17T09:28:45Z

examples/graphcast/plot.py

+    data: xarray.Dataset,
+    center: Optional[float] = None,
+    robust: bool = False,
+) -> tuple[xarray.Dataset, matplotlib.colors.Normalize, str]:


改成Tuple

lijialin03 · 2024-06-17T09:28:58Z

examples/graphcast/plot.py

+
+
+def plot_data(
+    data: dict[str, xarray.Dataset],


lijialin03 · 2024-06-17T09:29:51Z

examples/graphcast/plot.py

+
+import datetime
+import math
+from typing import Optional


增加from typing import Tuple和from typing import Dict

lijialin03 · 2024-06-17T09:30:30Z

examples/graphcast/plot.py

+    robust: bool = False,
+    cols: int = 4,
+    file: str = "result.png",
+) -> tuple[xarray.Dataset, matplotlib.colors.Normalize, str]:


同上改成Tuple

lijialin03 · 2024-06-17T09:40:35Z

ppsci/data/dataset/atmospheric_utils.py

+    grid_longitude: np.ndarray,
+    mesh: TriangularMesh,
+    radius: float,
+) -> tuple[np.ndarray, np.ndarray]:


改成Tuple

lijialin03 · 2024-06-17T13:28:06Z

ppsci/data/dataset/atmospheric_utils.py

这个文件似乎是为了定义一种新的数据类型“GraphGridMesh”，因此把这个文件整个挪到atmospheric_dataset.py中吧

好的，移到了atmospheric_dataset.py中

lijialin03 · 2024-06-17T13:29:14Z

ppsci/arch/graphcast.py

+import paddle
+import paddle.nn as nn
+
+import ppsci.data.dataset.atmospheric_utils as atmospheric_utils


这边改成：

from typing import TYPE_CHECKING if TYPE_CHECKING: import ppsci.data.dataset.atmospheric_utils as atmospheric_utils

然后将这个文件中所有atmospheric_utils.GraphGridMesh加上双引号，改为“atmospheric_utils.GraphGridMesh”

atmospheric_utils.py 内容移到了 atmospheri_dataset.py

lijialin03 · 2024-06-17T13:47:48Z

ppsci/data/dataset/atmospheric_utils.py

+        should_init = any(var is None for var in all_input_vars)
+
+        if should_init:
+            # 初始化构建


删除中文注释

lijialin03 · 2024-06-17T13:48:07Z

ppsci/data/dataset/atmospheric_utils.py

+            self._mesh2grid_graph_structure = self._init_mesh2grid_graph()
+        else:
+            # 直接构建图数据
+            # 图结构信息


同上删除中文注释

lijialin03 · 2024-06-17T13:48:23Z

ppsci/data/dataset/atmospheric_utils.py

+            self.grid2mesh_num_edges = grid2mesh_num_edges
+            self.mesh2grid_num_edges = mesh2grid_num_edges
+
+            # 图特征信息


lijialin03

调用jointContribution/graphcast/run中的compare
输入运行结果后得到对齐结果为：

10m_u_component_of_wind diff is -2.356016193110792e-05
10m_v_component_of_wind diff is 7.523250922509292e-06
2m_temperature diff is 2.4777781459127155e-05
geopotential diff is -0.000539646573504715
mean_sea_level_pressure diff is -0.0005250472340893714
specific_humidity diff is 6.035347363899133e-10
temperature diff is 8.158555083697042e-07
total_precipitation_6hr diff is 1.8224237754237203e-09
u_component_of_wind diff is 1.7749217200032017e-06
v_component_of_wind diff is 7.323719328271271e-06
vertical_velocity diff is -2.375314757111974e-07
All diff is -8.92277301779838e-05

参考#699，认为可以对齐

HydrogenSulfate

辛苦提交PR，代码内容修改完毕之后，可以同步更新一下文档的代码块定位位置

HydrogenSulfate · 2024-06-26T02:47:54Z

docs/zh/examples/graphcast.md

+=== "模型评估命令"
+    ``` sh
+


Suggested change

=== "模型评估命令"

``` sh

=== "模型评估命令"

``` sh

HydrogenSulfate · 2024-06-26T02:53:52Z

examples/graphcast/conf/graphcast_small.yaml

@@ -0,0 +1,63 @@
+hydra:


Suggested change

hydra:

defaults:

- ppsci_default

- TRAIN: train_default

- TRAIN/ema: ema_default

- TRAIN/swa: swa_default

- EVAL: eval_default

- INFER: infer_default

- hydra/job/config/override_dirname/exclude_keys: exclude_keys_default

- _self_

hydra:

HydrogenSulfate · 2024-06-26T02:54:13Z

examples/graphcast/conf/graphcast_small.yaml

+    config:
+      override_dirname:
+        exclude_keys:
+          - TRAIN.checkpoint_path
+          - TRAIN.pretrained_model_path
+          - EVAL.pretrained_model_path
+          - mode
+          - output_dir
+          - log_freq


Suggested change

config:

override_dirname:

exclude_keys:

- TRAIN.checkpoint_path

- TRAIN.pretrained_model_path

- EVAL.pretrained_model_path

- mode

- output_dir

- log_freq

HydrogenSulfate · 2024-06-26T02:55:45Z

examples/graphcast/conf/graphcast_small.yaml

+
+EVAL:
+  batch_size: 1
+  pretrained_model_path: "data/params/GraphCast_small---ERA5-1979-2015---resolution-1.0---pressure-levels-13---mesh-2to5---precipitation-input-and-output.pdparams"


pretrained_model_path默认值改为null，模型评估要求手动指定预训练模型

HydrogenSulfate · 2024-06-26T02:58:52Z

docs/zh/examples/graphcast.md

+    unzip -q stats.zip -d data/
+    unzip -q template_graph.zip -d data/
+
+    python graphcast.py


模型评估要求手动指定预训练模型，命令应该类似：
python graphcast.py mode=eval EVAL.pretrained_model_path=xxxx

HydrogenSulfate · 2024-06-26T03:29:15Z

ppsci/data/dataset/atmospheric_dataset.py

+def stacked_to_dataset(
+    stacked_array: "xarray.Variable",
+    template_dataset: "xarray.Dataset",
+    preserved_dims: typing.Tuple[str, ...] = ("batch", "lat", "lon"),


typing.XXX建议改为XXX

好的，已修改

HydrogenSulfate · 2024-06-26T03:29:21Z

ppsci/data/dataset/atmospheric_dataset.py

+    faces: np.ndarray
+
+
+def merge_meshes(mesh_list: typing.Sequence[TriangularMesh]) -> TriangularMesh:


HydrogenSulfate · 2024-06-26T03:29:26Z

ppsci/data/dataset/atmospheric_dataset.py

+
+def get_hierarchy_of_triangular_meshes_for_sphere(
+    splits: int,
+) -> typing.List[TriangularMesh]:


HydrogenSulfate · 2024-06-26T03:30:18Z

ppsci/data/dataset/atmospheric_dataset.py

+    )
+
+
+class _ChildVerticesBuilder(object):


python3不需要显式继承object

Suggested change

class _ChildVerticesBuilder(object):

class _ChildVerticesBuilder:

HydrogenSulfate · 2024-06-26T03:32:13Z

ppsci/data/dataset/spherical_swe_dataset.py

这类位置不需要加换行，其余文件同理，不需要添加换行

这里是删除多余换行的

… into gc

HydrogenSulfate

LGTM

HydrogenSulfate

LGTM

* add GraphGridMeshDataSet * add graphcast model and example * refine code and visualization * add license * fix bugs * fix ci errors * fix ci errors * fix ci errors * fix ci errors * fix ci errors * add docs * resolve conflicts * resolve conflicts * resolve conflicts * resolve conflicts * refine docs * fix comments * fix comments * fix comments * fix * delete atmospheric_utils.py * fix ci errors * fix * fix comments * fix * add transform in graphcast * fix model bugs * fix * fix docs

add GraphGridMeshDataSet

2011bce

paddle-bot bot added the contributor label May 14, 2024

luotao1 mentioned this pull request May 14, 2024

【Hackathon 6th】开源贡献个人挑战赛 PaddlePaddle/Paddle#62905

Closed

luotao1 added the PaddlePaddle Hackathon label May 15, 2024

luotao1 assigned luotao1 and lijialin03 May 15, 2024

MayYouBeProsperous added 10 commits May 21, 2024 13:55

add graphcast model and example

7ded30b

refine code and visualization

16c6750

add license

32ce261

fix bugs

e931ef2

fix ci errors

73fe5fe

fix ci errors

19ae480

fix ci errors

25685be

fix ci errors

8dedf25

fix ci errors

6821353

add docs

b94e697

MayYouBeProsperous added 5 commits June 6, 2024 21:15

resolve conflicts

3a1ec58

resolve conflicts

67771e9

resolve conflicts

575076f

resolve conflicts

c3833d2

resolve conflicts

cce0c9c

MayYouBeProsperous changed the title ~~[WIP]【Hackathon 6th No.37】GraphCastNet 代码迁移至 PaddleScience~~ 【Hackathon 6th No.37】GraphCastNet 代码迁移至 PaddleScience Jun 6, 2024

MayYouBeProsperous added 2 commits June 7, 2024 14:11

refine docs

3acbed8

fix comments

5e89327

MayYouBeProsperous added 2 commits June 13, 2024 12:46

fix comments

0c21995

fix comments

973d691

lijialin03 reviewed Jun 17, 2024

View reviewed changes

MayYouBeProsperous added 9 commits June 19, 2024 15:13

fix

1b05f68

delete atmospheric_utils.py

54f6058

resolve conflicts

00a7ba4

fix ci errors

d67305a

fix

fc4b730

fix comments

1aa6229

fix

6e5d370

add transform in graphcast

f680cba

fix model bugs

1b2cf7f

lijialin03 approved these changes Jun 25, 2024

View reviewed changes

HydrogenSulfate requested changes Jun 26, 2024

View reviewed changes

MayYouBeProsperous added 3 commits June 26, 2024 14:27

fix

2b73a88

Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleScience…

5ec00bf

… into gc

fix docs

75e1137

HydrogenSulfate reviewed Jun 26, 2024

View reviewed changes

HydrogenSulfate approved these changes Jun 26, 2024

View reviewed changes

HydrogenSulfate merged commit 2eb2639 into PaddlePaddle:develop Jun 26, 2024
2 of 3 checks passed

-hydra:
+defaults:
+  - ppsci_default
+  - TRAIN: train_default
+  - TRAIN/ema: ema_default
+  - TRAIN/swa: swa_default
+  - EVAL: eval_default
+  - INFER: infer_default
+  - hydra/job/config/override_dirname/exclude_keys: exclude_keys_default
+  - _self_
+hydra:

		faces: np.ndarray


		def merge_meshes(mesh_list: typing.Sequence[TriangularMesh]) -> TriangularMesh:

	class _ChildVerticesBuilder(object):
	class _ChildVerticesBuilder:

【Hackathon 6th No.37】GraphCastNet 代码迁移至 PaddleScience #897

【Hackathon 6th No.37】GraphCastNet 代码迁移至 PaddleScience #897

Conversation

MayYouBeProsperous commented May 14, 2024 • edited Loading

PR types

PR changes

Describe

paddle-bot bot commented May 14, 2024

MayYouBeProsperous commented Jun 6, 2024

lijialin03 commented Jun 7, 2024 • edited Loading

lijialin03 commented Jun 13, 2024

MayYouBeProsperous commented Jun 13, 2024

lijialin03 commented Jun 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lijialin03 left a comment • edited Loading

Choose a reason for hiding this comment

HydrogenSulfate left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HydrogenSulfate left a comment

Choose a reason for hiding this comment

HydrogenSulfate left a comment

Choose a reason for hiding this comment

MayYouBeProsperous commented May 14, 2024 •

edited

Loading

lijialin03 commented Jun 7, 2024 •

edited

Loading

lijialin03 left a comment •

edited

Loading