Abnormal MD result was obtained in FPGA with example data

[Chandler-Lu]

Dear Professor Liu and Team @LiuGroupHNU , Jinzhe @njzjz,

I am a staff at DP Technology. In recent tests, we attempted to use sample data to train a QNN network and run MD simulations on LAMMPS. However, we obtained abnormal temperature data during this process.

The train data are got from https://github.com/deepmodeling/deepmd-kit/tree/r2/examples/water/data. Next, we trained the QNN model by referring nvnmd.md.

When the model generated by the CNN network on nVidia V100, the results are as follows:

   Step         PotEng         KinEng         TotEng          Temp          Press          Volume    
         0  -29943.907      8.1472669     -29935.76       330            23271.104      1927.3176    
       100  -29943.655      7.9050077     -29935.75       320.18744      14723.762      1927.3176    
       200  -29943.871      8.1120794     -29935.759      328.57475      15785.894      1927.3176    
       300  -29943.881      8.1288295     -29935.752      329.2532       12787.38       1927.3176    
       400  -29943.866      8.1024646     -29935.764      328.18531      12193.077      1927.3176 
       ...

But when we use QNN model on FPGA, the results are as follows:

   Step         PotEng         KinEng         TotEng          Temp          Press          Volume    
         0  -29978.917      8.1472669     -29970.77       330            428752.59      1927.3176    
      1000  -30020.707      277.34287     -29743.364      11233.601      255554.26      1927.3176    
      2000  -30102.399      190.83664     -29911.563      7729.7198      28502.869      1927.3176    
      3000  -30105.378      187.03335     -29918.344      7575.6702      110547.03      1927.3176    
      4000  -30112.372      198.48515     -29913.887      8039.5182     -72028.766      1927.3176 
      ...

I am very eager to know why we get the abnormal temperature.

The test file are shown in the attachments Thanks!

example.zip

[LiuGroupHNU]

@Chandler-Lu
We now concern that the version of nvnmd of train_cnn.json and train_qnn.json differs in the example.zip file, please make sure the version is the same.

[Chandler-Lu]

@Chandler-Lu We now concern that the version of nvnmd of train_cnn.json and train_qnn.json differs in the example.zip file, please make sure the version is the same.

Thank you for your reply.

Based on the example file, I modified the train file and run the other train.

train_cnn

{
	"nvnmd": {
		"version": 0,
		"net_size": 128,
		"sel": [
			46,
			92
		],
		"rcut": 6.0,
		"rcut_smth": 0.5,
		"type_map": [
			"O",
			"H"
		]
	},
	"learning_rate": {
		"type": "exp",
		"start_lr": 0.001,
		"stop_lr": 3.51e-8,
		"decay_steps": 5000
	},
	"loss": {
		"start_pref_e": 0.02,
		"limit_pref_e": 1,
		"start_pref_f": 1000,
		"limit_pref_f": 1,
		"start_pref_v": 0,
		"limit_pref_v": 0
	},
	"training": {
		"seed": 1,
		"stop_batch": 10000,
		"numb_test": 1,
		"disp_file": "lcurve.out",
		"disp_freq": 1000,
		"save_ckpt": "model.ckpt",
		"save_freq": 1000,
		"training_data": {
			"systems": [
				"../water_training_and_validation_data/data_0/",
				"../water_training_and_validation_data/data_1/",
				"../water_training_and_validation_data/data_2/"
			],
			"set_prefix": "set",
			"batch_size": 2
		},
		"validation_data": {
			"systems": [
				"../water_training_and_validation_data/data_3"
			],
			"batch_size": 1,
			"numb_btch": 3,
			"_comment8": "that's all"
		}
	}
}

train_qnn

    {
        "nvnmd": {
            "version": 0,
            "net_size": 128,
            "sel": [
                46,
                92
            ],
            "rcut": 6.0,
            "rcut_smth": 0.5,
            "type_map": [
                "O",
                "H"
            ]
        },
        "learning_rate": {
            "type": "exp",
            "start_lr": 1e-8,
            "stop_lr": 1e-9,
            "decay_steps": 1
        },
        "loss": {
            "start_pref_e": 1,
            "limit_pref_e": 1,
            "start_pref_f": 1,
            "limit_pref_f": 1,
            "start_pref_v": 0,
            "limit_pref_v": 0
        },
        "training": {
            "seed": 1,
            "stop_batch": 1,
            "numb_test": 1,
            "disp_file": "lcurve.out",
            "disp_freq": 1,
            "save_ckpt": "model.ckpt",
            "save_freq": 1,
            "training_data": {
                "systems": [
                    "../water_training_and_validation_data/data_0/",
                    "../water_training_and_validation_data/data_1/",
                    "../water_training_and_validation_data/data_2/"
                ],
                "set_prefix": "set",
                "batch_size": 1
            }
        }
    }

And then I use dp train-nvnmd train_qnn.json -s s2 to train QNN model. However, I still get the abnormal temperature.

   Step         PotEng         KinEng         TotEng          Temp          Press          Volume    
         0  -29943.879      8.1472669     -29935.732      330            454318.77      1927.3176    
       100  -29859.132      606.37119     -29252.761      24560.689      360044.23      1927.3176    
       200  -29862.154      714.37787     -29147.776      28935.433      193939.09      1927.3176    
       300  -29672.741      693.17173     -28979.569      28076.491      437772.91      1927.3176    
       400  -29688.613      582.77763     -29105.836      23605.047      93116.335      1927.3176    
       500  -29661.332      424.18385     -29237.148      17181.304      273595.72      1927.3176    
       600  -29665.902      450.63722     -29215.265      18252.781      170818.26      1927.3176    
       700  -29600.497      376.80738     -29223.69       15262.35       29590.986      1927.3176    
       ...
     98600  -29832.447      84.03791      -29748.409      3403.9035      65480.033      1927.3176    
     98700  -29841.385      72.892998     -29768.492      2952.4857      129810.46      1927.3176    
     98800  -29893.014      84.368327     -29808.645      3417.2868      57728.409      1927.3176    
     98900  -29871.715      82.538206     -29789.177      3343.1589      71800.453      1927.3176    
     99000  -29859.822      84.669958     -29775.152      3429.5042      50410.746      1927.3176    
     99100  -29830.701      84.070728     -29746.63       3405.2328     -100451.3       1927.3176    
     99200  -29852.163      78.358233     -29773.805      3173.8517      154381.32      1927.3176    
     99300  -29846.791      89.01894      -29757.772      3605.657      -63441.031      1927.3176    
     99400  -29849.346      99.369204     -29749.977      4024.8881     -75507.54       1927.3176    
     99500  -29833.751      86.58763      -29747.163      3507.1783      54287.521      1927.3176    
     99600  -29870.763      95.530518     -29775.232      3869.4045     -12892.239      1927.3176    
     99700  -29887.898      91.666299     -29796.232      3712.8867      115958.65      1927.3176    
     99800  -29829.064      84.771368     -29744.293      3433.6118     -74785.42       1927.3176    
     99900  -29826.458      85.069885     -29741.388      3445.703      -49876.733      1927.3176    
    100000  -29868.992      81.182068     -29787.81       3288.2294     -6242.862       1927.3176

[Chandler-Lu]

@Chandler-Lu We now concern that the version of nvnmd of train_cnn.json and train_qnn.json differs in the example.zip file, please make sure the version is the same.

The training logs of QNN network are as follow.
nvnmd_qnn.zip