DeepMD training problem

[1997sankha]

Summary

I am using DeepMD right now.
Now in licurve.out file errors are reduced. I am giving last few steps of lcurve.out

step rmse_val rmse_trn rmse_e_val rmse_e_trn rmse_f_val rmse_f_trn lr
If there is no available reference data, rmse_*_{val,trn} will print nan
......
1492000 6.66e-01 9.37e-01 7.04e-02 5.71e-02 6.04e-01 8.78e-01 1.1e-08
1493000 4.68e-01 7.73e-01 9.73e-02 5.93e-02 3.60e-01 6.81e-01 1.1e-08
1494000 8.44e-01 4.53e-01 3.80e-02 7.81e-02 7.82e-01 2.31e-01 1.1e-08
1495000 6.62e-01 6.34e-01 1.09e-01 1.13e-01 5.57e-01 5.20e-01 1.1e-08
1496000 9.56e-01 1.39e+00 1.16e-01 2.26e-01 8.53e-01 8.05e-01 1.1e-08
1497000 4.40e-01 8.34e-01 2.61e-02 4.64e-02 4.00e-01 7.63e-01 1.1e-08
1498000 1.23e+00 8.01e-01 2.10e-01 1.23e-01 1.03e+00 5.00e-01 1.1e-08
1499000 5.88e-01 4.31e-01 6.72e-02 7.78e-02 4.64e-01 1.92e-01 1.1e-08
1500000 1.06e+00 8.67e-01 7.30e-02 1.59e-01 9.88e-01 7.07e-01 1.0e-08

i am using 14 data set for each of which 80 percent is training and 20 percent for validation. datasets are increased also by bootstrapping. Now problem is that when I plot dft data and pred value plots are like

my code is also given below
{
"_comment1": " model parameters",
"model": {
"type_map": [
"Nb",
"Co"
],
"descriptor": {
"type": "se_e2_a",
"sel":"auto",
"rcut_smth": 0.5,
"rcut": 8.00,
"neuron": [
100,
200,
400
],
"resnet_dt": true,
"axis_neuron": 20,
"type_one_side": false,
"precision": "float64",
"seed": 1,
"_comment2": " that's all"
},
"fitting_net": {
"neuron": [
100,
100,
100
],
"resnet_dt": true,
"precision": "float64",
"seed": 1,
"_comment3": " that's all"
},
"_comment4": " that's all"
},

"learning_rate": {
"type": "exp",
"decay_steps": 10000,
"start_lr":1e-4,
"stop_lr": 1e-8,
"_comment5": "that's all"
},
"loss": {
"type": "ener",
"start_pref_e": 0.02,
"limit_pref_e": 1,
"start_pref_f": 1000,
"limit_pref_f": 1,
"start_pref_v": 0,
"limit_pref_v": 0,
"_comment6": " that's all"
},
"training": {
"training_data": {
"systems": [
"Data_8new/data_0",
"Data_8new/data_1",
"Data_8new/data_2",
"Data_8new/data_3",
"Data_9new/data_0",
"Data_9new/data_1",
"Data_9new/data_2",
"Data_9new/data_3",
"Data_10new/data_0",
"Data_10new/data_1",
"Data_10new/data_2",
"Data_10new/data_3",
"Data_11new/data_0",
"Data_11new/data_2",
"Data_11new/data_3",
"Data_11new/data_4",
"Data_12new/data_0",
"Data_12new/data_2",
"Data_12new/data_3",
"Data_12new/data_4",
"Data_13new/data_0",
"Data_13new/data_2",
"Data_13new/data_3",
"Data_13new/data_4",
"Data_14new/data_0",
"Data_14new/data_1",
"Data_14new/data_3",
"Data_14new/data_4",
"Data_15new/data_0",
"Data_15new/data_1",
"Data_15new/data_3",
"Data_15new/data_4",
"Data_16new/data_0",
"Data_16new/data_1",
"Data_16new/data_3",
"Data_16new/data_4",
"Data_17new/data_0",
"Data_17new/data_1",
"Data_17new/data_2",
"Data_17new/data_4",
"Data_18new/data_0",
"Data_18new/data_1",
"Data_18new/data_2",
"Data_18new/data_4",
"Data_19new/data_0",
"Data_19new/data_1",
"Data_19new/data_2",
"Data_19new/data_4",
"Data_20new/data_0",
"Data_20new/data_1",
"Data_20new/data_2",
"Data_20new/data_4",
"Data_21new/data_0",
"Data_21new/data_1",
"Data_21new/data_2",
"Data_21new/data_4"
],
"batch_size": "auto",
"_comment7": "that's all"
},
"validation_data": {
"systems": [
"Data_8new/data_4",
"Data_9new/data_4",
"Data_10new/data_4",
"Data_11new/data_1",
"Data_12new/data_1",
"Data_13new/data_1",
"Data_14new/data_2",
"Data_15new/data_2",
"Data_16new/data_2",
"Data_17new/data_3",
"Data_18new/data_3",
"Data_19new/data_3",
"Data_20new/data_3",
"Data_21new/data_3"
],
"batch_size": "auto",
"numb_btch": 1,
"_comment8": "that's all"
},
"numb_steps": 1500000,
"seed": 10,
"disp_file": "lcurve.out",
"disp_freq": 1000,
"save_freq": 10000,
"_comment9": "that's all"
},
"_comment10": "that's all"
}

can you pls help where I am going wrong??

DeePMD-kit Version

3.1.1

Backend and its version

TensorFlow 2.19.1

Python Version, CUDA Version, GCC Version, LAMMPS Version, etc

No response

Details

Training problem occur, error decreases but deviation between DFT and potential predicted value. details are mentioned above

[dosubot]

Hi @1997sankha, good to see the errors have come down significantly from your earlier training run. However, your force RMSE values (0.4–1.0 eV/Å) are still quite high and are fluctuating a lot at the end of training, which explains the poor DFT-vs-predicted agreement in your plots. Here are several things to look at:

1. Learning rate is essentially zero — training has stalled.
By step 1.5M your learning rate is ~1e-8, which means the model is barely updating. The large step-to-step fluctuations in RMSE (e.g., 0.44 → 1.23 → 0.59) at a near-zero learning rate suggest the model hasn't converged to a good minimum. Consider increasing numb_steps (4M–8M+) and/or adjusting decay_steps so the learning rate doesn't decay so fast. A common recommendation is start_lr=1e-3 with decay_steps tuned so you still have a meaningful learning rate during later training [1].

2. Check data quality — this was the root cause previously.
In your earlier issue on this same thread, you identified that your force data had values in the range of −222 to 222 eV/Å and near-zero coordinates. Even though you've rebuilt the dataset, make sure you've removed all outlier frames. Scan your .npy force files and remove any frame where forces exceed ~10–20 eV/Å — even a few bad frames can prevent convergence [1].

3. Bootstrapping can introduce problems.
You mention increasing data by bootstrapping. Duplicating data does not add new information — it only changes sampling weights. If some of the bootstrapped data contains outliers, their influence is amplified. Instead of bootstrapping, consider using auto_prob_style to control system sampling weights [2].

4. Sampling imbalance across 56 training systems.
With 56 training systems of potentially different sizes, the default prob_sys_size weighting can cause the model to focus on larger systems and neglect smaller ones. Check that frame counts are roughly comparable across systems, or use "auto_prob": "prob_uniform" to weight all systems equally [2].

5. Loss prefactors look reasonable but consider tuning.
Your start_pref_f=1000, limit_pref_f=1 is standard. But with force errors this high, try also setting start_pref_e slightly higher (e.g., 0.1–1.0) and check if equal prefactors (start_pref_f=1, limit_pref_f=1, start_pref_e=1, limit_pref_e=1) work better as a diagnostic [1].

6. Verify units and data format.
Double-check that your converted data (from QE CPMD) has energies in eV, forces in eV/Å, and coordinates in Å. Use dpdata to convert and verify: d = dpdata.LabeledSystem('your_qe_output', fmt='qe/pw/scf') and inspect d['forces'] and d['energies'] ranges.

Recommended next steps:

1. Scan all training data for force outliers (any |force component| > 15 eV/Å) and remove those frames
2. Increase numb_steps to 4M+ or increase start_lr to 1e-3
3. Remove bootstrapped duplicates and use auto_prob for system weighting instead
4. Try training on a small, clean subset first to confirm the model can converge before scaling up

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[1997sankha]

Thanks @dosu. I am looking into it

[dosubot]

Sounds good! Feel free to share your findings or any updated results — happy to help you troubleshoot further.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[1997sankha]

@dosu, I want to bring to your notice that no outliers are there, every folder has 10,000 frames, out of which 8000 are for training and 2000 for validation, units are ok, after bootstraping no outkliers are there , then where should I change?