Problems with noisy energies

[KrumaKruma]

I did a path integration for the ASE calculator integrated the forces and calculated the energies along a circle. At first all was going very well but if the circle is drawn smaller and smaller the energies and forces don't fit anymore! The path integration is implemented in this repository: https://github.com/Jonas-Finkler/path-integration-force-test.git.

The model I used was trained on GPU and then deployed. The path integration was done locally on CPU where I expected to get double precision for both, energy and forces. The original reason I did that test is that I've been trying to do geometry optimization using a NequIP potential and it seemed to be a really challenging task to do.

So my question is now if this related to single precision or if there is a minor bug in the software?

Attached you can find three files with different sizes of the integration.

Cheers,
Marco

[Linux-cpp-lisp]

Hi @KrumaKruma ,

Thanks for your detailed report. What do orange and blue indicate in these plots?

The path integration was done locally on CPU where I expected to get double precision for both, energy and forces

nequip sets the dtype independently of the device, which helps provide as much consistency as possible between GPU and CPU. If you are using the main branch, float32 is the default and will be used on both CPU and GPU. On main you can change this by setting default_type: float64.

However, on the develop branch which has the beta of the next big version, version 0.6.0, I have implemented some changes to numerics that I think might resolve exactly this issue you have pointed out without imposing any meaningful performance cost, and without your needing to set any new options. I think the first thing would be for you to try develop and see if it resolves the issue.

[KrumaKruma]

Hi @Linux-cpp-lisp

Thanks you very much for your prompt reply! The orange line is the integrated force and the blue line is the corresponding energy which should match in a path integration ( as in the first plot).

As you suggested I will first try to do the same test on the develop branch and then maybe also retrain the model with float 64 and post my findings here.

[KrumaKruma]

Hi @Linux-cpp-lisp

So first of all: The new version (0.6.0) on the development branch is much much faster than the previous version. It converges faster and also is faster in evaluating energies and forces. I'm really happy with the speed up.

The noise is much less in this version, however, it didn't vanish. I did the same test again with the new version (training and prediction) with float64 so that everything should be consistent. The results you should find below. It is still confusing for me that the energy is so much more stable or less noisy than the directly predicted energy. Shouldn't this be consistent? What is concerning me a little bit for the geometry optimization is that disparity of noise in integrated energy vs. the predicted energy.

However, I will take a look into the geometry optimization if it gets better with less noise.