Fix Amp tests for tensorflow

[andrew_peterson]

Currently the nightly tests crash on the tensorflow module. I think this is just a lack of an import. This will need to be fixed by me (Andy).

[Muammar El Khatib]

I was able to compile tensorflow 0.11.0 to be used in the cluster. It is located at data/tensorflow-0.11.0-py2-none-any.whl. It can be used with test_gaussian_tflow.py:

[melkhati@login001 tflow]$ python test_gaussian_tflow.py 
/users/melkhati/git/ase/ase/lattice/surface.py:17: UserWarning: Moved to ase.build
  warnings.warn('Moved to ase.build')
/users/melkhati/.local/lib/python2.7/site-packages/tensorflow/python/ops/gradients.py:90: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/users/melkhati/.local/lib/python2.7/site-packages/tensorflow/contrib/opt/python/training/external_optimizer.py:321: OptimizeWarning: Unknown solver options: factr
  result = scipy.optimize.minimize(*minimize_args, **minimize_kwargs)
energy = [ 9.21283531]
forces = [[ -8.12581391e-04   1.16090901e-04   3.04703750e-02]
 [  8.12581391e-04   1.16090901e-04   3.04703750e-02]
 [ -8.12581391e-04  -1.16090901e-04   3.04703750e-02]
 [  8.12581391e-04  -1.16090901e-04   3.04703750e-02]
 [  1.24919782e-18  -1.42049307e-18  -3.20296399e-02]
 [ -3.75179802e-18   2.60864460e-19  -3.68724391e-02]
 [  3.10347232e-19  -5.24846058e-21  -3.24361138e-02]
 [ -4.44363741e-18   2.26201100e-18  -4.46551517e-02]
 [ -5.12352537e-20   6.24709494e-19   2.70054769e-02]
 [  6.39803744e-24  -1.81508428e-18  -2.89363205e-03]]
energy = [ 9.19882488]
forces = [[ 0.00221454 -0.00025891  0.02273602]
 [ 0.01464948  0.01054202  0.00714763]
 [ 0.01515592  0.00740176  0.01972192]
 [ 0.00023322  0.00281066  0.01715012]
 [ 0.00117597  0.00776827 -0.01787526]
 [-0.01630559 -0.01878563 -0.02480217]
 [-0.00786576 -0.01463179 -0.01156852]
 [-0.00928736  0.00585999 -0.01516279]
 [-0.00027493 -0.00533345 -0.00548602]
 [ 0.00030449  0.00462708  0.00813907]]

[Simon Muller]

Could you comment on the warning "OptimizeWarning: Unknown solver options: factr result = scipy.optimize.minimize(minimize_args, *minimize_kwargs)"?

[Muammar El Khatib]

That warning has to do with the scipy.optimize.minimize "wrapper" not accepting the factr solver option. The factrkeyword argument is set here from the module tflow.pyin Amp. TensorFlow uses ADAM or L-BFGS-B as optimizers. In Amp the latter is default. As I see it, the tensorflow's ScipyOptimizerInterface module is imported from tflow.py. Then it receives, ExternalOptimizerInterface as an argument, and does:

import scipy.optimize  # pylint: disable=g-import-not-at-top
result = scipy.optimize.minimize(*minimize_args, **minimize_kwargs)

and the scipy.optimize.minimize does not accept factr. I think the way around this issue would be changing line 1171 in tflow.py to use ftol instead of factr mentioned here.

[Muammar El Khatib]

I have observed some inconsistencies in the prediction from tensorflow NN. I slightly modified the test script for this module:

diff --git a/tests/test_gaussian_tflow.py b/tests/test_gaussian_tflow.py
index 69f2382..f83ec96 100644
--- a/tests/test_gaussian_tflow.py
+++ b/tests/test_gaussian_tflow.py
@@ -62,21 +62,28 @@ def train_test():

     from amp.model.tflow import NeuralNetwork
     label = 'train_test/calc'
-    train_images = generate_data(2)
+    train_images = generate_data(15)
+    train_images = train_images[0:10]
+    print('Length: {}' .format(len(train_images)))
     convergence = {
             'energy_rmse': 0.02,
             'force_rmse': 0.02
             }

     calc = Amp(descriptor=Gaussian(),
-               model=NeuralNetwork(hiddenlayers=(3, 3),
+               model=NeuralNetwork(hiddenlayers=(5, 5),
                                    convergenceCriteria=convergence),
                label=label,
                cores=1)

     calc.train(images=train_images,)
+    train_images = train_images[-5:]
+    print('Test Length: {}' .format(len(train_images)))
+
     for image in train_images:
+        print("Energy =", image.get_potential_energy())
         print("energy =", calc.get_potential_energy(image))
+        print("Forces =", image.get_forces())
         print("forces =", calc.get_forces(image))

Below, I am posting target values vs predictions for one of the unseen images using TensorFlow 0.11.0. Properties with a capital letter are those coming from the original calculator and the others are ML predictions:

Test Length: 5
('Energy =', 8.310588888972827)
('energy =', array([8.664198], dtype=float32))
('Forces =', array([[ 0.        ,  0.        ,  0.        ],
       [-1.25839239, -0.52908486, -0.44971333],
       [ 0.        ,  0.        ,  0.        ],
       [-0.3558867 ,  1.70085379, -0.58707117],
       [ 0.63315985, -0.92845858,  0.13016976],
       [-0.26600996,  0.45560162,  0.01447438],
       [ 1.93367376, -0.06405294,  0.65480354],
       [ 0.68849322, -0.24157533,  1.40880371],
       [ 0.12725669,  0.05685044,  0.38817817],
       [-0.08650451, -0.25796725, -0.62260205]]))
('forces =', array([[ 3.6847105e-03,  3.5128396e-03,  9.5515354e-03],
       [ 8.2092658e-03, -3.8295847e-04,  5.4547829e-03],
       [ 2.4447001e-03, -2.2697991e-03,  8.1457123e-03],
       [ 5.5896752e-03, -6.0966425e-03,  7.8025507e-03],
       [-4.7097844e-03,  3.6573287e-03, -6.8126558e-03],
       [ 1.0994184e-03, -6.6993287e-04, -6.6648466e-03],
       [-1.1098579e-02,  1.8842700e-03, -7.8503452e-03],
       [-5.3612911e-03,  8.1996399e-04, -1.6735876e-02],
       [ 1.8541722e-04, -1.7552178e-03,  7.1176267e-03],
       [-4.3533884e-05,  1.3001483e-03, -8.4821713e-06]], dtype=float32))

As you can see, energy predictions seem right, but for forces, it is not the case. Using TensorFlow 1.6.0 is the same:

Test Length: 5
('Energy =', 8.183153071890118)
('energy =', array([8.726102], dtype=float32))
('Forces =', array([[ 0.        ,  0.        ,  0.        ],
       [-0.09430469, -1.75691708, -0.74334414],
       [ 0.        ,  0.        ,  0.        ],
       [-0.32062687,  0.82518729, -0.20991994],
       [-0.00854701, -0.06651756,  0.76368747],
       [ 0.87476395, -0.26460163,  0.45419152],
       [ 1.15732471,  0.3518591 ,  0.50060917],
       [-1.00486088,  0.73141072,  0.78732044],
       [ 0.84537417, -0.2599461 ,  0.35097638],
       [-0.94463245,  0.28454476, -1.08690144]]))
('forces =', array([[ 0.00204374, -0.00087876, -0.00648994],
       [-0.00154252,  0.00188018, -0.00022534],
       [ 0.00164029,  0.00079411, -0.00620799],
       [-0.00169598, -0.00332731, -0.00135216],
       [-0.00205044, -0.00030446,  0.00346738],
       [ 0.00194829,  0.0006923 ,  0.00235331],
       [-0.00176497,  0.00147636,  0.00717454],
       [ 0.00087587, -0.00012318,  0.00458725],
       [-0.00537482, -0.00309596, -0.00819426],
       [ 0.00592053,  0.00288671,  0.00488721]], dtype=float32))

Finally, by reverting the changes above and printing each prediction for the two images in the original test we get:

('Energy =', 9.391512796710966)
('energy =', array([8.907822], dtype=float32))
('Forces =', array([[ 0.00000000e+00,  0.00000000e+00,  0.00000000e+00],
       [ 1.13878891e-01, -7.27048925e-02,  1.66379366e+00],
       [ 0.00000000e+00,  0.00000000e+00,  0.00000000e+00],
       [ 1.13878891e-01,  7.27048925e-02,  1.66379366e+00],
       [-2.21177243e-17, -2.95553512e-16, -1.73360473e+00],
       [ 0.00000000e+00,  2.26635248e-15, -1.59103977e+00],
       [-1.75207071e-15,  1.76594850e-15, -1.72730447e+00],
       [-5.34294831e-16,  3.55271368e-15, -7.41800987e-01],
       [-4.11996826e-18, -8.32667268e-17,  8.09683455e-01],
       [ 0.00000000e+00,  0.00000000e+00, -1.67110814e+00]]))
('forces =', array([[ 2.7275153e-03, -6.7170954e-04,  2.5449730e-03],
       [-2.7275153e-03, -6.7170954e-04,  2.5449730e-03],
       [ 2.7275153e-03,  6.7170954e-04,  2.5449730e-03],
       [-2.7275153e-03,  6.7170954e-04,  2.5449730e-03],
       [-3.9048289e-19, -9.1108568e-19, -7.5114728e-04],
       [-9.1027742e-19, -4.8418743e-19, -1.1872562e-03],
       [-3.9402485e-19,  2.0781786e-19, -4.4262956e-04],
       [-7.3480563e-19,  3.4504131e-19,  1.3120837e-03],
       [-4.1095448e-21,  3.0218169e-19, -1.3046423e-02],
       [ 1.2543149e-23, -1.6664003e-19,  3.9354805e-03]], dtype=float32))
('Energy =', 8.765803171959153)
('energy =', array([8.910725], dtype=float32))
('Forces =', array([[ 0.        ,  0.        ,  0.        ],
       [ 0.04438365, -0.0252446 ,  1.16583466],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.00512817,  0.0735986 ,  0.99858469],
       [ 0.13447516,  0.04628451, -1.25258365],
       [-0.16177521,  0.00476531, -1.12960904],
       [ 0.13352245, -0.06903842, -1.13518543],
       [-0.02881105, -0.11618577, -0.6047823 ],
       [ 0.01005221,  0.2448184 , -2.56701604],
       [-0.01026368, -0.23819793,  1.98938524]]))
('forces =', array([[-2.4364408e-04, -1.0109526e-03, -1.0723310e-04],
       [-7.4692571e-04,  1.5144423e-03, -2.9116340e-03],
       [-8.9384930e-04,  1.6709700e-03, -1.3705894e-03],
       [-8.4772491e-04, -9.0343715e-04, -4.4189831e-03],
       [ 3.4581765e-03,  4.7460230e-04,  2.1660191e-03],
       [-2.9057818e-03,  1.2856203e-03,  4.0753628e-03],
       [ 1.6220108e-03, -6.9638854e-04,  3.0755636e-03],
       [ 3.7753131e-04, -2.2243659e-03,  6.3252676e-04],
       [ 1.6954025e-04, -2.2936975e-04, -2.7846899e-03],
       [ 1.0666315e-05,  1.1887904e-04,  1.6436602e-03]], dtype=float32))

It seems to be a problem in the loss function minimization?