Autodiff for CPU-based programs

[Jacob Hinkle]

99% of my programming time is spent debugging derivatives, which are extremely hard to test without a baseline. I'm creating this issue to basically just bring up a topic that might be useful. I want to look into exposing the CPU-based algorithms to automatic differentiation in numpy. We already have the numpy array access from Field3Ds and Image3Ds. There are of course libraries available for performing ad in reverse mode (generally what we want) in either python/numpy or in C++. The trick is how do we make them talk to one another so that we can eventually just write our objective function and get back our gradient as a numpy array or list of Fields and Images?

To be clear, I'm mostly just interested in getting this working for the CPU code. With that we can write simple tests for our more complicated manual differentiation code that's fully parallelized and runs on cuda, etc.

@jspreston @nikhilsingh @crottman have you guys looked into this at all?

[Sam Preston]

can you be more specific about which libraries / functions you'd like to interoperate with? If it's just numpy, this should be simple -- as long as you just need it for testing (not worried about speed), PyCA.Common.AsNPCopy / PyCA.Common.FieldFromNPArr defines canonical conversions from a Field3D to a numpy array and vice versa, so we should then just be able to test on the numpy arrays -- but maybe I'm missing your point?

[Jacob Hinkle]

I think there are a few different packages that can do reverse mode ad w/ numpy arrays, for instance I think fenics has a library for it, and pyautodiff. I'm not sure if those libraries can make the jump to differentiating library functions like those we have swigged up though. So the reason I made the issue is really to explore whether there would be some manual intervention necessary to make those python tools able to differentiate a chain of pyca function calls.

It may be impractical, but I think it would save a ton of programmer time to just start with our objective function, write a test that compares to the autodiff, then implement an efficient gpu-friendly computation of the gradient until it passes the test.

[Nikhil Singh]

I have tried to give fenics and pyautodiff a shot in the past but was never successful to get it working with my arch distribution. But I agree a pyca implementation of auto-diff would be very useful.