THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
import mxnet.ndarray as nd
from mxnet import autograd
x = nd.array([1, 2, 3])
x.attach_grad()
with autograd.record():
y = nd.sin(x)
# y_grad is first order gradient of y and should be cos(x)
y_grad = autograd.grad(y, x, create_graph=True, retain_graph=True)[0]
# this call should calculate the second order of y w.r.t x which should be -sin(x)
y_grad.backward()
print(x.grad) # Should be -sin(x)
Goals/Usecases
...
This project will implement a set of operators to support adaptive learning rate optimization proposed by http://proceedings.mlr.press/v70/franceschi17a.html. The following operators will be supported in this project
Operator | Second order support? | ETA |
---|---|---|
exp | Y | |
elemwise_mul | N | 04/2019 |
sin | N | 04/2019 |
cos | N | 04/2019 |
relu | N | 04/2019 |
dot | N | 04/2019 |
negative | N | 04/2019 |
Open Questions
TBD
Proposed Approach
MXNet Java Inference API#ProposedApproach
MXNet Java Inference API#ClassDiagram
MXNet Java Inference API#SequenceDiagramThe reason that many operators currently do not support second or higher order gradient is because that
Addition of New APIs
Backward compatibility
...