Then, the fact that Newton Raphson is making an adapted step is actually bad you don't want to immediately optimize the approximated function because you will be jumping too far and will never converge. An interesting case of why you might want to is if you are doing something like mini-batch updates, where you are approximating the target function by only using a small portion of the full data (I mention this because the SGD tag was used). Adaptive step sizes for first order methods are strongly motivated by trying to adapt the step size by the Hessian, which Newton Raphson already does. In general, I cannot see any good reason for an adaptive step size outside using a line search. If the function is particularly unstable (i.e., Hessian is very non-stationary), it's often a good idea to use a line-search method to, for example, check whether half stepping actually decreases the target function more than the full step proposed by vanilla Newton-Raphson. However, if the problem is convex, we can show that the algorithm will still converge. Of course, generally the target function is not quadratic, or we don't need the iterative Newton Raphson algorithm as there is a closed form solution (i.e., the first step).
NEWTON RAPHSON METHOD FOURBAR UPDATE
If the function is quadratic, this the "optimal" update in that in converges in one step. In a sense, Newton Raphson is automatically doing the adaptive step size it's adapting the step in each dimension (which changes the direction) according to the rate of change of the gradient.
0 kommentar(er)
