Scribd Logo
Upload

Welcome to Scribd!

  • 50 views

    Second Order Method: Newton Method Quasi Newton Method

    Uploaded by

    indah_yanti
    The document discusses various second-order optimization methods, including Newton's method, quasi-Newton methods, and the BFGS method. Newton's method finds the minimum of a function by taking the second derivative (Hessian matrix) into account. Quasi-Newton methods approximate the Hessian matrix without calculating second derivatives. The BFGS method is a popular quasi-Newton method that updates the approximate Hessian matrix using gradient information from previous iterations.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd

    Second Order Method: Newton Method Quasi Newton Method

    Uploaded by

    indah_yanti
    50 views11 pages
    The document discusses various second-order optimization methods, including Newton's method, quasi-Newton methods, and the BFGS method. Newton's method finds the minimum of a function by taking the second derivative (Hessian matrix) into account. Quasi-Newton methods approximate the Hessian matrix without calculating second derivatives. The BFGS method is a popular quasi-Newton method that updates the approximate Hessian matrix using gradient information from previous iterations.

    Original Description:

    newton

    Original Title

    03 Newton Method

    Copyright

    © © All Rights Reserved

    Available Formats

    PPTX, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    The document discusses various second-order optimization methods, including Newton's method, quasi-Newton methods, and the BFGS method. Newton's method finds the minimum of a function by taking the second derivative (Hessian matrix) into account. Quasi-Newton methods approximate the Hessian matrix without calculating second derivatives. The BFGS method is a popular quasi-Newton method that updates the approximate Hessian matrix using gradient information from previous iterations.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Download as pptx, pdf, or txt
    50 views11 pages

    Second Order Method: Newton Method Quasi Newton Method

    Uploaded by

    indah_yanti
    The document discusses various second-order optimization methods, including Newton's method, quasi-Newton methods, and the BFGS method. Newton's method finds the minimum of a function by taking the second derivative (Hessian matrix) into account. Quasi-Newton methods approximate the Hessian matrix without calculating second derivatives. The BFGS method is a popular quasi-Newton method that updates the approximate Hessian matrix using gradient information from previous iterations.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Download as pptx, pdf, or txt
    of 11

    Second Order Method

    Newton Method
    Quasi Newton Method

    Newton Method
    It is a second order method.
    Let x(k) be the current point. The Taylor expansion of the objective function about
    x(k):
    f(x(k +1)) = f (x(k)) + f (x(k)) T x + 1/2 x T 2f (x(k)) x + O (x3)
    The quadratic approximation of f(x) is
    f (x) = f(x(k)) + f (x(k)) T x+ 1/2x T 2 f (x(k)) x
    We need to find the critical point of the approximation:
    f(x(k)) + 2f(x(k)) x = 0
    x = 2f(x(k)) 1 f(x(k))

    Newton Method
    The Newton optimization method is
    x(k + 1) = x(k) (2f(x(k))) 1 f(x(k))
    If the function f(x) is quadratic, the solution can be
    found in exactly one step.

    Modified Newtons Method


    Newton method can be unreliable for nonquadratic functions.
    The Newton step will often be large when x(0) is far
    from x*.
    To solve this problem, we add a step length:
    x(k + 1) = x(k) (k) (2f(x(k))) 1 f(x(k))

    Quasi-Newton Method
    Quasi-Newton methods use a Hessian-like matrix but without calculating
    second-order derivatives.
    Sometimes, these methods are referred to as the variable metric methods.
    Take the general formula:
    x(k + 1) = x(k) A(k) f(x(k))
    When A(k) = I (identity matrix), the formula becomes the formula of the
    steepest descent method.
    When A(k) = 2f(x(k))-1, the formula becomes the formula of the Newton
    method.

    Starting from a positive definite matrix, the quasiNewton methods gradually build up an approximate
    Hessian matrix by using gradient information from the
    previous iterations.
    The matrix A is kept positive definite; hence the
    direction
    s(k) = - A(k)f(x(k))
    remains a descent direction.
    There are several ways to update the matrix A.

    Davidon-Fletcher-Powel Formula
    Earliest (and one of the most clever) schemes for constructing the inverse
    Hessian was originally proposed by Davidon (1959) and later developed by
    Fletcher and Powell (1963).
    It has the interesting property that, for a quadratic objective, it
    simultaneously generates the directions of the conjugate gradient method
    while constructing the inverse Hessian.
    The method is also referred to as the variable metric method (originally
    suggested by Davidon).
    A(k +1) = A(k) (A(k)(k) ((k))TA(k)/ (k) A(k)) + ((k) ((k))T/ ((k))T (k))
    where A(0) = I, (k) = x(k+1) x(k) and (k) = f(x(k+1)) f(x(k))

    Quasi Newton Algorithm


    1. Input x(0) , A(0), termination criteria.
    2.For any k, set s(k) = - A(k)f(x(k))
    3.Compute a step size and set x(k+1) = x(k) + s(k)
    4.Compute the update matrix Au(k) according to a given formula
    5.Set A(k + 1) = A(k) + Au(k)
    6.Continue with next k until termination criteria are satisfied

    BFGS method
    Broyden-Fletcher-Goldfarb-Shanno.
    To solve a nonlinear optimization problem.
    Is derived from gradient descent.
    Using an approximate Hessian matrix, by analyzing successive gradient vectors.
    Allowing the application of quasi-Newton condition to update Hessian matrix.
    B(k + 1)(x(k +1) x(k)) = f(x(k + 1)) f(x(k))
    B(k +1) = B(k) (B(k)(k)(B(k)(k))T/((k))TB(k)(k)) + ((k) ((k))T/ ((k))T (k)) + ((k))TB(k)(k)vvT)
    Where
    (k) = f(x(k+1)) f(x(k)), (k) = x(k+1) x(k) , v = ( (k)/ ((k))T (k)) (B(k)(k)/((k))TB(k)(k))

    Algorithm
    1.

    Input x(0) , B(0), termination criteria.

    2. For any k, set s(k) = - B(k)f(x(k))


    3. Compute a step size and set x(k+1) = x(k) + s(k)
    4. Compute the update matrix
    5.

    If f(x(k)) < treshold, STOP. Else, go to step 2

    Group Assignment (Homework)


    Find an algorithm for newton method and give an example.
    Minimize f (x) = x12 + 10x22 using the quasi-Newton method with x(0) = [1,
    1]T .
    Stop when ||f(x(k))|| 0.0001
    Find the local minimum of
    f(x)= 5x12 + 2x1x2 + x22 + 7
    using BFGS with initial point x(0)= (1, 2), = 0.001
    Collect it on next Friday
    Place: my locker
    Max: 10.00 am

    You might also like

    pFad - Phonifier reborn

    Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

    Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


    Alternative Proxies:

    Alternative Proxy

    pFad Proxy

    pFad v3 Proxy

    pFad v4 Proxy