In mathematics, the Hessian matrix or Hessian is a square matrix of second-order partial derivatives of a scalar-valued function, or scalar field. It describes the local curvature of a function of many variables. Hessian Matrices are often used in optimization problems within Newton-Raphson's method.

Hf=[∂2f∂x2∂2f∂x∂y∂2f∂x∂z⋯∂2f∂y∂x∂2f∂y2∂2f∂y∂z⋯∂2f∂z∂x∂2f∂z∂y∂2f∂z2⋯⋮⋮⋮⋱]\mathbf{H} f=\left[ \begin{array}{cccc}{\frac{\partial^{2} f}{\partial x^{2}}} & {\frac{\partial^{2} f}{\partial x \partial y}} & {\frac{\partial^{2} f}{\partial x \partial z}} & {\cdots} \\ {\frac{\partial^{2} f}{\partial y \partial x}} & {\frac{\partial^{2} f}{\partial y^{2}}} & {\frac{\partial^{2} f}{\partial y \partial z}} & {\cdots} \\ {\frac{\partial^{2} f}{\partial z \partial x}} & {\frac{\partial^{2} f}{\partial z \partial y}} & {\frac{\partial^{2} f}{\partial z^{2}}} & {\cdots} \\ {\vdots} & {\vdots} & {\vdots} & {\ddots}\end{array}\right]

Example 1: Computing a Hessian

Problem: Compute the Hessian of f(x,y)=x3−2xy−y6f(x, y)=x^{3}-2 x y-y^{6}.

Solution:

First compute both partial derivatives:

fx(x,y)=∂∂x(x3−2xy−y6)=3x2−2yf_{x}(x, y)=\frac{\partial}{\partial x}\left(x^{3}-2 x y-y^{6}\right)=3 x^{2}-2 y

fy(x,y)=∂∂y(x3−2xy−y6)=−2x−6y5f_{y}(x, y)=\frac{\partial}{\partial y}\left(x^{3}-2 x y-y^{6}\right)=-2 x-6 y^{5}

With these, we compute all four second partial derivatives:

fxx(x,y)=∂∂x(3x2−2y)=6xf_{x x}(x, y)=\frac{\partial}{\partial x}\left(3 x^{2}-2 y\right)=6 x

fxy(x,y)=∂∂y(3x2−2y)=−2{f_{x y}(x, y)=\frac{\partial}{\partial y}\left(3 x^{2}-2 y\right)=-2}

fyx(x,y)=∂∂x(−2x−6y5)=−2{f_{y x}(x, y)=\frac{\partial}{\partial x}\left(-2 x-6 y^{5}\right)=-2}

fyy(x,y)=∂∂y(−2x−6y5)=−30y4f_{y y}(x, y)=\frac{\partial}{\partial y}\left(-2 x-6 y^{5}\right)=-30 y^{4}

The Hessian matrix in this case is a $ 2\times 2$ matrix with these functions as entries:

Hf(x,y)=[fxx(x,y)fxy(x,y)fyx(x,y)fyy(x,y)]=[6x−2−2−30y4]\mathbf{H} f(x, y)=\left[ \begin{array}{cc}{f_{x x}(x, y)} & {f_{x y}(x, y)} \\ {f_{y x}(x, y)} & {f_{y y}(x, y)}\end{array}\right]=\left[ \begin{array}{cc}{6 x} & {-2} \\ {-2} & {-30 y^{4}}\end{array}\right]

Example 2

Problem: the function f(x)=x⊤Ax+b⊤x+cf(x)=x^{\top} A x+b^{\top} x+c, where AA is a n×nn \times n matrix, bb is a vector of length nn and cc is a constant.

  1. Determine the gradient of ff: ∇f(x)\nabla f(x).
  2. Determine the Hessian of ff: Hf(x)H_{f}(x).

Solution:

  1. compute the gradient ∇f(x)\nabla f(x):

∇f(x)=∂xT∂x⋅(Ax)+xT⋅∂(Ax)∂x⎵product−rule+∂bTx∂x+∂c∂x=Ax+xT⋅A+b=Ax+x⋅AT+b=(A+AT)x+b\begin{aligned} \nabla f(x)&=\underbrace{\frac{\partial x^{T}}{\partial x}\cdot (Ax)+x^{T}\cdot \frac{\partial (Ax)}{\partial x}}_{product-rule}+\frac{\partial b^Tx}{\partial x}+\frac{\partial c}{\partial x}\\ &= Ax + x^{T}\cdot A+b \\ &= Ax + x\cdot A^{T} + b \\ &= (A+A^{T})x + b \end{aligned}

  1. compute the Hessian Hf(x)H_{f}(x):

Hf(x)=∂∇f(x)∂x=A+ATH_{f}(x) = \frac{\partial \nabla f(x)}{\partial x} = A + A^{T}