Differentiation

Partial Differentiation of Real-Valued Functions

let $f : D \subset R^{n} \to R$ be a real valued function, $x_{j}, h \in R$

\begin{aligned} \frac{\partial f}{\partial x_{j}} & = lim_{h \to 0} \frac{f (x_{1}, x_{2}, \dots, x_{j} + h, \dots, x_{n}) - f (x_{1}, \dots, x_{n})}{h} \\ = lim_{h \to 0} \frac{f (x + h \cdot \hat{e_{j}}) - f (x)}{h} x \in D \end{aligned}

Gradient

$\nabla f (x_{0}) = [\begin{matrix} \frac{\partial f}{\partial x_{1}} \\ ⋮ \\ \frac{\partial f}{\partial x_{n}} \end{matrix}]$

C1

A real-valued function is called $C^{1}$ at $x_{0}$ if all partial derivatives $\frac{\partial f}{\partial x_{j}}$ exist and are continuous at $x_{0}$

Iterated Partial Derivatives

A function is $C^{2}$ if all partial derivatives of $\frac{\partial f}{\partial x_{j}}$ exist and are continuous or:

$\frac{\partial^{2} f}{\partial x_{j}^{2}}$ exist and are continuous.

A function $f : R^{n} \to R$ is $C^{k}$ if all partial derivatives of $k$ -th order of $f$ exist and are continuous.

Clairaut's Theorem

if $f : R^{n} \to R$ is $C^{2}$ then:

\frac{\partial^{2} f}{\partial x \partial y} = \frac{\partial^{2} f}{\partial y \partial x}

The same idea holds for higher order $C^{k}$

Differentiability of Vector-Valued Functions

Let $f$ be a vector-valued function where $f : D \subset R^{n} \to R^{m}$
$f$ is differentiable at $x_{0} \in D$ if all the partial derivatives of the component functions of $f$ exist at $x_{0}$ and there exists a linear transformation $T : R^{n} \to R^{m}$ such that:

lim_{x \to x_{0}} \frac{| | f (x) - f (x_{0}) - T (x - x_{o}) | |}{| | x - x_{0} | |} = 0

Put otherwise:
$f$ is differentiable if the partial derivatives are $C^{1}$ at $x_{0}$

Jacobian

The Jacobian, which can be denoted $J_{f,} D f (x_{0})$ , or $\nabla f (x_{0})$ is such a linear transformation.

D f (x_{0}) = [\begin{matrix} \frac{\partial f_{1}}{\partial x_{1}} & \dots & \frac{\partial f_{1}}{\partial x_{n}} \\ ⋮ & ⋮ \\ \frac{\partial f_{m}}{\partial x_{n}} & \dots & \frac{\partial f_{m}}{\partial x_{n}} \end{matrix}]

Where $f_{i}$ are the component functions.

The Derivative

If If $f$ is $C^{1}$ at $x_{0}$ , the derivative $f : R^{n} \to R^{m}$ at a point $x_{0}$ , applied to a vector $v \in R^{n}$ , is given by:

v \mapsto D f (x_{0}) \cdot v,

If $f$ is vector-valued function then $D f (x_{0})$ the Jacobian matrix of $f$ at $x_{0}$ .
If $f$ is real-valued the $D f (x_{0})$ is the Gradient of $f$ at $x_{0}$

Properties of the Derivative

Let $f : R^{n} \to R^{m}$ be differentiable at $x_{0}$

Constant Multiple Rule

$c \in R$

D (c f) (x_{0}) = c (D f) (x_{0})

Sum Rule

let $g : R^{n} \to R^{m}$ be differentiable at $x_{0}$

Then $h (x) = g (x) + f (x)$ is differentiable at $x_{0}$ where:

D h (x_{0}) = D g (x_{0}) + D f (x_{0})

Product Rule

let $g : R^{m} \to R$ be differentiable at $x_{0} \in R^{n}$ and $h = g \cdot f$ so that $h : R^{n} \to R^{m}$ .
$h$ is therefore differentiable at $x_{0}$ where:

\begin{array}{r} D h (x_{0}) v = g (x_{0}) D f (x_{0}) v + D g (x_{0}) v f (x_{0}) v \in R^{n} \end{array}

is the derivative of $h$

Quotient Rule

Let $f : R^{n} \to R$ and $g : R^{n} \to R ∖ {0}$ be differentiable at $x_{0}$ and $h = \frac{f}{g}$ .
$h$ is therefore differentiable at $x_{0}$ where:

D h (x_{0}) = \frac{g (x_{0}) D f (x_{0}) - f (x_{0}) D g (x_{0})}{[g (x_{0})]^{2}}

Chain Rule

Let $U \subset R^{n}$ and $V \subset R^{m}$ be open sets.
Let $g : U \to R^{m}$ and $f : V \to R^{k}$

D (f \circ g) (x_{0}) = D f (g (x_{0})) D g (x_{0})

Tangent Plane Real Valued-Function

If $f$ is a real valued function such that $f : U \subset R^{2} \to R$ where $U$ is open. If $f$ is $C^{1}$ at $(x_{0}, y_{0}) \in U$

The tangent plane of the graph of $f$ is:

z = D f (x_{0}, y_{0}) [\begin{matrix} x - x_{0} \\ y - y_{0} \end{matrix}] + f (x_{0}, y_{0})

Linear Approximation/First-Order Taylor Expansion

if $f : D \subset R^{n} \to R^{m}$ is differentiable at $x_{0} \in D$ then for a given $ϵ > 0$ there exists some neighbourhood $U$ around $x_{0}$ so that $x \in U$ then $| | f (x) - D f (x_{0}) x - f (x_{0}) | | < ϵ$

Therefore we can approximate $f$ as:

f (x_{0}) = D f (x_{0}) (x - x_{0}) + f (x_{0})

Hessian

Let $f : R^{n} \to R$ be $C^{2}$

H_{f} = [\begin{matrix} \frac{\partial^{2} f}{\partial x_{1}^{2}} & \frac{\partial^{2} f}{\partial x_{1} \partial x_{2}} & \dots & \frac{\partial^{2} f}{\partial x_{1} \partial x_{n}} \\ \frac{\partial^{2} f}{\partial x_{2} \partial x_{1}} & \frac{\partial^{2} f}{\partial x_{2}^{2}} & \dots & \frac{\partial^{2} f}{\partial x_{2} \partial x_{n}} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ \frac{\partial^{2} f}{\partial x_{n} \partial x_{1}} & \frac{\partial^{2} f}{\partial x_{n} \partial x_{2}} & \dots & \frac{\partial^{2} f}{\partial x_{n}^{2}} \end{matrix}]

Second-Order Taylor Expansion

if $f : R^{n} \to R$ is $C^{2}$ at $x_{0}$
The second-order Taylor expansion is:

f (x_{0}) \approx f (x_{0}) + \nabla f (x_{0})^{⊤} (x_{} - x_{0}) + \frac{1}{2} (x - x_{0})^{⊤} H_{f} (x_{0}) (x - x_{0}),