Taylor Series

$f(x+ϵ)f(x)+ϵf^{\prime }(x)$

Power Series

${\sum }_{n=0}^{\infty }{C}_n(x-a{)}^n$

Taylor Series

${\sum }_{n=0}^{\infty }\frac{f^n}{n!}(x-a{)}^n$ $f(x)$

Global vs local (important contrast)

Global approximation

One formula tries to match the function everywhere
(example: Fourier series on an interval)

Local approximation

One formula matches the function at one point and nearby Taylor is local because:

• It matches derivatives at one point aa
• Accuracy degrades as you move away from 

Why gradient descent is local

Gradient descent assumes:

$f(x)\approx f(x_k)+\nabla f(x_k)T(x-x_k)f(x)\approx f(x_k)+\nabla f(x_k)T(x-x_k)$

This is only reliable if:

$\Vert x-xk\Vert$ is small $\Vert x-xk\Vert$ is small

That’s why:

• Learning rate must be small
• Large steps break GD

Core reason (most important)

Optimization is invariant to adding or removing constants.

Formally:

$arg{\min }_{x}f(x)=arg{\min }_x(f(x)+C)$

for any constant $C$ independent of $x$.

mk Beg