Numerical Analysis/Differentiation/Examples

When deriving a finite difference approximation of the $j$th derivative of a function $f:\mathbb {R} \rightarrow \mathbb {R}$, we wish to find $a_{1},a_{2},...,a_{n}\in \mathbb {R}$ and $b_{1},b_{2},...,b_{n}\in \mathbb {R}$ such that

$f^{(j)}(x_{0})=h^{-j}\sum _{i=1}^{n}a_{i}f(x_{0}+b_{i}h)+O(h^{k}){\text{ as }}h\to 0$

or, equivalently,

$h^{-j}\sum _{i=1}^{n}a_{i}f(x_{0}+b_{i}h)=f^{(j)}(x_{0})+O(h^{k}){\text{ as }}h\to 0$

where $O(h^{k})$ is the error, the difference between the correct answer and the approximation, expressed using Big-O notation. Because $h$ may be presumed to be small, a larger value for $k$ is better than a smaller value.

A general method for finding the coefficients is to generate the Taylor expansion of $h^{-j}\sum _{i=1}^{n}a_{i}f(x_{0}+b_{i}h)$ and choose $a_{1},a_{2},...,a_{n}$ and $b_{1},b_{2},...,b_{n}$ such that $f^{(j)}(x_{0})$ and the remainder term are the only non-zero terms. If there are no such coefficients, a smaller value for $k$ must be chosen.

For a function of $m$ variables $g:\mathbb {R} ^{m}\rightarrow \mathbb {R}$, the procedure is similar, except $x_{0},b_{1},b_{2},...,b_{n}$ are replaced by points in $\mathbb {R} ^{m}$ and the multivariate extension of Taylor's theorem is used.

Single-Variable

In all single-variable examples, $x_{0}\in \mathbb {R}$ and $f:\mathbb {R} \rightarrow \mathbb {R}$ are unknown, and $h\in \mathbb {R}$ is small. Additionally, let $f$ be 5 times continuously differentiable on $\mathbb {R}$.

First Derivative

Find $a,b,c\in \mathbb {R}$ such that ${\frac {af(x_{0}+h)+bf(x_{0}+ch)}{h}}$ best approximates $f'(x_{0})$.

Solution:

First, we find the Taylor series expansion of ${\frac {af(x_{0}+h)+bf(x_{0}+ch)}{h}}$ with remainder term to be

${\frac {af(x_{0}+h)+bf(x_{0}+ch)}{h}}={\frac {a+b}{h}}f(x_{0})+(a+bc)f'(x_{0})+{\frac {h}{2}}(a+bc^{2})f''(x_{0})+O(h^{2}){\text{ as }}h\to 0\,.$

If we can find a solution to the system

${\begin{aligned}0&=&a+&b&\\1&=&a+&bc&\\0&=&a+&bc^{2}&\end{aligned}}$

then we can substitute that solution into the Taylor expansion to obtain

${\frac {af(x_{0}+h)+bf(x_{0}+ch)}{h}}=f'(x_{0})+O(h^{2}){\text{ as }}h\to 0\,.$

The system of equations has exactly one solution: $a={\frac {1}{2}}$, $b={\frac {1}{-2}}$, $c=-1$, so

${\frac {f(x_{0}+h)-f(x_{0}-h)}{2h}}=f'(x_{0})+O(h^{2}){\text{ as }}h\to 0\,.$

Let $f:\mathbb {R} \rightarrow \mathbb {R}$ be 42 times continuously differentiable on $\mathbb {R}$. Find the largest $n\in \mathbb {N}$ such that

${\frac {df}{dx}}(x_{0})={\frac {-f(x_{0}+2h)+8f(x_{0}+h,y_{0})-8f(x_{0}-h)+f(x_{0}-2h)}{12h}}+O(h^{n}){\text{ as }}h\to 0$

In other words, find the order of the error of the method.

Solution:

The Taylor expansion of the method is

${\begin{aligned}{\frac {-f(x_{0}+2h)+8f(x_{0}+h,y_{0})-8f(x_{0}-h)+f(x_{0}-2h)}{12h}}&={\frac {-1+8-8+1}{12h}}f(x_{0})+{\frac {-2+8+8-2}{12}}f'(x_{0})\\&+h{\frac {(-4+8-8+4)}{24}}f''(x_{0})+h^{2}{\frac {(-8+8+8-8}{72}}f'''(x_{0})\\&+h^{3}{\frac {(-16+8-8+16}{288}}f^{(4)}(x_{0})+h^{4}{\frac {(-32+8+8-32)}{1440}}f^{(4)}(x_{0})+O(h^{5}){\text{ as }}h\to 0\,.\end{aligned}}$

Simplifying this algebraically gives

${\frac {-f(x_{0}+2h)+8f(x_{0}+h,y_{0})-8f(x_{0}-h)+f(x_{0}-2h)}{12h}}=f'(x_{0})+{\frac {h^{4}}{-30}}f^{(5)}(x_{0})+O(h^{5}){\text{ as }}h\to 0\,.$

The multiple of $h^{4}$ cannot be removed, so $n=4$ and by properties of Big-O notation,

${\frac {-f(x_{0}+2h)+8f(x_{0}+h,y_{0})-8f(x_{0}-h)+f(x_{0}-2h)}{12h}}=f'(x_{0})+O(h^{4}){\text{ as }}h\to 0\,.$

Second Derivative

Find $a,b,c\in \mathbb {R}$ such that ${\frac {af(x_{0}-h)+bf(x_{0})+cf(x_{0}+h)}{h^{2}}}$ best approximates $f''(x_{0})$.

Solution:

First, we find the Taylor series expansion of ${\frac {af(x_{0}-h)+bf(x_{0})+cf(x_{0}+h)}{h^{2}}}$, with remainder term to be

${\frac {af(x_{0}-h)+bf(x_{0})+cf(x_{0}+h)}{h^{2}}}={\frac {a+b+c}{h^{2}}}f(x_{0})+{\frac {c-a}{h}}f'(x_{0})+{\frac {a+c}{2}}f''(x_{0})+{\frac {h}{6}}(c-a)f'''(x_{0})+O(h^{2}){\text{ as }}h\to 0\,.$

If we can find a solution to the system

${\begin{aligned}0&=&&a&+&b&+&c&\\0&=&&-a&+&0&+&c&\\2&=&&a&+&0&+&c&\\0&=&&-a&+&0&+&c&\end{aligned}}$

then we can substitute that solution into the Taylor expansion and obtain

${\frac {af(x_{0}-h)+bf(x_{0})+cf(x_{0}+h)}{h^{2}}}=f''(x_{0})+O(h^{2}){\text{ as }}h\to 0\,.$

The system of equations has exactly one solution: $a=1$, $b=-2$, $c=1$ so

${\frac {f(x_{0}-h)-2f(x_{0})+f(x_{0}+h)}{h^{2}}}=f''(x_{0})+O(h^{2}){\text{ as }}h\to 0\,.$

Multivariate

In all two-variable examples, $x_{0},y_{0}\in \mathbb {R}$ and $f:\mathbb {R} ^{2}\rightarrow \mathbb {R}$ are unknown, and $h\in \mathbb {R}$ is small.

Non-Mixed Derivatives

Because of the nature of partial derivatives, some of them may be calculated using single-variable methods. This is done by holding constant all but one variable to form a new function of one variable. For example if $g_{y}(y)=f(x_{0},y)$, then ${\frac {df}{dy}}(x_{0},y_{0})={\frac {dg}{dy}}(y_{0})$.

Find an approximation of ${\frac {d}{dy}}f(x_{0},y_{0})$

Solution:

Because we are differentiating with respect to only one variable, we can hold x constant and use the result of one of the single-variable examples:

${\frac {f(x_{0},y_{0}+h)-f(x_{0},y_{0}-h)}{2h}}={\frac {df}{dy}}(x_{0},y_{0})+O(h^{2}){\text{ as }}h\to 0$

Mixed Derivatives

Mixed derivatives may require the multivariate extension of Taylor's theorem.

Let $f:\mathbb {R} ^{2}\rightarrow \mathbb {R}$ be 42 times continuously differentiable on $\mathbb {R} ^{2}$ and let $g:\mathbb {R} ^{3}\rightarrow \mathbb {R}$ be defined as

$g(x_{0},y_{0},h)={\frac {f(x_{0}+h,y_{0}+h)+f(x_{0}-h,y_{0}-h)-f(x_{0}-h,y_{0}+h)-f(x_{0}+h,y_{0}-h)}{4h^{2}}}\,.$

Find the largest $n\in \mathbb {N}$ such that

${\frac {d^{2}f}{dxdy}}(x_{0},y_{0})=g(x_{0},y_{0},h)+O(h^{n}){\text{ as }}h\to 0\,.$

In other words, find the order of the error of the approximation.

Solution:

The first few terms of the multivariate Taylor expansion of $f$ around $(x_{0},y_{0})$ are

${\begin{aligned}f(x_{0}+x,y_{0}+y)&=f(x_{0},y_{0})\\&+x{\frac {df}{dx}}(x_{0},y_{0})+y{\frac {df}{dy}}(x_{0},y_{0})\\&+{\frac {x^{2}}{2}}{\frac {d^{2}f}{dx^{2}}}(x_{0},y_{0})+xy{\frac {d}{dy}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {y^{2}}{2}}{\frac {d^{2}f}{dy^{2}}}(x_{0},y_{0})\\&+{\frac {x^{3}}{6}}{\frac {d^{3}f}{dx^{3}}}(x_{0},y_{0})+{\frac {x^{2}y}{2}}{\frac {d}{dy}}{\frac {d^{2}f}{dx^{2}}}(x_{0},y_{0})+{\frac {xy^{2}}{2}}{\frac {d^{2}}{dy^{2}}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {y^{3}}{6}}{\frac {d^{3}f}{dy^{3}}}(x_{0},y_{0})\\&+{\frac {x^{4}}{24}}{\frac {d^{4}f}{dx^{4}}}(x_{0},y_{0})+{\frac {x^{3}y}{6}}{\frac {d}{dy}}{\frac {d^{3}f}{dx^{3}}}(x_{0},y_{0})+{\frac {x^{2}y^{2}}{4}}{\frac {d^{2}}{dy^{2}}}{\frac {d^{2}f}{dx^{2}}}(x_{0},y_{0})+{\frac {xy^{3}}{6}}{\frac {d^{3}}{dy^{3}}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {y^{4}}{24}}{\frac {d^{4}f}{dy^{4}}}(x_{0},y_{0})\\&+O(x^{5})+O(y^{5}){\text{ as }}x,y\to 0\,.\end{aligned}}$

We substitute the expansion for $f$ into the approximation $g$ to obtain

${\begin{aligned}g(x_{0},y_{0},h)&={\frac {1+1-1-1}{4h^{2}}}f(x_{0},y_{0})\\&+{\frac {1-1+1-1}{4h}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {1-1-1+1}{4h}}{\frac {df}{dy}}(x_{0},y_{0})\\&+{\frac {1+1-1-1}{8}}{\frac {d^{2}f}{dx^{2}}}(x_{0},y_{0})+{\frac {1+1+1+1}{4}}{\frac {d}{dy}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {1+1-1-1}{8}}{\frac {d^{2}f}{dy^{2}}}(x_{0},y_{0})\\&+h\left({\frac {1-1+1-1}{24}}{\frac {d^{3}f}{dx^{3}}}(x_{0},y_{0})+{\frac {1-1-1+1}{8}}{\frac {d}{dy}}{\frac {d^{2}f}{dx^{2}}}(x_{0},y_{0})+{\frac {1-1+1-1}{8}}{\frac {d^{2}}{dy^{2}}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {1-1-1+1}{24}}{\frac {d^{3}f}{dy^{3}}}(x_{0},y_{0})\right)\\&+h^{2}\left({\frac {1+1-1-1}{96}}{\frac {d^{4}f}{dx^{4}}}(x_{0},y_{0})+{\frac {1+1+1+1}{24}}{\frac {d}{dy}}{\frac {d^{3}f}{dx^{3}}}(x_{0},y_{0})+{\frac {1+1-1-1}{16}}{\frac {d^{2}}{dy^{2}}}{\frac {d^{2}f}{dx^{2}}}(x_{0},y_{0})+{\frac {1+1+1+1}{24}}{\frac {d^{3}}{dy^{3}}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {1+1-1-1}{96}}{\frac {d^{4}f}{dy^{4}}}(x_{0},y_{0})\right)\\&+O(h^{3}){\text{ as }}h\to 0\,.\end{aligned}}$

Because of the careful choices of coefficients, we can simplify this to

$g(x_{0},y_{0},h)={\frac {d}{dy}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {h^{2}}{6}}\left({\frac {d}{dy}}{\frac {d^{3}f}{dx^{3}}}(x_{0},y_{0})+{\frac {d^{3}}{dy^{3}}}{\frac {df}{dx}}(x_{0},y_{0})\right)+O(h^{3}){\text{ as }}h\to 0\,.$

We note that Big-O notation permits us to write the last 3 terms as $O(h^{2}){\text{ as }}h\to 0$. Thus,

$g(x_{0},y_{0},h)={\frac {d}{dy}}{\frac {df}{dx}}(x_{0},y_{0})+O(h^{2}){\text{ as }}h\to 0\,.$

Because the multiples of $h^{2}$ are unaffected by adding more terms to the Taylor expansion, $n=2$ is the greatest natural number satisfying the conditions given in the problem.

Example Code

Implementing these methods is reasonably simple in programming languages that support higher-order functions. For example, the method from the first example may be implemented in C++ using function pointers, as follows:

//  Returns an approximation of the derivative of f at x.
double derivative (double (*f)(double), double x, double h =0.01) {
  return (f(x + h) - f(x - h)) / (2 * h);
}