When deriving a finite difference approximation of the j {\displaystyle j} th derivative of a function f : R → R {\displaystyle f:\mathbb {R} \rightarrow \mathbb {R} } , we wish to find a 1 , a 2 , . . . , a n ∈ R {\displaystyle a_{1},a_{2},...,a_{n}\in \mathbb {R} } and b 1 , b 2 , . . . , b n ∈ R {\displaystyle b_{1},b_{2},...,b_{n}\in \mathbb {R} } such that
f ( j ) ( x 0 ) = h − j ∑ i = 1 n a i f ( x 0 + b i h ) + O ( h k ) as h → 0 {\displaystyle f^{(j)}(x_{0})=h^{-j}\sum _{i=1}^{n}a_{i}f(x_{0}+b_{i}h)+O(h^{k}){\text{ as }}h\to 0} or, equivalently,
h − j ∑ i = 1 n a i f ( x 0 + b i h ) = f ( j ) ( x 0 ) + O ( h k ) as h → 0 {\displaystyle h^{-j}\sum _{i=1}^{n}a_{i}f(x_{0}+b_{i}h)=f^{(j)}(x_{0})+O(h^{k}){\text{ as }}h\to 0} where O ( h k ) {\displaystyle O(h^{k})} is the error, the difference between the correct answer and the approximation, expressed using Big-O notation . Because h {\displaystyle h} may be presumed to be small, a larger value for k {\displaystyle k} is better than a smaller value.
A general method for finding the coefficients is to generate the Taylor expansion of h − j ∑ i = 1 n a i f ( x 0 + b i h ) {\displaystyle h^{-j}\sum _{i=1}^{n}a_{i}f(x_{0}+b_{i}h)} and choose a 1 , a 2 , . . . , a n {\displaystyle a_{1},a_{2},...,a_{n}} and b 1 , b 2 , . . . , b n {\displaystyle b_{1},b_{2},...,b_{n}} such that f ( j ) ( x 0 ) {\displaystyle f^{(j)}(x_{0})} and the remainder term are the only non-zero terms. If there are no such coefficients, a smaller value for k {\displaystyle k} must be chosen.
For a function of m {\displaystyle m} variables g : R m → R {\displaystyle g:\mathbb {R} ^{m}\rightarrow \mathbb {R} } , the procedure is similar, except x 0 , b 1 , b 2 , . . . , b n {\displaystyle x_{0},b_{1},b_{2},...,b_{n}} are replaced by points in R m {\displaystyle \mathbb {R} ^{m}} and the multivariate extension of Taylor's theorem is used.
Single-Variable
edit
In all single-variable examples, x 0 ∈ R {\displaystyle x_{0}\in \mathbb {R} } and f : R → R {\displaystyle f:\mathbb {R} \rightarrow \mathbb {R} } are unknown, and h ∈ R {\displaystyle h\in \mathbb {R} } is small. Additionally, let f {\displaystyle f} be 5 times continuously differentiable on R {\displaystyle \mathbb {R} } .
First Derivative
edit
Find a , b , c ∈ R {\displaystyle a,b,c\in \mathbb {R} } such that a f ( x 0 + h ) + b f ( x 0 + c h ) h {\displaystyle {\frac {af(x_{0}+h)+bf(x_{0}+ch)}{h}}} best approximates f ′ ( x 0 ) {\displaystyle f'(x_{0})} .
Let f : R → R {\displaystyle f:\mathbb {R} \rightarrow \mathbb {R} } be 42 times continuously differentiable on R {\displaystyle \mathbb {R} } . Find the largest n ∈ N {\displaystyle n\in \mathbb {N} } such that
d f d x ( x 0 ) = − f ( x 0 + 2 h ) + 8 f ( x 0 + h , y 0 ) − 8 f ( x 0 − h ) + f ( x 0 − 2 h ) 12 h + O ( h n ) as h → 0 {\displaystyle {\frac {df}{dx}}(x_{0})={\frac {-f(x_{0}+2h)+8f(x_{0}+h,y_{0})-8f(x_{0}-h)+f(x_{0}-2h)}{12h}}+O(h^{n}){\text{ as }}h\to 0} In other words, find the order of the error of the method.
Solution:
The Taylor expansion of the method is
− f ( x 0 + 2 h ) + 8 f ( x 0 + h , y 0 ) − 8 f ( x 0 − h ) + f ( x 0 − 2 h ) 12 h = − 1 + 8 − 8 + 1 12 h f ( x 0 ) + − 2 + 8 + 8 − 2 12 f ′ ( x 0 ) + h ( − 4 + 8 − 8 + 4 ) 24 f ″ ( x 0 ) + h 2 ( − 8 + 8 + 8 − 8 72 f ‴ ( x 0 ) + h 3 ( − 16 + 8 − 8 + 16 288 f ( 4 ) ( x 0 ) + h 4 ( − 32 + 8 + 8 − 32 ) 1440 f ( 4 ) ( x 0 ) + O ( h 5 ) as h → 0 . {\displaystyle {\begin{aligned}{\frac {-f(x_{0}+2h)+8f(x_{0}+h,y_{0})-8f(x_{0}-h)+f(x_{0}-2h)}{12h}}&={\frac {-1+8-8+1}{12h}}f(x_{0})+{\frac {-2+8+8-2}{12}}f'(x_{0})\\&+h{\frac {(-4+8-8+4)}{24}}f''(x_{0})+h^{2}{\frac {(-8+8+8-8}{72}}f'''(x_{0})\\&+h^{3}{\frac {(-16+8-8+16}{288}}f^{(4)}(x_{0})+h^{4}{\frac {(-32+8+8-32)}{1440}}f^{(4)}(x_{0})+O(h^{5}){\text{ as }}h\to 0\,.\end{aligned}}} Simplifying this algebraically gives
− f ( x 0 + 2 h ) + 8 f ( x 0 + h , y 0 ) − 8 f ( x 0 − h ) + f ( x 0 − 2 h ) 12 h = f ′ ( x 0 ) + h 4 − 30 f ( 5 ) ( x 0 ) + O ( h 5 ) as h → 0 . {\displaystyle {\frac {-f(x_{0}+2h)+8f(x_{0}+h,y_{0})-8f(x_{0}-h)+f(x_{0}-2h)}{12h}}=f'(x_{0})+{\frac {h^{4}}{-30}}f^{(5)}(x_{0})+O(h^{5}){\text{ as }}h\to 0\,.} The multiple of h 4 {\displaystyle h^{4}} cannot be removed, so n = 4 {\displaystyle n=4} and by properties of Big-O notation,
− f ( x 0 + 2 h ) + 8 f ( x 0 + h , y 0 ) − 8 f ( x 0 − h ) + f ( x 0 − 2 h ) 12 h = f ′ ( x 0 ) + O ( h 4 ) as h → 0 . {\displaystyle {\frac {-f(x_{0}+2h)+8f(x_{0}+h,y_{0})-8f(x_{0}-h)+f(x_{0}-2h)}{12h}}=f'(x_{0})+O(h^{4}){\text{ as }}h\to 0\,.}
Second Derivative
edit
Find a , b , c ∈ R {\displaystyle a,b,c\in \mathbb {R} } such that a f ( x 0 − h ) + b f ( x 0 ) + c f ( x 0 + h ) h 2 {\displaystyle {\frac {af(x_{0}-h)+bf(x_{0})+cf(x_{0}+h)}{h^{2}}}} best approximates f ″ ( x 0 ) {\displaystyle f''(x_{0})} .
Multivariate
edit
In all two-variable examples, x 0 , y 0 ∈ R {\displaystyle x_{0},y_{0}\in \mathbb {R} } and f : R 2 → R {\displaystyle f:\mathbb {R} ^{2}\rightarrow \mathbb {R} } are unknown, and h ∈ R {\displaystyle h\in \mathbb {R} } is small.
Non-Mixed Derivatives
edit
Because of the nature of partial derivatives , some of them may be calculated using single-variable methods. This is done by holding constant all but one variable to form a new function of one variable. For example if g y ( y ) = f ( x 0 , y ) {\displaystyle g_{y}(y)=f(x_{0},y)} , then d f d y ( x 0 , y 0 ) = d g d y ( y 0 ) {\displaystyle {\frac {df}{dy}}(x_{0},y_{0})={\frac {dg}{dy}}(y_{0})} .
Find an approximation of d d y f ( x 0 , y 0 ) {\displaystyle {\frac {d}{dy}}f(x_{0},y_{0})}
Solution:
Because we are differentiating with respect to only one variable, we can hold x constant and use the result of one of the single-variable examples:
f ( x 0 , y 0 + h ) − f ( x 0 , y 0 − h ) 2 h = d f d y ( x 0 , y 0 ) + O ( h 2 ) as h → 0 {\displaystyle {\frac {f(x_{0},y_{0}+h)-f(x_{0},y_{0}-h)}{2h}}={\frac {df}{dy}}(x_{0},y_{0})+O(h^{2}){\text{ as }}h\to 0}
Mixed Derivatives
edit
Mixed derivatives may require the multivariate extension of Taylor's theorem .
Let f : R 2 → R {\displaystyle f:\mathbb {R} ^{2}\rightarrow \mathbb {R} } be 42 times continuously differentiable on R 2 {\displaystyle \mathbb {R} ^{2}} and let g : R 3 → R {\displaystyle g:\mathbb {R} ^{3}\rightarrow \mathbb {R} } be defined as
g ( x 0 , y 0 , h ) = f ( x 0 + h , y 0 + h ) + f ( x 0 − h , y 0 − h ) − f ( x 0 − h , y 0 + h ) − f ( x 0 + h , y 0 − h ) 4 h 2 . {\displaystyle g(x_{0},y_{0},h)={\frac {f(x_{0}+h,y_{0}+h)+f(x_{0}-h,y_{0}-h)-f(x_{0}-h,y_{0}+h)-f(x_{0}+h,y_{0}-h)}{4h^{2}}}\,.} Find the largest n ∈ N {\displaystyle n\in \mathbb {N} } such that
d 2 f d x d y ( x 0 , y 0 ) = g ( x 0 , y 0 , h ) + O ( h n ) as h → 0 . {\displaystyle {\frac {d^{2}f}{dxdy}}(x_{0},y_{0})=g(x_{0},y_{0},h)+O(h^{n}){\text{ as }}h\to 0\,.} In other words, find the order of the error of the approximation.
Solution:
The first few terms of the multivariate Taylor expansion of f {\displaystyle f} around ( x 0 , y 0 ) {\displaystyle (x_{0},y_{0})} are
f ( x 0 + x , y 0 + y ) = f ( x 0 , y 0 ) + x d f d x ( x 0 , y 0 ) + y d f d y ( x 0 , y 0 ) + x 2 2 d 2 f d x 2 ( x 0 , y 0 ) + x y d d y d f d x ( x 0 , y 0 ) + y 2 2 d 2 f d y 2 ( x 0 , y 0 ) + x 3 6 d 3 f d x 3 ( x 0 , y 0 ) + x 2 y 2 d d y d 2 f d x 2 ( x 0 , y 0 ) + x y 2 2 d 2 d y 2 d f d x ( x 0 , y 0 ) + y 3 6 d 3 f d y 3 ( x 0 , y 0 ) + x 4 24 d 4 f d x 4 ( x 0 , y 0 ) + x 3 y 6 d d y d 3 f d x 3 ( x 0 , y 0 ) + x 2 y 2 4 d 2 d y 2 d 2 f d x 2 ( x 0 , y 0 ) + x y 3 6 d 3 d y 3 d f d x ( x 0 , y 0 ) + y 4 24 d 4 f d y 4 ( x 0 , y 0 ) + O ( x 5 ) + O ( y 5 ) as x , y → 0 . {\displaystyle {\begin{aligned}f(x_{0}+x,y_{0}+y)&=f(x_{0},y_{0})\\&+x{\frac {df}{dx}}(x_{0},y_{0})+y{\frac {df}{dy}}(x_{0},y_{0})\\&+{\frac {x^{2}}{2}}{\frac {d^{2}f}{dx^{2}}}(x_{0},y_{0})+xy{\frac {d}{dy}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {y^{2}}{2}}{\frac {d^{2}f}{dy^{2}}}(x_{0},y_{0})\\&+{\frac {x^{3}}{6}}{\frac {d^{3}f}{dx^{3}}}(x_{0},y_{0})+{\frac {x^{2}y}{2}}{\frac {d}{dy}}{\frac {d^{2}f}{dx^{2}}}(x_{0},y_{0})+{\frac {xy^{2}}{2}}{\frac {d^{2}}{dy^{2}}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {y^{3}}{6}}{\frac {d^{3}f}{dy^{3}}}(x_{0},y_{0})\\&+{\frac {x^{4}}{24}}{\frac {d^{4}f}{dx^{4}}}(x_{0},y_{0})+{\frac {x^{3}y}{6}}{\frac {d}{dy}}{\frac {d^{3}f}{dx^{3}}}(x_{0},y_{0})+{\frac {x^{2}y^{2}}{4}}{\frac {d^{2}}{dy^{2}}}{\frac {d^{2}f}{dx^{2}}}(x_{0},y_{0})+{\frac {xy^{3}}{6}}{\frac {d^{3}}{dy^{3}}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {y^{4}}{24}}{\frac {d^{4}f}{dy^{4}}}(x_{0},y_{0})\\&+O(x^{5})+O(y^{5}){\text{ as }}x,y\to 0\,.\end{aligned}}} We substitute the expansion for f {\displaystyle f} into the approximation g {\displaystyle g} to obtain
g ( x 0 , y 0 , h ) = 1 + 1 − 1 − 1 4 h 2 f ( x 0 , y 0 ) + 1 − 1 + 1 − 1 4 h d f d x ( x 0 , y 0 ) + 1 − 1 − 1 + 1 4 h d f d y ( x 0 , y 0 ) + 1 + 1 − 1 − 1 8 d 2 f d x 2 ( x 0 , y 0 ) + 1 + 1 + 1 + 1 4 d d y d f d x ( x 0 , y 0 ) + 1 + 1 − 1 − 1 8 d 2 f d y 2 ( x 0 , y 0 ) + h ( 1 − 1 + 1 − 1 24 d 3 f d x 3 ( x 0 , y 0 ) + 1 − 1 − 1 + 1 8 d d y d 2 f d x 2 ( x 0 , y 0 ) + 1 − 1 + 1 − 1 8 d 2 d y 2 d f d x ( x 0 , y 0 ) + 1 − 1 − 1 + 1 24 d 3 f d y 3 ( x 0 , y 0 ) ) + h 2 ( 1 + 1 − 1 − 1 96 d 4 f d x 4 ( x 0 , y 0 ) + 1 + 1 + 1 + 1 24 d d y d 3 f d x 3 ( x 0 , y 0 ) + 1 + 1 − 1 − 1 16 d 2 d y 2 d 2 f d x 2 ( x 0 , y 0 ) + 1 + 1 + 1 + 1 24 d 3 d y 3 d f d x ( x 0 , y 0 ) + 1 + 1 − 1 − 1 96 d 4 f d y 4 ( x 0 , y 0 ) ) + O ( h 3 ) as h → 0 . {\displaystyle {\begin{aligned}g(x_{0},y_{0},h)&={\frac {1+1-1-1}{4h^{2}}}f(x_{0},y_{0})\\&+{\frac {1-1+1-1}{4h}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {1-1-1+1}{4h}}{\frac {df}{dy}}(x_{0},y_{0})\\&+{\frac {1+1-1-1}{8}}{\frac {d^{2}f}{dx^{2}}}(x_{0},y_{0})+{\frac {1+1+1+1}{4}}{\frac {d}{dy}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {1+1-1-1}{8}}{\frac {d^{2}f}{dy^{2}}}(x_{0},y_{0})\\&+h\left({\frac {1-1+1-1}{24}}{\frac {d^{3}f}{dx^{3}}}(x_{0},y_{0})+{\frac {1-1-1+1}{8}}{\frac {d}{dy}}{\frac {d^{2}f}{dx^{2}}}(x_{0},y_{0})+{\frac {1-1+1-1}{8}}{\frac {d^{2}}{dy^{2}}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {1-1-1+1}{24}}{\frac {d^{3}f}{dy^{3}}}(x_{0},y_{0})\right)\\&+h^{2}\left({\frac {1+1-1-1}{96}}{\frac {d^{4}f}{dx^{4}}}(x_{0},y_{0})+{\frac {1+1+1+1}{24}}{\frac {d}{dy}}{\frac {d^{3}f}{dx^{3}}}(x_{0},y_{0})+{\frac {1+1-1-1}{16}}{\frac {d^{2}}{dy^{2}}}{\frac {d^{2}f}{dx^{2}}}(x_{0},y_{0})+{\frac {1+1+1+1}{24}}{\frac {d^{3}}{dy^{3}}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {1+1-1-1}{96}}{\frac {d^{4}f}{dy^{4}}}(x_{0},y_{0})\right)\\&+O(h^{3}){\text{ as }}h\to 0\,.\end{aligned}}} Because of the careful choices of coefficients, we can simplify this to
g ( x 0 , y 0 , h ) = d d y d f d x ( x 0 , y 0 ) + h 2 6 ( d d y d 3 f d x 3 ( x 0 , y 0 ) + d 3 d y 3 d f d x ( x 0 , y 0 ) ) + O ( h 3 ) as h → 0 . {\displaystyle g(x_{0},y_{0},h)={\frac {d}{dy}}{\frac {df}{dx}}(x_{0},y_{0})+{\frac {h^{2}}{6}}\left({\frac {d}{dy}}{\frac {d^{3}f}{dx^{3}}}(x_{0},y_{0})+{\frac {d^{3}}{dy^{3}}}{\frac {df}{dx}}(x_{0},y_{0})\right)+O(h^{3}){\text{ as }}h\to 0\,.} We note that Big-O notation permits us to write the last 3 terms as O ( h 2 ) as h → 0 {\displaystyle O(h^{2}){\text{ as }}h\to 0} . Thus,
g ( x 0 , y 0 , h ) = d d y d f d x ( x 0 , y 0 ) + O ( h 2 ) as h → 0 . {\displaystyle g(x_{0},y_{0},h)={\frac {d}{dy}}{\frac {df}{dx}}(x_{0},y_{0})+O(h^{2}){\text{ as }}h\to 0\,.} Because the multiples of h 2 {\displaystyle h^{2}} are unaffected by adding more terms to the Taylor expansion, n = 2 {\displaystyle n=2} is the greatest natural number satisfying the conditions given in the problem.
Example Code
edit
Implementing these methods is reasonably simple in programming languages that support higher-order functions . For example, the method from the first example may be implemented in C++ using function pointers , as follows:
// Returns an approximation of the derivative of f at x.
double derivative ( double ( * f )( double ), double x , double h = 0.01 ) {
return ( f ( x + h ) - f ( x - h )) / ( 2 * h );
}