E. T. S. I. Caminos, Canales y Puertos 1
Lecture 2
EngineeringComputation
E. T. S. I. Caminos, Canales y Puertos 2
Smearing Occurs when individual terms are larger than summation itself. Consider the exponential series with x = -10
Consider formulas such as:
With 7-decimal-digit accuracy:exact answer = 4.54 10-05
computed answer = – 6.26 10-05
(45 terms) wrong sign !
Largest intermediate terms are:9th = –2,755.732 & 10th = 2,755.732
2 3 4 5x x x x x
e 1 x2! 3! 4! 5!
Round-off Error due to Arithmetic Operations
E. T. S. I. Caminos, Canales y Puertos 3
Error caused by the nature of the numerical technique employed to approximate the solution.
Example:
Maclaurin series expansion of ex
2 3 4 5x x x x x
e 1 x2! 3! 4! 5!
2x x
e 1 x2!
3 4 5x x x
3! 4! 5!
If we use a truncated version of the series:
Then the Truncation Error is:
Truncation Error
E. T. S. I. Caminos, Canales y Puertos 4
Approximations and Rounding Errors
• Precautions:– Sums of large and small numbers: due to equaling the
exponent. They are common in sums of infinite series where the individual terms are very small when compared with the accumulated sum. This error can be reduced by summing first the small terms and using double precision.
– Cancellation of the subtraction: The subtraction of very similar numbers.
– Smearing: The individual terms are larger than the total sum.
– Inner products: They are prone to rounding errors. Thus, it is convenient to use double precision in this type of calculations.
n
1inn2211ii yxyxyxyx
E. T. S. I. Caminos, Canales y Puertos 5
Error PropagationErrors which appear because we are basing current calculations on previous calculations which also incurred some form of error
Stability and Condition NumberNumerically Unstable: Computations which are so sensitive to round-off errors that errors grow uncontrollably during calculations.Condition: sensitivity to such uncertainty; "well conditioned" vs. "ill conditioned"Condition Number: measure of the condition; i.e., extent to which uncertainty in x is amplified by ƒ(x)
C.N. 1 ===> "well-conditioned" C.N. >> 1 ===> "ill-conditioned"
E. T. S. I. Caminos, Canales y Puertos 6
Basic Idea:
Predict the value of a function, ƒ, at a point xi+1 based on the value of the function and all of its derivatives, ƒ, ƒ', ƒ",… at a neighboring point xi
Given xi, ƒ(xi), ƒ'(xi), ƒ"(xi), ... ƒn+1(xi),
we can predict or approximate ƒ(xi+1)
Taylor Series Expansion
E. T. S. I. Caminos, Canales y Puertos 7
General Form:
h = "step size" = xi+1 – xi
Rn = remainder to account for all other terms
= O (hn+1) with x not exactly known "on the order of hn+1 "
Note: f(x) must be a function with n+1 continuous derivatives
2 3 nn
i 1 i i i i i nh h h
f (x ) f (x ) hf (x ) f (x ) f (x ) f (x ) R2! 3! n!
n 1n 1h
f ( )(n 1)!
with xi xi+1
Taylor Series Expansion
E. T. S. I. Caminos, Canales y Puertos 8
0th order T.S. approx. (n = 0): f(xi+1) = f(xi) + O (h1)
1st order T.S. approx. (n = 1): f(xi+1) = f(xi) + hf '(xi) + O (h2)
2nd order T.S. approx. (n = 2):
nth order T.S. approximation will be exact for an nth order polynomial
2 nn n 1
i 1 i i i ih h
f (x ) f (x ) h f (x ) f (x ) f (x ) (h )2! n!
O
2n 1
i 1 i i ih
f (x ) f (x ) h f (x ) f (x ) (h )2!
O
Taylor Series Expansion
E. T. S. I. Caminos, Canales y Puertos 9
Zero orderFirst orderSecond order
f(xi )f(xi+1 ) f(xi )
f(xi+1 ) f(xi )+f '(xi )h
f(xi+1 ) f(xi )+f '(xi )h+ )+f "(xi )h2/2!
f(xi+1 )
True
f(x )
xi+1xi
h
x
Taylor Series Expansion
E. T. S. I. Caminos, Canales y Puertos 10
Objective:Evaluate the derivatives of function, ƒ(xi), without doing it analytically.
When would we want to do this?1. function is too complicated to differentiate analytically:
2. function is not defined by an equation,
i.e., given a set of data points (xi, ƒ(xi)), i=1,…,n
i 0 1 2 3 4
xi 1.0 3.0 5.0 7.0 9.0ƒ(xi) 2.3 4.1 5.5 5.7 5.9
0.5x2 cos(1 x )e
1 0.5x
Numerical Differentiation from Taylor Series Expansion
E. T. S. I. Caminos, Canales y Puertos 11
Numerical Differentiation from Taylor Series Expansion
– First derivative with backward difference.
)xx)(x('f)x(f)x(f 1iiii1i
)xx()x(f)x(f
)x('f1ii
1iii
E. T. S. I. Caminos, Canales y Puertos 12
Backward Difference Approx.:
First Derivative:
first backward difference
2
i 1 i i 1 i ih
f (x ) f (x ) (x x )f '(x ) f "( )2
2
i 1 i ih
f (x ) f (x ) hf '(x ) f "( )2
Letting h = xi - xi-1
2
i i i 1h
hf '(x ) f (x ) f (x ) f "( )2
i i 1f (x ) f (x )f '(x) (h)
h
O
Numerical Differentiation from Taylor Series Expansion
E. T. S. I. Caminos, Canales y Puertos 13
Using data below calculate ƒ'(x1) :
i 0 1 2 3 4xi 1.0 3.0 5.0 7.0 9.0
ƒ(xi) 2.3 4.1 5.5 5.7 5.9
First Backward Finite-Divided-Difference at x1:
1 0f (x ) f (x )f '(x) (h)
h
O
14.1 2.3
f '(x ) (h)2
O
f ' (x1) 0.9 { + O (h) }
Example of 1st Backward FDD
E. T. S. I. Caminos, Canales y Puertos 14
Second Derivative:
2i 2 i
i 2 i i 2 i ix x
f (x ) f (x ) x x f '(x) f "(x )2!
+ O([xi-2– xi]3)
with h = xi– xi-1 and 2h = xi – xi-2
The 2nd order approximation to ƒ(xi-2) becomes:
ƒ(xi-2) = ƒ(xi) – 2hƒ'(xi) + 2h2 ƒ"(xi) +O (h3) [1]
2nd order approximation to ƒ(xi-1):2
3i 1 i i
h(x ) (x ) h '(x) "(x ) (h )
2!f f f f O [2]
Backward Difference Approximation
E. T. S. I. Caminos, Canales y Puertos 15
Subtracting 2*[2] from [1] yields:
f(xi-2) – 2f(xi-1) = –f(xi) + h2f"(xi) + O (h3)
Rearranging:
h2ƒ"(xi) = f(xi) – 2f(xi-1) + f(xi-2) + O (h3)
Second backward difference
3i i 1 i 2
i 2
f (x ) 2f (x ) f (x ) O(h )f "(x )
h
i i 1 i 2i 2
f (x ) 2f (x ) f (x )f "(x ) O(h)
h
Backward Difference Approximation
E. T. S. I. Caminos, Canales y Puertos 16
Using data below calculate ƒ"(x2) :i 0 1 2 3 4
xi 1.0 3.0 5.0 7.0 9.0ƒ(xi) 2.3 4.1 5.5 5.7 5.9
Second Backward Finite-Divided-Difference at x2:
2 1 02 2
f (x ) 2f (x ) f (x )f "(x ) (h)
h
O
2 2
5.5 2*4.1 2.3f "(x ) (h)
2
O
f " (5.0) - 0.1 { + O (h) }
Example of 2nd Backward FDD
E. T. S. I. Caminos, Canales y Puertos 17
What points are used for each form?
Backward:
…, ƒ(xi-2), ƒ(xi-1), ƒ(xi), ƒ(xi+1), ƒ(xi+2), …
Forward:
…, ƒ(xi-2), ƒ(xi-1), ƒ(xi), ƒ(xi+1), ƒ(xi+2), …
Centered:
…, ƒ(xi-2), ƒ(xi-1), ƒ(xi), ƒ(xi+1), ƒ(xi+2), …
Other Forms of Numerical Differentiation
E. T. S. I. Caminos, Canales y Puertos 18
Taylor Series and Truncation errors
- Higher order divided differences.
2iii1i
2iii1i
h!2
)x(''fh)x('f)x(f)x(f
h!2
)x(''fh)x('f)x(f)x(f
- Second finite central divided difference
E. T. S. I. Caminos, Canales y Puertos 19
Forward:
i 1 ii
(x ) (x )f '(x ) (h)
h
O
i 2 i 1 ii 2
(x ) 2 (x ) (x )f "(x ) (h)
h
O
Centered:
2i 1 i 1i
(x ) (x )f '(x ) (h )
2h
O
2i 1 i i 1i 2
(x ) 2 (x ) (x )f "(x ) (h )
h
O
22 1i
- ( ) 4 ( ) -3 ( )f '(x ) = + O(h )
2hi i if x f x f x
Other Forms of Numerical Differentiation
E. T. S. I. Caminos, Canales y Puertos 20
Taylor Series and Truncation errors
• Use of the Taylor series to calculate derivatives.– First derivative with forward difference.
)xx)(x('f)x(f)x(f i1iii1i
)xx()x(f)x(f
)x('fi1i
i1ii
E. T. S. I. Caminos, Canales y Puertos 21
Taylor Series and Truncation errors
– First derivative with central differences.
h2)x(f)x(f
)x('f 1i1ii
2iii1i
2iii1i
h!2
)x(''fh)x('f)x(f)x(f
h!2
)x(''fh)x('f)x(f)x(f
3i
)3(
i1i1i h!3
)x(fh)x('f2)x(f)x(f
E. T. S. I. Caminos, Canales y Puertos 22
Questions:
• Which is a better approximation? Forward, Centered, or Backward?
• Why?
• When would you use which?
Note:
We also can get higher order forward, centered, and backward difference derivative approximations
[C&C Chapter 23, tabulated in Figs. 23.1-3]
Taylor Series and Truncation errors
E. T. S. I. Caminos, Canales y Puertos 23
Determine h to minimize the total error of a forward finite-divided difference approximation for:
i 1 if (x ) f (x )f '(x)
h
xi xi+1
• Round-off Error:
• Truncation Error:
i 1 if (x ) f (x ) hf '(x) f "( )
h 2
ˆx x x f f f with = machine epsilon.
. .i i(x h) (1 ) (x ) (1 ) hˆ "( )
h 2
f ' =
i i i(x h) (x ) 2 (x )RoundoffError h h
As a result:
Example Combining Roundoff and Truncation Error
E. T. S. I. Caminos, Canales y Puertos 24
Total error = truncation error + roundoff error
E = | Total Error | h
f "( )2
NOTE: Truncation error decreases as h decreases Round-off error increases as h decreases
i2 f (x )
h
+
Example Combining Roundoff and Truncation Error
E. T. S. I. Caminos, Canales y Puertos 25
Example Combining Roundoff and Truncation Error
E. T. S. I. Caminos, Canales y Puertos 26
To minimize total error E with respect to h, set the first derivative to zero:
i2
f "( ) 2 f (x )dE0
dh 2 h
Solve for h and approximate f "() as f
"(xi):
i
i
4 f (x )h
f "(x )
Example Combining Roundoff and Truncation Error
E. T. S. I. Caminos, Canales y Puertos 27
Using the first forward-divided-difference approximation with
error O(h) and a 5-decimal-digit machine:
e = b1-t = 101-5 = 10-4 = 0.0001
f '(x) = ; f "(x) = 0
i
i
4 f (x )h infinity
f "(x )
Linear Application:
Determine h that will minimize total error for calculating f’(x) for
f(x) = x at x = 1
Example Combining Roundoff and Truncation Error
E. T. S. I. Caminos, Canales y Puertos 28
f(x+h)= {exact: 3.1415} h (x+h) f(x+h)-f(x) [f(x+h)-f(x)]/h0 3.1415
0.000001 3.1415 0 00.00001 3.1416 0.0001 100.0001 3.1419 0.0004 4.00.001 3.1447 0.0032 3.20.01 3.1730 0.0315 3.150.1 3. 4557 0.3142 3.1421 6.2831 3.1416 3.146
Underlined digits are subject to round-off error. They are likely to be in error by ± one or two units. This does not cause much problem when h = 1, but causes large errors in the final result when h < 10-4.
Example Combining Roundoff and Truncation Error
E. T. S. I. Caminos, Canales y Puertos 29
Nonlinear Application:
Determine h for minimizing the total error for computing f’(x) for
ƒ(x) = ex at x = 3
Using the first forward-divided-difference approximation with
error O(h) and a 5-decimal-digit machine:
e = b1-t = 101-5 = 10-4 = 0.0001
f(x) = f '(x) = f "(x) = ex = 20.0855;
i
i
4 (x )h 0.02
"(x )
or about 0.01
Example Combining Roundoff and Truncation Error
E. T. S. I. Caminos, Canales y Puertos 30
Underlined digits subject to roundoff error.Bold digits in error due to truncation.
full precisionh f(x+h)=ex+h f(x+h)-f(x) [f(x+h)-f(x)] [f(x+h)-f(x)]
h h0 20.085 {exact = 20.085}
0.00001 20.085 0.0 0 20.0860.0001 20.087 0.002 20 20.0860.001 20.105 0.020 20 20.0960.01 20.287 0.202 20.2 20.180.1 22.198 2.113 21.13 21.121 54.598 34.513 34.523 34.512
Roundoff Truncation
Example Combining Roundoff and Truncation Error