cos 323: computing for the physical and social sciences szymon rusinkiewicz
Post on 19-Dec-2015
215 views
TRANSCRIPT
COS 323: Computing for theCOS 323: Computing for thePhysical and Social SciencesPhysical and Social Sciences
Szymon RusinkiewiczSzymon Rusinkiewicz
What’s This Course About?What’s This Course About?
• Numerical AlgorithmsNumerical Algorithms
• Analysis of DataAnalysis of Data
• SimulationSimulation
Numerical AnalysisNumerical Analysis
• Algorithms for solving numerical problemsAlgorithms for solving numerical problems– Calculus, algebra, data analysis, etc.Calculus, algebra, data analysis, etc.
– Applications in all scientific and engineering Applications in all scientific and engineering fieldsfields
• Analyze/design algorithms based on:Analyze/design algorithms based on:– Running time, memory usageRunning time, memory usage
(both asymptotic and constant factors)(both asymptotic and constant factors)
– Applicability, stability, and accuracyApplicability, stability, and accuracyfor different classes of problemsfor different classes of problems
Why Is This Hard/Interesting?Why Is This Hard/Interesting?
• ““Numbers” in computers Numbers” in computers numbers in numbers in mathmath– Limited precision and rangeLimited precision and range
• Algorithms sometimes don’t give right Algorithms sometimes don’t give right answeranswer– Iterative, randomized, approximateIterative, randomized, approximate
– UnstableUnstable
• Running time / accuracy / stability Running time / accuracy / stability tradeoffstradeoffs
Numbers in ComputersNumbers in Computers
• ““Integers”Integers”– Implemented in hardware: fastImplemented in hardware: fast
– Mostly sane, except for limited rangeMostly sane, except for limited range
• Floating pointFloating point– Implemented in most hardwareImplemented in most hardware
– Much larger rangeMuch larger range(e.g. (e.g. 223131.. 2.. 23131 for integers, vs. for integers, vs. 22127127.. 2.. 2127127 for FP) for FP)
– Lower precision (e.g. 9 digits vs. 7)Lower precision (e.g. 9 digits vs. 7)
– ““Relative” precision: actual accuracy depends Relative” precision: actual accuracy depends on sizeon size
Floating Point NumbersFloating Point Numbers
• Like scientific notation: e.g., Like scientific notation: e.g., cc is is2.997924582.99792458 101088 m/s m/s
• This has the formThis has the form(multiplier)(multiplier) (base)(base)(power)(power)
• In the computer,In the computer,– MultiplierMultiplier is called mantissa is called mantissa
– BaseBase is almost always 2 is almost always 2
– PowerPower is called exponent is called exponent
Modern Floating Point FormatsModern Floating Point Formats
• Almost all computers use IEEE 754 standardAlmost all computers use IEEE 754 standard
• ““Single precision”:Single precision”:– 24-bit mantissa, base = 2, 8-bit exponent, 1 bit sign24-bit mantissa, base = 2, 8-bit exponent, 1 bit sign
– All fits into 32 bits (!)All fits into 32 bits (!)
• ““Double precision”:Double precision”:– 53-bit mantissa, base = 2, 11-bit exponent, 1 bit sign53-bit mantissa, base = 2, 11-bit exponent, 1 bit sign
– All fits into 64 bitsAll fits into 64 bits
• Sometimes also have “extended formats”Sometimes also have “extended formats”
Other Number RepresentationsOther Number Representations
• Fixed pointFixed point– AbsoluteAbsolute accuracy doesn’t vary with magnitude accuracy doesn’t vary with magnitude
– Represent fractions to a fixed precisionRepresent fractions to a fixed precision
– Not supported directly in hardware, but can hack Not supported directly in hardware, but can hack itit
• ““Infinite precision”Infinite precision”– Integers or rationals allocated dynamicallyIntegers or rationals allocated dynamically
– Can grow up to available memoryCan grow up to available memory
– No direct support in hardware, but libraries No direct support in hardware, but libraries availableavailable
Consequences of Floating PointConsequences of Floating Point
• ““Machine epsilon”: smallest positive number Machine epsilon”: smallest positive number you can add to 1.0 and get something other you can add to 1.0 and get something other than 1.0than 1.0
• For single precision: For single precision: 10 1077
– No such number as 1.000000001No such number as 1.000000001
– Rule of thumb: “almost 7 digits of precision”Rule of thumb: “almost 7 digits of precision”
• For double: For double: 2 2 10 101616
– Rule of thumb: “not quite 16 digits of precision”Rule of thumb: “not quite 16 digits of precision”
• These are all These are all relativerelative numbers numbers
So What?So What?
• Simple example: add Simple example: add 11//1010 to itself 10 to itself 10
timestimes
Oops!Oops!
• Result: Result: 11//1010 + + 11//1010 + … + … 1 1
• Reason: 0.1 can’t be represented Reason: 0.1 can’t be represented exactly in binary floating pointexactly in binary floating point– Like Like 11//33 in decimal in decimal
• Rule of thumbRule of thumb: comparing floating point : comparing floating point numbers for equality is always wrongnumbers for equality is always wrong
More Subtle ProblemMore Subtle Problem
• Using quadratic formulaUsing quadratic formula
to solve to solve xx22 – 9999 – 9999xx + 1 = 0 + 1 = 0– Only 4 digits: single precision should be OK, right?Only 4 digits: single precision should be OK, right?
• Correct answers: 0.0001… and 9998.999…Correct answers: 0.0001… and 9998.999…
• Actual answers in single precision: 0 and 9999Actual answers in single precision: 0 and 9999– First answer is 100% off!First answer is 100% off!
– Total cancellation in numerator because Total cancellation in numerator because bb22 >> 4 >> 4acac
a
acbbx
2
42
a
acbbx
2
42
Catalog of ErrorsCatalog of Errors
• Roundoff error – caused by limitations ofRoundoff error – caused by limitations offloating-point “numbers”floating-point “numbers”
• Truncation error – caused by stopping an Truncation error – caused by stopping an approximate technique earlyapproximate technique early– e.g., too few terms of Taylor series for sin(e.g., too few terms of Taylor series for sin( ))
• Inherent error – limitation on data Inherent error – limitation on data availableavailable– GIGOGIGO
• Statistical error – too few random samplesStatistical error – too few random samples
Running TimeRunning Time
• Depending on algorithm, we’ll look at:Depending on algorithm, we’ll look at:– Convergence orderConvergence order for iterative approximate for iterative approximate
algorithmsalgorithms(e.g., an answer to precision (e.g., an answer to precision might require might require iterations proportional to 1/ iterations proportional to 1/ or 1/ or 1/ ))
– Asymptotic analysisAsymptotic analysis for noniterative for noniterative algorithmsalgorithms(e.g., inverting an (e.g., inverting an nnnn matrix requires time matrix requires time proportional to proportional to nn33))
This CourseThis Course
• Basic techniques:Basic techniques: root finding, root finding, optimization, linear systemsoptimization, linear systems
• Data analysis and modeling:Data analysis and modeling: least least squares, dimensionality reduction, squares, dimensionality reduction, visualization, statisticsvisualization, statistics
• Signal processing:Signal processing: sampling, filtering sampling, filtering
• IntegrationIntegration and differential equations and differential equations
• SimulationSimulation
MechanicsMechanics
• Programming assignmentsProgramming assignments– Typically more thought/analysis than codingTypically more thought/analysis than coding
– Some in MATLAB, some in Java or CSome in MATLAB, some in Java or C
MechanicsMechanics
• Precept time scheduled, but no regular Precept time scheduled, but no regular preceptsprecepts– Occasionally will have extra material on Occasionally will have extra material on
assignmentsassignments
– Otherwise, will hold office hoursOtherwise, will hold office hours
• Course webpageCourse webpage– (infinitely long address – find from CS dept (infinitely long address – find from CS dept
webpage)webpage)
• Contact me at Contact me at [email protected]@princeton.edu