physically based sound
DESCRIPTION
Physically Based Sound. COMP259Nikunj Raghuvanshi. Overview. Background FEM Simulation Modal Synthesis (FoleyAutomatic) Comparison/Conclusions. Motivation. Sounds could in-principle be produced automatically, just like graphics: Sound Rendering - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/1.jpg)
Physically Based Sound
COMP259 Nikunj Raghuvanshi
![Page 2: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/2.jpg)
Overview
Background
FEM Simulation
Modal Synthesis (FoleyAutomatic)
Comparison/Conclusions
![Page 3: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/3.jpg)
Motivation
Sounds could in-principle be produced automatically, just like graphics: Sound Rendering
Sound Rendering has not received much research effort
Main Goal: Automatic generation of non-music, non-dialogue sound
![Page 4: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/4.jpg)
Sound Production Today
Movies: Foley Artistshttp://www.marblehead.net/foley/index.html
Games: Anyone noticed the huge sound directory in Unreal Tournament?
![Page 5: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/5.jpg)
PBS: Sound Production in Nature
Collisions/Other interactions lead to surface vibrations
Vibrations create pressure waves in airPressure waves sensed by ear
Surface Vibration Pressure Wave Ear
Vibration Propagation Perception
![Page 6: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/6.jpg)
Main Aims of PBS
Physics simulator gives contact/collision information
Assign material properties for sound, Wood, concrete, metal etc.
Sound simulator generates sound using this data (in real time?)
![Page 7: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/7.jpg)
Challenges
Sound must be produced at a minimum of ~44,000 Hz
Extremely High Temporal Resolution (timesteps in the range of 10-6-10-8 s)
Stiffness of underlying systems (eg. Metallic sounds. K/m~=108)
Stability may require even smaller timesteps
![Page 8: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/8.jpg)
Two Approaches
FEM deformable simulationO'Brien, J. F. et. al., “Synthesizing Sounds from Physically Based Motion.” SIGGRAPH 2001.
FoleyAutomatic (Modal Synthesis)Kees van den Doel et. Al., “FoleyAutomatic: Physically-based Sound Effects for Interactive Simulation and Animation.” SIGGRAPH 2001.
![Page 9: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/9.jpg)
Main ideas
Deformable Simulation (arguably) much more “physically based”
Foley Automatic: Additive Synthesis
Component Sinusoids
Sound Signal
![Page 10: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/10.jpg)
Overview
Background
FEM Simulation
Modal Synthesis (FoleyAutomatic)
Comparison/Conclusions
![Page 11: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/11.jpg)
Simulation Requirements
Temporal ResolutionSimulate Vibration as well as PropagationVibration Modeling: Deformable Model for
ObjectsPropagation Modeling: Explicit Surface
RepresentationPhysical/Perceptual Realism
![Page 12: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/12.jpg)
System Structure
![Page 13: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/13.jpg)
Vibration Modelling
FEM with Tetrahedral Elements Linear Basis Functions, green’s strain Explicit Time Integration Typically #nodes = 500, #elements = 1500,
dt = 10-6-10-7 s
![Page 14: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/14.jpg)
Sound Propagation Modelling
Fluid Dynamic FEM simulation of surrounding air? Very expensive. Instead…
Employ Huygen’s Principle: Pressure Wave may be seen as sum of pressure wavelets
ReceiverReceiver
Pressure Wave Pressure
“Wavelets”
![Page 15: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/15.jpg)
n̂ v
ds
nvzp ˆ
msPacz /415 Acoustic Impedance of Air
Surface Vibrations and Sound
Pressure contribution of a patch,
Velocity
Density of Air
Sound Propagation Speed in Air
Unit Normal
![Page 16: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/16.jpg)
Surface Vibrations and Sound
Approximate differential elements with surface triangles
Apply band pass filters: Low pass: windowed sinc filter High pass: DC blocking filter
Result: Pressure known for all surface triangles
![Page 17: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/17.jpg)
Putting it all together
)cos(~
)( rx
apts rx
Pressure/Signal at Receiver
Filtered Average Pressure
Area of Triangle
Visibility Term
Approximation of Beam Pattern
Distance Falloff
n̂
Receiver
r
Vibrationx̂
![Page 18: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/18.jpg)
Propagation Delay
Accumulation Buffer
c
dDelay
Receiver
d1
d2
Source
t=0
t1= d1/c
t2= d2/c
1
2
Receiver Distance from Source
Sound Propagation Speed
![Page 19: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/19.jpg)
Results: Capabilities
General models
Generated sounds are accurate
Stereo Sound
Doppler’s Effect
![Page 20: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/20.jpg)
Demo
![Page 21: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/21.jpg)
Results: Accuracy
![Page 22: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/22.jpg)
Results: Speed
Scene TimeStep(s) Nodes/Elems Time/Audio Time
Bowl 10-6 387/1081 91.3/4.01 mins
Clamped Bar 10-7 125/265 240.4/1.26 mins
Vibraphone 10-7 539/1484 1309.7/5.31 mins
(~1 day)
Timings on a 350MHz SGI Origin MIPS R12K processor
![Page 23: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/23.jpg)
Overview
Background
FEM Simulation
Modal Synthesis (FoleyAutomatic)
Comparison/Conclusions
![Page 24: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/24.jpg)
Features
Modal resonance model of solids Location dependent sounds Impact, slide, roll excitation models Real-time, low latency Easy integration with simulation/animation Practical Do not model propagation of sound from source
to receiver
![Page 25: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/25.jpg)
Synthesis Method
ForceForceVibrationVibrationEmissionEmission
PropagationPropagation ListenerListener SpeakersSpeakers
Sound SamplesSound Samples
User
![Page 26: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/26.jpg)
Vibration
),(),(]1
),([2
2
2txFtxu
tcx
xg iii
i
Surface u(x,t) of body responds to external contact force F(x,t)
u(x,t)F(x,t)
Strain Functional Speed of Sound
Under suitable boundary conditions, the solution to the PDE is a sum of sinusoids
![Page 27: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/27.jpg)
Emission
Sound pressure s(t) linear functional L of surface vibration u(x,t)
)],([)( txuLts i
u(x,t)Ls(t)
nvzp ii ˆ~
Note that propagation is not modeled in above
![Page 28: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/28.jpg)
The Modal Synthesis Model
u(x,t) F(p,t)Ls(t)
Impulse response/modal model
“The response u(x,t) of an arbitrary solid object to an external force can be described as a weighted sum of damped sinusoids”
Since L is linear, it implies at s(t) must be a sum of damped sinusoids too
![Page 29: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/29.jpg)
Example: A 1D string
1st Mode 2nd Mode Frequency = f0
…Higher modes Frequency = f1= 2*f0 Frequency = fk= k*f0
)2sin( 000 tfea td )2sin( 11
1 tfea td )2sin( tfea ktd
kk
Main Idea: Sum contributions of all the modes
The point of impact decides the proportions in which the modes are to be mixed: ak. Therefore, ak is a function of p, the point of impact
The frequencies and damping parameters are a property of the object, and independent of how the object is hit
+ +...+
a0a1 ak
![Page 30: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/30.jpg)
The Modal Synthesis Model
u(x,t) F(p,t)Ls(t)
)2sin()()(1
tfepats ktd
N
kk
k
Impulse response,
modal model
Parameters measured experimentally
Kth mode: Gain Factor Point Damping Vibration of impact Term Frequency
![Page 31: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/31.jpg)
Force Modeling
ImpactSlidingRolling
Wavetable
Stochastic
At runtime: Find gain parameters given the location, strength and kind of force.
Synthesize sound from previous equation.
![Page 32: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/32.jpg)
Impact Forces
•Duration: hardness (T)•Magnitude: energy transfer (w)•Multiple micro-collisions
TtTtwtF 0)),/2cos(1()( Example:
![Page 33: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/33.jpg)
Sliding/Scraping
Micro-collisions lead to noisy audio-force
![Page 34: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/34.jpg)
Sliding/Scraping
Wavetable approach Store force parameters Modulate amplitude with energy transfer Modulate rate with contact speed
Synthesis Approach Fractal noise represents roughness Filter through reson filter Resonance ~ contact speed Width ~ randomness of surface
![Page 35: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/35.jpg)
Rolling
No relative surface motion
Differences with sliding:•Smoother: Use low pass•More damping•Harder to create•Less understood•Essential coupling?
![Page 36: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/36.jpg)
Rolling: Smooth Surfaces
Polyhedral objects do not lead to smooth rolling forces
Instead use smooth surfaces directly
![Page 37: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/37.jpg)
Rolling: Contact Evolution
Evolve the contact in Reduced coordinates
q = (u,v,s,t, )
q q q .. .
c(u,v)
d(s,t)
![Page 38: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/38.jpg)
Rolling: Contact Evolution
Piecewise parametric surfaces, loop subdivision surfaces
Explicit integration, no stabilization Multiple contacts and conforming contacts
are not handled Used only when multiple contacts in close
spatio-temporal proximity
![Page 39: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/39.jpg)
Demo
![Page 40: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/40.jpg)
Dynamic Forces
Contact force
Rolling speed
Slipping speed
Impulses
…and locations
Pebble-in-Wok Demo
![Page 41: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/41.jpg)
Results
0.1% CPU time per mode Graceful degradation of quality The bell demo is interactive Uses a PHANToM for interaction Authors do not report any real timings State that “sound quality” is perception-
based and has no metric as of now
![Page 42: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/42.jpg)
Overview
Background
FEM Simulation
Modal Synthesis (FoleyAutomatic)
Comparison/Conclusions
![Page 43: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/43.jpg)
Discussion
FEM: Physically Rigorous and GeneralToo slow for interactive applicationsDoesn’t scale wellInappropriate to apply a 30fps technique to
44000fps?Maybe too general for the problem
domain?
![Page 44: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/44.jpg)
Discussion
Modal model exploits the vibrational nature
Higher EfficiencyBut, not rigorously physically basedFinding the parameters requires
experimentation and “earballing”No rigorous correlation between physical
and perceptual parameters
![Page 45: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/45.jpg)
Discussion
For Realtime: Need for a technique to cover the middle ground
Extracting modal parameters in general requires solving PDEs
Not possible to do in an automated manner
Approximate modal parameters and then use modal synthesis?
![Page 46: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/46.jpg)
Conclusion
PBS involves orders of magnitude smaller temporal and spatial scales
Research is sparse, problems are denseMain contributions of the two papers
besides vibration modeling: FEM: Efficient modeling of sound propagation FoleyAutomatic: Efficient, Approximate models
to handle surface properties and contact forces
![Page 47: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/47.jpg)
References
O'Brien, J. F., Cook, P. R., Essl G., "Synthesizing Sounds from Physically Based Motion." The proceedings of ACM SIGGRAPH 2001, Los Angeles, California, August 11-17, pp. 529-536.
Kees van den Doel, Paul G. Kry and Dinesh K. Pai, “FoleyAutomatic: Physically-based Sound Effects for Interactive Simulation and Animation” Computer Graphics (ACM SIGGRAPH 01 Conference Proceedings), pp. 537-544, 2001.
![Page 48: Physically Based Sound](https://reader033.vdocuments.site/reader033/viewer/2022061605/568145dc550346895db2df70/html5/thumbnails/48.jpg)
Acknowledgements
Some images were taken from the referred papers and the corresponding SIGGRAPH slides