spatio-temporal quincunx sub-sampling.. and how we get there david lyon
TRANSCRIPT
Spatio-Temporal Quincunx Sub-Sampling
. . and how we get there
David Lyon
Overview
Sampling in Television and Film The problems of aliasing Filtering requirements Conversion between differing formats Problems that can occur How we can mitigate some of the problems and maintain
or improve quality
Sampling Theory
Harry Nyquist – 1889 to 1976 “The number of independent pulses that can be put through a
telegraph channel per unit time is limited to twice the bandwidth of the channel”
Sampling Theory
Harry Nyquist – 1889 to 1976 “The number of independent pulses that can be put through a
telegraph channel per unit time is limited to twice the bandwidth of the channel”
Later Nyquist-Shannon “Exact reconstruction of a continuous-time baseband signal from
its samples is possible if the signal is bandlimited and the sampling frequency is greater than twice the signal bandwidth”
Sampling Theory
Frequency
Amplitude
Fs
Sampling Theory
Audio:20kHz bandwidth, Fs = 44.1kHz, 48kHz
Frequency
Amplitude
Fs
Sampling Theory
Audio:20kHz bandwidth, Fs = 44.1kHz, 48kHz
Video:5.75MHz bandwidth, Fs = 13.5MHz30MHz bandwidth, Fs = 74.25MHz
Frequency
Amplitude
Fs
Aliasing
Frequency
Amplitude
Fs
Nyquist
Frequency
Aliasing
Frequencies above Fs/2 are “reflected” into the lower portion of the spectrum and become entangled with the low-frequency signals
Frequency
Amplitude
Fs
Nyquist
Frequency
Aliasing
Frequencies above Fs/2 are “reflected” into the lower portion of the spectrum and become entangled with the low-frequency signals
These signals CANNOT be removed afterwards
Frequency
Amplitude
Fs
Nyquist
Frequency
Aliasing
Frequencies above Fs/2 are “reflected” into the lower portion of the spectrum and become entangled with the low-frequency signals
These signals CANNOT be removed afterwards Filtering BEFORE sampling is needed
Frequency
Amplitude
Fs
Nyquist
Frequency
Image Sampling
Horizontal - pixels
Vertical - lines
Temporal – frames
Image Sampling
Horizontal resolutionSampling rate of 720, 1280, 1920 or 2048 samples/picture width
• Resulting resolution of 360, 640, 960 or 1024 cycles/pw
Image Sampling
Horizontal resolutionSampling rate of 720, 1280, 1920 or 2048 samples/picture width
• Resulting resolution of 360, 640, 960 or 1024 cycles/pw
Vertical resolutionSampling rate of 480, 576, 720, 1080 samples/picture height
• Resulting resolution of 240, 288, 360 or 540 cycles/ph
Image Sampling
Horizontal resolutionSampling rate of 720, 1280, 1920 or 2048 samples/picture width
• Resulting resolution of 360, 640, 960 or 1024 cycles/pw
Vertical resolutionSampling rate of 480, 576, 720, 1080 samples/picture height
• Resulting resolution of 240, 288, 360 or 540 cycles/ph
Temporal resolutionSampling rate of 24, 25, 30, 50, 60 . . . samples/second
• Resulting resolution of 12, 15, 25, 30 cycles/sec
Re-sampling
Image size changes are common
Re-sampling
Image size changes are commonSimple example of interpolating a 1080 picture to 480:
• Input resolution is 540 cycles/ph
• Output resolution is 240 cycles/ph (division by 2.25)
Vertical Frequency
Am
plitu
de 1080
Vertical Frequency
Am
plitu
de 480
FilterPotential Alias
Re-sampling
Interpolation is only one part of the problemFiltering is needed to control the signal spectrum and avoid the
introduction of aliasesSimple interpolators are generally poor filters
Re-sampling
Interpolation is only one part of the problemFiltering is needed to control the signal spectrum and avoid the
introduction of aliasesSimple interpolators are generally poor filters
Alias terms are “folded” about the Nyquist point Inverted in frequency, inverted “movement”Highly noticeable to the human eye, which references its own
internal 3D model
Re-sampling
Interpolation is only one part of the problemFiltering is needed to control the signal spectrum and avoid the
introduction of aliasesSimple interpolators are generally poor filters
Alias terms are “folded” about the Nyquist point Inverted in frequency, inverted “movement”Highly noticeable to the human eye, which references its own
internal 3D model
Alias terms left in the image will be shifted again in any subsequent operationsPotentially cumulative problems
3D Sampling
Horizontal - pixels
Vertical - lines
Temporal – frames Restricted by
practical limitations
Linked by aspect ratio
and pixel shape
Spatio-Temporal Sampling
Spatial - lines
Temporal – frames
Temporal Frequency
Spatial Frequency
Frame Rate
No of Lines
Spectrum
Potential alias
Potential alias
Spatio-Temporal Sampling
Filtering:Spatial – optical LPF and lens MTF
Spatial - lines
Temporal – frames
Temporal Frequency
Spatial Frequency
Frame Rate
No of Lines
Spectrum
Potential alias
Potential alias
Spatio-Temporal Sampling
Filtering:Spatial – optical LPF and lens MTFTemporal – integration time of sensor system
Spatial - lines
Temporal – frames
Temporal Frequency
Spatial Frequency
Frame Rate
No of Lines
Spectrum
Potential alias
Potential alias
Spatio-Temporal Sub-Sampling
Temporal – frames
Temporal Frequency
Spatial Frequency
Frame Rate
No of Lines
Spectrum
Spatial - lines
Where is the filter?
Potential alias
Potential alias
Up-conversion
Temporal Frequency
Spatial Frequency
Frame Rate
No of Lines
?
Spectrum
Temporal
Ver
tical
Horizontal
Up-conversion
Temporal Frequency
Spatial Frequency
Frame Rate
No of Lines
Adaptive filtering
?
Spectrum
Temporal
Ver
tical
Horizontal
Up-conversion
Temporal Frequency
Spatial Frequency
Frame Rate
No of Lines
Adaptive filtering Motion compensation
?
Spectrum
Temporal
Ver
tical
Horizontal
Format Interchange
Temporal Frequency
Spatial Frequency
1080p
720p
Film
1080i480i
1080p (24)
15c/s 30c/s0c/s
0c/ph
250c/ph
500c/ph
Format Interchange
Temporal Frequency
Spatial Frequency
Conversion between formats requires care
1080p
720p
Film
1080i480i
1080p (24)
15c/s 30c/s0c/s
0c/ph
250c/ph
500c/ph
Format Interchange
Temporal Frequency
Spatial Frequency
Conversion between formats requires care
Mixing formats such as film and video is to be avoided
1080p
720p
Film
1080i480i
1080p (24)
15c/s 30c/s0c/s
0c/ph
250c/ph
500c/ph
Format Interchange
Temporal Frequency
Spatial Frequency
Conversion between formats requires care
Mixing formats such as film and video is to be avoided
1080p down-conversion might raise new challenges
1080p
720p
Film
1080i480i
1080p (24)
15c/s 30c/s0c/s
0c/ph
250c/ph
500c/ph
Over-sampling
Commonly applied to audio – eg 96kHz down to 48kHzAllows the use of a high performance digital filter:
FrequencyAm
plitu
de 96
Filter
FrequencyAm
plitu
de 48
Over-sampling
Commonly applied to audio – eg 96kHz down to 48kHzAllows the use of a high performance digital filter:
Over-sampling
Commonly applied to audio – eg 96kHz down to 48kHzAllows the use of a high performance digital filter:
1080p allows similar gains for outputs of 720p and 1080iGood temporal filtering must introduce delay
Over-sampling
Commonly applied to audio – eg 96kHz down to 48kHzAllows the use of a high performance digital filter:
1080p allows similar gains for outputs of 720p and 1080iGood temporal filtering must introduce delay
Film sampling at >1080 lines/ph also allows controlled down-sampling
Conclusion
Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time
Conclusion
Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time
Modern cameras and processing can stress the format unless care is taken
Conclusion
Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time
Modern cameras and processing can stress the format unless care is taken Imprinted alias is difficult (or impossible) to removeCamera integration is an important filter for interlace
Conclusion
Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time
Modern cameras and processing can stress the format unless care is taken Imprinted alias is difficult (or impossible) to removeCamera integration is an important filter for interlace
Poor anti-alias filtering leads to additional compression concatenation artefacts
Conclusion
Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time
Modern cameras and processing can stress the format unless care is taken Imprinted alias is difficult (or impossible) to removeCamera integration is an important filter for interlace
Poor anti-alias filtering leads to additional compression concatenation artefacts
1080p down-conversion could make the stress worse