3D Techniques

Univ.Prof. Dr.-Ing. Markus RuppLVA 389.141 Fachvertiefung Telekommunikation

(LVA: 389.137 Image and Video Compression)

Last change: Jan 20, 2020

Outline

• Binocluar Vision

• Stereo Images: From first approaches tostandardisation

• 3D TV

2Univ.-Prof. Dr.-Ing. Markus Rupp

So far we did only this…

This is aneye-pad

not an IPad

Binocular vision

• Binocular vision is vision in which both eyes are used together. The word binocular comes from two Latin roots, bini for double, and oculus for eye.[1] Having two eyes confers at least four advantages over having one. First, it gives a creature a spare eye in case one is damaged. Second, it gives a wider field of view. For example, humans have a maximum horizontal field of view of approximately 200 degrees with two eyes, approximately 120 degrees of which makes up the binocular field of view (seen by both eyes) flanked by two uniocular fields (seen by only one eye) of approximately 40 degrees. [2] Third, it gives binocular summation in which the ability to detect faint objects is enhanced.[3] Fourth it can give stereopsis in which parallax provided by the two eyes' different positions on the head give precise depth perception.[4] Such binocular vision is usually accompanied by singleness of vision or binocular fusion, in which a single image is seen despite each eye's having its own image of any object.[4]

Binocular vision

• Some animals, usually prey animals, have their two eyes positioned on opposite sides of their heads to give the widest possible field of view. Examples include rabbits, buffaloes, and antelopes. In such animals, the eyes often move independently to increase the field of view. Even without moving their eyes, some birds have a 360-degree field of view.

• Other animals, usually predatory animals, have their two eyes positioned on the front of their heads, thereby allowing for binocular vision and reducing their field of view in favour of stereopsis. Examples include humans, eagles, wolves, and snakes.

• In animals with forward-facing eyes, the eyes usually move together.

The field of view of a pigeon compared to that of an owl.

Stereoscopy

• Stereoscopy (also called stereoscopic or 3-D imaging) refers to a technique for creating or enhancing the illusion of depth in an image by presenting two offset images separately to the left and right eye of the viewer. Both of these 2-D offset images are then combined in the brain to give the perception of 3-D depth. Three strategies have been used to accomplish this: have the viewer wear eyeglasses to combine separate images from two offset sources, have the viewer wear eyeglasses to filter offset images from a single source separated to each eye, or have the lightsource split the images directionally into the viewer's eyes (no glasses required; known as Autostereoscopy).

Camera History

• The first photography was taken by Joseph Nicéphore Nièpce in 1826 by a so called Heliografie-method. In 1837 Louis Jacques Mandé Daguerre improved the method, developing the films by mercury steam (Quecksilber) and final fixation in a hot salt solution (Kochsalzlösung) or normal temperatured Natriumthiosulfatlösung. The so obtained pictures were all on copper plates named Daguerreotypien. In 1835 the Englishman William Fox Talbot developed the Negativ-Positiv-method.

Univ.-Prof. Dr.-Ing. Markus Rupp 8

Camera History

• Constanze Mozart, wife ofWolfgang Amadeus Mozart, at 78 years of age, pictured front left in black 2 years before her death. (Her maiden name was Weber). Bavarian composer Max Keller is seated center front, and to his right is his wife Josefa. From left to right in rear: family cook, Philip Lattner (Keller's brother in law), Keller's daughters Luise and Josefa. The print is a 19th century copy of the original daguerrotype photograph taken October 1840, at the home of composer Max Keller.

Univ.-Prof. Dr.-Ing. Markus Rupp 9

Stereoscopy

• Stereoscopy is the enhancement of the illusion of depth in a photograph, movie, or other two-dimensional image by presenting a slightly different image to each eye, and thereby adding the first of these cues (stereopsis) as well. It is important to note that the second cue is still not satisfied and therefore the illusion of depth is incomplete.

• Many 3D displays use this method to convey images. It was first invented by Sir Charles Wheatstone in 1838.[2][3]

• Wheatstone originally used his stereoscope (a rather bulky device)[4] with drawings because photography was not yet available, yet his original paper seems to foresee the development of a realistic imaging method[5]:

Sir Charles Wheatstone in 1838

• “For the purposes of illustration I have employed only outline figures, for had either shading or colouring been introduced it might be supposed that the effect was wholly or in part due to these circumstances, whereas by leaving them out of consideration no room is left to doubt that the entire effect of relief is owing to the simultaneous perception of the two monocular projections, one on each retina. But if it be required to obtain the most faithful resemblances of real objects, shadowing and colouring may properly be employed to heighten the effects. Careful attention would enable an artist to draw and paint the two component pictures, so as to present to the mind of the observer, in the resultant perception, perfect identity with the object represented. Flowers, crystals, busts, vases, instruments of various kinds, &c., might thus be represented so as not to be distinguished by sight from the real objects themselves.”[2]

Univ.-Prof. Dr.-Ing. Markus Rupp 11

1860: stereoscopic postcards becamea fashion

But you need a „viewer“

Not so much different today

• Anaglyph (red and blue)

• Also shutter glasses, polarisation glasses…

Autostereoscopy

• Autostereoscopy is any method of displaying stereoscopic (3D) images without the use of special headgear or glasses on the part of the viewer. Because headgear is not required, it is also called "glasses-free 3D". The technology includes two broad classes of displays: those that use head-tracking to ensure that each of the viewer's two eyes sees a different image on the screen, and those that display multiple views so that the display does not need to know where the viewers' eyes are directed. Examples of autostereoscopic displays include parallax barrier, lenticular, volumetric, electro-holographic, and light field displays.

• Some autostereoscopic displays are also capable of recreating a perception of movement parallax, which is not possible with any of the active or passive technologies discussed above. "Movement parallax" refers to the fact that the view of a scene changes with movement of the head. Thus, different images of the scene are seen as the head is moved from left to right, and from up to down.

• This is the method used by the Nintendo 3DS video game system and the Optimus 3D and LG Thrill by cellphone manufacturer LG Electronics MobileComm, HTC EVO 3D.

Parallax Barrier

• The principle of the parallax barrier has been invented by AugusteBerthier[4]in 1896 but was later popularized by the independent invention by Frederic E. Ives.[5] Sharp developed the technology to commercialization, briefly selling two laptops with the world's only 3D LCD screens.[6] These displays are no longer available from Sharp but still being manufactured and further developed from other companies like Tridelity and SpatialView. Similarly, Hitachi has released the first 3D mobile phone for the Japanese market under distribution by KDDI.[7][8] In 2009, Fujifilm released the Fujifilm FinePix Real 3D W1 digital camera, which featured a built-in autostereoscopic LCD display measuring 2.8" diagonal. Nintendohas also implemented this technology on their latest portable gaming console, the Nintendo 3DS.

Integral Photography and LenticularArrays

• The principle of integral photography, which uses a two-dimensional array of lenslets to capture a 3-D scene, is usually attributed to Gabriel Lippmann in his paper of 1908.[9][10] Lenticular arrays were invented later; according to Benton, "[i]n an attempt to overcome the brightness limitations of raster barrier screens, Hess, [in a patent filed in 1913,[11]] showed that tiny cylindrical lenslets could be used for the same purpose."[12] Pierre Allio produced some of the first patents in lenticular displays in the mid-1980s. Philips solved a significant problem with these displays in the mid-1990s by slanting the lenticular lenses with respect to the underlying pixel grid.[13] Philips produced its WOWvx line, based on this idea, until 2009, running up to a 2160p resolution of 3840×2160 pixel 46 viewing angles.[14]

Lenny Lipton's company, StereoGraphics, produced displays based on the same idea, citing a much earlier patent for the slanted lenticulars. Magnetic3d and Zero Creative have also been involved.[15] The hardware overlay for iPhone and iPod touch named 3DeeSlide also adopts this technology to convert the standard screen into an auto 3D display.[16]

Autostereoscopic displays

• Movement parallax refers to the fact that the view of a scene changes with movement of the head. Thus, different images of the scene are seen as the head is moved from left to right, and from up to down.

• Many autostereoscopic displays are single-view displays and are thus not capable of reproducing the sense of movement parallax, except for a single viewer in systems capable of eye tracking.

• Some autostereoscopic displays, however, are multi-view displays, and are thus capable of providing the perception of left-right movement parallax.[26] Eight and sixteen views are typical for such displays. While it is theoretically possible to simulate the perception of up-down movement parallax, no current display systems are known to do so, and the up-down effect is widely seen as less important than left-right movement parallax. One consequence of not including parallax about both axes becomes more evident as objects increasingly distant from the plane of the display are presented, for as the viewer moves closer to or farther away from the display such objects will more obviously exhibit the effects of perspective shift about one axis but not the other, appearing variously stretched or squashed to a viewer not positioned at the optimum distance from the display.

Stereo cameras

• Stereo Realist The original "Realist Format" camera, first sold in 1947, which inspired many imitators who introduced cameras capable of producing the 5P stereo slides which remain fairly popular to this day.

• Kodak Stereo Camera Kodak's own offering in the field of Realist format cameras which actually outsold the Realist during the five years it was available and might have eclipsed it in all time sales had it been introduced prior to the end of 1954.

• View-Master Personal Stereo Camera Introduced in 1952, this camera allowed individuals to make their own personal View-Master reels, either by using the commercial processing services available at the time or by using the custom cutter and blank reel mounts.

• Loreo 3D Lens in a Cap (Hong Kong) - an accessory device, which incorporates a pair of small closely spaced lens, and a simple mirror box as an attachment for many modern SLR digital cameras. The latest version has 25mm wider angle lenses. Loreo also makes currently, a cross-view 35mm film only, 3D CAMERA, (model 321) which takes "deeper" stereo images, with a wider mirror system, sold with a folding print viewer included.

• Nimslo 3D The first compact consumer level lenticular camera, designed to take 3D prints that are viewable without glasses or special technique. Though it didn't "catch on" and was soon discontinued, it inspired many 3 and 4 lens clones marketed well into the 1990s.

• Fujifilm FinePix Real 3D W1, a digital stereo camera, 2009.• In 2009, 3D technologies experienced a resurgence,[3][4][5] including stereo cameras, with continuing

developments in plenoptic camera technologies, as well as the emergence of stereo digital camera products such as the Fujifilm FinePix Real 3D series[6] and the Minoru 3D Webcam.[7]

1954

2009

3D Video

Univ.-Prof. Dr.-Ing. Markus Rupp 22

Univ.-Prof. Dr.-Ing. Markus Rupp 23

Accomodation: Einstellbereich Vergence: Brechwert

Univ.-Prof. Dr.-Ing. Markus Rupp 24

Univ.-Prof. Dr.-Ing. Markus Rupp 25

Univ.-Prof. Dr.-Ing. Markus Rupp 26

Univ.-Prof. Dr.-Ing. Markus Rupp 27

Univ.-Prof. Dr.-Ing. Markus Rupp 28

Univ.-Prof. Dr.-Ing. Markus Rupp 29

Univ.-Prof. Dr.-Ing. Markus Rupp 30

MVC= Multi View Coding

Univ.-Prof. Dr.-Ing. Markus Rupp 31

1dB

2D-plus-Depth

• 2D-plus-Depth or also called 2D +'Z' format is a Stereoscopic Video Coding format that is used for 3D displays, such as Philips WOWvx. Philips discontinued work on the WOWvx line in 2009, citing "current market developments".[1] The 2D-plus-Depth format is described in a Philips' white paper[2] and articles[3].

• Each 2D image frame is supplemented with a greyscale depth map which indicates if a specific pixel in the 2D image needs to be shown in front of the display (white) or behind the screen plane (black). The 256 greyscales can build a smooth gradient of depth within the image. Processing within the monitor used this input to render the multiview images.

• Supported by various companies across the display industry, 2D-plus-Depth has been standardized in MPEG as an extension for 3D filed under ISO/IEC FDIS 23002-3:2007(E).

• Advantages: – 2D-plus-Depth has the advantage that it has a limited bandwidth increase compared to 2D

(compressed greyscale increases bandwidth 5–20%) so that it can be used in existing distribution infrastructures.

– 2D-plus-Depth offers flexibility and compatibility with existing production equipment and compression tools.

– It allows applications to use different 3D display screen sizes and designs in the same system.

• Disadvantages:

– 2D-plus-Depth is not compatible with existing 2D or 3D-Ready displays. The format has been criticized due to the limited amount of depth that can be displayed in an 8-bit greyscale.

– 2d-plus-Depth cannot handle transparency (semi-transparent objects in the scene) and occlusion (an object blocking the view of another). The 2d plus DOT format takes these factors into account.[4] Additionally, it cannot handle reflection, refraction (beyond simple transparency) and other optical phenomena.

– Creation of accurate 2D-plus-Depth is generally costly and difficult, though recent advances in range imaging have made this process more accessible.[5][6]

– 2d-plus-Depth lacks the potential increase in resolution of using two complete images.

– Depth cannot be reliably estimated for a monocular video in most cases. Notable exceptions are camera motion scenes when object motion is static or almost absent. This allows automatic depth estimation[7]. In general case only semi-automatic approach is viable for 2D to 2D-plus-depth conversion. Philips developed a 3D content creation software suite named BlueBox[8] which includes semi-automated conversion of 2D content into 2D-plus-Depth format and automatic generation of 2D-plus-Depth from stereo.

– Stereoscopic to 2D-plus-Depth conversion involves several algorithms including scene change detection, segmentation, motion estimation and image matching. But automatic stereo to 2D+Depth conversion is possible due to new high performance software and

CPU even in a live mode.[9]

2D-plus-Depth

2D Plus Delta (AKA 2D+Delta)

• is a standards listed methodology as part of MPEG2, and MPEG4, specifically on the H.264 implementation of Multiview Video Codingextension. This technology originally started as a proprietary method for Stereoscopic Video Coding and content deployment that utilizes the Left or Right channel as the 2D version and the optimized difference or disparity (Delta) between that image channel view and a second eye image view is injected into the videostream as user data, secondary stream, independent stream, enhancement layer or NALU for deployment. The Delta data can be either a spatial stereo disparity, temporal predictive, bidirectional or optimized motion compensation.

• The technology was originally filed for protection World Wide via WIPO in 2003, submitted the patent statements to ISO in 2007 and listed as part of the MVC standard in 2008 and is now considered as an Open Standard available for licensing and usage under a world wide, non-discriminatory basis and on reasonable terms and conditions.

2D Plus Delta (AKA 2D+Delta)

• The MVC initiative was started in June 2006.• The resulting videostream has the following characteristics:

– Contains the full information to reconstruct the Left and Right stereoscopic views in full resolution per eye

– Requires an additional average bandwidth ranging from 30% to 60% regarding a 2D only stream depending on the encoding implementation

– Can be deployed over existing pipelines, starting from 16Mbits/s for 1080p@60fps per eye

– One single videostream services both: 3D viewers and 2D Legacy users

– Can be seamlessly decoded by 2D legacy decoders in full 2D HD without the need of any change or additional hardware

Univ.-Prof. Dr.-Ing. Markus Rupp 36

3D-TV History

• In the late-1890's, the British film pioneer William Friese-Greene filed a patent for a 3-D movie process. When viewed stereoscopically, it showed that the two images are combined by the brain to produce 3-D depth perception. On June 10, 1915, Edwin S. Porter and William E. Waddell presented tests to an audience at the Astor Theater in New York City. In red-green anaglyph, the audience was presented three reels of tests, which included rural scenes, test shots of Marie Doro, a segment of John Mason playing a number of passages from Jim the Penman (a film released by Famous Players-Lasky that year, but not in 3-D), Oriental dancers, and a reel of footage of Niagara Falls. However, not much was produced in this process after these tests.

3D-TV History

• The first anaglyph (use of red-and-blue glasses, invented by L.D. DuHauron) movie was produced in 1915 and in 1922 the first public 3D movie was displayed. Stereoscopic 3D television was demonstrated for the first time on August 10, 1928, by John Logie Baird in his company's premises at 133 Long Acre, London.[1] Baird pioneered a variety of 3D television systems using electro-mechanical and cathode-ray tube techniques. In 1935 the first 3D color movie was produced.

• In the 1950s many 3D movies were produced. The first such movie was Bwana Devil from United Artists that could be seen all across the US in 1952. One year later, in 1953, came the 3D movie House of Wax which also featured stereophonic sound. Alfred Hitchcock produced his film Dial M for Murder in 3D, but for the purpose of maximizing profits the movie was released in 2D because not all cinemas were able to display 3D films. The Soviet Union also developed 3D films, with Robinzon Kruzo being its first full-length 3D movie, in 1946.[2]

• Subsequently, television stations started airing 3D serials in 2009 based on the same technology as 3D movies.

3D-TV News

• DVB 3D-TV will be a new standard coming partially out at the end of 2010 which will include techniques and procedures to send a three dimensional video signal through actual DVB transmission standards (Cable, Terrestrial or Satellital). Currently there is a commercial requirement text for 3D TV broadcasters and Set-top box manufacturers, but no technical information is in there.

• Recent proposal (Feb. 2011)

• http://www.dvb.org/technology/standards/a154_DVB-3DTV_Spec.pdf