enreviist-1999-11185 enrevi enhanced reality for the video october 4 concertation

ENREVIENREVIIST-1999-11185IST-1999-11185

ENREVIENREVIENENhanced hanced REREality for the ality for the

VIVIdeodeoOctober 4October 4

ConcertationConcertation


ENREVI :The partnersENREVI :The partners

• Symah Vision(.fr) as project co-ordinator and bringing expertise in tracking and video systems.

• Peak(.au) bringing expertise in 3D real time synthesis.

• Crystal Vision(.uk) in advanced keying know-how.

• Fraunhofer Institut (.de) for advance tracking research.


The development objectives:The development objectives:

• Improving real time 3D camera tracking in an undefined environment

• Improve keying techniques in particular in undefined environment such as outside the studio.

• Provide real time rendering software for 3D objects with broadcast quality, in particular in respect to anti-aliasing techniques, on affordable hardware platforms.

• Provide system integration and ergonomics in concordance to the professional standards.


Technologies involved:Technologies involved:

• 2D and 3D graphics design.

• Computer vision.

• Robotics and sensors.

• Video professional system integration.


Objectives reformulation:Objectives reformulation:Toward scene description authoringToward scene description authoring

• Tracking/Keying : Scene description ==>

MPEG4 authoring.

• Low cost 3D rendering addresses MPEG4

player graphic capabilities

• ENREVI system layer: how to store transport and

retrieve specific metadata: camera position, camera

calibration, camera movement, object position in

camera FOV.


Summary of project StatusSummary of project Status

• System specifications and interfaces (SV)• Tracking options and final specification

(IGD)• Status on low cost rendering options

(Peak+SV)• Detailed specifications on Keyer (Crystal)• Achievements, problems and prospects


System specifications and interfaces (SV)


Camera(Real Scene)

TrackingModule

ControlModule

Keyer

Renderer

Display of Enhanced scene Video

Data

Network

Storage

Transport

Virtual ObjectsDesign

Inside project

Outside Project

ENREVI General System ENREVI General System ArchitectureArchitecture


Communications library :Communications library :The development status (2)The development status (2)

• Communication with PEAK software (Rendering):

Shared MEMORY

Peak rendering

Tracking independent interfaceTracking independent

interface

Daemonfor “ X ”Tracking

Tracking independent interface

SV communication

libraryDaemon for ENREVI Tracking


Communication library :Communication library :The development status (3)The development status (3)

• Communication with IGD software (Tracking):

IGD Tracking Algorithm

IGD tracking API

SV tracking application

SV communication

library

Remote control+data


Communication library :Communication library :The development status (4)The development status (4)

• Communication with Crystal Vision software (CK):

Cristal Vision dedicated Hardware

Crystal Vision TCP/IP or RS485 Communication protocol

SV Chroma Key control GUI

SV communication

library (Optionnal)

Remote control+data


• We tried : – Native libraries : XVT DSC++, Twin, YAAF, Zinc, Microsoft

MFC, WxWindows, Open UI, CPAT and XVT DSC++ but the conclusion is : no native library is simultaneously platform independent, easy to use and complete.

– Interpreted languages : Two serious candidates are available : TCL/TK and JAVA. The two complete the requirements

.

Graphics Library : the choiceGraphics Library : the choice


Specifications for communication LibrarySpecifications for communication Library

Bi-directional data transmission. Real time transmission. Synchronisation of data by time stamping Real time losses compensation. Multiple application on the same

machine. User defined data


Standardisation activitiesStandardisation activities

• Focused on MPEG4 during 6 first months• French delegation registration• WG11 meetings participation• Outside Enrevi:

– Development of an Authoring Tool for scene description using BIFS syntax

– Development of a Production Tool for « on line » scene updates


Tracking options and final specifications (IGD)


General StepsGeneral Steps

• Calibration of Stereo System• Calibration of Hybrid System• Tracking

– Detection/tracking of 2D-coordinates of scene features

– Calculation of 3D-coordinates of scene features

– Calculation of camera parameter for each frame


Tracking IITracking II

• Tracking is optimization of camera parameter for each frame

• Prediction can be done in several ways:– polynomial extrapolation – motion models (e.g.kalman filter)– external sensors (inertial, magnetic)

• Outlier detection by using M-estimators


3D Reconstruction I3D Reconstruction I

• From matrix F the camera projection matrices for each image can be computed : P and P’ , w.r.t the chosen projective basis.

• Reconstruction:• E matrix is obtained from F matrix

• E is decomposed in a rotation (R) and translation (T) matrix

• Pi is established.


System DesignSystem Design

Stereo Calibration

C1

C2

Feat 1

Feat 2

n frames

n frames

Calibext./int.

Comm Modul



Reconstruction

C1

C2

n frames

n frames

Calibext./ int.

Comm Modul

3D-Recon-struction

3D-Coor-dinates

Feat 1

Feat 2



Tracking

C1

C2n frames

Optimi-zation

Comm Modul

3D-Coor-dinates

Calibext./int.

Feat


Next StepsNext Steps

• Finalising and testing of implementation of calibration method (stereo, hybrid)

• Work on self calibration• Improving image processing• Implementing and testing reconstruction

methods• First tests of tracking in natural scenes


Status on low cost rendering options (Peak+SV)


Low cost HW for rendererLow cost HW for renderer

• The renderer HW main requirements:– Low cost (inferior to 20 kE).– Compatible with Linux.– Powerful graphics accelerator.– Real time mix 3D/video.

• The candidates are :– SGI Linux workstations : no video device is planned– integrated video/3D cards like MATROX SG2000 and the

wildcat 4210.– “Home made” solution based on nVidia GeForce II.

• SV explores the home made solution in order to:– have a back up solution.– Have rackable HW– be independent from a integrator (like SGI or Intergraph)


HW for renderer :HW for renderer :the “home made” solution (1)the “home made” solution (1)

• The video environment

Video OutKeyer Channel

422 out3D Renderer

Tracking Data

( Video Texture In )

Background video


HW for renderer : HW for renderer : the “home made” solution (2)the “home made” solution (2)

The principle: Video and 3D acceleration are processed by separated devices.

2 D1 boards provide video output in D1 4224 format. One card is dedicated to the 422 video and the other one to the alpha channel. These cards load their video content in real time from the mother board’s RAM by DMA transfers.

A nVidia GeForce II accelerates the 3D rendering. The result buffer is transferred to the mother board.



• The architecture

System

Memory

Textures

1.0 GB/s

Pentium® III processor

Core2.0 GB/sL2 Cache

Intel i840 AGPset

1.0

G

B/

s

Nvidia Quadro Graphic Board 1.0 GB/s

AGP 4x

4.4

7

Gb

/s

Local Memory

ResidentTextures

Graphic Board

DVSAlpha Out

DVSVideo Out

PC

I1

32

M

B/s

Personnal Computer

Keyer

Liv

e

Vid

eo

In

Broadcast

Video Out



Conclusion :

This architecture is sufficient for simple 3D rendering. But for advanced 3D products like Everest the lost of 50% of the graphics processing power may be unacceptable.


Detailed specifications on Keyer (Crystal)


ENREVI Keyer Functionality Block DiagramsENREVI Keyer Functionality Block Diagrams

(Mark 2 - June 2000)(Mark 2 - June 2000)

• Top Level• SDI Input• SDI Output• OPEN Architecture CPU and

Communications • Linear Key Generator

Functionality of ENREVI chroma

keyer (Mark 2 prototype)


GPI Input

RS422 To Control Panel

RS422 From Control Panel

RS422 Receiver

Hitachi H8 Microcontroller

RS422 Transmitter

16K BitsNon-volatile RAM

32K Bytes SRAM

1M Byte EPROM

Watchdog and Reset Generator

To/From Coefficient Stores

Program Port For Programmable Logic

GPI Input Processing

CPU and Communications Block CPU and Communications Block Diagram of ENREVI KeyerDiagram of ENREVI Keyer

Functionality of ENREVI chroma

keyer (Mark 2 prototype)


Main Out

Foreground In

Unipolar to Bipolar Data Conversion

Wipe Generator

Rectangular Mask Generator

Matte Generator

Select Foreground

Main Key Processing Engine

Background Multiplier

Select, Add andMultiplex Main Output

Preview OutSelect and

Multiplex Preview Output

Coefficients StoreFrom CPU

Background In

External Key In

Suppressed Foreground In

Chroma Key In

Unipolar to Bipolar Data Conversion

External Key Processing

Non Additive Mix

Strip and delay Non-active Picture Information

Combine Active and Non-active Picture Information

Mix Generator

Fade Generator

Self Key Generator

Black Generator

Chroma Key Processing

ForegroundMultiplier

Linear Key Generator Block Linear Key Generator Block Diagram of ENREVI KeyerDiagram of ENREVI Keyer


Detailled Project statusDetailled Project status

• WP1 Management

• WP2 System architecture

• WP3 Tracking

• WP4 Keying

• WP5 Rendering

• WP6 System and Integration


• Hiring of engineers in direct relation with project accounts to 5 people

• Equipment related to project was purchased by partners.

• Web site of project has been completed and permanently updated.

• Symah Vision entered into the national initiative for standardisation (AFNOR).

• Contact with main national active bodies have been initiated and SV personnel attented several meetings

• Review documents after month 6 have been posted to private section of web site.

• Activity is generally in line with plan.

WP1 Project management WP1 Project management


WP1 (Project management)

• Initial payment was transferred to consortium partners • Consortium agreement is being circulated

PISTE project has been contacted to study concertation potential

• Concertation subgroup on AR and VR imaging has been formed under the leadership of Fabio LAVAGETTO from University of Genova.

• Dissemination strategy:. Technical paper should be ready by october to be submitted to adequate conferences.


WP2 (System studies, specifications)WP2 (System studies, specifications)

• First version of system specifications being revised ans updated.

• Preliminary experiments on new hardware plateforms have been performed.

• Linux investigations are completed.

• Evaluation of various Digital Video I/O cards for industry standard PCs in under way.

• Concept of how to antialias 3D objects in OpenGL has been studied.

• Documentation of interface tracker-renderer finalized.

• Documentation of external control for renderer finalised.

• Documentation of Keyer external control finalised.


WP3 (Tracking development)

• Selection and ordering of components for the tracking subsystem• Continuing investigation of hybrid tracking systems; selecting of promising

approach and starting of development of detailed mathematical formulation• Starting of work on extending of existing marker-based tracking algorithm

to achieve more stability, robustness and speed. • The work that will be performed includes:

Improvement of robustness of marker detection Making algorithm platform independentImproving speed by parallizing

• The marker-based algorithm will be as a reference algorithm for verifying the new non-marker based tracking algorithm and the used algorithms for stereo and hybrid calibration.

• Continuing investigation of stable, reliable and practicable approaches for calibration of stereo- systems (e.g. self calibration, calibration from planar patterns)

• Continuing of investigation of techniques and algorithms for feature tracking (points, lines, line segments)

• Some High frequency tracking research contribution performed by SV.


WP4 (Keying development)

• Mathematical model for the chroma keyer completed in JanuaryProgrammable logic coding for 1st version started in February

• 1st Prototype was built in AprilChroma keying demonstrated on new hardware.

• All basic functions working. Select colour, acceptance angle, chrominance suppression. Y suppression, suppression angle.

• Purchase Altera FIR filter software to design better interpolation filters. Start using new generation Altera design software (Quartus) despite it being unfinished. Better release promised in September.

• Update hardware design in June to include new Altera logic family. PCB layout for new design.Demonstrate new Altera design using 2 prototypes (1/2 processing on each)Transfer new Altera design to 20K logic ready for new hardware is completed .


WP5 (3D real time rendering)WP5 (3D real time rendering)

• Peak and Symah have initiated discussion on various tracks to generate broadcast quality genlocked video signal from general purpose platforms.

• Implementation of a first step for realtime antialiasing in OpenGL on a PC.

• First system studies for porting renderer to Linux.

• First tests with video cards from Matrox and Pinnacle

• Symah Vision has obtained some results in preliminary architecture of low cost 3D real time renderer.

• Peak described how they plug their renderer to external trackers : a deamon external to the application takes the data coming via the tracking provider’s protocol (ex : SV communication library) and send them into a memory shared by the renderer.

• The market for low cost renderers has been shacked by discontinuation of SGI NT solutions and change of Intergraph/Wildcat strategy.SV is proposing to try a “ home made ” solution where video and 3D acceleration are processed by separated devices. The corresponding architecture has been reviewed at consortium meeting in Cambridge.


WP6 (System Integration)

• Final demo scenario has been agreed:

• A real time insertion of 2D and 3D objects will be positioned on a mock-up of a stadium.

The size would be approx 10 square meters.

This solution has been preferred to an outdoor demo to allow possibility to implement the demo for IBC 2001.


IssuesIssues

• Consortium Agreement late

• Cost statements dry run late

• Report on standardisation activity late

• No visibility on low cost renderer from the trade

• SV proposes to test home made solution with no global budget

impact.


Meetings AttendedMeetings Attended

• Concertation meeting in Brussels on 24 and 25 Jan 2000

• Kickoff meeting on 26 and 27 Jan 200 in Insbrück

• National meeting of MPEG initiative in Nice on 7 Feb 2000

• Meeting between Crystal and BBC on 14 Feb for preliminary specifications.

• Many meetings between Crystal, Peak and SV in Las Vegas during NAB in april ,

IGD was unfortunately unable to attend – 10th to 13th April 2000.

• European MPEG meeting in Noordwijkerhout on 22nd and 23rd March 2000.

• MPEG application meeting in Montreux from 23rd to 25th May 2000.

• Concertation meeting in Brussels 24th and 25th May 2000.

• Meeting SV/Peak in Innsbrück on 28th June 2000.

• Meeting SV/IGD in Darmstad on 3rd July 2000.

• Consortium meeting in Cambridge on 5th and 6th July 2000, all partners attented.

enreviist-1999-11185 enrevi enhanced reality for the video october 4 concertation

Documents

enrevi tracking slide

d camera tracking

interfaces sv slide

igd software tracking

concertation slide

choice slide

interfaces sv tracking

advance tracking research