variable viewpoint reality...pitches with men on base and less than 2 outs mit ai lab viola &...
TRANSCRIPT
![Page 1: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/1.jpg)
MIT AI LabViola & Grimson
Variable Viewpoint Reality
Professor Paul Viola & Professor Eric Grimson
Collaborators: Jeremy De Bonet, John Winn, Owen Ozier,Chris Stauffer, John Fisher, Kinh Tieu,Dan Snow, Tom Rikert, Lily Lee,Raquel Romano, Huizhen Yu, Mike Ross,Nick Matsakis, Jeff Norris, Todd AtkinsMark Pipes
![Page 2: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/2.jpg)
MIT AI LabViola & Grimson
The BIG picture:User selected viewing of sporting events.
• show me that play from theviewpoint of the goalie
•… from the viewpoint of the ball
•… from a viewpoint along thesideline
• what offensive plays does Brazilrun from this formation
• how often has Italy hadpossession in the offensive zone
![Page 3: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/3.jpg)
MIT AI LabViola & Grimson
The BIG picture:User selected viewing of sporting events.
• Let me see my son’s motion fromthe following viewpoint
• Let me see what has changed inhis motion in the past year
• Show me his swing now and aweek ago
• How often does he swing atpitches low and away
• What is his normal sequence ofpitches with men on base and lessthan 2 outs
![Page 4: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/4.jpg)
MIT AI LabViola & Grimson
A wish list of capabilities
• Construct a system that will allow each/every userto observe any viewpoint of a sporting event.
• Provide high level commentary/statistics– analyze plays
![Page 5: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/5.jpg)
MIT AI LabViola & Grimson
A wish list of capabilities
• Recover human dynamics• Search databases for similar events
![Page 6: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/6.jpg)
MIT AI LabViola & Grimson
VVR Spectator Environment
• Build an exciting, fun, high-profile system– Sports: Soccer, Hockey, Tennis, Basketball, Baseball
– Drama, Dance, Ballet
• Leverage MIT technology in:– Vision/Video Analysis
• Tracking, Calibration, Action Recognition
• Image/Video Databases
– Graphics
• Build a system that provides data available nowhere else…– Record/Study Human movements and actions
– Motion Capture / Motion Generation
![Page 7: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/7.jpg)
MIT AI LabViola & Grimson
Window of Opportunity
• 20-50 cameras in a stadium– Soon there will be many more
• US HDTV is digital– Flexible, very high bandwidth digital transmissions
• Future Televisions will be Computers– Plenty of extra computation available
– 3D Graphics hardware will be integrated
• Economics of sports– Dollar investments by broadcasters is huge (Billions)
• Computation is getting cheaper
![Page 8: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/8.jpg)
MIT AI LabViola & Grimson
For example …
Computed using a single view…
some steps by hand
![Page 9: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/9.jpg)
MIT AI LabViola & Grimson
ViewCube: Reconstructing action & movement
• Twelve cameras, computers, digitizers
• Parallel software for real-time processing
![Page 10: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/10.jpg)
MIT AI LabViola & Grimson
The View from ViewCube
Mul
ti-ca
mer
a M
ovie
![Page 11: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/11.jpg)
MIT AI LabViola & Grimson
Robust adaptive tracker
PixelConsistent
withBackground?
Video Frames Adaptive Background Model
X,Y,Size,Dx,DyX,Y,Size,Dx,Dy
X,Y,Size,Dx,Dy
LocalTrackingHistories
![Page 12: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/12.jpg)
MIT AI LabViola & Grimson
Examples of tracking moving objects
• Example of tracking results
![Page 13: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/13.jpg)
MIT AI LabViola & Grimson
Dynamic calibration
![Page 14: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/14.jpg)
MIT AI LabViola & Grimson
Multi-camera coordination
![Page 15: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/15.jpg)
MIT AI LabViola & Grimson
Mapping patterns to groundplane
![Page 16: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/16.jpg)
MIT AI LabViola & Grimson
Projecting Silhouettes to form3D Models
3D R
econ
stru
ctio
n M
ovieReal-time 3D
Reconstructionis computed by intersecting silhouettes
![Page 17: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/17.jpg)
MIT AI LabViola & Grimson
First 3D reconstructions ...
3D M
ovem
ent
Rec
onst
ruct
ion
Mov
ie
![Page 18: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/18.jpg)
MIT AI LabViola & Grimson
A more detailed reconstruction…
Model
![Page 19: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/19.jpg)
MIT AI LabViola & Grimson
Finding an articulate human body
Segment
3D Model
VirtualHuman
Human
![Page 20: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/20.jpg)
MIT AI LabViola & Grimson
Automatically generated result:
Bod
y T
rack
ing
Mov
ie
![Page 21: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/21.jpg)
MIT AI LabViola & Grimson
Analyzing Human Motion
• Key Difficulty: Complex Time Trajectories
Complex Inter-dependencies
• Our Approach: Multi-scale statistical models
![Page 22: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/22.jpg)
MIT AI LabViola & Grimson
Detect Regularities & Anomalies in Events?
![Page 23: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/23.jpg)
MIT AI LabViola & Grimson
Example track patterns
• Running continuously for almost 3 years– during snow, wind, rain, dark of night, …
– have processed 1 Billion images
• one can observe patterns over space and over time
• have a machine learning method that detectspatterns automatically
![Page 24: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/24.jpg)
MIT AI LabViola & Grimson
Automatic activity classification
![Page 25: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/25.jpg)
MIT AI LabViola & Grimson
Example categories of patterns
• Video of sorted activities
![Page 26: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/26.jpg)
MIT AI LabViola & Grimson
Analyzing event sequencesHistogram of activity over a single day
12am 6am 12pm 6pm 12pm
12am 6am 12pm 6pm 12pm
12am 6am 12pm 6pm 12pm
12am 6am 12pm 6pm 12pm
Resultingclassifier
cars(1564 total
with3.4% FP)
groups of people
(712 total with2.2% FP)
people(1993 total with
.1% FP)
clutter/lighting effects
(647 total with 10.5% FP)
![Page 27: Variable Viewpoint Reality...pitches with men on base and less than 2 outs MIT AI Lab Viola & Grimson A wish list of capabilities • Construct a system that will allow each/every](https://reader030.vdocuments.site/reader030/viewer/2022041120/5f346a3e1811470af41b20d4/html5/thumbnails/27.jpg)
MIT AI LabViola & Grimson
…and this works for other problems
• Sporting events
• Eldercare monitoring
• Disease progression tracking– Parkinson’s
• … anything else that involves capturing,archiving, recognizing and reconstructing events!