making tele-presence work or making the ag work better
DESCRIPTION
Making tele-presence work or Making the AG work better. [email protected] Head, ANU Internet Futures. It’s all about tele-presence. The feeling of being there with somebody or something Requires sufficient “quality” to make the technology invisible Quality: - PowerPoint PPT PresentationTRANSCRIPT
Making tele-presence workMaking tele-presence workoror
Making the AG work Making the AG work betterbetter
[email protected]@anu.edu.au
Head, ANU Internet FuturesHead, ANU Internet Futures
It’s all about tele-presenceIt’s all about tele-presence The feeling of being there with somebody or The feeling of being there with somebody or
somethingsomething
Requires sufficient “quality” to make the Requires sufficient “quality” to make the technology invisibletechnology invisible
Quality:Quality:• Image resolution, colour, framerate, …Image resolution, colour, framerate, …• Audio realismAudio realism• Latency, jitter, errors, …Latency, jitter, errors, …• ““Feel” for remote site(s)Feel” for remote site(s)• Minimise cognitive distractionsMinimise cognitive distractions• Avoid loss of functionalityAvoid loss of functionality
‘‘Sorry, you can’t do that with this system’Sorry, you can’t do that with this system’
www.accessgrid.orgwww.accessgrid.org (open-source ANL project)(open-source ANL project) Multicast-based Multicast-based (mostly)(mostly) Many-2-many interaction Many-2-many interaction (in principle)(in principle) Multiple A/V streams per site Multiple A/V streams per site (usually)(usually) Some Application sharing Some Application sharing (sort of)(sort of)
• PPT, WWWPPT, WWW• Text Edit, Whiteboard, InstrumentsText Edit, Whiteboard, Instruments
No bandwidth constraints No bandwidth constraints (ideally)(ideally)• Today (h.261 x [2 - 4] + audio) ~ 2Mb/s per siteToday (h.261 x [2 - 4] + audio) ~ 2Mb/s per site
Can scale down to desktop and PDACan scale down to desktop and PDA
Access Grid usageAccess Grid usage
Formal multisite eventsFormal multisite events• Seminars, distributed classes, virtual Seminars, distributed classes, virtual
conferencesconferences• Board/committee meetingsBoard/committee meetings
Casual usageCasual usage• Walk-in discussions, ad-hoc meetingsWalk-in discussions, ad-hoc meetings• Desktop monitoringDesktop monitoring
Wrapper to larger applicationsWrapper to larger applications• Visualisation/Virtual Reality, Visualisation/Virtual Reality, • Computational SteeringComputational Steering• Data browsingData browsing
Who, Where?Who, Where? 400+ sites today 400+ sites today
150+ U.S., 40+ Europe, 30+ Asia150+ U.S., 40+ Europe, 30+ Asia 30 live in Oz, 15+ in plan/build (4+ at ANU)30 live in Oz, 15+ in plan/build (4+ at ANU) Mostly at Universities, so farMostly at Universities, so far
Developers:Developers: 20+ groups on shared applications20+ groups on shared applications 10+ groups on visualisation/VR10+ groups on visualisation/VR 10+ groups on high-quality video10+ groups on high-quality video 3+ groups on user interfaces3+ groups on user interfaces
An AG An AG (room sized)(room sized) Node Node
Screen
Proj Proj ProjCam
Cam Cam
Mike Mike
Spkr
Display PC
VideoCap PC
Audio PC
Control PC
Now: Now: Making the AG work Making the AG work betterbetter
The power of the AG is its flexibilityThe power of the AG is its flexibility
The problem with the AG is too much The problem with the AG is too much flexibilityflexibility• Lots of effort on new functionalityLots of effort on new functionality
Often in domain-specific areasOften in domain-specific areas
• Limited effort on Limited effort on Robust-ifying existing functionalityRobust-ifying existing functionality Enhancing the common functionalityEnhancing the common functionality
The problemThe problem 4+ AG (room) Nodes at ANU4+ AG (room) Nodes at ANU
• 3 almost complete 3 almost complete Computer Science, HPC, ObservatoryComputer Science, HPC, Observatory
• 1 under way 1 under way Medical SchoolMedical School
• 4 more being discussed4 more being discussed Including a 1500 seat theatre!Including a 1500 seat theatre!
• Plus various desktops and test systemsPlus various desktops and test systems
Pressure to make the AG nodes Pressure to make the AG nodes • Production quality (robust)Production quality (robust)• Cheaper to operate (less qualified operators)Cheaper to operate (less qualified operators)• Remotely operableRemotely operable• USER FRIENDLY !!!!!!USER FRIENDLY !!!!!!
Maximise the feeling of tele-presenceMaximise the feeling of tele-presence
AG problemsAG problems
1.1. Performance, benchmarkingPerformance, benchmarking2.2. MulticastMulticast3.3. User InterfaceUser Interface4.4. Capture, codecsCapture, codecs5.5. Integration with “standard” videoconf.Integration with “standard” videoconf.6.6. Recording, recoding and playbackRecording, recoding and playback7.7. Hardware add-ons, control, and Hardware add-ons, control, and
applicationsapplications
1-Performance measurement1-Performance measurement
RTPreplicateRTPreplicate Stress your display/audio machinesStress your display/audio machines One stream goes in, N come outOne stream goes in, N come out Use feedback from rtcp to benchmarkUse feedback from rtcp to benchmark
Derive a meaningful numberDerive a meaningful number
Full throughput testFull throughput test network, interfaces, CPU/RAM/graphics busnetwork, interfaces, CPU/RAM/graphics bus
2-Multicast2-Multicast
Multicast isn’t everywhereMulticast isn’t everywhere Bandwidth isn’t everywhereBandwidth isn’t everywhere Multicast/unicast bridges Multicast/unicast bridges
don’t scale well in don’t scale well in managementmanagement
Host must reconfigure all the Host must reconfigure all the timetime
RCbridgeRCbridge Web interface, per request bridge Web interface, per request bridge
set up and configured on set up and configured on demand, any number of demand, any number of clients/sessionsclients/sessions
rtp stream selection (home use, rtp stream selection (home use, wireless use, …)wireless use, …)
RCBridgeRCBridge
•http is scriptable – command line clients
•Can build a P2P network of unicast fowarders, e.g. for failover, or application-layer multicast – but prefer multicast to be FIXED!!!
3-User Interface3-User Interface Vic (video) and rat (audio)Vic (video) and rat (audio)
old, unfriendlyold, unfriendly Few new codecs (DV, MPEG-x, MJPEG, HDTV, Few new codecs (DV, MPEG-x, MJPEG, HDTV,
surround/hi-fi sound, …)surround/hi-fi sound, …) And don’t scale all that wellAnd don’t scale all that well
• No lip-synch No lip-synch • No audio-localisation, to match displayNo audio-localisation, to match display• Manageability is a problemManageability is a problem
Major effort of node operatorMajor effort of node operator
• No ability to scale hardware further, No ability to scale hardware further, across clustersacross clusters
Performance and reliability advantagesPerformance and reliability advantages
VP site-tiles and layout managersVP site-tiles and layout managers
VP – a new startVP – a new start
Site-tiles Site-tiles (grouping and prioritising)(grouping and prioritising) Display manager Display manager (layout of site tiles)(layout of site tiles) Use of OpenGL capabilitiesUse of OpenGL capabilities
• Smooth (not just quantised) scaling, done in hardwareSmooth (not just quantised) scaling, done in hardware
• Much more flexible user interfaces possibleMuch more flexible user interfaces possible
Other metadataOther metadata• Stream layout Stream layout (e.g. panorama)(e.g. panorama)
• Local time Local time (why is that person asleep??)(why is that person asleep??)
• Network issues Network issues (have we lost multicast again??)(have we lost multicast again??)
• Flags, icons Flags, icons (graphics to identify a site)(graphics to identify a site)
VP – a new startVP – a new start
Remote OperationRemote Operation• ““VP-master” controls the “VP-client”, over the networkVP-master” controls the “VP-client”, over the network
Cluster display: Cluster display: • MULTIPLE VP clientsMULTIPLE VP clients, still single VP-master, still single VP-master
• Works across different operating systems at the same time!Works across different operating systems at the same time!
Audio Audio (via existing audio tool for now)(via existing audio tool for now) • lip-synchronisation, lip-synchronisation,
• Visual highlightsVisual highlights
• Virtual audio localisation, Virtual audio localisation,
Runs on Windows, Linux; FreeBSD and MacOSXRuns on Windows, Linux; FreeBSD and MacOSX
VP – with DVVP – with DV
and MPEG4 and H263 in the next release
4 – Capture, Encoding 4 – Capture, Encoding
““VC” – new project in 2004VC” – new project in 2004 Supported by AUDF; Mac first!Supported by AUDF; Mac first!
Any number of input devicesAny number of input devices Various filtersVarious filters
WatermarksWatermarks Captions, bannersCaptions, banners ChromakeyChromakey
MetadataMetadata Layout, site information, priorities, …Layout, site information, priorities, …
Various codecsVarious codecs H.261, m(j)peg-X?, DV?, H.263, H.264H.261, m(j)peg-X?, DV?, H.263, H.264
5 - Integration with H.323, …5 - Integration with H.323, … H.323 is wide-spread, AG H.323 is wide-spread, AG
is not (yet)is not (yet) Bridging between two Bridging between two
worldsworlds• Analogue Analogue • VRVSVRVS
Silver323Silver323• Open source, software Open source, software
gatewaygateway• Some MCU-like functionalitySome MCU-like functionality
• Hardware constrained, but Hardware constrained, but cluster-ablecluster-able
• http managed, very flexible http managed, very flexible and “automate-able” and “automate-able” stream selectionstream selection
• Provides an easy path into Provides an easy path into VoIP and PSTNVoIP and PSTN
• SIP, streaming easy to addSIP, streaming easy to add
6 – Recording, recoding, playback6 – Recording, recoding, playback
Current practice: rtpdump, rtpplayCurrent practice: rtpdump, rtpplay• Poor design, does not scale, Poor design, does not scale, • no remote controlno remote control
New toolkit New toolkit • Recording, playout daemons, remote controlRecording, playout daemons, remote control• Scalable file formatScalable file format• Additional media typesAdditional media types
Whiteboard, shared-apps, annotations, …Whiteboard, shared-apps, annotations, …
• Tools to extract subset of streamsTools to extract subset of streams• Tools to convert to e.g. DVD formatsTools to convert to e.g. DVD formats
7 – Hardware, Applications7 – Hardware, Applications Integrate electronic pen-tracking whiteboardIntegrate electronic pen-tracking whiteboard
Astronomers want blackboard/chalk integration!Astronomers want blackboard/chalk integration! E-beam and Mimio are excellentE-beam and Mimio are excellent Existing whiteboard apps are mouse driven, and oldExisting whiteboard apps are mouse driven, and old
Remote control of nodesRemote control of nodes Design nodes to be remotely managed, user managed Design nodes to be remotely managed, user managed
(cameras, projectors, vcr/dvd, …)(cameras, projectors, vcr/dvd, …) Maintain rapid user support in rooms (operator-cam, with Maintain rapid user support in rooms (operator-cam, with
dedicated audio)dedicated audio)
More AG-literate applicationsMore AG-literate applications Shared Acrobat Shared Acrobat Library for generic apps (cross-platform) to code toLibrary for generic apps (cross-platform) to code to Most generic: VGA capture!Most generic: VGA capture!
Conclusion: AG developmentConclusion: AG development
1.1. Performance, benchmarkingPerformance, benchmarking2.2. MulticastMulticast3.3. User InterfaceUser Interface4.4. Capture, codecsCapture, codecs5.5. Integration with “standard” videoconferencingIntegration with “standard” videoconferencing6.6. Recording, recoding and playbackRecording, recoding and playback7.7. Hardware add-ons, control, applicationsHardware add-ons, control, applications
But we need your help, your code, But we need your help, your code, your ideas, and your advice – you, the users!your ideas, and your advice – you, the users!
http://if.anu.edu.auhttp://if.anu.edu.au