telepresence microscopy at csiro minerals

TelePresence Microscopy at CSIRO Minerals

Colin MacRae

CSIRO Minerals - Microbeam Laboratorywww.minerals.csiro.au/microbeamlab

Remote Access and automation Workshop 2005

Talk outline

What and why? History of TelePresence at CSIRO Minerals 3rd generation TelePresence system

System architecture Control interfaces Video streaming technology Security features Administration Modular installation of new functions

Conclusions, where to from here?


What is TelePresence? Tele-conferencing

Video + Audio

Tele-control Microscope controls On-line data collection + analysis Tele-control is provided by some

manufacturers

Ideally, TelePresence should be: Interoperable with many platforms ‘Shrink-wrapped’ software

• Looks professional, with easy to use interfaces

Fast and responsive Stable and secure


Why do we want TelePresence? Internal clients (Brisbane, Hobart, Perth) External clients (Melbourne, Sydney, Brisbane) For analyses, our clients must either:

Mail / courier samples, and wait for our report (slow) Travel to our laboratory (expensive)

TelePresence can help Provide results to client in real-time, and also allows instant

feedback from client Attract new clients

CSIRO

Marine Research

CSIRO

Minerals

Comalco & RioTinto

CSIRO

Minerals, Exploration & MiningBHP-Billiton


What do we want to connect?

Scanning Electron Microscopes Electron Microprobes & Proton Microprobe X-ray detectors –EDS & WDS Cathodoluminescence systems Optical microscopes


What do we want to offer

Clear fast images On-line microanalysis On-line x-ray spectra Large area optical views Streaming maps Security Voice over IP Data storage Data

analysis/presentation software


What do we need from TelePresence?

Provision of: Real-time video from multiple sources (SEM, room camera) Streaming EDS spectra and other data ( + controls) Control of microscope stage + column (lower priority)

Minimal hardware requirementsPC or laptop (no workstations)ADSL or cable modem, ~1Mbit/s (not Internet2, GRID)

‘Shrink-wrap’ softwareEasy to set-upVery easy to learn and use, with generic controls/interfaceVery, very stableExtraordinary securityLow maintenance

Must not affect normal operation of instruments!


TelePresence history

TelePresence Microscopy 1999 Nestor J. Zaluzec, Argonne National

Laboratory Web-based, ‘pervasive’ Web based notebook Video with “GTS” streaming

technology


TelePresence history

TelePresence Microscopy 1999 Nestor J. Zaluzec, Argonne National

Laboratory Web-based, ‘pervasive’

Java system 2001 Platform independent, ‘safe’ code MJPEG video (TCP-IP) Web based notebook with

embedded Java applets for Images, Spectra and maps

Successful, but limited


TelePresence historyTelePresence Microscopy - 1999

Nestor J. Zaluzec, Argonne National Laboratory

Web-based, ‘pervasive’

Java system - 2001 Platform independent, ‘safe’ code MJPEG video Web based notebook with embedded

Java applets for Images, Spectra and maps

Successful, but limited

C++ system – 2004 Widely used language, with broad

library support MPEG-4 video compression in software,

changeable video size New, secure architecture


Internet protocols: UDP versus TCP

RouterTCP packet

Sender

acknowledgement

Lost packet?

UDP packet

TCP

UDP

TCP = Transport Control Protocol

UDP = User Datagram Protocol


Video latency Definition:

Latency = time displayed – time captured What do we want?

<500ms for focussing (or better) <200ms preferred

Causes of latency1. Video compression / decompression

• Hardware accelerated rendering (done)• Hardware compression (cheap MPEG-4 hardware compression

coming soon)

2. Router delays (congestion)• Ultra-broadband internet (eg GRID)• Reduce stream size (better codec)

3. inter-router delays (speed / distance)• More bandwidth wont help• Use faster internet protocol (i.e. TCP UDP)


Another real life problem : The Firewall

Client

Host

Server

‘DMZ’

Client network

Internet

Video

Our networkFirewall Firewall

port 8080 (UDP), from specified addresses

port 8080 (TCP)


Comparison of platforms

Web browser

Java C++

Trusted? Somewhat Yes Yes, with security

Platform independent?

Yes Yes No

Functionality HighModerateLow

Execution speed HighHighLow

Development pain ModerateHighLow

The vibe Yeah groovySo soPoor


Password

System architecture

Client

Server

Drivers

Host

Video

(UDP stream)

Server

Server

commands & data

(TCP socket)

Instrument specific

Generic interface

JEOL 8900RElectron Microprobe

FEI QuantaFEG-Environmental

SEM

Hitachi S-5000In-Lens Cold FEG

SEM

PIXEProton Microprobe


DirectShow video streaming

‘GraphEdit’ component of Microsoft DirectX software development kit (SDK)

Video capture

Timing system

(frame-rate)

Compression codec

Network transmission

sockets


DirectShow video streaming Very easy to configure and re-configure ‘graphs’ Other stream designs are possible, eg:

Saving stream to HDD Motion detection triggering Adaptable to as yet unforseen requirements

All components are upgradeable and replaceable New codecs can be adopted as they become available ‘Future proof’

We didn’t have to write any of these components (only wire them together)

Some DirectShow components are hardware-accelerated

Fast video rendering


TelePresence interfaces

Video Camera control

Tilt-pan-zoom controlsUsed in conjunction with video

window

EDX spectrometersGeneric, easy to use interface Fast (2-5 updates/s)

Text streams Miscellaneous others Stage and Electron

optical System


Video streaming technology

Microsoft DirectShow Widely supported standard

with pre-compiled modules Hardware-accelerated

rendering

MPEG-4 encoding 320 240 at 25 fps is

~500kbit/s (~ADSL) (~12 Mbit/s by MJPEG) (~59 Mbit/s for raw video)

UDP transmission Lowest possible latency over

IP (~200ms)

Themeda Australis

Kangaroo grass

Variations in humidity cause grass to spiral and “dig” into the soil


Video cameras

Canon VC-C1 and VC-C3 cameras

450 x 350 pixels at 30 frames/s.

±90º pan, ±25º tilt, 10 zoom

Good low-light performance


Bench-top camera

‘ELMO’ camera XGA video, 20 frames/s www.elmousa.com/presentation


Client feedback

Problem: Video stream is one-way. How do clients tell us what they want to see?

Verbal feedback (telephone) “that thing, over to the left. Up,

say, half an inch…” Not very effective

Graphical feedback Video overlay marker Visible to all other clients and the

operator More tools to follow


Security Firewalls

Blocks all direct communication with servers and drivers

Host and servers are in separate ‘DMZ’ network, not mounted on CSIRO file system

Socket encryption: Sockets are wrapped with ‘Rijndael’ cipher,

NIST advanced encryption standard (AES 2001).

• Certified for all levels of classification, up to and including ‘top-secret’.

Hackers must break AES to read data, send false data, or issue commands


On-line help


Administration


Web based administration


Software based Administration tools


Administration tools – Modular

New instruments can be added and

configured


Summary

TelePresence allows Microscopy facilities providing a alternative service to clients

High-quality real-time video is achievable over cable modem using MPEG-4 codec's and UDP transmission

Internet security is ensured using firewalls and encryption of communications

Architecture is extensible and applicable to any instrument for which DLLs or control code are available


Acknowledgements CSIRO Minerals

Aaron TorpyNick Wilson

CSIRO ITSTony SandersonAndrew Smith

Engineering Firm CymontkowskiMarc Cymontkowski

Further informationTPM web reference

http://minerals.tpm.csiro.au/software/telepresence/


Questions?

telepresence microscopy at csiro minerals

Documents