the magic project final demonstration and briefing

Corporation For NationalResearch Initiatives

The MAGIC ProjectFinal Demonstration and Briefing

Ira Richer

29 July 1999

DARPADARPA


MAGIC Scenario

1 m aerial terrain10/95

10 m satellite terrain“yesterday”

1 km satellite weather“today”

10 m aerial troop movement“now”

10 m satellite terrain1/96


Datasets

Globe– San Francisco Bay area

Menlo Park

– Ft. Irwin– Camp Pendleton (low resolution)

Camp Pendleton (high resolution)– Laguna Beach

– Yosemite– Colorado– Lawrence, KS– Ohio– Washington, DC– Ft. Benning


Background

The MAGIC scenario is based on applications such as mission planning and crisis management

These applications need to:– locate, process, and display real-time and/or archived datasets

on-the-fly– fuse multiple types of data (e.g., weather, terrain)– be able to transfer large volumes of data for interactive

visualization– support a range of user capabilities (e.g., command center,

mobile user)

MAGIC focuses on a terrain visualization application that uses geo-referenced datasets


Objectives

To develop an information system that provides application users with fast, interactive access to very large volumes of remote data distributed over a wide geographic area

To demonstrate the above capability in the context of a 3-D terrain visualization application


MAGIC Participants

Corp. for National Research Initiatives USGS/EROS Data Center Lawrence Berkeley National Laboratory Sprint SRI International University of Kansas


Distinguishing Characteristics

Distributed information system, designed from the outset to work over a network– adaptation to network conditions and to system load

No limit on size of viewable datasets No restrictions on location of datasets Fusion of multiple datasets Near real-time visualization of terrain data


MAGIC Components

High-speed internetwork Distributed parallel storage system (DPSS) Software for terrain dataset preparation 3-D interactive terrain visualization

application: TerraVision


DPSS

DPSS

DPSS

DPSS

Generic Configuration

High-speedinternetwork

Repository

Processor

Processor

User

User

Repository


7172

7374

70 4331

8788

4445

4632

3334

3536

86

2524

System Operation

D1 D2

733335

45

SERVER 2Server 1

D1 D2

7143 34

SERVER 1

Internetwork

User

7143 33 34 35 73 45

Tiles located onDPSS server/disk

S1D1

S1D2

S1D1

S2D1

S2D2

S2D2

S2D1

Tiles onpath of travel

Tiles retrievedand transmitted


Today’s Demonstration and Briefing

3-D, interactive visualization of large volumes of remote data

Platform scalability On-demand data preparation and 3-D

flyover Fusion of multiple types of data from

multiple sources DPSS fault tolerance


Today’s Configuration

DARPA ISI-E

LBNL

ESnet GPN

NREN

ATDnet

Abilene

EDC

NASAGoddard

KU

DPSS

DPSS

DPSSDPSS


Dataset Characteristics

Resolution (m) Size (MB)

Globe ~1000 4,400San Francisco Bay area 25 670Menlo Park 1 680Ft. Irwin 1 330Camp Pendleton 30 44Camp Pendleton 1 12,300Laguna Beach 1 140Yosemite 1 2,100Colorado 30 200Lawrence, KS 1 680Ohio 30 1,000Washington, DC 1 8,600Ft. Benning 0 .15 380


Data Flow (1)

DARPA ISI-E

LBNL

GPN

NREN

ATDnet

DPSS

DPSS

Abilene

EDC

DPSS

NASAGoddard

KUDPSS

ESnet


Dataset Characteristics

Resolution (m) Size (MB)

Globe ~1000 4,400San Francisco Bay area 25 670Menlo Park 1 680Ft. Irwin 1 330Camp Pendleton 30 44Camp Pendleton 1 12,300Laguna Beach 1 140Yosemite 1 2,100Colorado 30 200Lawrence, KS 1 680Ohio 30 1,000Washington, DC 1 8,600Ft. Benning 0 .15 380


Platform Scalability

SGI Octane (~$50k)– OC-3, IRIX, DPSS– 60 frame/sec, ~80Mb/s

Dual-processor PC with graphics accel. (~$8k)– Fast Ethernet, Windows NT, HTTP– 30 frame/sec, ~0.5Mb/s (Internet-limited)

Dual-processor PC (~$4k)– OC-3, Linux, DPSS– 10 frame/sec, ~10Mb/s

Laptop (~$3k)– Fast Ethernet, Windows 98, HTTP– ~4 frame/sec, ~0.5Mb/s (Internet-limited)


TerraVision II

Composite datasets: data from multiple sources, at different resolutions

VRML overlays: wind vectors, buildings, roads; viewable with standard browser– Wind-vector dataset hosted at NAVO

Co-registered maps for navigation Pre-defined flight paths for operations

support, testing


C lient A pp lica tion

P ara lle lD isks

D P S S S erve r

P ara lle lD isks

D P S S S erve r

P ara lle lD isks

D P S S S erve r

D PSS M aster

da ta b locks

da ta b locks

da ta b locksLog ica l B lock

R eques ts

log ica l to phys ica lb lock lookup

access con tro l load ba lanc ing

P hys ica l B lockR eques ts

DPSS Cache Architecture


DPSS Costs (Jan. ‘99)

Throughput Capacity Cost(Mb/s) (GB) servers disks ($k)

80 33 1 2 6

400 165 5 10 32

400 1000 5 64 80

800 1000 10 64 100

Configuration


Monitoring and measurement

Real-time traffic monitor DPSS system monitor NetLogger

– Logs events at critical points of distributed applications– Provides complete view of system operation– Visualization tools allow for real-time monitoring

Agent-based management of measurement tools


NetLogger Instrumentation

ClientApplication

Shared Memory Cache

BlockRequestThread

DiskRead

Thread

DiskRead

Thread

DiskRead

Thread

DiskRead

Thread

DPSS Master

from o ther D P S Sservers

*

= monitoring point

DPSS Data Server BlockW riterThread

to o ther c lien ts

Disk Disk DiskDisk

*

**

*

**

*

*

*

**


DPSS Capabilities

Load balancing– Dataset must be replicated – Tile requests sent to server that will provide the best

performance, based on current server load and network conditions

Automatic TCP tuning– Window size determined by throughput and delay

information

Fault tolerance– Automatic detection of failed server, switchover to

alternate servers, and reconnection after recovery


Data Flow (2)

DARPA ISI-E

LBNL

ESnet GPN

NREN

ATDnet

DPSS

DPSS

Abilene

EDC

DPSS

NASAGoddard

KUDPSS


Technology Transfer and “Spinoffs”

DPSS– Earth Systems Grid (climate modeling)

– Kaiser Permanente project

– Combustion Corridor (diesel modeling)

– Matisse (MEMS/Supernet)

NetLogger– Being used by many network and grid projects


Technology Transfer and “Spinoffs” (concluded)

USGS archive – Web-based access to digital earth-science data

– Gateway to the Earth: Ohio Pilot (automatic dissemination of Landsat imagery)

TerraVision– BADD (Dissemination of VRML terrain datasets to warfighter)

– Digital Earth (fast indexing of geo-data, multimodal capabilities, and education applications)

– Netscape plug-in

the magic project final demonstration and briefing

Documents

large volumes of data

flyfuse multiple types

application users

interactive access

system loadno limit

laguna beach1140yosemite12

magic participantscorp

storage servers repositories