joint capability technology demonstration osd (rfd) – usstratcom – nrl – nga – inscom –...
TRANSCRIPT
Joint Capability Technology DemonstrationOSD (RFD) – USSTRATCOM – NRL – NGA – INSCOM – DISA
22
AgendaAgenda
▪ Warfighter Problem▪ Large Data Concept of Operations▪ Operational Utility Assessment▪ Why LD JCTD Works▪ Transition ▪ Summary
33
Other Enterprise
Data
CALA
Global enterprise data available “as if on local desktop”
Warfighter Data
Intelligence Community (IC)
Enterprise
Combatant Command Ops & Intel Enterprise
Medical, Weather, Logistics, etc
44
Large Data Reference ModelLarge Data Reference Model
Open Standards• OFED • Key Interfaces • InfiniBand DDR/SDR• Ethernet• Software Standards• SuSE Linux• OMAR (OGS WMS)• Lustre (POSIX)• Open Source Software• Lustre File SystemCommodity Hardware• Intel Servers• Multiple suppliers for modular componentsFully MOSA Compliant
Architecture
Open Standards• OFED • Key Interfaces • InfiniBand DDR/SDR• Ethernet• Software Standards• SuSE Linux• OMAR (OGS WMS)• Lustre (POSIX)• Open Source Software• Lustre File SystemCommodity Hardware• Intel Servers• Multiple suppliers for modular componentsFully MOSA Compliant
Architecture
Scalable to 1000’s of Nodes — Petabyte File System today…Exabytes tomorrow
FileServers
WANBridge
Workgroup Ethernet
IP, SONET, orLambda Links
Wide Area Networks
Disk ArraysLinks to Disks:• Fibre Channel• SCSI• SATA II• SAS
Examples:• Standard Workstations• Thin Clients• File System Clients
WorkStations
Storage Appliance
Storage Appliance
Core Large-Data
Domain
Ethernet Switches
InfiniBandSwitches
Shared File System
Type 1Encryptor
Client ApplicationDomain
File System Client
GraphicStations
File System Client
DatabaseServers
File System Client
ApplicationServers
File System Client
WebServers
File System Client
Net-FileServers
File System Client
Key Interfaces:
InfiniBand
Ethernet
IP or SONET
55
Large Data DISN InterfaceLarge Data DISN Interface
DISN
Voice, Video, and Data
All security domains (e.g. Unclassified etc.)
Lambda Access
DWDM
IP - Centric
FIBER
IP
Large Data JCTD
Info Services, Collaboration, Visualization, Applications
IB & Ethernet
Data - Centric
IPRDMA
Storage Virt.
Global File System
Servers
• LD architecture efficiently exploits GIG-BE/DISN• Cost effective system interfaces can take full advantage of 10 Gbps• Flexibility to utilize all encryption solutions• Options to leverage lambda, IP/MPLS, Ethernet services
• Options to connect to current DISN Interfaces
• 10 Gbps lambda• 1 Gbps IP for smaller clients• 2.5 Gbps PT-to-PT
• Options to connect to future DISN Interfaces
• 40** Gbps lambda• 10/40** Gbps IP/MPLS service*• 10/40** Gbps Ethernet service*
* Requires no new development; not yet validated in LD** 40 Gbps hardware development underway
66
Standard 2-Rack Node
• Scalable & Flexible
• Extremely Dense
• High Performance
• Low Latency
• Open Standards
• Open Systems
• Open Source
• Commodity Parts
• Scalable & Flexible
• Extremely Dense
• High Performance
• Low Latency
• Open Standards
• Open Systems
• Open Source
• Commodity Parts
77
Availability: Access to global data – “as if as if on desktopon desktop”Availability: Drive failure tolerance, typically 8+2 parity, Raid6 technologyAvailability: Journaling, providing the Reduction / Elimination of LUN rebuilds through I/O bit mappingReliability: Detection and protection against silent data corruption
Parity calculation on all reads On-the-fly-data-correction
Space: Extreme disk density (150TB/ft2)Performance: Delivers sustainable real-time raw large block I/O at near line rates – 8 GBps
• Economics: Sleep mode for power/cooling savings
LD Storage-Processing Solutions
75 TB
22,400 cores – 1540u up to 11.8 PB
5-16u90TB
Man-portable
1.2 PB
High Performance = Low Latency High Performance = Low Latency Access to Critical DataAccess to Critical Data
88
LD Web ServicesLD Web Services
• LD provides “point and click” access to globally distributed data
• Thin-client access with thumbnail previews
• Provides near real-time updates to planning and intelligence applications
99
Use of Open Technology in JCTDUse of Open Technology in JCTD
▪ Lustre▫ Providing scalable, global distributed file
system
▫ Leverages DoE investment and Sun Microsystems backing
▪ OSSIM Mapping Archive (OMAR)▫ Provides dynamic web services
▫ Leverages IC investment with Radiant Blue
▪ Open Federation Enterprise Development (OFED)▫ Industry consortium for an open standard
high performance transport, storage and processing protocol stack
Lustre Key benefits• Open source, multi-platform and multi-
vendor under GPL• Aggregates petabytes of storage and tens of
thousands of nodes into one file system• Production-quality stability and high
availability• POSIX-compliant• Object-based architecture• Windows support with pCIFS• Heterogeneous networking• RDMA performance enhancements with InfiniBand networks• Manual and automated failover• All designs and the internals course are on
lustre.org web site• CVS tree (repository of all code as it
develops) is open to all• Architecture discussions open to all on
lustre development mail list• Managed by a Fortune 500 major technology
company (Oracle Sun) and in wide use among high performance systems
Lustre Key benefits• Open source, multi-platform and multi-
vendor under GPL• Aggregates petabytes of storage and tens of
thousands of nodes into one file system• Production-quality stability and high
availability• POSIX-compliant• Object-based architecture• Windows support with pCIFS• Heterogeneous networking• RDMA performance enhancements with InfiniBand networks• Manual and automated failover• All designs and the internals course are on
lustre.org web site• CVS tree (repository of all code as it
develops) is open to all• Architecture discussions open to all on
lustre development mail list• Managed by a Fortune 500 major technology
company (Oracle Sun) and in wide use among high performance systems
1010
LD JCTD Operational Utility AssessmentLD JCTD Operational Utility Assessment
LDLDLDLD
“While the first of about 10 files was still being transferred to the legacy work station, Large Data
had all ten files.”
LD JCTD IOUA Report, Nov 09, JHU/APL
“While the first of about 10 files was still being transferred to the legacy work station, Large Data
had all ten files.”
LD JCTD IOUA Report, Nov 09, JHU/APL
Globally Synchronized, Shared Data and High Resolution Collaboration
LDLD
LDLD
LDLD
LDLD
Large Large Data Data
Large Large Data Data
1111
0
5
10
15
20
25
7/21/2009 7/26/2009 7/31/2009 8/5/2009 8/10/2009 8/15/2009
Tim
e t
o c
rea
te o
n-d
em
an
d p
rod
uc
ts (
Se
co
nd
s)
LD Site Backend Processing ComparisonLD Site Backend Processing Comparison
Sites 1 & 2 orthorectify and display Imagery to thin clients (on-demand); Site 3 converts MPG-2 to FLV (on-demand) and streams video to clients
Site 1 AVG = 3.77 secSite 2 AVG = 4.44 sec
For Imagery ProcessingSite 3 AVG = 9.35 secFor Video Processing
No discernible difference to user for remote (~1200 miles*) vs. local imagery processing
LD Web Services (Backend)LD Web Services (Backend)
* ~2000 fiber miles
1212
Operational Demonstration ResultsOperational Demonstration Results
Suitability• Demonstrated TRL-7/8/9• Cost effective, open source• Commodity components• Stability/availability on par with operational systems
“A quick overview of the system was all that was required for ease of use”
-- LD User
Operational Impact• GEOINT access & web services for warfighters• UNCLAS US Gov’t and NGO support• Remote access to large, distributed ISR files• Data virtualization & near real-time failover
“Simply put, the system NRL has in place for delivering large, AOI detailed imagery is outstanding and truly a model for the
DoD/IC.” – Sean Wohltman, Google Inc.
Effectiveness: LD met or exceeded Transition Thresholds
1313
LD System Performance LD System Performance
Network TypePt-to-Pt
ConnectionLine Rate
Type of File Transfer(Single or Multiple)
Results Achieved:% of Theoretical/Max Rate (MBps)
MAN (2.5 ms latency)
OC-192
Single file62.8%590.6
Aggregate Multiple Simultaneous Files
99.6%
935.7
WAN(~2000 fiber mi; 34.5 ms latency)
OC-192
Single File59.1%555.2
Aggregate Multiple Simultaneous Files
91.1%
856.4
Long Haul WAN(~13,000 fiber
miles;206 ms latency)
Partial OC-48
Single File86.0%182.3
Aggregate Multiple Simultaneous Files
94.6%
200.7• Maximum Theoretical Data Transfer Rate for OC-192 = 939.7 MBps;
for Partial OC-48 = 212.0 MBps; 1 MB = 1,048,576 bits (2^20)
Test 1: File System-to-File System Data Results:91%-99% of Max Theoretical Data Transfer Rate
Test 2: Remote Access Results:User “accessing” vs. “copying” file from remote file system to local workstation• RDMA-enabled• Access “as if local” to large ISR files over MAN/WAN/Long Haul WAN• Matches local performance• Saves 99.9% of time over “copy to local storage & open”!
Test 3: Data Stream Results:File System-to-User Workstations• OC-192 Link: -- MAN: 808 MBps (85.9% Max) -- WAN: 759 MBps (80.8% Max)• OC-48 Partial Link: -- LH Wan: 160 MBps (75.4% Max)
OFED RDMA / Lustre file system enables remote access and/or very rapid file delivery to user applications
1414
LD Network Performance LD Network Performance
TESTS ON 1 Gbps CIRCUIT (~8000 miles)[ ~13,000 fiber miles]
TESTS ON 1 Gbps CIRCUIT (~8000 miles)[ ~13,000 fiber miles]
TESTS ON 8 Gbps CIRCUIT (~1200 miles)[~2000 fiber miles]
TESTS ON 8 Gbps CIRCUIT (~1200 miles)[~2000 fiber miles]
TypicalRDMA/IB
Performance
TypicalTCP/IP/ETHERNET
Performance• RDMA/Infiniband (IB) provides highly
efficient use of available bandwidth • IB scales well with multiple, concurrent
data flows• RDMA/IB performance ≥ 80%• TCP/IP performance ≤ 40%• RDMA CPU usage estimated 4x less• IB is lossless and provides “fair share”
of bandwidth
TypicalRDMA/IB
Performance
1515
Large Data DISN InterfaceLarge Data DISN Interface
DISN
Voice, Video, and Data
All security domains (e.g. Unclassified etc.)
Lambda Access
DWDM
IP - Centric
FIBER
IP
Large Data JCTD
Info Services, Collaboration, Visualization, Applications
IB & Ethernet
Data - Centric
IPRDMA
Storage Virt.
Global File System
Servers
• LD architecture efficiently exploits GIG-BE/DISN• Cost effective system interfaces can take full advantage of 10 Gbps• Flexibility to utilize all encryption solutions• Options to leverage lambda, IP/MPLS, Ethernet services
• Options to connect to current DISN Interfaces
• 10 Gbps lambda• 1 Gbps IP for smaller clients• 2.5 Gbps PT-to-PT
• Options to connect to future DISN Interfaces
• 40** Gbps lambda• 10/40** Gbps IP/MPLS service*• 10/40** Gbps Ethernet service*
* Requires no new development; not yet validated in LD** 40 Gbps hardware development underway
1616
Shared IP/MPLS Test ConfigurationShared IP/MPLS Test Configuration
IP/MPLS Configuration Testing goals:
1. Validate carrying LD JCTD
traffic over a HAIPE/ IP/MPLS network
2. Validate feasibility of operating at scalable provisioning levels (2, 3, 4, 5, … Gbps)
3. Test LD JCTD in the presence of competing IP traffic
CienaCiena
CienaCiena
WAN GWencryp
Site 1
encryp
Site 2
NRL LabDISA Lab
CienaCiena
WAN GW
Site 2
1717
Validated:RDMA/IB traffic over a shared IP/MPLS core network in a DISA lab
RDMA/IB network transactions performed well at a range of operating levels
RDMA/IB Performance on DISN CoreRDMA/IB Performance on DISN Core
RDMA performs well within DISN assigned bandwidth
RDMA/IB performance with competing traffic on 10 Gbps IP/MPLS network
This is the approx. volume of datathat the node attempted to send.
Excellent RDMA results over IP core up to assigned BW
Excellent RDMA results over IP core up to assigned BW
Expected RDMA results over IP core
beyond assigned BW
Expected RDMA results over IP core
beyond assigned BW
1818
Final Configuration
Site 1 Site 2
VM4VM3
Site 2Server
Site 1Server
MirroredVMStorage
FinalDatabase
Host
VM0
VM1
VM2
VM5
VM6
Initial Configuration
Site 1 Site 2
VM4VM3
Site 2Server
Site 1Server
MirroredVMStorage
InitialDatabase
Host
VM0
VM1
VM2
VM5
VM6
WAN Bridge WAN Bridge
EthernetSwitch
Storage (IB)Switch
EthernetSwitch
Storage (IB)Switch
Virtual StorageController
VMStorage“Distributed RAID 1” Mirrored Storage
VM Hypervisor
Linux VM
Database
Site 1 Site 2
VM Hypervisor
Linux VM
Applications
Virtual StorageController
VMStorage
Linux VM
DatabaseOperator Initiated Migration
EthernetInfiniBandSite 1 DataSite 2 Data
High performance computing/storage for virtualized enterprise
Remote mirroring allows VM clusters to include servers from different remote data centers
Various combinations of data
migration from locations ~1200
miles apart
LD Data Virtualization/Failover
Shared Resource Cluster Shared Resource Cluster
1919
Why the LD JCTD worksWhy the LD JCTD works
The LD JCTD demonstrated the use of RDMA and a clustered Global File
System over long distances to create a globally accessible storage
and compute cloud
1. LD JCTD used RDMA and parallel file systems to build multiple scalable, cost effective data centers
2. LD JCTD extended RDMA over a high bandwidth WAN to virtualize the data centers
3. Clients connect to virtual data center via cost effective, low bandwidth (1 Gbps) IP networks
LD JCTD used standards-based COTS technology and components to demonstrate a secure, cloud computing infrastructure operating over the DISN
RDMA interconnectRDMA interconnect
Global File SystemGlobal File System
+
• Data available to clients anywhere in the world• Remote clients may disconnect at will• Centralized apps available to clients
clientclient
clientclientclientclient
clientclientData CenterData Center Data CenterData Center
Data CenterData Center
2020
LD JCTD – Instant Data Access LD JCTD – Instant Data Access
RDMA interconnect
Global File System
Instantaneous data access model – “as though the data were on your desktop”
• Fast data access times (seconds, not hours)• Efficient WAN usage, fast data transfers• Scalable, cost effective, low power data centers
• Scalable cost effective storage - Inherent COOP• Globally distributed clustered file system• Local caching (for performance) • Direct file access (file transfer not required)
Structured Data Access
• Global ingest, processing, distribution of data• Data is “known” from the moment of ingest• Direct file access (file search not required)
2121
Demonstrated TRL-7, with major elements of TRL-8 and TRL-9
Successful LD operations in mission environment
LD Technical Readiness Level
2222
LD Transition LD Transition
▪ DoD and IC programs of record are adopting LD benefits and capabilities in FY10/11 for:
Rapid, global data access and federated exploitation for very large files such as imagery and wide area persistent surveillance
Operationally responsive data dissemination/transfer Data federation & synchronization for planning Support to global intelligence operations Enhancing net-centric data delivery to warfighters
2323
SC09 Demonstration for Laboratory for Advanced Computing @ UIC / NRLSC09 Demonstration for Laboratory for Advanced Computing @ UIC / NRL
Super Computing 2009 Bandwidth Challenge Winner(“Overall” and “Manifold-Process Implementation” Categories)
18 November 2009
TeraFlowLayer 2
TeraFlowLayer 2
NLRLayer 3(IPv6)
NLRLayer 3(IPv6)
InfiniBandSwitchFabric
InfiniBandSwitchFabric
NRL Data
Center
NRLWashington, DC
LAC/UICChicago, IL
NRL/SC09Portland, OR
WAN Gateway
WAN Gateway
Workstation
10-GigEMonitor
InfiniBand4x SDR
IPv4oE
IPv4oE
IPv4oE
10-G
igE
IBoIPv6oE IBoIPv6oE
Infin
iBan
d
4x S
DR
InfiniBand4x DDR
10-GigE 10-GigE
10-
Gig
E
IBoIPv6oE
UDToIPv6oE
UDToE
LD HPC Testbed
Open CloudTestbedLayer 3 (IPv4)
Open CloudTestbedLayer 3 (IPv4)
InfiniBand
4x SDR
InfiniBand
4x DDR
Data Center
WAN Gateway
WAN Gateway
IBoIPv6oE
WAN Gateway
WAN Gateway
WAN Gateway
WAN Gateway
UDToIPv6oE
UDToE
Infin
iBan
d
4x S
DR
LAC/SC09Portland, OR
1 PB Disk
Array
Data Center
Workstation
• IBoIPV6oE, switched and routed including RDMA
• Displayed high-res (2560x1600x70fps video stream in Portland from Washington
• Rendered images in Portland, processed in DC, then retransmitted back to Portland
• Adjusted BW in presence of congestion and loss to achieve max rates
2424
Dr. Hank Dardy, 1943-2010Dr. Hank Dardy, 1943-2010
2525
Questions?
2626
LD JCTD Concept of OperationsLD JCTD Concept of Operations
Moving Data to Users
Moving Data to Users
2727
Encryptor
StorageController
LAN GW LustreFile System
Web Servers
Application Servers
WorkstationsWorkstations
Core Domain
Client Domain(RDMA/IB)
InfiniBandSwitch
Client Domain(IP/Eth)
InfiniBand
Encryptor
WAN GW
WAN GW
RemoteSystems
RemoteSystems
WAN Domain
Open StandardsOpen SystemsOpen SourceCommodity Parts
Disk Drives
InfiniBandSwitch
DISN
Scalable to 1000’s of Nodes — Petabyte File System today…Exabytes tomorrow
Large Data Reference ModelLarge Data Reference Model
2828
Warfighter ProblemWarfighter Problem
UUVs
Sensor Data Volume
Sensor Data Volume
2000 Today 2010 2015 & Beyond
PREDATOR UAV VIDEO
GLOBAL HAWK DATAGLOBAL HAWK DATA
Future Sensor XFuture Sensor X
Future Sensor YFuture Sensor Y
Larg
e Data JC
TD
Future Sensor ZFuture Sensor Z
1018
1012
Exabytes
Terabytes
1015
Petabytes
FIRESCOUT VTUAV DATAFIRESCOUT VTUAV DATA
Notional Gap
Warfighter Problem
GIG Data Capacity (Services, Transport & Storage)
2929
SummarySummary
▪ LD underpins net-centric warfighting by providing a data-centric DoD information enterprise
▫ LD seeded in key programs
▫ Next generation performance (scalable to exabytes) in smaller footprint at lower cost
▪ Working with Transition Partners to ensure integrated enterprise implementation
3030
LD JCTD – What Does It Mean?LD JCTD – What Does It Mean?
Standard DVD = 4.7 GB 1 Hr of WAOS* raw data = 1 TB
Shared File Systems and Distributed SAN structure – no “FTP” latency• DVD• WAOS
Nearly instant ≤ 5 seconds effective remote access time
▪ 5 Mbps (Home Connection)▫ DVD▫ WAOS
▪ OC-3 (155 Mbps – decent business connection)▫ DVD▫ WAOS
▪ 1 Gbps (Standard GIG/DISN LAN)▫ DVD▫ WAOS
▪ 10 Gbps (OC-192 – 8 Gbps effective)▫ DVD▫ WAOS
2.1 Hrs 18.5 Days
4 Min14 Hrs
37.6 Seconds2.2 Hrs
4.7 Seconds16.7 Min…still too slow…
* Wide Area Optical Surveillance
Large Data JCTDGlobal Access, Global Visualization
Large Data JCTDGlobal Access, Global Visualization
TMJim Hofmann202.404.3132
OEFritz Schultz703.697.3443
OMRandy Heth
XMMike O’Brien703.735.2721
XMMike Laurine703.882.1358