infiniband: today and tomorrow

Download InfiniBand: Today and Tomorrow

Post on 06-May-2015

1.724 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • 1.InfiniBand: Today and Tomorrow Jamie Riotto Sr. Director of Engineering Cisco Systems (formerly Topspin Communications) [email_address]

2. Agenda

  • InfiniBand Today
    • State of the market
    • Cisco and InfiniBand
    • InfiniBand products available now
    • Open source initiatives
  • InfiniBand Tomorrow
    • Scaling InfiniBand
    • Future Issues
  • Q&A

3. InfiniBand Maturity Milestones

  • High adoption rates
    • Currently shipping > 10,000 IB ports / Qtr
  • Cisco acquisition will drive broader market adoption
  • End-to-end price points of 16X PCIe

24. Future InfiniBand Cables

  • InfiniBand over CAT5 / CAT6 / CAT7
    • Shielded cable distances up to ???
    • Leverage existing 10-GigE cabling
    • 10-GigE too expensive?

25. IB Distance Scaling

  • IB Short Haul
    • New Copper drivers
    • 25 50 Meters (KeyEye)
    • 75 - 100 Meters (IEEE 10Ge)
  • IB Wan
    • Same Subnet over distance (300 KM target)
    • Buffer / Credit / Timeout issues
    • Applications: Disaster Recover, Data Mirroring
  • IB Long Haul
    • IB over IP (over SONET?)
    • utilizes existing public plant (WDM, Debugging, etc)

26. Scaling InfiniBand

  • Subnet Management
  • Host-side Drivers
    • MPI
    • IPoIB
    • SRP
  • Memory Utilization

27. IB Subnet Manager

  • Subnets are getting bigger
    • 4,000 -> 10,000 nodes
    • Topology convergence times
      • Topology disturbance times
      • Topology disturbance minimization

28. Subnet Management Challenges

  • Cluster Cold Start times
    • Template Routing
    • Persistent Routing
  • Cluster Topology Change Management
    • Intentional Change - Maintenance
    • Unintentional Change Dealing with Faults
      • How to impact minimum number of connections
      • Predetermine fault reaction strategy?
  • Topology Diagnostic Tools
    • Link/Route Verification
    • Built-in BERT testing
  • Partition Management

29. Multiple Routing Models

  • Minimum Latency Routing:
    • Load-Balanced Shortest-Path Routing
  • Minimum Contention Routing:
    • Lowest-Interference Divergent-Path Routing
  • Template Driven Routing:
    • Supports Pre-Determined Routing Topology
    • For example: Clos Routing, Matrix Row/Column, etc
    • Automatic Cabling Verification for Large Installations

30. IB Routing Challenges

  • Static / Dynamic Routing
    • IB impliments Static Routing through Linear Forwarding Tables at each chip
    • Multi-LID Routing enables Dynamic Routing
  • Credit Loops
  • Cost Base Routing
    • Speed mismatches cause Store & Forward (vs. cut through)
    • SDR DDR QDR
    • 4X 12X
    • Short Haul Long Haul

31. Multi-LID Source-Based Routing Support

  • Applications can implement Dynamic Routing for Contention Avoidance, Failover, Parallel Data Transfer

1 , 2 , 3 ,4 Spine Switches Leaf Switches Leaf Switches 32. New IB Peripherals

  • CPUs?
  • Storage
    • SAN
    • NFS-RDMA
  • Memory (coherent / non-coherent)
  • Purpose built Processors?
    • Floating Point Processors
    • Graphics Processors
    • Pattern Matching Hardware
    • XML Processor

33. THANK YOU!

  • Questions & Answers