March 12, 2003 Networks of Workstations 1 Prabal Dutta

Networks of Workstations

Prabal Duttadutta.4@osu.edu

Electrical Engineering 864Advanced Computer Design

March 12, 2003 Networks of Workstations 2 Prabal Dutta

Introduction

• Much of this class has been about the details of advanced and parallel computing.– Interconnection Networks

– Routing.

– Pipelining.

– Scheduling.

– Algorithms.

• The purpose of this presentation is to look at the trends in parallel computing with a focus on networking…

• And to understand the technical and economic forces driving changes in parallel computing today.

March 12, 2003 Networks of Workstations 3 Prabal Dutta

Taxonomy

• Massively Parallel Processors (MPP)– Built with specialized networks– Specifically intended for use as parallel computers

• Clusters– Clusters: small number of processors/node.– Constellations: large number of processors/node.

• Networks of Workstations (NOW)– Scalable interconnection networks.– Clusters of commodity PCs/workstations and high-performance

network gear.– Beowulf = Linux + Bonded Ethernet or Myrinet.– UCB NOW: HP9000 + FDDI, ATM, or Myrinet.

• Other– Single Instruction Multiple Data: Connection Machine 2– Vector Processors: Cray

• Let’s take a look at some of the once-high-flyers of MPPs…

March 12, 2003 Networks of Workstations 4 Prabal Dutta

Bankrupt, Acquired, Spun Out, Sold Off!

March 12, 2003 Networks of Workstations 5 Prabal Dutta

What Happened?

• In the 1990s, the supercomputing market had completely changed:

"The cold war was over, the commercial world was so competitive, and companies couldn’t afford the machines. Meanwhile, microprocessors became more powerful. It was the beginning of commodity technology instead of proprietary technology. There was the move to vertically integrate everything. We were in a new age. Supercomputers were too expensive. They were not only hard to build, but they took a long time to build.”

- Steve Chen, Ph.D. 1975, UIUC Chief Architect of the Cray X-MP

March 12, 2003 Networks of Workstations 6 Prabal Dutta

Granularity: Computation-Communications Tradeoff

Communications

Com

puta

tions

MPP

Constellations

Clusters

DistributedComputing

DesktopComputing

SMP

$$$

$

$$

$$

WorkstationComputing

Networks of W

orkstations

Networks of W

orkstations

March 12, 2003 Networks of Workstations 7 Prabal Dutta

Top 500 Supercomputer Sites (Metric: GFlops)

Breakdown of Top 500 Supercomputers

0

10

20

30

40

50

60

70

80

1995 1996 1997 1998 1999 2000 2001 2002 2003

Year

Pe

rce

nt

Constellation MPP Cluster SMP

Source data: http://www.top500.org/

March 12, 2003 Networks of Workstations 8 Prabal Dutta

Top 500 Supercomputer Sites (Metric: Processor Arch)

Breakdown of Top 500 Supercomputers

0102030405060708090

100

1995 1996 1997 1998 1999 2000 2001 2002 2003

Year

Pe

rce

nt

Scalar Vector

Source data: http://www.top500.org/

March 12, 2003 Networks of Workstations 9 Prabal Dutta

Why are MPPs Falling Behind?

• Massively Parallel Processors– Cost too much.– Required over-engineering due to high integration.– Focused on niche applications with small markets.– Did not take into account networking commoditization.– Always a year or two behind the current state of the art.

• On the other hand, Networks of Workstations– Were marginally more expensive than workstations.– Leveraged emerging fast, scalable, and switched LANs– Leveraged high-volume, low-cost, super-pipelined

commodity processors.– Extended existing operating systems with global process

management and shared memory/message passing abstractions.

March 12, 2003 Networks of Workstations 10 Prabal Dutta

Emergence of the “Killer” Network

• Massively Parallel Processing took advantage of:– Switched Networks.– Scalable bandwidth.– Low-latency communications.– Low processor overhead.

• These technologies are becoming available in LANs:– Myrinet.

• Used in 140 of the top 500 supercomputers.

– Asynchronous Transfer Mode (ATM).– Fiber Distributed Data Interconnect (FDDI).– Fast Ethernet/Gigabit Ethernet.

March 12, 2003 Networks of Workstations 11 Prabal Dutta

Fast, Scalable, Switched Networks: Myrinet

• Myrinet is based on the networks from MPPs and is used in 140 of the top 500 supercomputers.

• By decoupling network from workstation, allows each to evolve at its own pace.

• General Features– Full-duplex 2x2Gbps data rate (250MBps).– Self-initializing.– Low-latency.– Cut-through crossbar switches.– Direct communications between user processes and network.– Scales to tens of thousands of hosts.– Provide alternative communication paths between hosts. – X-Y Dimension-order routing.– Flow control on every link obviates packet buffering.– Error control and "heartbeat" continuity monitoring every link.– Linux,Windows,Solaris,Mac OSX,True64, FreeBSB,VxWorks

March 12, 2003 Networks of Workstations 12 Prabal Dutta

Fast, Scalable, Switched Networks: Myrinet

• Sustained One-Way Data Rates

March 12, 2003 Networks of Workstations 13 Prabal Dutta

Fast, Scalable, Switched Networks: Myrinet

• Example Myrinet Topologies

March 12, 2003 Networks of Workstations 14 Prabal Dutta

Fast, Scalable, Switched Networks: Myrinet

• PCI Host Interface Card: $995

• Modular Switch Enclosure: $12,800

• 8-port Fiber Switch Line Card: $2,400

Affordable !!!

March 12, 2003 Networks of Workstations 15 Prabal Dutta

The “Killer” Workstation (and Desktop)

• Workstations have become extraordinarily powerful and affordable.

• Example: Intel Pentium 4 - 3.06GHz System: <$1,200

March 12, 2003 Networks of Workstations 16 Prabal Dutta

The I/O Bottleneck

• Processors are getting faster and disks are improving mostly in capacity, not performance.

• So…we’ve reached a point of diminishing returns– More and more time spent waiting for I/O.– Little benefit for the end user.

• Fast networks enable aggregate DRAM as a giant cache for disks.

• Network DRAM access time is 1/10 of local disk.• I/O can be striped across multiple nodes like

RAID.

March 12, 2003 Networks of Workstations 17 Prabal Dutta

Key Lessons

• Never bet against the power of commoditization.• Market forces are unstoppable.• Frequently, we must decide between novel

research or customer needs.• Companies formed to commercialize innovative

research without compelling market economics are doomed– There are plenty of good research problems.– There are plenty of good business opportunities.– There are far fewer good research problems that are

also good business opportunities.

March 12, 2003 Networks of Workstations 18 Prabal Dutta

Closing Thoughts

March 12, 2003 Networks of Workstations 19 Prabal Dutta

References

1. T. Anderson, D. Culler, and D. Patterson, “A Case of NOW (Networks of Workstations),” IEEE Micro, Vol. 15, Issue 1, February 1995.

2. T. Anderson, D. Culler, and D. Patterson, “The Berkeley Networks of Workstations (NOW) Project,” CompCon ’95, March 1995.

3. N. Boden, D. Cohen, R. Feldman, A. Kulawik, C. Seitz, J. Seizovic, and W. Su, “Myrinet: A Gigabit-per-Second Local Area Network,” IEEE Micro, Vol. 15, Issue 1, February 1995.

4. Douglas P. Ghormley, David Petrou, Steven H. Rodrigues, Amin M. Vahdat, and Thomas E. Anderson, “GLUnix: A Global Layer Unix for a Network of Workstations,” Software Practice and Experience, Special Issue on Experience with Distributed Systems, 1998.

5. http://www.myrinet.com

6. http://www.columbusmicro.com/