das 1-4: experiences with the distributed asci supercomputersmotivation for das-1 cs/asci needs its...
TRANSCRIPT
![Page 1: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/1.jpg)
DAS 1-4: Experiences with the
Distributed ASCI Supercomputers
Henri Bal [email protected]
Kees Verstoep [email protected]
Vrije Universiteit Amsterdam
![Page 2: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/2.jpg)
Introduction
● DAS: shared distributed infrastructure for experimental computer science research ● Controlled experiments for CS, not production
● 16 years experience with funding & research ● Huge impact on Dutch CS
DAS-1 DAS-2 DAS-3 DAS-4
![Page 3: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/3.jpg)
Overview
● Historical overview of the DAS systems ● How to organize a national CS testbed ● Examples of research done on DAS at the VU
● Ask Adam Belloum for an UvA perspective ☺
![Page 4: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/4.jpg)
Outline ● DAS (pre-)history ● DAS organization ● DAS-1 – DAS-4 systems ● DAS-1 research ● DAS-2 research ● DAS-3 research ● DAS-4 research ● DAS conclusions
![Page 5: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/5.jpg)
Historical perspective
1980 1990 2000 2010
Collections of workstations (COWs) Networks of workstations (NOWs)
Processor pools, !Clusters
Metacomputing paper
Flocking Condors Grid blueprint book
Grids e-Science
Optical networks
Clouds Accelerators
Green IT Big Data
Mainframes
DAS Pre-DAS
Beowulf cluster Minicomputers
![Page 6: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/6.jpg)
VU (pre-)history
● Andy Tanenbaum already built a cluster around 1984 ● Pronet token ring network ● 8086 CPUs ● (no pictures available)
● He built several Amoeba processor pools ● MC68000, MC68020, MicroSparc ● VME bus, 10 Mb/s Ethernet, Myrinet, ATM
![Page 7: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/7.jpg)
Amoeba processor pool (Zoo, 1994)
![Page 8: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/8.jpg)
DAS-1 background: ASCI
● Research schools (Dutch product from 1990s) ● Stimulate top research & collaboration ● Organize Ph.D. education
● ASCI (1995): ● Advanced School for Computing and Imaging ● About 100 staff & 100 Ph.D. students from TU Delft,
Vrije Universiteit, U. of Amsterdam, Leiden, Utrecht, TU Eindhoven, TU Twente, …
![Page 9: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/9.jpg)
Motivation for DAS-1 ● CS/ASCI needs its own infrastructure for
● Systems research and experimentation ● Distributed experiments ● Doing many small, interactive experiments
● Need distributed experimental system, rather than centralized production supercomputer
![Page 10: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/10.jpg)
Funding ● DAS proposals written by ASCI committees
● Chaired by Tanenbaum (DAS-1), Bal (DAS 2-5)
● NWO (national science foundation) funding for all 4 DAS systems (100% success rate) ● About 900 K€ funding per system, 300 K€
matching by participants, extra funding from VU ● "Currently preparing for DAS-5
● ASCI committee also acts as steering group
![Page 11: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/11.jpg)
Goals of DAS-1 ● Goals of DAS systems:
● Ease collaboration within ASCI ● Ease software exchange ● Ease systems management ● Ease experimentation
● Want a clean, laboratory-like system ● Keep DAS simple and homogeneous
● Same OS, local network, CPU type everywhere ● Single (replicated) user account file
# [ACM SIGOPS 2000] (paper with 50 authors)
![Page 12: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/12.jpg)
Behind the screens ….
Source: Tanenbaum (ASCI’97 conference)
![Page 13: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/13.jpg)
DAS-1 (1997-2002) A homogeneous wide-area system
VU (128) Amsterdam (24)
Leiden (24) Delft (24)
6 Mb/s ATM
200 MHz Pentium Pro 128 MB memory Myrinet interconnect BSDI ! RedHat Linux Built by Parsytec
![Page 14: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/14.jpg)
DAS-2 (2002-2007) a Computer Science Grid
VU (72) Amsterdam (32)
Leiden (32) Delft (32)
SURFnet 1 Gb/s
Utrecht (32)
two 1 GHz Pentium-3s ≥1 GB memory 20-80 GB disk
Myrinet interconnect Redhat Enterprise Linux Globus 3.2 PBS ! Sun Grid Engine Built by IBM
![Page 15: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/15.jpg)
VU (85)
TU Delft (68) Leiden (32)
UvA/MultimediaN (40/46)
DAS-3 (2007-2010) An optical grid
dual AMD Opterons 4 GB memory 250-1500 GB disk More heterogeneous: 2.2-2.6 GHz
Single/dual core nodes Myrinet-10G (exc. Delft) Gigabit Ethernet Scientific Linux 4 Globus, SGE Built by ClusterVision
SURFnet6
10 Gb/s lambdas
![Page 16: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/16.jpg)
DAS-4 (2011) Testbed for clouds, diversity & Green IT
Dual quad-core Xeon E5620 24-48 GB memory 1-10 TB disk Infiniband + 1Gb/s Ethernet Various accelerators (GPUs, multicores, ….) CentOS Linux 6 Bright Cluster Manager Built by ClusterVision
VU (74)
TU Delft (32) Leiden (16)
UvA/MultimediaN (16/36)
SURFnet6
10 Gb/s lambdas ASTRON (23)
![Page 17: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/17.jpg)
Performance
![Page 18: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/18.jpg)
Impact of DAS
● Major incentive for VL-e " 20 M€ funding ● Virtual Laboratory for e-Science
● Collaboration SURFnet on DAS-3 & DAS-4 ● SURFnet provides dedicated 10 Gb/s light paths
● About 10 Ph.D. theses per year use DAS
● Many other research grants
![Page 19: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/19.jpg)
Outline ● DAS (pre-)history ● DAS organization ● DAS-1 – DAS-4 systems ● DAS-1 research ● DAS-2 research ● DAS-3 research ● DAS-4 research ● DAS conclusions
![Page 20: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/20.jpg)
DAS research agenda
• Pre-DAS: Cluster computing
• DAS-1: Wide-area computing
• DAS-2: Grids & P2P computing
• DAS-3: e-Science & optical Grids
• DAS-4: Clouds, accelerators & green IT
![Page 21: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/21.jpg)
Overview VU research Algorithms & applications
Programming systems
DAS-1 Albatross Manta MagPIe
DAS-2 Search algorithms Awari
Satin
DAS-3 StarPlane Model checking
Ibis
DAS-4 Multimedia analysis Semantic web
Ibis
![Page 22: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/22.jpg)
DAS-1 research
● Albatross: optimize wide-area algorithms ● Manta: fast parallel Java ● MagPIe: fast wide-area collective operations
![Page 23: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/23.jpg)
Albatross project
● Study algorithms and applications for wide-area parallel systems
● Basic assumption: wide-area system is hierarchical ● Connect clusters, not individual workstations
● General approach ● Optimize applications to exploit hierarchical
structure " most communication is local # [HPCA 1999]
![Page 24: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/24.jpg)
Wide-area algorithms ● Discovered numerous optimizations that
reduce wide-area overhead ● Caching, load balancing, message combining …
● Performance comparison between ● 1 small (15 node) cluster, 1 big (60 node) cluster,
wide-area (4*15 nodes) system
![Page 25: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/25.jpg)
Manta: high-performance Java ● Native compilation (Java " executable) ● Fast RMI protocol
● Compiler-generated serialization routines
● Factor 35 lower latency than JDK RMI ● Used for writing wide-area applications
# [ACM TOPLAS 2001]
![Page 26: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/26.jpg)
MagPIe: wide-area collective communication
● Collective communication among many processors ● e.g., multicast, all-to-all, scatter, gather,
reduction
● MagPIe: MPI’s collective operations optimized for hierarchical wide-area systems
● Transparent to application programmer # [PPoPP’99]
![Page 27: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/27.jpg)
Spanning-tree broadcast
Cluster 1 Cluster 2 Cluster 3 Cluster 4
● MPICH (WAN-unaware) ● Wide-area latency is chained ● Data is sent multiple times over same WAN-link
● MagPIe (WAN-optimized) ● Each sender-receiver path contains ≤1 WAN-link ● No data item travels multiple times to same cluster
![Page 28: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/28.jpg)
DAS-2 research
● Satin: wide-area divide-and-conquer ● Search algorithms ● Solving Awari
![Page 29: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/29.jpg)
Satin: parallel divide-and-conquer
● Divide-and-conquer is inherently hierarchical
● More general than master/worker
● Satin: Cilk-like primitives (spawn/sync) in Java
![Page 30: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/30.jpg)
Satin
● Grid-aware load-balancing [PPoPP’01]
● Supports malleability (nodes joining/leaving) and fault-tolerance (nodes crashing) [IPDPS’05]
● Self-adaptive [PPoPP’07]
● Range of applications (SAT-solver, N-body simulation, raytracing, grammar learning, ….) [ACM TOPLAS 2010]
# Ph.D theses: van Nieuwpoort (2003), Wrzesinska (2007)
![Page 31: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/31.jpg)
Satin on wide-area DAS-2
![Page 32: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/32.jpg)
Search algorithms
● Parallel search: partition search space
● Many applications have transpositions ● Reach same position through different moves
● Transposition table: ● Hash table storing positions that have been
evaluated before ● Shared among all processors
![Page 33: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/33.jpg)
Distributed transposition tables ● Partitioned tables
● 10,000s synchronous messages per second
● Poor performance even on a cluster
● Replicated tables ● Broadcasting doesn’t
scale (many updates)
![Page 34: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/34.jpg)
● Send job asynchronously to owner table entry ● Can be overlapped with computation ● Random (=good) load balancing ● Delayed & combined into fewer large messages
" Bulk transfers are far more efficient on most networks
Transposition Driven Scheduling
![Page 35: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/35.jpg)
Speedups for Rubik’s cube Single Myrinet cluster
TDS on wide-area DAS-2
● Latency-insensitive algorithm works well even on a grid, despite huge amount of communication
# [IEEE TPDS 2002]
![Page 36: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/36.jpg)
Solving awari ● Solved by John Romein [IEEE Computer, Oct. 2003]
● Computed on VU DAS-2 cluster, using similar ideas as TDS
● Determined score for 889,063,398,406 positions
● Game is a draw Andy Tanenbaum: ``You just ruined a perfectly fine 3500 year old game’’
![Page 37: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/37.jpg)
DAS-3 research
● Ibis – [IEEE Computer 2010]
● StarPlane ● Wide-area Awari ● Distributed Model Checking
![Page 38: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/38.jpg)
StarPlane
![Page 39: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/39.jpg)
● Multiple dedicated 10G light paths between sites ● Idea: dynamically change wide-area topology
![Page 40: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/40.jpg)
Wide-area Awari
● Based on retrograde analysis ● Backwards analysis of search space (database)
● Partitions database, like transposition tables ● Random distribution good load balance
● Repeatedly send results to parent nodes ● Asynchronous, combined into bulk transfers
● Extremely communication intensive: ● 1 Pbit of data in 51 hours (on 1 DAS-2 cluster)
![Page 41: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/41.jpg)
Awari on DAS-3 grid ● Implementation on single big cluster
● 144 cores ● Myrinet (MPI)
● Naïve implementation on 3 small clusters ● 144 cores ● Myrinet + 10G light paths (OpenMPI)
![Page 42: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/42.jpg)
Initial insights
● Single-cluster version has high performance, despite high communication rate ● Up to 28 Gb/s cumulative network throughput
● Naïve grid version has flow control problems ● Faster CPUs overwhelm slower CPUs with work ● Unrestricted job queue growth
" Add regular global synchronizations (barriers)
![Page 43: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/43.jpg)
Optimizations
● Scalable barrier synchronization algorithm ● Ring algorithm has too much latency on a grid ● Tree algorithm for barrier&termination detection
● Reduce host overhead ● CPU overhead for MPI message handling/polling
● Optimize grain size per network (LAN vs. WAN) ● Large messages (much combining) have lower host
overhead but higher load-imbalance # [CCGrid 2008]
![Page 44: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/44.jpg)
Performance ● Optimizations improved grid performance by 50% ● Grid version only 15% slower than 1 big cluster
● Despite huge amount of communication (14.8 billion messages for 48-stone database)
![Page 45: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/45.jpg)
From Games to Model Checking ● Distributed model checking has very
similar communication pattern as Awari ● Search huge state spaces, random work distribution, bulk
asynchronous transfers
● Can efficiently run DiVinE model checker on wide-area DAS-3, use up to 1 TB memory [IPDPS’09]
![Page 46: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/46.jpg)
Required wide-area bandwidth
![Page 47: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/47.jpg)
DAS-4 research examples
● Distributed reasoning ● (Jacopo Urbani)
● Multimedia analysis on GPUs ● (Ben van Werkhoven)
![Page 48: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/48.jpg)
WebPIE
● The Semantic Web is a set of technologies to enrich the current Web
● Machines can reason over SW data to find best results to the queries
● WebPIE is a MapReduce reasoner with linear scalability
● It significantly outperforms other approaches by one/two orders of magnitude
![Page 49: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/49.jpg)
Performance previous state-of-the-art
0
5
10
15
20
25
30
35
0 2 4 6 8 10
Thro
ughp
ut (K
trip
les/
sec)
Input size (billion triples)
BigOWLIM
Oracle 11g
DAML DB
BigData
![Page 50: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/50.jpg)
Performance WebPIE
0 100 200 300 400 500 600 700 800 900
1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000
0 10 20 30 40 50 60 70 80 90 100 110
Thro
ughp
ut (K
Trip
les/
sec)
Input size (billion triples)
BigOWLIM
Oracle 11g
DAML DB
BigData
WebPIE (DAS3)
WebPIE (DAS4)
Now we are here (DAS-‐4)!
Our performance at CCGrid 2010 (SCALE Award, DAS-‐3)
![Page 51: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/51.jpg)
Parallel-Horus: User Transparent Parallel Multimedia Computing on GPU Clusters
![Page 52: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/52.jpg)
Naive 2D Convolution Kernel on DAS-4 GPU
![Page 53: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/53.jpg)
Our best performing 2D Convolution Kernel
![Page 54: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/54.jpg)
Conclusions
● Having a dedicated distributed infrastructure for CS enables experiments that are impossible on production systems ● Need long-term organization (like ASCI)
● DAS has had a huge impact on Dutch CS, at moderate cost ● Collaboration around common infrastructure led to
large joint projects (VL-e, SURFnet) & grants
![Page 55: DAS 1-4: Experiences with the Distributed ASCI SupercomputersMotivation for DAS-1 CS/ASCI needs its own infrastructure for Systems research and experimentation Distributed experiments](https://reader033.vdocuments.site/reader033/viewer/2022052816/60a917399498634f274f2020/html5/thumbnails/55.jpg)
Acknowledgements VU Group: Niels Drost
Ceriel Jacobs Roelof Kemp
Timo van Kessel Thilo Kielmann Jason Maassen
Rob van Nieuwpoort Nick Palmer
Kees van Reeuwijk Frank Seinstra Jacopo Urbani Kees Verstoep
Ben van Werkhoven & many others
DAS Steering Group: Lex Wolters Dick Epema
Cees de Laat Frank Seinstra John Romein
Rob van Nieuwpoort
DAS management: Kees Verstoep
DAS grandfathers: Andy Tanenbaum Bob Hertzberger
Henk Sips
Funding: NWO, NCF, SURFnet, VL-e, MultimediaN