development environments for hpc: the view from...
TRANSCRIPT
![Page 1: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/1.jpg)
Development Environments for HPC: The View from NCSA
Jay Alameda National Center for Supercomputing Applications, University of Illinois at Urbana-Champaign
DEHPC ‘15
San Francisco, CA 18 October 2015
![Page 2: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/2.jpg)
Acknowledgements ª Portions of this material are supported by or based upon work supported by the
Defense Advanced Research Projects Agency (DARPA) under its Agreement No. HR0011-07-9-0002, the United States Department of Energy under Contract No. DE-FG02-06ER25752, the Blue Waters sustained petascale computing project, which is supported by the National Science Foundation under award number OCI 07-25070, and the SI2-SSI Productive and Accessible Development Workbench for HPC Applications, which is supported by the National Science Foundation under award number OCI 1047956
ª The SI2-SSI team is lead by Jay Alameda (NCSA), Greg Watson (IBM), Steven Brandt (LSU), and Allen Malony (U Oregon). Team members and senior personnel include Beth Tibbitts (IBM), Ralph Johnson (U Illinois), Chris Navarro (NCSA), Sameer Shende (U Oregon), Wyatt Spear (U Oregon), Brian Jewett (U Illinois), Galen Arnold (NCSA), and Rui Liu (NCSA)
ª Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation
1
![Page 3: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/3.jpg)
Outline
ª Overview of Eclipse and the Parallel Tools Platform (PTP)
ª Motivation for Workbench for High Performance Computing (WHPC)
ª Advantages of Eclipse for HPC Software Developers
ª Sustainability, Investment, Collaboration: Building Community
ª Challenges with DEHPC
ª Opportunities (for discussion)
2
![Page 4: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/4.jpg)
Eclipse
ª Integrated development environment (IDE) ª Edit code, compile, run, debug without leaving Eclipse ª Graphical user interface
3
![Page 5: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/5.jpg)
Eclipse
4
![Page 6: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/6.jpg)
Eclipse
ª Integrated development environment (IDE) ª Edit code, compile, run, debug without leaving Eclipse ª Graphical user interface
ª Plug-ins add new functionality to Eclipse
ª Support languages: C/C++, Fortran, Python, … ª Support version control systems: Git, Subversion, … ª Support issue tracking systems: Bugzilla, Jira, … ª Support HPC development (?!)
5
![Page 7: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/7.jpg)
Parallel Tools Platform (PTP)
ª PTP: a set of plug-ins that extend Eclipse to support HPC development ª Write code on your laptop
ª Build it on a remote HPC system using its compilers
ª Run it on the remote HPC system (qsub)
ª Debug it on the remote system (MPI debugger)
ª Tune it for performance on the remote system
ª …all inside Eclipse! 6
![Page 8: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/8.jpg)
7
Eclipse Parallel Tools Platform (PTP) Coding & Analysis
Code completion
![Page 9: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/9.jpg)
8
Launching & Monitoring
Eclipse Parallel Tools Platform (PTP) Coding & Analysis
![Page 10: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/10.jpg)
9
Launching & Monitoring
Eclipse Parallel Tools Platform (PTP)
Debugging
Coding & Analysis
![Page 11: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/11.jpg)
10
Launching & Monitoring
Eclipse Parallel Tools Platform (PTP)
Debugging
Coding & Analysis
Performance Tuning
![Page 12: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/12.jpg)
11
Launching & Monitoring
Eclipse Parallel Tools Platform (PTP)
Debugging
Coding & Analysis
Performance Tuning
![Page 13: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/13.jpg)
Motivation for Workbench for High Performance Computing (WHPC)
ª Context ª NSF 1047956: SI2-SSI: A Productive and Accessible
Development Workbench for HPC Applications Using the Eclipse Parallel Tools Platform
ª Stable, portable platform for tool development ª Focus on tool functionality, manage rapid evolution of HPC
platforms ª Encourage consistent tool look and feel ª Support for HPC application development practices
ª Why Parallel Tools Platform? ª High potential to meet needs of a WHPC. ª Target next generation of HPC developers growing up with
IDEs (Eclipse, Visual Studio, ...) ª Need to cultivate community of users!
12
![Page 14: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/14.jpg)
Improvements
ª Work within Eclipse release cycle ª Major (API-breaking) improvements with coordinated
June release ª Last major release Eclipse 4.5 “Mars” released
June 2015 ª Minor enhancements and bug-fixes with two
coordinated service releases in September and February ª Eclipse 4.5 SR1 Released September, 2015.
ª Foci of improvements ª Improve usability ª Improve productivity
13
![Page 15: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/15.jpg)
Consider 2 possible types of users of Eclipse Parallel Tools … ª Science code users/modelers
ª Need to build science code ª May need to modify science code (and rebuild)
ª Software specialists enabling modeling projects ª Lots of software engineering concerns
ª Next set of slides address some of those concerns.
14
![Page 16: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/16.jpg)
Science code users/modelers
ª Some of the challenges ª Complex codes (eg WRF) ª Codes + HPC architectures can be daunting ª Adding user code not always easy
ª WRF: from http://wrf-model.org/PRESENTATIONS/2000_04_18_Klemp/sld007.htm
15
![Page 17: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/17.jpg)
Navigating Computational Science Codes
16
Code navigation
Syntax-aware editing (navigate to program units and declarations)
Code Outline
![Page 18: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/18.jpg)
Eclipse aiding in a typical code workflow…
ª May want to add a model output variable ª Eclipse PTP makes it easy to navigate source,
make changes ª Drive remote builds on HPC resources
ª Make, autotools…
ª Run – can generate a run configuration for particular system, batch environment
17
![Page 19: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/19.jpg)
Software Specialists enabling modeling projects
ª Need a wider array of software engineering tools ª Source repository ª Issue tracking ª Documentation ª Performance tuning…
ª Eclipse Parallel Tools can help with many of these concerns
18
![Page 20: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/20.jpg)
Source Code Control: “Team” Features
ª Eclipse supports integration with multiple version control systems (VCS) ª CVS, SVN, Git, and others ª Collectively known as “Team” services
ª Many features are common across VCS ª Compare/merge ª History ª Check-in/check-out
ª Some differences ª Version numbers ª Branching
19
![Page 21: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/21.jpg)
Issue Tracking
ª Mylyn Bridge ª Tracks tasks, links to source and bug repositories
20
Connections to Jira, bugzilla, …
![Page 22: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/22.jpg)
Eclipse Documentation
ª Eclipse Help System – built in and standalone (http://help.eclipse.org)
21
![Page 23: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/23.jpg)
Performance Tuning: PTP TAU plug-ins http://www.cs.uoregon.edu/research/tau
ª TAU (Tuning and Analysis Utilities) ª First implementation of External Tools Framework (ETFw) ª Eclipse plug-ins wrap TAU functions, make them available
from Eclipse ª Full GUI support for the TAU command line interface ª Performance analysis integrated with development
environment
22
![Page 24: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/24.jpg)
Sustainability, Investment, Collaboration: Building Community
ª Stakeholders/investors ª Funding agencies à innovative capability ª HPC vendors
ª contribute to open source foundational software ª basis for proprietary or open-source add-ons
ª Supercomputing centers ª user support and training ª xml documents for batch system and local policy
integration
23
![Page 25: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/25.jpg)
Sustainability, Investment, Collaboration: Building Community II ª Stakeholders/investors continued
ª Supercomputing integrating organizations (eg, XSEDE): ª user support ª single sign-on support (?) ª training ª education support
ª Tool providers ª xml documents for integration of command line
tools ª extra value-add from tool-specific plugins?
ª Anyone else?
24
![Page 26: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/26.jpg)
Challenges for DEHPC
ª Budgetary pressure on supercomputing centers and integrating organizations
ª Challenges successfully competing proposals in open calls (eg, NSF SI2). ª Innovation, impact are paramount here
ª User community is large, disperse (eg lots of downloads of PTP, hard to connect to user community)
25
![Page 27: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/27.jpg)
Opportunities (for discussion)
ª Can we build a community to support a DEHPC? ª PIs proposing to funding agencies for innovative
additions? ª Vendor contributions to DEHPC? ª Supercomputing centers and integrating
organizations support? ª Eg, User support, training, configuration (xml)
documents
26
![Page 28: Development Environments for HPC: The View from NCSAdehpc15.github.io/proceedings/alameda-slides.pdf · 2015-12-16 · number OCI 07-25070, and the SI2-SSI Productive and Accessible](https://reader034.vdocuments.site/reader034/viewer/2022042403/5f156b03b553b6379838be2c/html5/thumbnails/28.jpg)
Eclipse Parallel Tools Platform
www.eclipse.org/ptp
27