LLNL-PRES-806064This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC

github.com/spack/spack

ManagingHPCSoftwareComplexitywithSpack

Supercomputing2017Full-dayTutorialNovember,2017Denver,Colorado

LLNL-PRES-8060642github.com/spack/spack

Materials: Downloadthelatestversionofslidesandhandoutsat:

http://spack.readthedocs.io

Slidesandhands-onscriptsareinthe“Tutorial”Sectionofthedocs.

§ Spack GitHubrepository: http://github.com/spack/spack

§ Spack Documentation: http://spack.readthedocs.io

TutorialMaterials

Tweet at us!@spackpm #SC17

LLNL-PRES-8060643github.com/spack/spack

ToddGamblinGregoryBeckerGregLeeMattLegendre

MassimilianoCulpo

AdamStewart

MarioMelara

TutorialPresenters

LLNL-PRES-8060644github.com/spack/spack

Thecomplexityoftheexascale ecosystemthreatensproductivity.

§ Everyapplicationhasitsownstackofdependencies.§ Developers,users,andfacilitiesdedicate(many)FTEstobuilding&porting.§ Oftentradereuseandusabilityforperformance.

80+ software packagesx5+ target architectures/platforms

Xeon Power KNLNVIDIA ARM Laptops?

x

Up to 7 compilersIntel GCC Clang XL

PGI Cray NAGx

= up to 1,260,000 combinations!

15+ applications

x10+ Programming Models

OpenMPI MPICH MVAPICH OpenMP CUDAOpenACC Dharma Legion RAJA Kokkos

2-3 versions of each package + external dependenciesx

We must make it easier to rely on others’ software!

LLNL-PRES-8060645github.com/spack/spack

§ Someone’shomedirectory?

§ LLNL?LANL?Sandia?ANL?LBL?TACC?— Environmentsatlarge-scalesitesareverydifferent.

§ WhichMPIimplementation?

§ Whichcompiler?

§ Whichdependencies?

§ Whichversionsofdependencies?— Manyapplicationsrequirespecificdependencyversions.

Whatisthe“production”environmentforHPC?

Real answer: there isn’t a single production environment or a standard way to build.Reusing someone else’s software is HARD.

LLNL-PRES-8060646github.com/spack/spack

§ HowtoinstallSpack:

§ Howtoinstallapackage:

§ HDF5anditsdependenciesareinstalledwithintheSpack directory.

§ Unliketypicalpackagemanagers,Spack canalsoinstallmanyvariantsofthesamebuild.— Differentcompilers— DifferentMPIimplementations— Differentbuildoptions

Spack isaflexiblepackagemanagerforHPC

GetSpack!http://github.com/spack/spack

$ git clone https://github.com/spack/spack$ . spack/share/spack/setup-env.sh

$ spack install hdf5

@spackpm

LLNL-PRES-8060647github.com/spack/spack

PeoplewhowanttouseordistributesoftwareforHPC!

1. EndUsersofHPCSoftware— InstallandrunHPCapplicationsandtools

2. HPCApplicationTeams— Managethird-partydependencylibraries

3. PackageDevelopers— Peoplewhowanttopackagetheirownsoftwarefordistribution

4. UsersupportteamsatHPCCenters— PeoplewhodeploysoftwareforusersatlargeHPCsites

WhocanuseSpack?

LLNL-PRES-8060648github.com/spack/spack

Spack isusedworldwide

Sessions at spack.readthedocs.io by location, since April 2017

LLNL-PRES-8060649github.com/spack/spack

§ InNovember2015,LLNLprovidedmostofthecontributionstoSpack

§ Sincethen,we’vegonefrom300toover2,100packages

§ Mostpackagesarefromexternalcontributors!

§ Manycontributionsincore,aswell.

§ WearecommittedtosustainingSpack’s opensourceecosystem!

ContributionstoSpack aregrowing!

LLNL-PRES-80606410github.com/spack/spack

§ 30+organizations140+contributorsSharingover1,500packages andgrowing

§ Otherusecases:— ARMusingforentirecompilerregressionsuite.— LIGO collaborationusingfordeployment— Intel usingSpack topackageMLsoftware— NERSC usingSpack onCori:Craysupport.— EPFL (Switzerland)contributingcorefeatures.— Fermi,CERN,BNL:highenergyphysics.— ANL usingSpack onproductionLinuxclusters.— NASA packaginganIcemodelcode.— ORNL workingwithusonSpack forCORAL.— Kitware:corefeatures,ParaView,Qt,UV-CDAT

support

TheSpack communitynowspansDOEandbeyond

3IBM Brand ElementsIBM Impact 2010 DESIGN GUIDELINES

IBM Brand Elements

IBM Logo

IBM has one of the world’s most recognizable logos. Maintaining its integrity is an essential part of protecting the brand. The 8-bar blue logois more than a signature to our ads or a name on our products andservices. To customers, it is a promise that represents trust, value, quality and advanced technology. In a world of confusing, competitiveads and product claims, the IBM logo is uniquely reassuring, conveyingauthenticity, reliability and quality. The IBM logo is present on every formof IBM communication.

Color of the IBM Logo

IBM has one logo, which may appear only in black, white, or IBM Blue.The logo should appear consistently and without alteration, as shownhere. On a white or light background, use the IBM logo in black or IBMBlue. On a black or dark background, use the IBM logo in white or IBMBlue. The color itself is an important symbol of IBM, and should bespecified for printed documents using Pantone 2718 as a spot colorwhenever possible. Use of this spot color is necessary when setting thelogo in IBM Blue on a colored background.

IBM Blue

PMS 2718CMYK C75 M43 Y0 K0RGB R75 G107 B175WEB 006699

1 white on black or dark background

2 black on white or light background

3 IBM blue on white or light background

4 IBM blue on black or dark background

1 2

3 4

LLNL-PRES-80606411github.com/spack/spack

§ Majornewfeatures:1. Binarypackagingsupport(we’llusethistoday)2. Aninterfaceforpassingbuildinfobetweenpackages(demoedtoday)3. Testdependencies4. Multi-valuedvariants5. Morecontrolovercompilerflags6. BetterhelpoutputforSpack’s manycommands7. Betterbuildoutputhandlinganderrordisplay8. Numerousotherbugfixesandspeedimprovements

§ Over2,100packages

§ Spack nowhasitsownGitHuborganization(https://github.com/spack)

Spack v0.11.0wasjustreleased

LLNL-PRES-80606412github.com/spack/spack

§ Features:— Build/testinfrastructuretobuildandvalidatepackagebuildsregularly— Hostinginfrastructuretomakeselectpackageconfigurationsavialable asbinaries— Supportforcreatingcustomisolated“environments”(cf.Conda,Nix,Virtualenv)— Betterdependencyresolution(concretization)andcompilersupport— Distributed-parallelbuilds

§ Collaborations:— WorkwithUSExascale Programcodeteamstocoordinatesoftwarereleases— WorkwithDOElabstobuildandtestsoftwareonlargesupercomputers— Continueworkingwithbroadercommunitytointegratecontributions

WhatisontheSpack roadmap?

LLNL-PRES-80606413github.com/spack/spack

Spack isnotthefirsttooltoautomatebuilds— Inspiredbycopiouspriorwork

1. “Functional”PackageManagers— Nix https://nixos.org/— GNUGuix https://www.gnu.org/s/guix/

2. Build-from-sourcePackageManagers— Homebrew http://brew.sh— MacPorts https://www.macports.org

OthertoolsintheHPCSpace:

§ Easybuild http://hpcugent.github.io/easybuild/— Aninstallation toolforHPC— FocusedonHPCsystemadministrators– differentpackagemodelfromSpack— Reliesonafixedsoftwarestack– hardertotweakrecipesforexperimentation

§ Hashdist https://hashdist.github.io

RelatedWork

LLNL-PRES-80606414github.com/spack/spack

1. Foreveryone: Welcome&Overview 8:30- 8:50

2. Foreveryone: BasicsofBuildingandLinking 8:50- 9:00

3. Foreveryone: Spack Basics(handson) 9:00- 9:45

4. Foreveryone: CoreSpack Concepts 9:45- 10:00

-- Break-- 10:00- 10:30

5. Foreveryone: Configuration(hands-on) 10:3011:00

6. Forpackagers: MakingyourownPackages(handson) 11:00– 12:00

-- TutorialsLunch(FourSeasonsBallroom)-- 12:00- 1:30

7. Forpackagers: AdvancedPackaging(buildsystems) 1:30– 2:15

8. Forpackagers: AdvancedPackaging(virtualdependencies:BLAS/LAPACK/MPI) 2:15-3:00

-- CoffeeBreak-- 3:30- 3:30

9. ForHPCcenters: DeployingSpack withLmod atHPCFacilities(hands-on) 3:30- 4:15

10. Foreveryone: BuildyourownpackageswithhelpfromtheSpack team! 4:15– 5:00

TutorialOverview(timesareestimates)

LLNL-PRES-80606415github.com/spack/spack

Pleasesubmityourfeedbackonthistutorialat:

Tutorialreviewsareonlinethisyear

http://bit.ly/sc17-evalYour reviews help us keep the tutorial going!

LLNL-PRES-80606416github.com/spack/spack

Building&LinkingBasics

LLNL-PRES-80606417github.com/spack/spack

What’sapackagemanager?

§ Spack isapackagemanager— Doesnot areplaceCmake/Autotools— PackagesbuiltbySpack canhaveany

buildsystemtheywant

§ Spack managesdependencies— Drivespackage-levelbuildsystems— Ensuresconsistentbuilds

§ Determiningmagicconfigurelinestakestime— Spack isacacheofrecipes

• Manages package installation• Manages dependency relationships• Drives package-level build systems

Package Manager

• Cmake, Autotools• Handle library abstractions• Generate Makefiles, etc.

High Level Build

System

• Make, Ninja• Handles dependencies among commands in a single build

Low Level Build

System

LLNL-PRES-80606418github.com/spack/spack

§ Staticlibraries:libfoo.a— .afilesarearchivesof.ofiles(objectfiles)— Linkerincludesneededpartsofastaticlibraryintheoutputexecutable— Noneedtofinddependenciesatruntime– onlyatbuildtime.— Canleadtolargeexecutables— Oftenhardtobuildacompletelystaticexecutableonmodernsystems.

§ Sharedlibraries:libfoo.so (Linux),libfoo.dylib (MacOS)— Morecomplexbuildsemantics,typicallyhandledbythebuildsystem— Mustbefoundbyld.so ordyld (dynamiclinker)andloadedatruntime

• Cancauselotsofheadacheswithmultipleversions— 2mainways:

• LD_LIBRARY_PATH:environmentvariableconfiguredbyuserand/ormodulesystem• RPATH:pathsembeddedinexecutablesandlibraries,sothattheyknowwheretofindtheirowndependencies.

Staticvs.sharedlibraries

LLNL-PRES-80606419github.com/spack/spack

§ API:ApplicationProgrammingInterface— Sourcecodefunctionsandsymbolnamesexposedbyalibrary— IfAPIofadependencyisbackwardcompatible,sourcecodeneednotbechangedtouseit— May needtorecompilecodetouseanewversion.

§ ABI:ApplicationBinaryInterface— Callingconventions,registersemantics,exceptionhandling,etc.— Definedbyhowthecompilerbuildsalibrary

• BinariesgeneratedbydifferentcompilersaretypicallyABI-incompatible.— Mayalsoincludethingslikestandardruntimelibrariesandcompilerintrinsicfunctions— Mayalsoincludevaluesofhard-codedsymbols/constantsinheaders.

§ HPCcode,includingMPI,istypicallyAPI-compatiblebutnotABI-compatible.— Causesmanybuildproblems,especiallyfordynamicallyloadedlibraries— OftenneedtorebuildtogetaroundABIproblems— LeadstocombinatorialbuildsofsoftwareatHPCsites.

APIandABICompatibility

LLNL-PRES-80606420github.com/spack/spack

1. Make(usuallyGNUMake)— https://www.gnu.org/software/make/

2. GNUAutotools— Automake: https://www.gnu.org/software/automake/— Autoconf: https://www.gnu.org/software/autoconf/— Libtool: https://www.gnu.org/software/libtool/

3. CMake:— https://cmake.org

3majorbuildsystemstobeawareof

Spack has built-in support for these plus Waf, Perl, Python, and R

LLNL-PRES-80606421github.com/spack/spack

§ ManyprojectsopttowritetheirownMakefiles.— Canrangefromsimpletovery

complicated

§ Makedeclaressomestandardvariablesforvariouscompilers— ManyHPCprojectsdon’trespectthem— Nostandardinstallprefixconvention— Makefiles maynothaveinstalltarget

§ AutomatingbuildswithMakeusuallyrequireseditingfiles— Typicaltousesed/awk/someother

regularexpressiontoolonMakefile— Canalsousepatches

MakeandGNUMake

<edit Makefile>make PREFIX=/path/to/prefix

Typical build incantation

None. Typically must edit Makefiles.

Configure options

Environment variablesCC CFLAGS LDFLAGSCXX CXXFLAGS LIBSFC FFLAGS CPPF77 FFLAGS

LLNL-PRES-80606422github.com/spack/spack

§ Threepartsofautotools:— autoconf:generatesaportable

configurescripttoinspectbuildhost— automake:high-levelsyntaxfor

generatinglower-levelMakefiles.— libtool:abstractionforsharedlibraries

§ Typicalvariablesaresimilartomake

§ Muchmoreconsistencyamongautotools projects— Wideuseofstandardvariablesand

configureoptions— Standardinstalltarget,staging

conventions.

Autotools

./configure --prefix=/path/to/install_dirmakemake install

Typical build incantation

./configure \--prefix=/path/to/install_dir \--with-package=/path/to/dependency \--enable-foo \--disable-bar

Configure options

Useful environment variablesCC CFLAGS LDFLAGSCXX CXXFLAGS LIBSFC FFLAGS CPPF77 FFLAGS

Environment variables

LLNL-PRES-80606423github.com/spack/spack

§ Gainingpopularity

§ Arguablyeasiertouse(fordevelopers)thanautotools

§ Similarstandardoptionstoautotools— differentvariablenames— Moreconfigurationoptions— Abstractsplatform-specific

detailsofsharedlibraries

§ MostCMake projectsshouldbebuilt“outofsouce”— Separatebuilddirectoryfrom

sourcedirectory

CMake

mkdir BUILD && cd BUILDcmake –DCMAKE_INSTALL_PREFIX=/path/to/install_dir ..makemake install

Typical build incantation

cmake \–D CMAKE_INSTALL_PREFIX=/path/to/install_dir \-D ENABLE_FOO=yes \-D ENABLE_BAR=no \..

Configure options

Useful environment variablesCMAKE_C_COMPILER CMAKE_C_FLAGSCMAKE_CXX_COMPILER CMAKE_CXX_FLAGSCMAKE_Fortran_COMPILER CMAKE_Fortran_FLAGSCMAKE_SHARED_LINKER_FLAGS CMAKE_EXE_LINKER_FLAGSCMAKE_STATIC_LINKER_FLAGS

Common –D options

LLNL-PRES-80606424github.com/spack/spack

Spack Basics

LLNL-PRES-80606425github.com/spack/spack

§ Eachexpressionisaspecforaparticularconfiguration— Eachclauseaddsaconstrainttothespec— Constraintsareoptional– specifyonlywhatyouneed.— Customizeinstallonthecommandline!

§ Specsyntaxisrecursive— Fullcontroloverthecombinatorialbuildspace

Spack providesaspecsyntaxtodescribecustomizedDAGconfigurations

$ spack install mpileaks unconstrained$ spack install mpileaks@3.3 @ custom version$ spack install mpileaks@3.3 %gcc@4.7.3 % custom compiler$ spack install mpileaks@3.3 %gcc@4.7.3 +threads +/- build option$ spack install mpileaks@3.3 cppflags="-O3 –g3" setting compiler flags$ spack install mpileaks@3.3 os=CNL10 target=haswell setting target for X-compile$ spack install mpileaks@3.3 ^mpich@3.2 %gcc@4.9.3 ^ dependency information

LLNL-PRES-80606426github.com/spack/spack

`spack list`showswhatpackagesareavailable

$ spack list==> 303 packages.activeharmony cgal fish gtkplus libgd mesa openmpi py-coverage py-pycparser qt tcladept-utils cgm flex harfbuzz libgpg-error metis openspeedshop py-cython py-pyelftools qthreads texinfoapex cityhash fltk hdf libjpeg-turbo Mitos openssl py-dateutil py-pygments R the_silver_searcherarpack cleverleaf flux hdf5 libjson-c mpc otf py-epydoc py-pylint ravel thriftasciidoc cloog fontconfig hwloc libmng mpe2 otf2 py-funcsigs py-pypar readline tkatk cmake freetype hypre libmonitor mpfr pango py-genders py-pyparsing rose tmuxatlas cmocka gasnet icu libNBC mpibash papi py-gnuplot py-pyqt rsync tmuxinatoratop coreutils gcc icu4c libpciaccess mpich paraver py-h5py py-pyside ruby trilinosautoconf cppcheck gdb ImageMagick libpng mpileaks paraview py-ipython py-pytables SAMRAI uncrustifyautomaded cram gdk-pixbuf isl libsodium mrnet parmetis py-libxml2 py-python-daemon samtools util-linux automake cscope geos jdk libtiff mumps parpack py-lockfile py-pytz scalasca valgrindbear cube gflags jemalloc libtool munge patchelf py-mako py-rpy2 scorep vimbib2xhtml curl ghostscript jpeg libunwind muster pcre py-matplotlib py-scientificpython scotch vtkbinutils czmq git judy libuuid mvapich2 pcre2 py-mock py-scikit-learn scr wgetbison damselfly glib julia libxcb nasm pdt py-mpi4py py-scipy silo wxboost dbus glm launchmon libxml2 ncdu petsc py-mx py-setuptools snappy wxpropgridbowtie2 docbook-xml global lcms libxshmfence ncurses pidx py-mysqldb1 py-shiboken sparsehash xcb-protoboxlib doxygen glog leveldb libxslt netcdf pixman py-nose py-sip spindle xerces-c bzip2 dri2proto glpk libarchive llvm netgauge pkg-config py-numexpr py-six spot xzcairo dtcmp gmp libcerf llvm-lld netlib-blas pmgr_collective py-numpy py-sphinx sqlite yasmcallpath dyninst gmsh libcircle lmdb netlib-lapack postgresql py-pandas py-sympy stat zeromqcblas eigen gnuplot libdrm lmod netlib-scalapack ppl py-pbr py-tappy sundials zlibcbtf elfutils gnutls libdwarf lua nettle protobuf py-periodictable py-twisted swig zshcbtf-argonavis elpa gperf libedit lwgrp ninja py-astropy py-pexpect py-urwid szipcbtf-krell expat gperftools libelf lwm2 ompss py-basemap py-pil py-virtualenv tarcbtf-lanl extrae graphlib libevent matio ompt-openmp py-biopython py-pillow py-yapf taskcereal exuberant-ctags graphviz libffi mbedtls opari2 py-blessings py-pmw python taskdcfitsio fftw gsl libgcrypt memaxes openblas py-cffi py-pychecker qhull tau

§ Spack hasover2,100packagesnow.

LLNL-PRES-80606427github.com/spack/spack

§ Alltheversionscoexist!— Multipleversionsof

samepackageareok.

§ Packagesareinstalledtoautomaticallyfindcorrectdependencies.

§ Binariesworkregardlessofuser’senvironment.

§ Spack alsogeneratesmodulefiles.— Don’thave tousethem.

`spack find`showswhatisinstalled

$ spack find==> 103 installed packages.-- linux-redhat6-x86_64 / gcc@4.4.7 --------------------------------ImageMagick@6.8.9-10 glib@2.42.1 libtiff@4.0.3 pango@1.36.8 qt@4.8.6SAMRAI@3.9.1 graphlib@2.0.0 libtool@2.4.2 parmetis@4.0.3 qt@5.4.0adept-utils@1.0 gtkplus@2.24.25 libxcb@1.11 pixman@0.32.6 ravel@1.0.0atk@2.14.0 harfbuzz@0.9.37 libxml2@2.9.2 py-dateutil@2.4.0 readline@6.3boost@1.55.0 hdf5@1.8.13 llvm@3.0 py-ipython@2.3.1 scotch@6.0.3cairo@1.14.0 icu@54.1 metis@5.1.0 py-nose@1.3.4 starpu@1.1.4callpath@1.0.2 jpeg@9a mpich@3.0.4 py-numpy@1.9.1 stat@2.1.0dyninst@8.1.2 libdwarf@20130729 ncurses@5.9 py-pytz@2014.10 xz@5.2.0dyninst@8.1.2 libelf@0.8.13 ocr@2015-02-16 py-setuptools@11.3.1 zlib@1.2.8fontconfig@2.11.1 libffi@3.1 openssl@1.0.1h py-six@1.9.0freetype@2.5.3 libmng@2.0.2 otf@1.12.5salmon python@2.7.8gdk-pixbuf@2.31.2 libpng@1.6.16 otf2@1.4 qhull@1.0

-- linux-redhat6-x86_64 / gcc@4.8.2 --------------------------------adept-utils@1.0.1 boost@1.55.0 cmake@5.6-special libdwarf@20130729 mpich@3.0.4adept-utils@1.0.1 cmake@5.6 dyninst@8.1.2 libelf@0.8.13 openmpi@1.8.2

-- linux-redhat6-x86_64 / intel@14.0.2 -----------------------------hwloc@1.9 mpich@3.0.4 starpu@1.1.4

-- linux-redhat6-x86_64 / intel@15.0.0 -----------------------------adept-utils@1.0.1 boost@1.55.0 libdwarf@20130729 libelf@0.8.13 mpich@3.0.4

-- linux-redhat6-x86_64 / intel@15.0.1 -----------------------------adept-utils@1.0.1 callpath@1.0.2 libdwarf@20130729 mpich@3.0.4boost@1.55.0 hwloc@1.9 libelf@0.8.13 starpu@1.1.4

LLNL-PRES-80606428github.com/spack/spack

Userscanquerythefulldependencyconfigurationofinstalledpackages.

§ Architecture,compiler,versions,andvariantsmaydifferbetweenbuilds.

$ spack find callpath==> 2 installed packages.-- linux-x86_64 / clang@3.4 ———————— -- linux-x86_64 / gcc@4.9.2 -------------callpath@1.0.2 callpath@1.0.2

Expand dependencies with spack find -d

$ spack find -dl callpath==> 2 installed packages.-- linux-x86_64 / clang@3.4 ----------- -- linux-x86_64 / gcc@4.9.2 -----------xv2clz2 callpath@1.0.2 udltshs callpath@1.0.2ckjazss ^adept-utils@1.0.1 rfsu7fb ^adept-utils@1.0.13ws43m4 ^boost@1.59.0 ybet64y ^boost@1.55.0ft7znm6 ^mpich@3.1.4 aa4ar6i ^mpich@3.1.4qqnuet3 ^dyninst@8.2.1 tmnnge5 ^dyninst@8.2.13ws43m4 ^boost@1.59.0 ybet64y ^boost@1.55.0g65rdud ^libdwarf@20130729 g2mxrl2 ^libdwarf@20130729cj5p5fk ^libelf@0.8.13 ynpai3j ^libelf@0.8.13cj5p5fk ^libelf@0.8.13 ynpai3j ^libelf@0.8.13g65rdud ^libdwarf@20130729 g2mxrl2 ^libdwarf@20130729cj5p5fk ^libelf@0.8.13 ynpai3j ^libelf@0.8.13cj5p5fk ^libelf@0.8.13 ynpai3j ^libelf@0.8.13ft7znm6 ^mpich@3.1.4 aa4ar6i ^mpich@3.1.4

$ spack find -dl callpath==> 2 installed packages.-- linux-x86_64 / clang@3.4 ----------- -- linux-x86_64 / gcc@4.9.2 -----------xv2clz2 callpath@1.0.2 udltshs callpath@1.0.2ckjazss ^adept-utils@1.0.1 rfsu7fb ^adept-utils@1.0.13ws43m4 ^boost@1.59.0 ybet64y ^boost@1.55.0ft7znm6 ^mpich@3.1.4 aa4ar6i ^mpich@3.1.4qqnuet3 ^dyninst@8.2.1 tmnnge5 ^dyninst@8.2.13ws43m4 ^boost@1.59.0 ybet64y ^boost@1.55.0g65rdud ^libdwarf@20130729 g2mxrl2 ^libdwarf@20130729cj5p5fk ^libelf@0.8.13 ynpai3j ^libelf@0.8.13cj5p5fk ^libelf@0.8.13 ynpai3j ^libelf@0.8.13g65rdud ^libdwarf@20130729 g2mxrl2 ^libdwarf@20130729cj5p5fk ^libelf@0.8.13 ynpai3j ^libelf@0.8.13cj5p5fk ^libelf@0.8.13 ynpai3j ^libelf@0.8.13ft7znm6 ^mpich@3.1.4 aa4ar6i ^mpich@3.1.4

LLNL-PRES-80606429github.com/spack/spack

Spack managesinstalledcompilers

§ Compilersareautomaticallydetected— AutomaticdetectiondeterminedbyOS— Linux:PATH— Cray:`moduleavail`

§ Compilerscanbemanuallyadded— IncludingSpack-builtcompilers

compilers:- compiler:

modules: []operating_system: ubuntu14paths:cc: /usr/bin/gcc/4.9.3/gcccxx: /usr/bin/gcc/4.9.3/g++f77: /usr/bin/gcc/4.9.3/gfortranfc: /usr/bin/gcc/4.9.3/gfortran

spec: gcc@4.9.3- compiler:

modules: []operating_system: ubuntu14paths:cc: /usr/bin/clang/6.0/clangcxx: /usr/bin/clang/6.0/clang++f77: nullfc: null

spec: clang@6.0 - compiler:

...

$ spack compilers==> Available compilers-- gcc ----------------------------------gcc@4.2.1 gcc@4.9.3

-- clang --------------------------------clang@6.0

compilers.yaml

LLNL-PRES-80606430github.com/spack/spack

CoreSpack Concepts

LLNL-PRES-80606431github.com/spack/spack

§ Traditionalbinarypackagemanagers— RPM,yum,APT,yast,etc.— Designedtomanageasinglestack.— Installone versionofeachpackageinasingleprefix(/usr).— Seamlessupgradestoastable,welltested stack

§ Portsystems— BSDPorts,portage,Macports,Homebrew,Gentoo,etc.— Minimalsupportforbuildsparameterizedbycompilers,dependencyversions.

§ VirtualMachinesandLinuxContainers(Docker)— Containersallowuserstobuildenvironmentsfordifferentapplications.— Doesnotsolvethebuildproblem(someonehastobuildtheimage)— Performance,security,andupgradeissuespreventwidespreadHPCdeployment.

Mostexistingtoolsdonotsupportcombinatorialversioning

LLNL-PRES-80606432github.com/spack/spack

§ Eachuniquedependencygraphisauniqueconfiguration.

§ Eachconfigurationinstalledinauniquedirectory.— Configurationsofthesamepackagecancoexist.

§ Hash ofentiredirectedacyclicgraph(DAG)isappendedtoeachprefix.

§ Installedpackagesautomaticallyfinddependencies— Spack embedsRPATHsinbinaries.— NoneedtousemodulesorsetLD_LIBRARY_PATH— Thingsworkthewayyoubuiltthem

Spack handlescombinatorialsoftwarecomplexity.

spack/opt/linux-x86_64/

gcc-4.7.2/mpileaks-1.1-0f54bf34cadk/

intel-14.1/hdf5-1.8.15-lkf14aq3nqiz/

bgq/xl-12.1/

hdf5-1-8.16-fqb3a15abrwx/...

mpileaks

mpi

callpath dyninst

libdwarf

libelf

InstallationLayout

DependencyDAG

Hash

LLNL-PRES-80606433github.com/spack/spack

mpileaks

mpi

callpath dyninst

libdwarf

libelf

§ Spack ensuresoneconfigurationofeachlibraryperDAG— EnsuresABIconsistency.— UserdoesnotneedtoknowDAGstructure;onlythedependencynames.

§ Spack canensurethatbuildsusethesamecompiler,oryoucanmix— WorkingonensuringABIcompatibilitywhencompilersaremixed.

Spack Specscanconstrainversionsofdependencies

$ spack install mpileaks %intel@12.1 ^libelf@0.8.12

LLNL-PRES-80606434github.com/spack/spack

Spack handlesABI-incompatible,versionedinterfaceslikeMPI

$ spack install mpileaks ^mvapich@1.9 $ spack install mpileaks ^openmpi@1.4:

$ spack install mpileaks ^mpi@2

mpileaks

mpi

callpath dyninst

libdwarf

libelf

§ mpi isavirtualdependency

§ InstallthesamepackagebuiltwithtwodifferentMPIimplementations:

§ LetSpack chooseMPIimplementation,aslongasitprovidesMPI2interface:

LLNL-PRES-80606435github.com/spack/spack

Concretizationfillsinmissingconfigurationdetailswhentheuserisnotexplicit.

mpileaks ^callpath@1.0+debug ^libelf@0.8.11 User input: abstract spec with some constraints

Concrete spec is fully constrainedand can be passed to install.

mpileaks@2.3%gcc@4.7.3=linux-ppc64

mpich@3.0.4%gcc@4.7.3=linux-ppc64

callpath@1.0%gcc@a4.7.3+debug

=linux-ppc64

dyninst@8.1.2%gcc@4.7.3=linux-ppc64

libelf@0.8.11%gcc@4.7.3=linux-ppc64

libdwarf@20130729%gcc@4.7.3=linux-ppc64

Abstract, normalized specwith some dependencies.

mpileaks

mpi

callpath@1.0+debug

dyninst

libelf@0.8.11

libdwarf

Norm

alize

Concretize Store

spec:- mpileaks:

arch: linux-x86_64compiler:name: gccversion: 4.9.2

dependencies:adept-utils: kszrtkpbzac3ss2ixcjkcorlaybnptp4callpath: bah5f4h4d2n47mgycej2mtrnrivvxy77mpich: aa4ar6ifj23yijqmdabeakpejcli72t3

hash: 33hjjhxi7p6gyzn5ptgyes7sghyprujhvariants: {} version: '1.0'

- adept-utils:arch: linux-x86_64compiler:name: gccversion: 4.9.2

dependencies:boost: teesjv7ehpe5ksspjim5dk43a7qnowlqmpich: aa4ar6ifj23yijqmdabeakpejcli72t3

hash: kszrtkpbzac3ss2ixcjkcorlaybnptp4variants: {} version: 1.0.1

- boost:arch: linux-x86_64compiler:name: gccversion: 4.9.2

dependencies: {} hash: teesjv7ehpe5ksspjim5dk43a7qnowlqvariants: {} version: 1.59.0

...

spec.yaml

Detailed provenance is storedwith the installed package

LLNL-PRES-80606436github.com/spack/spack

Use`spack spec`toseetheresultsofconcretization

$ spack spec mpileaksInput spec------------------------------

mpileaks

Normalized------------------------------

mpileaks^adept-utils

^boost@1.42:^mpi

^callpath^dyninst

^libdwarf^libelf

Concretized------------------------------

mpileaks@1.0%gcc@5.3.0 arch=darwin-elcapitan-x86_64^adept-utils@1.0.1%gcc@5.3.0 arch=darwin-elcapitan-x86_64

^boost@1.61.0%gcc@5.3.0+atomic+chrono+date_time~debug+filesystem~graph~icu_support+iostreams+locale+log+math~mpi+multithreaded+program_options~python+random +regex+serialization+shared+signals+singlethreaded+system+test+thread+timer+wave arch=darwin-elcapitan-x86_64

^bzip2@1.0.6%gcc@5.3.0 arch=darwin-elcapitan-x86_64^zlib@1.2.8%gcc@5.3.0 arch=darwin-elcapitan-x86_64

^openmpi@2.0.0%gcc@5.3.0~mxm~pmi~psm~psm2~slurm~sqlite3~thread_multiple~tm~verbs+vt arch=darwin-elcapitan-x86_64^hwloc@1.11.3%gcc@5.3.0 arch=darwin-elcapitan-x86_64

^libpciaccess@0.13.4%gcc@5.3.0 arch=darwin-elcapitan-x86_64^libtool@2.4.6%gcc@5.3.0 arch=darwin-elcapitan-x86_64

^m4@1.4.17%gcc@5.3.0+sigsegv arch=darwin-elcapitan-x86_64^libsigsegv@2.10%gcc@5.3.0 arch=darwin-elcapitan-x86_64

^callpath@1.0.2%gcc@5.3.0 arch=darwin-elcapitan-x86_64^dyninst@9.2.0%gcc@5.3.0~stat_dysect arch=darwin-elcapitan-x86_64

^libdwarf@20160507%gcc@5.3.0 arch=darwin-elcapitan-x86_64^libelf@0.8.13%gcc@5.3.0 arch=darwin-elcapitan-x86_64

LLNL-PRES-80606437github.com/spack/spack

§ Spack installspackagesinownprefix

§ Somepackagesneedtobeinstalledwithindirectorystructureofotherpackages— i.e.,Pythonmodulesinstalledin$prefix/lib/python-<version>/site-packages— Spack supportsthisviaextensions

ExtensionsandPythonSupport

class PyNumpy(Package):"""NumPy is the fundamental package for scientific computing with Python."""

homepage = "https://numpy.org"url = "https://pypi.python.org/packages/source/n/numpy/numpy-1.9.1.tar.gz"version('1.9.1', ' 78842b73560ec378142665e712ae4ad9’)

extends('python')

def install(self, spec, prefix):setup_py("install“, "--prefix={0}".format(prefix))

LLNL-PRES-80606438github.com/spack/spack

Spack extensions

spack/opt/linux-redhat6-x86_64/

gcc-4.7.2/python-2.7.12-/

python/py-numpy

py-numpy-1.10.4-oaxix36/py-numpy/

...

§ SymboliclinktoSpack installlocation

§ Automaticallyactivateforcorrectversionofdependency— ProvenanceinformationfromDAG— Activatealldependenciesthatareextensionsaswell

spack/opt/linux-redhat6-x86_64/

gcc-4.7.2/python-2.7.12-6y6vvaw/

python/py-numpy-1.10.4-oaxix36/

py-numpy/...

$ spack activate py-numpy @1.10.4

§ Examplesofextensionpackages:— pythonlibrariesareagoodexample— R,Lua,perl— Needtomaintaincombinatorialversioning

LLNL-PRES-80606439github.com/spack/spack

Extensionsmustbeactivatedintoextendee

$ spack extensions python==> python@2.7.12% gcc@4.9.3 ~tk~ucs4 arch=linux-redhat7-x86_64-i25k4oi==> 118 extensions:…py-autopep8 py-docutils py-mako py-numpy py-py2neo …

==> 3 installed:-- linux-redhat7-x86_64 / gcc@4.9.3 -----------------------------py-nose@1.3.7 py-numpy@1.11.0 py-setuptools@20.7.0

==> None activated.

$ spack load python $ python Python 2.7.12 (default, Aug 26 2016, 15:12:42)[GCC 4.9.3] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> import numpyTraceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named numpy>>>

$ spack activate py-numpy==> Activated extension py-numpy@1.11.0% gcc@4.9.3 +blas+lapackarch=linux-redhat7-x86_64-77im5ny for python@2.7.12 ~tk~ucs4 % gcc@4.9.3$ spack extensions python==> python@2.7.12% gcc@4.9.3 ~tk~ucs4 arch=linux-redhat7-x86_64-i25k4oi…==> 1 currently activated:-- linux-redhat7-x86_64 / gcc@4.9.3 -----------------------------

$ python Python 2.7.12 (default, Aug 26 2016, 15:12:42)[GCC 4.9.3] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> import numpy>>> x = numpy.zeros(10)>>>

§ Pythonunawareofnumpy installation

▪ activate symlinks entirenumpy prefixintoPythoninstallation▪ Canalternativelyloadextension

$ spack load python$ spack load py-numpy

LLNL-PRES-80606440github.com/spack/spack

mpileaks@2.3gcc@4.7.3

arch=linux-redhat6-ppc64

callpath@1.0gcc@4.7.3

arch=linux-redhat6-ppc64+debug

openmpi@2.0.0gcc@4.7.3

arch=linux-redhat6-ppc64

dyninst@8.1.2gcc@4.7.3

arch=linux-redhat6-ppc64

hwloc@1.11.3gcc@4.7.3

arch=linux-redhat6-ppc64

libpciaccess@0.13.4gcc@4.7.3

arch=linux-redhat6-ppc64

libtool@2.4.6gcc@4.7.3

arch=linux-redhat6-ppc64

m4@1.4.17gcc@4.7.3

arch=linux-redhat6-ppc64

libsigsegv@2.10gcc@4.7.3

arch=linux-redhat6-ppc64

libelf@0.8.11gcc@4.7.3

arch=linux-redhat6-ppc64

libdwarf@20130729gcc@4.7.3

arch=linux-redhat6-ppc64

Buildingagainstexternallyinstalledsoftware

mpileaks@2.3gcc@4.7.3

arch=linux-redhat6-ppc64

callpath@1.0gcc@4.7.3

arch=linux-redhat6-ppc64+debug

openmpi@2.0.0gcc@4.7.3

arch=linux-redhat6-ppc64

dyninst@8.1.2gcc@4.7.3

arch=linux-redhat6-ppc64

libelf@0.8.11gcc@4.7.3

arch=linux-redhat6-ppc64

libdwarf@20130729gcc@4.7.3

arch=linux-redhat6-ppc64

/path/to/external/gcc/openmpi-2.0.0

packages:mpi:buildable: False

openmpi:buildable: Falsepaths:openmpi@2.0.0 %gcc@4.7.3 arch=linux-redhat6-ppc64:/path/to/external/gcc/openmpi-2.0.0

openmpi@1.10.3 %gcc@4.7.3 arch=linux-redhat6-ppc64:/path/to/external/gcc/openmpi-1.10.3

openmpi@2.0.0 %intel@16.0.0 arch=linux-redhat6-ppc64:/path/to/external/intel/openmpi-2.0.0

openmpi@1.10.3 %intel@16.0.0 arch=linux-redhat6-ppc64:/path/to/external/intel/openmpi-1.10.3

...

packages.yaml

A user registers external packages with Spack.

If a node in the DAG matches an registered external package, Spack prunes the DAG at that node and replaces the node with a reference to the external package.

mpileaks ^callpath@1.0+debug ^openmpi ^libelf@0.8.11

LLNL-PRES-80606441github.com/spack/spack

$ spack repo create /path/to/my_repo$ spack repo add my_repo$ spack repo list==> 2 package repositories.my_repo /path/to/my_repobuiltin spack/var/spack/repos/builtin

§ Somepackagescannotbereleasedpublicly

§ Someusershaveusecasesthatrequirebizarre custombuilds

§ PackagingissuesshouldnotpreventusersfromupdatingSpack— Solution:separaterepositories— Arepositoryissimplyadirectoryofpackagefiles

§ Spack supportsexternalrepositoriesthatcanbelayeredontopofthebuilt-inpackages

§ Custompackagescandependonbuilt-inpackages(orpackagesinotherrepositories)

Spack packagerepositories

my_repo var/spack/repos/builtin

Proprietary packages

Pathological build cases

“Standard” packages

LLNL-PRES-80606442github.com/spack/spack

§ UserCache— Userscreatea

“mirror”oftararchivesforpackages

— Removeinternetdependence

Fetchingsourcecodeforspack

Concretize

spack install mpileaks

Recursively install dependencies

Fetch package source

Build software

Load package from repository

User Cache

Spack Cache

The Internet§ Spack Cache

— Spack automaticallycachestararchivesforpreviouslyinstalledsoftware

§ TheInternet— Spack packagescanfindsourcefilesonline

LLNL-PRES-80606443github.com/spack/spack

Addingcustomcompilerflags

$ spack install hdf5 cflags='-O3 –g –fast –fpack-struct'

§ ThisinstallsHDF5withthespecifiedflags— FlagsareinjectedviaSpack’s compilerwrappers(discussedlater).

§ Flagsarepropagatedtodependenciesautomatically— FlagsareincludedintheDAGhash— Eachbuildwithdifferentflagsisconsideredadifferentversion

§ Thisprovidesaneasyharnessfordoingparameterstudiesfortuningcodes— Previouslyworkingwithlargecodeswasverytedious.

§ Supportscflags,cxxflags,fflags,cppflags,ldflags,andldlibs— AddedfromCLIorconfig file

LLNL-PRES-80606444github.com/spack/spack

Hands-onTime:Configuration

Follow script at http://spack.rtfd.ioUnder “Tutorial: Spack 101"

LLNL-PRES-80606445github.com/spack/spack

MakingyourownSpack Packages

LLNL-PRES-80606446github.com/spack/spack

CreatingyourownSpackPackages

from spack import *

class Zlib(Package):"""A free, general-purpose, legally unencumbered lossless

data-compression library."""

homepage = "http://zlib.net"url = "http://zlib.net/zlib-1.2.8.tar.gz"

version('1.2.8', '44d667c142d7cda120332623eab69f40')

depends_on(‘cmake’, type=‘build’)

def install(self, spec, prefix):configure('--prefix={0}'.format(prefix))

make()make('install')

▪ Packageisarecipeforbuilding

▪ EachpackageisaPythonclass— Downloadinformation— Versions/Checksums— Buildoptions— Dependencies— Buildinstructions

▪ Packagecollectionsarerepos— Spackhasa“builtin”repoin

var/spack/repos/builtin

$REPO/packages/zlib/package.py

LLNL-PRES-80606447github.com/spack/spack

Spack packagesaretemplates forbuilds

def install(self, spec, prefix):config_opts=[‘--prefix=‘+prefix]

if '~shared' in self.spec:config_opts.append('--disable-shared')

else:config_opts.append('--enable-shared')

configure(config_opts)make()make('install')

▪ Eachpackagehasoneclass— zlib forIntelcompilerandzlib forGCCcompilerare

builtwiththesamerecipe.

▪ Canaddconditionallogicusingspecsyntax— Thinkofpackageastranslating aconcreteDAGto

buildinstructions.— Dependenciesarealreadybuilt— Nosearchingortesting;justdowhattheDAGsays

▪ Compilerwrappershandlemanydetailsautomatically.— Spack feedscompilerwrappersto(cc,c++,f90,…)

toautoconf,cmake,gmake,…— Wrappersselectcompilers,dependencies,and

optionsunderthehood.

package.py

LLNL-PRES-80606448github.com/spack/spack

Spackbuildseachpackageinitsowncompilationenvironment

SpackProcess

Setupenvironment

CC = spack/env/spack-cc SPACK_CC = /opt/ic-15.1/bin/iccCXX = spack/env/spack-c++ SPACK_CXX = /opt/ic-15.1/bin/icpcF77 = spack/env/spack-f77 SPACK_F77 = /opt/ic-15.1/bin/ifortFC = spack/env/spack-f90 SPACK_FC = /opt/ic-15.1/bin/ifort

PKG_CONFIG_PATH = ... PATH = spack/env:$PATHCMAKE_PREFIX_PATH = ...LIBRARY_PATH = ...

do_install()

Install dep1 Install dep2 Install package…

BuildProcess

Fork

install() configure make makeinstall

-I /dep1-prefix/include-L /dep1-prefix/lib-Wl,-rpath=/dep1-prefix/lib

Compilerwrappers(spack-cc, spack-c++, spack-f77, spack-f90)

icc icpc ifort

▪ Forkedbuildprocessisolatesenvironmentforeachbuild.Usescompilerwrappersto:— Addinclude,lib,andRPATHflags— Ensurethatdependenciesarefoundautomatically— LoadCraymodules(userightcompiler/systemdeps)

LLNL-PRES-80606449github.com/spack/spack

WritingPackages- VersionsandURLs

class Mvapich2(Package):homepage = "http://mvapich.cse.ohio-state.edu/"url = "http://mvapich.cse.ohio-state.edu/download/mvapich/mv2/mvapich2-2.2rc2.tar.gz"

version('2.2rc2', 'f9082ffc3b853ad1b908cf7f169aa878')version('2.2b', '5651e8b7a72d7c77ca68da48f3a5d108')version('2.2a', 'b8ceb4fc5f5a97add9b3ff1b9cbe39d2')version('2.1', '0095ceecb19bbb7fb262131cb9c2cdd6')

▪ PackagedownloadsarehashedwithMD5bydefault— AlsosupportsSHA-1,SHA-256,SHA-512— We’llbeswitchingtoSHA-256orhighersoon.

▪ DownloadURLscanbeautomaticallyextrapolatedfromURL.— ExtraoptionscanbeprovidedifSpack can’textrapolateURLs

▪ OptionscanalsobeprovidedtofetchfromVCSrepositories

$REPO/packages/mvapich/package.py

LLNL-PRES-80606450github.com/spack/spack

WritingPackages– VariantsandDependencies

class Petsc(Package):variant('mpi', default=True, description='Activates MPI support')variant('complex', default=False, description='Build with complex numbers')variant('hdf5', default=True, description='Activates support for HDF5 (only parallel)')

depends_on('blas')depends_on('python@2.6:2.7')

depends_on('mpi', when='+mpi')

▪ Variantsarenamed,havedefaultvaluesandhelptext

▪ Otherpackagescanbedependencies— whenclauseprovidesconditionaldependencies— Candependonspecificversionsorothervariants

$REPO/packages/petsc/package.py

LLNL-PRES-80606451github.com/spack/spack

WritingPackages– BuildRecipes

def install(self, spec, prefix):with working_dir("build", create=True):

cmake("..", *std_cmake_args)make()make("install")

@when('@:8.1')def install(self, spec, prefix):

configure("--prefix=" + prefix)make()make("install")

▪ Functionswrapcommonops— cmake,configure,patch,make,…— Executable andwhichfornewwrappers.

▪ Commandsexecutedincleanenvironment

▪ FullPythonfunctionality— Patchupsourcecode— Makefilesanddirectories— Calculateflags— …

$REPO/packages/dyninst/package.py

LLNL-PRES-80606452github.com/spack/spack

Createnewpackageswithspack create

class Zlib(Package):# FIXME: Add a proper url for your package's homepage here. homepage = "http://www.example.com"url = "http://zlib.net/zlib-1.2.8.tar.gz"version('1.2.8', '44d667c142d7cda120332623eab69f40')

def install(self, spec, prefix):# FIXME: Modify the cmake line to suit your build system here.

▪ spack create<url>willcreateaskeletonforapackage— Spack reasonsaboutURL,hash,version,buildrecipe.— GeneratesboilerplateforCmake,Makefile,autotools,Python,R,Waf,Perl

— Notintendedtocompletelywritethepackage,butgetsyou80%ofthewaythere.

▪ spack edit<package>forsubsequentchanges

$ spack create http://zlib.net/zlib-1.2.8.tar.gz

$REPO/packages/zlib/package.py

LLNL-PRES-80606453github.com/spack/spack

Hands-onTime:CreatingPackages

Follow script at http://spack.rtfd.ioUnder “Tutorial: Spack 101"

LLNL-PRES-80606454github.com/spack/spack

Hands-onTime:AdvancedPackaging

Follow script at http://spack.rtfd.ioUnder “Tutorial: Spack 101"

LLNL-PRES-80606455github.com/spack/spack

Hands-onTime:ModuleFiles

Follow script at http://spack.rtfd.ioUnder “Tutorial: Spack 101"

LLNL-PRES-80606456github.com/spack/spack

§ ContributingpackagestoSpack issimple— Makepackagesonyourownsystem— SendapullrequesttoSpack toletothersusethem— GitHubguidetopullrequests:

https://help.github.com/articles/creating-a-pull-request/— SeecontributorguideintheSpack repository— Spack islicensedunderLGPLv2.1

§ Wewantmorethanjustpackages— Newfeatures— Newdocumentation— Anycontributioncanhelpthecommunity!

§ Spack hasahelpfulonlinecommunity— TypicallyhappytorespondtoGitHubissues— ActivemailinglistonGoogleGroups

JointheSpack community!

@spackpm

github.com/spack/spack

We hope to make distributing & using HPC software easy!