chameleon: operational lessons › media › filer_public › 9c › 54 › ...we live to serve:...

www. chameleoncloud.org

CHAMELEON: OPERATIONAL LESSONS Kate Keahey, Jason Anderson (ANL, UC)

Paul Ruth (RENCI), Jacob Colleran (UC, ANL), Cody Hammock (TACC), Joe Stubbs (TACC), Zhuo Zhen (UC, ANL)

{keahey, jasonanderson}@uchicago.edu July 29, 2019 HARC Workshop, Chicago, IL


CHAMELEONINANUTSHELL� Weliketochange:testbedthatadaptsitselftoyourexperimentalneeds

�  Deepreconfigurability(baremetal)andisolation(CHI)–butalsoeaseofuse(KVM)�  CHI:poweron/off,reboot,customkernel,serialconsoleaccess,etc.

� Wewanttobeallthingstoallpeople:balancinglarge-scaleanddiverse�  Large-scale:~largehomogenouspartition(~15,000cores),5PBofstoragedistributedover

2sites(now+1!)connectedwith100Gnetwork…�  …anddiverse:ARMs,Atoms,FPGAs,GPUs,Corsaswitches,etc.

�  Cloudoncloud:leveragingmainstreamcloudtechnologies�  PoweredbyOpenStackwithbaremetalreconfiguration(Ironic)+“specialsauce”�  ChameleonteamcontributionrecognizedasofficialOpenStackcomponent

� Welivetoserve:open,productiontestbedforComputerScienceResearch�  Startedin10/2014,testbedavailablesince07/2015,renewedin10/2017�  Currently3,000+users,500+projects,100+institutions


CLOUDSVERSUSHPCRESOURCES�  TraditionalHPCresources:interfaces,complexity,efficiency&cost�  Clouds:interfaces,complexity,efficiency&cost�  Differencesincomplexity:

�  Operationalcomplexity:networking,security,andothers

�  Greatersharingofartifacts:appliancemanagement

�  Relativeimmaturityoftheparadigm

�  Cloudsystemsaremorecomplexbecausetheysolveamorecomplexproblem


EXPERIMENTALINSTRUMENTSVERSUSCLOUDS

Bare-Metal Infrastructures security, fewer layers of abstraction, relative immaturity of infrastructure

Networking Access to L2 for all, complexity/automation, integration with commercial offerings

Chasing the Research Frontier and Adaptation Emphasis on development/adding new features, closer collaboration with user

community


…ANDITNEEDSTOSCALE

# of diverse CS experiments you can run

acce

ssib

ility

Traditional open HPC resources Open cloud

resources

manually configurable closed testbeds

Chameleon


WHATDOESITMEANTOPEOPLE?�  Operators

�  Veryhighlevelofskill:morediverseanddeeperexpertise

�  Significantlearningcurve

�  Teamsofoperatorswithdifferentspecialties

�  Developmentexperienceiscritical

� Moreeffort�  Manymovingparts,immatureparts,newparts,unexpectedparts

�  Closeinteractionwithusercommunity�  Usersareincreasinglylesscustomersandincreasinglymorepartners


HELPINGHUMANSINTHELOOP�  Researchersandinstructors(users)

�  Makeinterfacestocloudmoreintuitive(oratleastsimilartocommercialclouds)�  Facilitatecreationofecosystemforsharingknowledge�  Directinstructionandguidance

�  Hostinstitutions,serviceproviders(operators)�  ReducecostofrunningChameleonaslowaspossible�  Enablepluggingintoexistingecosystems

�  Ourselves�  Enableteammembersofvariableexpertisetobeproductive�  GiveinsightintousageandhealthofChameleon�  Addforce-multiplierstomaketeamhaveoutsizedimpact


MONITORING:THREEPILLARSQuantify�  Symptom-based

metrics�  Prometheus

�  Chameleon-specificmetrics

�  Logindexingandsearch�  Elasticsearch,Fluentd

�  Kolla-Ansible

Detect�  Metric-basedalerts

�  Prometheus,Alertmanager

�  ”Black-box”probes�  Periodicchecksfor

externalconnectivitytopublicAPIs

�  “Smoketests”�  SuiteofJenkinstests,

runnightly(expensive)orhourly(cheap)

�  Checks“happypath”throughsystem

React�  Runbooks

�  Documentationofknownerrorsandmitigations(foroperators)

�  Helpsnewteammembersbeproductive

�  Hammers�  Automatedsolutions

forknownerrors


ISITWORTHTHETIMETOAUTOMATE?


AUTOMATION�  Newappliancereleases

�  No“snowflake”images:expressedincodeandbuiltwithdiskimage-builder

�  Newsystemreleases�  Patchesaretested,builtintoanew(Docker)container,thenpushedtoalocal

registryforrelease

�  Ajobistriggeredtodeploynewcontainerversionusingcontrolledprocess(Kolla-Ansible)

�  Nobodyhastolearnhowtobuild/installpackages!Downside:somebodyshouldknowhowtofixproblemswithpipeline.

� Maintenanceprocesses�  Takingnodeoutofproduction,attachingmetadataforoperators


PACKAGINGCHAMELEON� WhatisCHI-in-a-Box?

�  InstallChameleononyourowninfrastructurewithsetofprovisioningscripts+softwarebundles

Traditional software OpenStack


CHI-IN-A-BOXUSECASES�  ChameleonAssociate

�  ResourcesaddeddirectlytoChameleon,whileretainingprojectidentity

�  Chameleonprovidesusermanagement(andusersupport!),resourcediscoveryandappliancecatalog

�  JointlymaintainedbyChameleonstaffandassociatesitepartnership

�  ChameleonPart-timeAssociate�  Similartoabove,butallresourcesareexpectedtobetakenofflineattimes

�  IndependentTestbed�  AssociatesitedeploysChameleon,butoperatesusermanagementandsupport

themselves.

�  FirstsitealreadydeployedatNU


PACKAGING:OURAPPROACH�  Distributeassetofprovisioningscripts,buildoncommodity

technology�  Kolla,Kolla-Ansible,Ansible

�  Allinfrastructureexpressedasversionedcode.Infrastructurecanbebuiltfromscratchrepeatably(goodfordisasterrecovery.)

�  Provideinstallationandsupportdocumentation�  Installguide,troubleshooting,runbooks

� We“dogfood”CHI-in-a-Boxinternally�  Beingconsumersofourownproductimprovesquality

�  Focusonreducingcouplingbetweensitesforreliability


USERS:THEFINALFRONTIER�  Ticketsvs.supportlists

�  Dedicated,trackablecommunicationversusdiscoverable,noisycommunications

�  Covenantbetweenusersandoperators�  Everythingworksbetterwhenusersareeducatedabout“properuse”

�  Educationandoutreach�  OpenStackdocumentationismixedblessing

�  Chameleondocsareupdatedwitheachrelease

�  Livewebinarsandface-to-facemeetupsmostimpactful

�  Incentivizinganecosystem�  Sharingismuchmorepowerfulinthecloud!


PARTINGTHOUGHTS�  Physicalenvironment:Chameleonisarapidlyevolvingexperimental

platform�  From“adaptstotheneedsofyourexperiment”…�  …to“adaptstothechangingresearchfrontier”

�  Cloudsarehardtooperatebecausetheysolveacomplexproblem–andexperimentalfacilitiesevenmoreso�  Moreskilledpersonnel,moreeffort–andespeciallyintestbedsmoredevelopment

�  Towardsanecosystem:ameetingplaceofuserssharingresourcesandresearch�  Testbedsaremorethanjustexperimentalplatforms:common/sharedplatformisa

“commondenominator”thatcaneliminatemuchcomplexitythatgoesintosystematicexperimentation,sharing,andreproducibility

�  WorkingwithotheroperatorsviaCHI-in-a-BoxandBYOHinitiatives�  Workingwithusersviaprovidingsharingmechanismsandfosteringcommunity

development


WE’REHERETOCHANGE

CHAMELEON:

www.chameleoncloud.org

chameleon: operational lessons › media › filer_public › 9c › 54 › ...we live to serve:...

Documents