scenarios for the deployment of indigo services
TRANSCRIPT
Scenarios forthedeploymentofINDIGOServices
RIA-653549Giacinto Donvito
INDIGO-DataCloud WP5LeaderandTCSeptember2016
Sampleusecases:
1. EnhancedResourceVirtualization->Computing2. EnhancedResourceVirtualization->Storage3. InteractiveusageofaDockercontainerwithssh4. Awebportalthatusesabatchsystemtorunapplications5. VirtualinfrastructuresforMedicalImagingBiobanks6. AnapplicationtoCMS7. RunningDockercontainerswithoutDocker
September2016 G.Donvito - TheINDIGO-DataCloudMidnightBlueRelease 2
EnhancedResourceVirtualization->Computing(OpenNebula)
September2016 G.Donvito - TheINDIGO-DataCloudMidnightBlueRelease 3
OpenNebula
OneDock
Orchestrator+TOSCASupport(IM)
OCCISupport
1. IM:Providesa) AdvancedIaaS Orchestrator
capabilitiesb) TOSCASupport
2. OCCI:a) EnhancedNetworkcapabilitiesb) Docker support
3. OneDock:a) SupportfornativeDocker (onbare-
metal)
EnhancedResourceVirtualization->Computing(OpenStack)
September2016 G.Donvito - TheINDIGO-DataCloudMidnightBlueRelease 4
OpenStack
NovaDocker
Orchestrator+TOSCASupport(HEAT)
OCCISupport
1. TOSCAonHEAT2. OCCI:
a) EnhancedNetworkcapabilitiesb) Docker support
3. NovaDocker:a) SupportfornativeDocker (onbare-
metal)4. Synergy:
a) Fair-shareoncloudresourceusage5. Spot-istances
Synergy
SpotIstances
EnhancedResourceVirtualization->Storage(QoS)
September2016 G.Donvito - TheINDIGO-DataCloudMidnightBlueRelease 5
CEPH
CDMI
1. CDMIservice providesthecapabilitytomanagetheQoS ofstorage
2. Indendently fromthetechnologyused
3. CDMIisnotusedtoaccessfilesa) Thefilesstillcouldbe
accessed/storedusingtheoriginalprotocols
POSIX dCache
CDMI CDMI
DataFederation through INDIGOOnedata
AmazonS3
DNS:p-aws-useast
INFNItaly
DockerOneclient
Docker
AWSUSA
DockerOnezone
VMonezone
DockerOneclient
Docker
NFSServer
VMoneprovider
VMnfs
VMoneclient
POSIXVolume
DockerOneclient
DockerUPVSpain
VM:demo-onedata-upv-provider
DockerOneclient
LaptopOSX
SAMBAExport
boot2docker
20D.Salomoni- TheINDIGO-DataCloudPlatformJuly20,2016- JinanCloudSchool
InteractiveusageofaDockercontainerwithssh - Overview
September2016 G.Donvito - TheINDIGO-DataCloudMidnightBlueRelease 7
3
FutureGatewayAPIServer
Orchestrator
OneDock nova-docker
WP6
WP5
WP4
TOSCADocumentsandDockerfilesperUseCase OtherPaaS
CoreServices
CloudSite
DockerContainerPublicIP
SSHdINDIGO-DataCloud
DockerHubOrganizaLon
Provider
Champion+JRA
User
1.a.1)build,push
1.a.2)Dockerfile(commit)
1.b)AutomatedBuild
3)DeployTOSCA
2)StageData
5)Mount
4)Access
App
IM
InteractiveusageofaDocker containerwithssh - Services
1. TOSCATemplatetodescribetheuserservice2. FutureGayeway to“configureandsubmit”TOSCATemplateinaneasy
way3. Orchestrator+PaaS Coreservices+CloudProviderRanker +SLAM/QoS:
a) TofindtheavailableIaaSb) Thatarecorrectlyworkingc) ThathasSLAwiththegivenuserd) Andsupportsthehw+sw requirements
4. InfrastructureManageratthePaaS levelincasetheIaaS donotsupportsnativeTOSCAenabledorchestrator
5. IaaS Orchestrator(Heat/IM)supportingTOSCA6. OneDock orNovaDocker torunDocker onbaremetalatIaaS level
September2016 G.Donvito - TheINDIGO-DataCloudMidnightBlueRelease 8
G.Donvito - TheINDIGO-DataCloudMidnightBlueRelease
Future GatewayAPIServer
WP6
WP5
Front-EndPublic IP
Provider
User2)Deploy TOSCAwithVanilla VM/Container
1)Stage Data
5)Mount
6)AccessWebPortal
GalaxyWNWNWN …
VirtualElastic LRMSCluster
Orchestrator
IM
OpenNebula
WP4
Other PaaSCore Services
CloudSite
OpenStack
HeatClues
IM
TOSCADocuments andDockerfiles perUseCase
INDIGO-DataCloudDocker Hub Organization
Champion+JRA
1.a.1)build,push
1.a.2)Dockerfile(commit)
1.b)AutomatedBuild
September2016
Awebportalthatusesabatchsystemtorunapplications- Overview
OneZone
TOSCA TOSCA
Awebportalthatusesabatchsystemtorunapplications- Services
1. TOSCATemplatetodescribetheuserservice2. FutureGayeway to“configureandsubmit”TOSCATemplateinaneasyway3. Orchestrator+PaaS Coreservices+CloudProviderRanker +SLAM/QoS:
a) TofindtheavailableIaaSb) Thatarecorrectlyworkingc) ThathasSLAwiththegivenuserd) Andsupportsthehw+sw requirementse) Thathoststherequireddata
4. InfrastructureManageratthePaaS levelincasetheIaaS donotsupportsnativeTOSCAenabledorchestrator
5. IaaS Orchestrator(Heat/IM)supportingTOSCA6. Onedata forsharedanddistributeddataaccess7. Clues fordrivingtheautomaticresourceprovisioningbasedontheusage
September2016 G.Donvito - TheINDIGO-DataCloudMidnightBlueRelease 10
AnapplicationtoLHC/CMS
• Thegoal istodevelopasolutionforgeneratingautomaticallyanon-demand,container-basedclusterforCMSinordertoallow:
• Theeffectiveuseofopportunisticresources,suchasgeneralpurposescampusfacilities.• Thedynamicextension ofanalreadyexistingdedicatedfacility.
• Bysimplifyingandautomatingtheprocessofcreating,managingandaccessingapoolofcomputingresourcestheprojectaimstoimprove:
• Sitesmanagement:• Asimplesolutionfordynamic/elasticT2extensionson“opportunistic”/stableresources• Afriendlyproceduretodynamicallyinstantiateaspot“Tier3-likeresourcecenter”
• Usersexperience:• Generationofanephemeralon-demandT3seenbytheExperimentcomputinginfrastructureasa
personalWLCG-typefacility,inordertoserveagroupofcollaborators.Thesystemmustallowtheuseofstandard/regularCMSToolssuchasCRAB.
• Experiment-Collaborationresources:• Acomprehensiveapproachtoopportunisticcomputing.Asolutiontoaccessandorchestratee.g.
multiplecampuscenters,harvestingallthefreeCPUcycleswithoutmajordeploymentefforts.
September2016 G.Donvito - TheINDIGO-DataCloudMidnightBlueRelease 11
ApplicationtoCMS,fourpillars
• ClusterManagement:• Mesos clustersasasolutioninordertoexecutedocker foralltheservicesrequiredbyaregularCMSsite
(WorkerNodes,HTCondor Schedd andsquids).• Marathon guaranteesusthedynamicscalingupanddownofresources,akeypoint.
• AuthN/Z&CredentialManagement:• TheINDIGOIdentityAccessManagement(IAM)serviceisresponsibleforAuthN/Ztotheclustergeneration.• TheTokenTranslationService(TTS)enablestheconversionofIAMtokensintoanX.509certificate
• NOTE:ThisallowsMesos slaves(runningHTCondor_startd daemon)tojoin theCMScentralqueue(HTCondor_schedd) asaregularGridWN
• DataManagement:• Dynafed +FTSistheapproachcurrentlyfollowedbytheproject.Afurtherpossibilitywewillinvestigateis
Oneclient (fromOnedata)asatoolallowingtomountremotePosix filesystems.• Automation:
• TOSCAtemplates,meanttobemanagedbyINDIGOPaaSOrchestrator,allowtheautomationoftheoverallsetup.
• TheaimistoproduceasingleYAMLfiledescribingthesetupofallrequiredservicesanddeps.• Clues isabletoscaletheMesos clusterasneededbytheloadoftheusersjobs
September2016 G.Donvito - TheINDIGO-DataCloudMidnightBlueRelease 12
ApplicationtoCMS,architecture
September2016 G.Donvito - TheINDIGO-DataCloudMidnightBlueRelease 13
USER
Schedd(CMScentralorprivate)
CRABCFGUseranalysisjobdescriptionpointingtoSITENAME
VM#1
Squid1
VM#2
Docker1
VM#3
Docker2
VM#4
Docker3
Cloud#1
Mesos clusterSITENAME#/typeofservices
SQUIDsSchedd ifneededWNs(rangedesired)
Onedata /Dynafed attachedStorageTFCrules
FallbackstrategyTempstoragetobeused
Cloud#2
DataManagementDataplacement andaccess(Onedata,Dynafed,FTS)
PaaSOrchestrator
+PaaS Service
TTSIAM
MesosCluster
MesosCluster
Clues
Clues
RunningDockercontainers…withoutDockerJ
• AdoptionofDockerisveryslowinHPCcenters• ThusthetypicalsituationisthatDockerisnotinstalledandonecannotruncontainerswithoutsomesupportfromsystemsoftware.
• Ingeneral,Dockeradoptionwillbeslowinanycomputingfarmorinteractivelinux systemsharedbymanyusers.
• Itwilltaketimeforsysadminstoovercometheconcernsoftheirsecurityteams.
• Itisyetanotherservicetomaintain…• ….younameit.
G.Donvito - TheINDIGO-DataCloudMidnightBlueRelease 14September2016
INDIGOudocker
• Atooltoexecutecontentofdocker containersinuserspacewhendocker isnotavailable• enablesdownloadofdocker containersfromdockerhub• enablesexecutionofdocker containersbynon-privilegedusers
• Itcanbeusedtoexecutethecontentofdocker containersinLinuxbatchsystemsandinteractiveclustersmanagedbyothers
• Awrapperaroundothertoolstomimicdocker capabilities• currentversionusesproot toprovideachroot likeenvironmentwithoutprivileges(itrunsonCentOS6,CentOS7,Fedora,Ubuntu)
• Moreinfoanddownloadsat:• https://www.gitbook.com/book/indigo-dc/udocker/details• https://indigo-dc.gitbooks.io/udocker/content/doc/user_manual.html
G.Donvito - TheINDIGO-DataCloudMidnightBlueRelease 15September2016
INDIGOudocker
• Examples:# download, but could also import or load a container exported/save by docker$ udocker.py pull ubuntu:latest$ udocker.py create --name=myubuntu ubuntu:latest
# make the host homedir visible inside the container and execute something$ udocker.py run -v $HOME myubuntu /bin/bash <<EOFcat /etc/lsb-releasels -l $HOMEEOF
udocker isNOTanalternativetodocker:weneedthecontainerimagebuiltbydocker.
Itisatooltohandleandruncontainerswithregularuserprivileges and/orwhendocker isnotavailableforsomereason:itisveryconvenienttoaccessclustersandGridresources
G.Donvito - TheINDIGO-DataCloudMidnightBlueRelease 16September2016
INDIGOudocker
• Everythingisstoredintheuserhomedir orsomeotherlocation• Containerlayersaredownloadtotheuserhome• Directorytreescanbecreated/extractedfromthesecontainerlayers• proot usesthedebuggerptrace mechanismtochangepathnamesandexecutetransparentlyinsideadirectorytree
• Noimpactonread/writeorexecution,onlyimpactonsystemcallsusingpathnames(ex.open,chdir,etc)
• Doesnotrequireinstallationofsoftwareinthehostsystem:• udocker isapythonscript• proot isstaticallycompiled
G.Donvito - TheINDIGO-DataCloudMidnightBlueRelease 17September2016
Conclusions
• ThefirstpublicINDIGOreleasecameoutatthebeginningofAugust2016.
• Itsservicesarealreadyavailableinseveraltestbeds.• ConcreteusecasesarecurrentlybeingimplementedbymanyINDIGOscientificcommunities.
• Alotofimportantdevelopmentsarebeingcarriedonwiththeoriginaldeveloperscommunity,sothatcodemaintenanceisnot(only)inourhands.
• Nowlookingforearlyadopters/peoplewillingtotesttheINDIGOcomponentswiththeirapplicationsorrequirements- Ifinterested,pleasecontactus.
18G.Donvito - TheINDIGO-DataCloudMidnightBlueReleaseSeptember2016