INDIGO PAAS

TUTORIAL !

Marica Antonacci

!!

marica.antonacci@ba.infn.it !

INFN-Bari

RIA-653549

e-Research Summer Hackfest - 04 July 2016, Catania

INDIGOPAASTutorial

▪ IntroductoryConcepts• TOSCA

• Ansible

• Docker

• OrchestratorAPIs

• INDIGOTOSCAcustomtypesandtemplates

▪Demos

2

e-Research Summer Hackfest - 04 July 2016, Catania

TOSCA

▪TopologyandOrchestrationSpecificationforCloudApplications▪Standardizesthelanguagetodescribe• ThestructureofanITService(itstopologymodel)

• Howtoorchestrateoperationalbehavior(planssuchasbuild,deploy,patch,shutdown,etc.)

- LeveragingtheBPMNstandard

• Declarativemodelthatspansapplications,virtualandphysicalinfrastructure

3

e-Research Summer Hackfest - 04 July 2016, Catania

TOSCAinanutshell

4

e-Research Summer Hackfest - 04 July 2016, Catania

TOSCATopology

5

TOSCA-Simple-Profile-YAML-v1.0-csprd02 04 February 2016 Standards Track Work Product Copyright © OASIS Open 2016. All Rights Reserved. Page 19 of 267

will execute the script automatically to create the node on db_server, but first it will prepare the local environment with the declared inputs for the operation. In this case, the orchestrator would see that the db_data input is using the get_artifact function to retrieve the file (my_db_content.txt) which is associated with the db_content artifact name prior to executing the db_create.sh script. The logical diagram for this example would appear as follows:

Note that while it would be possible to define one node type and corresponding node templates that represent both the DBMS middleware and actual database content as one entity, TOSCA normative node types distinguish between middleware (container) and application (containee) node types. This allows on one hand to have better re-use of generic middleware node types without binding them to content running on top of them, and on the other hand this allows for better substitutability of, for example, middleware components like a DBMS during the deployment of TOSCA models.

2.5 TOSCA template for a two-tier application The definition of multi-tier applications in TOSCA is quite similar to the example shown in section 2.2, with the only difference that multiple software node stacks (i.e., node templates for middleware and application layer components), typically hosted on different servers, are defined and related to each other. The example below defines a web application stack hosted on the web_server “compute” resource, and a database software stack similar to the one shown earlier in section 6 hosted on the db_server compute resource.

Example 6 - Basic two-tier application (web application and database server tiers)

tosca_definitions_version: tosca_simple_yaml_1_0 description: Template for deploying a two-tier application servers on two

▪3layers•Infrastructure(CloudorDCobjects)

•PlatformorMiddleware (Appcontainers)

•Applicationmodules,schemasandconfigurations

!

▪Relationshipsbetweencomponents:

•What’shostedonwhatorinstalledonwhat

•What’sconnectedtowhat

e-Research Summer Hackfest - 04 July 2016, Catania

TOSCATopology

▪ComponentsinthetopologyarecalledNodes

▪EachNodehasaType(e.g.Host,BD,Webserver).

•TheTypeisabstractandhenceportable•TheTypedefinesPropertiesandInterfaces▪AnInterfaceisasetofhooks(namedOperations)

▪NodesareconnectedtooneanotherusingRelationships▪BothNodeTypesandRelationshipTypescanbederived

6

e-Research Summer Hackfest - 04 July 2016, Catania

NodeTypeExamples

▪DefinespropertiesasYAMLmaps

▪Mightdefinescapabilities(Whatitcanprovidetoothernodes)

7

e-Research Summer Hackfest - 04 July 2016, Catania

NodeTypeExamples

▪MightDefineRequirements(whatitneedsfromothernodes)

8

e-Research Summer Hackfest - 04 July 2016, Catania

NodeTemplate

▪Aninstanceofatype(likeObjecttoClass)▪Hasspecificproperties▪Hasartifacts:•Whattoinstall

• Howtoinstall(mappedtointerfacehooks)

▪Hasrequirementsandcapabilities(orrelationships)

9

e-Research Summer Hackfest - 04 July 2016, Catania

TopologyTemplateExample

10

e-Research Summer Hackfest - 04 July 2016, Catania

INDIGOcustomtypes

11

▪ExtendtheTOSCAnormativetypes

E.g.:tosca.nodes.indigo.GalaxyPortal,tosca.nodes.indigo.GalaxyWN

Ansible playbook

e-Research Summer Hackfest - 04 July 2016, Catania

AutomatedDeploymentwithAnsible

▪Ansibleisaconfigurationmanagementandprovisioningtool,similartoChef,PuppetorSalt.

▪Agentless• usesSSHbydefaulttomaketheconnectiontothetargetnode

• canberunlocally

▪ConfigurationinYAML

▪Veryeasytolearn• http://docs.ansible.com/ansible/index.html

12

e-Research Summer Hackfest - 04 July 2016, Catania

Buildingblocks:Playbook

▪Definessequencesoftasks(Plays)tobeexecutedonagroupofhosts

•describespoliciesmachinesundermanagementshallenforce

•containsvariables,tasks, handlers,files,templatesandroles

•expressedinYAML

•Idempotent

13

--- - hosts: webservers vars: http_port: 80 max_clients: 200 remote_user: root tasks: - name: ensure apache is at the latest version yum: name=httpd state=latest - name: write the apache config file template: src=/srv/httpd.j2 dest=/etc/httpd.conf notify: - restart apache - name: ensure apache is running (and enable it at boot) service: name=httpd state=started enabled=yes handlers: - name: restart apache service: name=httpd state=restarted

e-Research Summer Hackfest - 04 July 2016, Catania

Buildingblocks:Roles

▪Thebestwaytoorganizeyourplaybooks.•structurecontentintorelatedvars,tasks,files,handlers,etc.•filestructureforautomatedinclusionofrolespecificcontent

•rolescanbesharedandpulledfromAnsibleGalaxy,GitHub,etc.

14

--- - hosts: webservers vars: http_port: 80 max_clients: 200 roles: - webserver

e-Research Summer Hackfest - 04 July 2016, Catania

AnsibleGalaxy:indigo-dcroles

15

ansible-galaxy install indigo-dc.<role>

e-Research Summer Hackfest - 04 July 2016, Catania

Docker:Build-Ship-Run

▪Dockerisanopen-sourceenginetoeasilycreatelightweight,portable,self-sufficientcontainersfromanyapplication.

▪Thesamecontainerthatadeveloperbuildsandtestonalaptopcanrunatscale,inproduction,onVMs,OpenStackcluster,publiccloudsandmore.

▪Dockerfeatures:• versioning(git-like)

• componentre-use

• sharing(publicrepository)

16

e-Research Summer Hackfest - 04 July 2016, Catania

Dockerfile

▪ADockerfileisascript,composedofvariouscommands(instructions)andargumentslistedsuccessivelytoautomaticallyperformactionsonabaseimageinordertocreate(orform)anewone

17

FROM ubuntu MAINTAINER Romin Irani (email@domain.com) RUN apt-get update RUN apt-get install -y nginx ENTRYPOINT [“/usr/sbin/nginx”,”-g”,”daemon off;”] EXPOSE 80

e-Research Summer Hackfest - 04 July 2016, Catania

Build-Dockerfile

▪dockerbuild-tindigodatacloudapp/ambertools:1.0.0

!

▪AutomatedbuildonDockerHub

18

e-Research Summer Hackfest - 04 July 2016, Catania

ShipandRun

▪ShipaDockerImage

• dockerpushindigodatacloudapp/ambertools

!

▪FetchaDockerImage

•dockerpullindigodatacloudapp/ambertools

▪downloaddockerimagefromDockerHubtolocalDockerrepository

!

▪Runadockercontainer• dockerrun[…]indigodatacloudapp/ambertools

- createsadockercontaineroutofthedockerimage

19

e-Research Summer Hackfest - 04 July 2016, Catania

OrchestratorAPIs

▪Createadeployment:

• POSTrequestto/deployments-parameters:

- template:stringcontainingaTOSCAYAML-formattedtemplate

- parameters:theinputparametersofthedeployment(mapofstrings)

▪Getdeploymentdetails

• GETrequestto/deployments:

- curl'http://localhost:8080/deployments/<uuid>'

▪Deletedeployment

• DELETErequest

- curl‘http://localhost:8080/deployments/<uuid>'

▪Documentation:http://indigo-dc.github.io/orchestrator/restdocs/#overview

20

e-Research Summer Hackfest - 04 July 2016, Catania

▪SubmitaJob:

• POSTrequestto/scheduler/iso8601

- acceptsJSONbody

▪Addadependentjob• POSTto/scheduler/dependency

- acceptJSONhash

▪Listjobs• GETtoscheduler/jobs

- filteronname:GET/scheduler/jobs/search?name=<jobId>

▪Deleteajob• POSTto/scheduler/job/<jobId>

ChronosJobdefinitionandAPIs

21

Exam

ple

e-Research Summer Hackfest - 04 July 2016, Catania

INDIGOTOSCAcustomtypesfordescribingChronosjobs

22

The Orchestrator interprets the TOSCA

template and orchestrates the

deployment on Chronos

e-Research Summer Hackfest - 04 July 2016, Catania

INDIGOChronosJob:TOSCATemplates

▪ScenarioI:thejobdataarestagedin/outusingtheuser’sonedataspaces(providingproperaccesstoken(s))

• exampletemplate:https://github.com/indigo-dc/tosca-types/blob/master/examples/indigo_job_onedata.yaml

▪ScenarioII:thejobdataarefetchedfrompublicURLsanduploadedtoarepository(web,swift/S3,etc.)usingthecredentialsspecifiedbytheuser

• exampletemplate:https://github.com/indigo-dc/tosca-types/blob/master/examples/indigo_job_output_upload_swift.yaml

23

e-Research Summer Hackfest - 04 July 2016, Catania

ChronosJob:ParameterSweep

▪TheTOSCAtemplatedescribesmultiplejobs

▪Eachjobisrunwithaspecificsetofinputparameters

▪ThejobsareruninparallelontheMesoscluster

• thescalingserviceensuresthatnewslavenodesareaddedtothecluster,ifneeded.

24

e-Research Summer Hackfest - 04 July 2016, Catania

DemosOutline

▪ ElasticGalaxyCluster(Recorded)Demo:

▪ aGalaxyportalisautomaticallydeployedfromTOSCAandconfiguredtouseaSLURMelasticcluster

▪ workernodesareaddedassoonasjobsaresubmittedthroughtheportal

▪ High-AvailableMesosCluster(Recorded)Demo:

▪ acompleteHAMesosclusterwithChronos/MarathonframeworkisautomaticallydeployedfromaTOSCAtemplate

▪ Applicationexecution(Live)Demos:

▪ Amber

▪ EMSO-INGV(Rscript)

▪ Lifewatch(barridocker)

25

e-Research Summer Hackfest - 04 July 2016, Catania

DemoOutline(cont.)

▪Long-RunningServicedeploymentDemo:

• deployLong-RunningServices:nginx(webservice),rstudio

• auto-scalingPoC

• load-balancing

26