iwsg2014: developing science gateways using apache airavata
Post on 25-Jan-2015
55 Views
Preview:
DESCRIPTION
TRANSCRIPT
Developing Computational Science Gateways using Apache Airavata
Marlon PierceScience Gateway Group, Indiana University
marpierc@iu.edu
1. We Are Hiring
Apply: http://goo.gl/LbiXNO
http://www.catonmat.net/blog/what-would-john-mccarthy-say-if-he-saw-you-programming/
2. dev-subscribe@airavata.apache.org
3. We’ll get to this one later….
http://www.bayoubelles.com/2011/08/night-before-big-day-was-not-what-was.html
Airavata’s Philosophy• There a lots of ways to build Web interfaces for
Science Gateways.– By Hand: PHP, Twitter Bootstrap, AngularJS, …– Turnkey Frameworks: Liferay, Drupal, Plone, Joomla, …– Science Gateway Frameworks: the SDSC Workbench,
HUBzero, gUse/WS-PGrade• Gateway developers should concentrate on
building interfaces that serve their community.• And outsource the general purpose services to
Airavata.
Airavata Role and Goals: Improve sustainability by converging on a single set of hosted infrastructure services
Apache Airavata ComponentsComponent Description
Airavata API Server
Apache Thrift-generated server skeletons of the API and data models; directs traffic to appropriate components
Registry Insert and access application, host machine, workflow, and provenance data.
Orchestrator Handles experiment execution request validation, scheduling, and decision making; selects GFAC instances that can fulfill a given request
GFAC Manages the execution and monitoring of an individual application.
Workflow Interpreter
Execute the workflow on one or more resources.
Messaging System
WS-Notification and WS-Eventing compliant publish/subscribe messaging system for workflow events
Airavata Components
Airavata’s Future Directions: SciGaP
• SciGaP: Airavata as a multi-tenanted Gateway Platform as a Service
• Goal: We run Airavata so you don’t have to.– Scalable support
• Challenges: –Centralize system state–Make Airavata more cloud friendly, elastic
http://scigap.org
Some Contribution OpportunitiesComponent Research Opportunities
Registry Better support for Thrift-generated objects; NoSQL and other backend data stores; fault tolerance
Orchestrator Pluggable scheduling; load balancing and elasticity
GFAC ZooKeeper-like strategies for configuring and managing.
Messenger Investigate AMQP, Kafka, and other newer messaging systems
Workflow Interpreter
Alternative workflow processing engines.
Overall Message-based rather than direct CPI calls.
Airavata components expose Component Programming Interfaces (CPIs) that allow you to switch out implementations. GFAC is also designed to be
pluggable.
Airavata and the Apache Software Foundation
Experiences with Open Governance
Cyberinfrastructure: How open is open source?
• What’s missing?– Open source licensing– Open Standards– Open Code (GitHub,
SourceForge, Google Code, etc)
We also need Open Governance
Open Source Software and Governance
• Open source projects need diversity, governance.– Sustainability
• Incentives for projects to diversify their developer base.
• Govern– Software releases– Contributions– Credit sharing.– Members are added– Project direction decisions.– IP, legal issues
• Our approach: Apache Software Foundation
Collaborate
Compete
“Apache” Means “Open”
Join the Airavata developer or architecture mailing lists, get involved, submit patches, contribute.
Use Give Back
More Information
• Contact Us:– marpierc@iu.edu, smarru@iu.edu– Join dev@airavata.apache.org,
users@airavata.apache.org, architecture@airavata.apache.org
• Websites:– Apache Airavata: http://airavata.apache.org – SciGaP Project: http://scigap.org – Science Gateway Institute: http://sciencegateways.org
Airavata AudiencesAudience Description
Gateway Developers
Use the Airavata API through and SDK in their favorite programming language.
Airavata Developers
Want to change Airavata components, experiment with different implementations.
Middleware Developers
Want to extend Airavata to talk to their middleware clients.
Resource Providers Want to configure Airavata to work with their middleware.
Apache Contributions Aren’t Just Software
• Apache committers and PMC members aren’t just code writers.
• Successful communities also include– Important users– Project evangelists – Content providers: documentation, tutorials– Testers, requirements providers, architects, and
constructive complainers• Using Jira and mailing lists
– Anything else that needs doing.
Getting Involved, Contributing Back
• Airavata is open source, open community software.
• Open Community: you can contribute back– Patches, suggestions, wiki
documentation, etc• We reward contributors
– Committers: write access to master Git repo
– Project Management Committee members: full, binding voting rights
Airavata AudiencesAudience Description
Gateway Developers
Use the Airavata API through and SDK in their favorite programming language.
Airavata Developers
Want to change Airavata components, experiment with different implementations.
Middleware Developers
Want to extend Airavata to talk to their middleware clients.
Resource Providers Want to configure Airavata to work with their middleware.
The Apache Software Foundation
• Apache software powers 65% of web sites worldwide
• 501(c)3 non-profit foundation
• Reasons for creating ASF– Create legal entity– Protect contributors from
liability– Protect Apache assets
• Membership: individual• Apache Incubator
• Governance and Staffing– Board of Directors– Project Management
Committees– ASF Members– Committers– Contributors
• Funding– All-volunteer
staffing/development resources
– Donations– Corporate investment
Where Is Airavata 1.0?
• Airavata 1.0 will be the stable version of the API.
• Version 1.0 determined by developer community vote.
• Semantic versioning
Airavata Thrift Definitions
airavata/airavata-api/thrift-interface-descriptions
Thrift File Contents
airavataAPI Definitions of services; top level file for other Thrift files.
airavataErrors Definitions of exceptions thrown by the API services.
airavataDataModel Top level container for data models.
workspaceModel Definitions of the structs (Project, User, Gateway, Group); includes experimentModel
experimentModel Definitions of data structures that constitute the experimentModel.
applicationCatalogAPI App catalog service definitions.
applicationCatalogDataModel App catalog data model definitions.
Airavata API and Apache Thrift
• We use Apache Thrift to define the API.• Advantages of Thrift– Supports well-defined, typed messages.– Custom defined errors, exceptions– Generators for many different programming languages.– Some shielding from API versioning problems.
• Downsides of Thrift– No message headers, so everything must be explicitly
defined in the API.
A Few Observations on Successful Gateways
• Support familiar community applications.• Make HPC systems easy for new user
communities who need HPC.• Keep it simple.• Have champions who build and support the
community.• Have a lot of common features.
Airavata API and Apache Thrift
• We use Apache Thrift to define the API.– TCP/IP typically– Not WS, not REST
• Advantages of Thrift– Supports well-defined, typed messages.– Custom defined errors, exceptions– Generators for many different programming languages.– Some shielding from API versioning problems.
• Downsides of Thrift– No message headers, so everything must be explicitly defined in
the API.
Airavata API Thrift Definitions
airavata/airavata-api/thrift-interface-descriptions
Thrift File Contents
airavataAPI Definitions of services; top level file for other Thrift files.
airavataErrors Definitions of exceptions thrown by the API services.
airavataDataModel Top level container for data models.
workspaceModel Definitions of the structs (Project, User, Gateway, Group); includes experimentModel
experimentModel Definitions of data structures that constitute the experimentModel.
applicationCatalogAPI App catalog service definitions.
applicationCatalogDataModel App catalog data model definitions.
top related