mistral openstack meetup feb 5
DESCRIPTION
Presentation about Mistral (Workflow Service) from SF Bay Area OpenStack meetup held at Yahoo! on Feb 6th.TRANSCRIPT
© MIRANTIS 2014© MIRANTIS 2014 !1
Mistral Workflow as a Service
Renat AkhmerovSenior Software Engineer
© MIRANTIS 2014 !2
Agenda
• Project Mission
• Project users
• Use Cases
• Main Features
• DSL, Data Flow concept, Architecture
• Current status
• Live Demo
• Roadmap and Contribution
© MIRANTIS 2014 !4
Project Mission
Provide a mechanism to define and execute tasks and workflows without writing the code
© MIRANTIS 2014 !5
Who are Mistral Users?
• OpenStack developers: Nova, Heat, Murano, Solum etc.
• Administrators: capabilities like Cloud Cron to schedule periodic helper cloud tasks (local VM processes, backups etc.)
• Enterprise application developers: automate business processes consisting of multiple distributed processing steps
© MIRANTIS 2014 !6
Use Cases
• Cloud Cron: a system administrator can schedule cloud tasks for periodical execution
• Cloud Environment Deployment: tools like Heat can represent deployment steps as a workflow and use Mistral to run it
• Live Migration: on CPU close to 100% run specific migration workflow
• Long-Running Business Process
• Big Data Analysis & Reporting
© MIRANTIS 2014 !7
Cloud Cron
• Start a workflow on event (timer)
• Single point of configuration and control for helper periodic jobs
• Wide range of tasks: local processes, OpenStack services (Nova, Cinder, Swift etc.), email, SMS, custom actions
• Monitoring & Management
• HA & Scalability
© MIRANTIS 2014 !8
Main Features
• RESTful Web service
• Based on Domain Specific Language (DSL)
• Runs workflows consisting of synchronous and asynchronous tasks
• Tasks associated with pluggable actions
• Built-in actions: HTTP request, AMQP message
• Runs workflows on event (timer, ceilometer alarm)
• Provides history and state of currently running workflows
• HA and Scalability for workflow executions
© MIRANTIS 2014 !9
Mistral Workflow
• Graph of tasks
• Task Actions (HTTP, AMQP, SSH, email, etc.)
• Transitions
• Dependencies
• Data Flow
• Conditions
• Events (triggers)
T1
T5
T3
T4T2
start
T6
T7
depends
T8
T9
© MIRANTIS 2014 !10
Mistral DSL Capabilities
• Task dependencies (task3 depends on task2 and task1)
• Direct transitions (on task1 success execute task2, on error task3)
• Conditional expressions in dependencies and direct transitions (based on Data Flow)
• Task action declarations (signals over HTTP, AMQP)
• Standard actions (std:repeat, std:email)
• Cron events (run workflow hourly)
© MIRANTIS 2014 !11
DSL Snippet
Workflow: tasks: runJob: requires: createVM: $.vm_ip != null action: DemoApp:runJob on-success: deleteVM on-error: sendJobError ! deleteVM: action: Nova:deleteVM on-finish: end ! sendJobError: action: std:email parameters: address: [email protected] subject: Workflow error body: Error occurred while running a VM job in execution {$.execution.id} on-finish: deleteVM
© MIRANTIS 2014 !12
Data Flow (concept)
RunVM input:
image_id
SendMsg input:
person.email
IsMale input:
person.name
GoOn “image_id”: “123”, “person”: { “name”: “John”, “email”: “[email protected]” }
“image_id”: “123”, “person”: { “name”: “John”, “email”: “[email protected]” }, “vm_ip”: “10.0.0.3”
“image_id”: “123”, “person”: { “name”: “John”, “email”: “[email protected]” }, “vm_ip”: “10.0.0.3”
“image_id”: “123”, “person”: { “name”: “John”, “email”: “[email protected]” }, “vm_ip”: “10.0.0.3”, “male”: “true”
“image_id”: “123”, “person”: { “name”: “John”, “email”: “[email protected]” }, “vm_ip”: “10.0.0.3”, “sent_msg”: “true”
“image_id”: “123”, “person”: { “name”: “John”, “email”: “[email protected]” }, “vm_ip”: “10.0.0.3”, “male”: “true”, “sent_msg”: “true”
- task input - task result (added into context)
© MIRANTIS 2014 !13
Architecture
Scheduler
Task Executor
API ServerEngine
Task Executor
!...
workflows executions
tasks events
Workflow Queue
Task Queue
© MIRANTIS 2014 !14
Current Status
• Basic task dependency based model works
• Pilot is currently being worked on
• Launchpad home page o https://launchpad.net/mistral
• Code in stackforge code repos o https://github.com/stackforge/mistral o https://github.com/stackforge/mistral-extra o https://github.com/stackforge/python-mistralclient
• Description on OpenStack Wiki o wiki.openstack.org/wiki/Mistral
• Contributors: Mirantis, StackStorm
© MIRANTIS 2014 !15
Live Demo
MistralDemo App
1. Upload workbook
2. Run
work
flow (ta
sk4)
HTTP
3
4
4
5
REST endpoint
Execute Backup
Backup User Data
Backup Service Data
Put Service on
hold
© MIRANTIS 2014 !16
Mistral Icehouse Roadmap
• End of February, 2014. Pilot. o Basic DSL capabilities (dependencies, direct transitions, conditionals,
cron events) o Keystone Integration o REST API o Python Client
• April, 2014. Release 0.1 o Overall stability o Refined DSL o Mistral CLI o Horizon Dashboard (Workbook CRUD operations, workflow execution
state, history)
© MIRANTIS 2014 !17
Contribution Opportunities
• Blueprints for additional capabilities
• Seed initial dev team
• Web UI (history, workflow lifecycle management)
• Ceilometer integration
• Integration with deployment tools (Puppet, Chef)
• Advanced capabilities of Python Client (building workflow step-by-step locally, callbacks etc.)
• Workflow rollback
• CLI for Mistral API
• CLI for Mistral action handlers
© MIRANTIS 2014 !18
How to Become a Contributor?
Sign OpenStack
CLA
Buy a computer
Learn Python
Register at Launchpad
Install Python, git etc.
clone mistral repos
Make changes
Send patches
no
Tired?
Have coffee
yes
Cont
ribut
or lif
e-cy
cle
© MIRANTIS 2014 !19
Thanks for Attending!
Q & AProject: launchpad.net/mistral
IRC: #openstack-mistral
Me: Renat Akhmerov
Email: [email protected]