Developing High Performing Parallel Application Services on Windows AzureWen-ming YeSr. Technical EvangelistMicrosoft Corporation
AZR303
Session Goals and Agenda
Understand Compute Intensive Workload in Windows AzureLearn Build, Deploy, and Run Services using the HPC SchedulerGet ideas for teaching your old CI apps some new Cloud tricks!
What’s the Problem?
Some apps run too slowly on a single computer
Splitting them up into chunks, then running those chunks in parallel on multiple computers can speed them up
Doing this effectively requires creating a compute cluster, with:
Tools to create and manage the clusterA batch scheduler for running apps on the cluster
How Does Azure Extend High Performance Computing to the Masses?
Example Workloads That Need a Cluster
Digital media rendering and encodingFinancial and insurance risk calculationEngineering modeling and simulationComputational life sciencesEarth sciencesData analytics
video
High resolution Computing
Earthquake and Weather Simulations
demo
A 20,000 x 20,000 pixel Tobacco Mosaic Virus render
Embarrassingly Parallel AppsAn illustration
1. Submit work
3. Get results
2. (Optionally) Interact with application
Compute Node
Application Logic
Compute Node
Application Logic
Compute Node
Application Logic
Compute Node Compute Node Compute Node
Compute Node Compute Node Compute Node
Application Logic
Application Logic
Application Logic
Application Logic
Application Logic
Application Logic
Compute Node
Application Logic
Compute Node
Application Logic
Compute Node
Application Logic
Compute Node Compute Node Compute Node
Compute Node Compute Node Compute Node
Application Logic
Application Logic
Application Logic
Application Logic
Application Logic
Application Logic
Tightly Coupled App (MPI)An illustration
1. Submit work
2. Get results
IO Intensive ApplicationsAn illustration
Compute Node
Application
Compute Node
Application
Compute Node
Application
Compute Node Compute Node Compute Node
Compute Node Compute Node Compute Node
Application Application Application
Application Application Application
1. Submit work
2. Get results
Compute Nodes
HPC Server Makes This Easy – offloading long running jobs
Logic
Logic
Logic
Windows HPC Server 2008 R2 SP3
Cloud - Applications in AzureJune 2012 Release
Application in hosted entirely in Azure No on-premise Head Node
Application is accessed fromRich client on premisePortalWeb Application
Multiple business modelsPackaged application is sold to a customer Application is available as a service in the cloud
Scheduler and Runtimes Supported in AzureWorkloads: Parametric Sweep, SOA, MPINo Head Node (Scheduler Service, no Admin Services)
Building a Scheduler Enabled Deployment
Azure SubscriptionYour applicationConstruct a base CS PackageCustomize and Deploy
Windows Azure Scheduler In Action
Application Front End (Web Page)
Application
Visual Studio Sample Solution – Easy first development experience for WAHS parallel applications in Azure
Deployment ApplicationAuto-create and initialize hosted service, storage, HPC database in same locationProgress bar and status
PSH Scripts available too
Example: Cloud Numerics…
demo
Deploy a cluster using Visual Studio
Scenario: Full desktop client offloading long running tasks to the Azure HPC Scheduler
Simplicity of Microsoft HPC Compute Intensive Paradigms Run on:
ClientClusterCloud
Transparently
demo
Rich client application connecting to windows Azure scheduler for offloading a fluid dynamics model
Openfoam compute offload
Moving files to and from Azure Nodes in a Windows HPC 2008 R2 cluster
Azure NodesWindows Azure Storage
3 Copy all packages from storage to Azure Nodes and extract files
Windows Azure
On-premises
hpcsync
HPC Head NodeComputer with
HPC client utilities installed
OR
Delete package from storage
hpcpack upload
2 Upload package to storage
hpcpack remove
Delete file from Azure Node
hpcfile get
hpcfile put
hpcfile del
Application files
hpcpack create
Zip package
Package your application files1
Upload or download a file or folder from a specific
Azure Node (up to 4 GB)
Scenario: More code to run: Moving code form Linux/Unix platforms
Develo
per
Tools
Compilers:• PGI • Intel• GCCBuild:• CMake• Visual StudioUnix Env:• Cygwin• SUA (win8)• MinGWPerf Tools• Vampir• TAU
Lib
rari
es
Solvers:• PETSc• Trilinos• FFTWIO:• HDF5MPI• NetCDF
Pop
ula
r H
PC
Ap
ps
ChemistryBio ScienceCFDStructureClimatePhysicsData MiningComputer Science
An
aly
sis
Tools
Rich client Parallel Visualization:• VisIt• ParaView
Python:• HTML5 front• Ipython• NumPY• SciPY
OSS Eco-system To create Scalable TC Azure services
Scenario: Building a Service for sharing simulation engine and data using the Azure HPC Scheduler
demo
Weather as a Service
Roadmap Spring 2012 Hybrid Clusters Across Enterprise and Cloud• Support for Client->Job Scheduler .NET API
tunneled over HTTP (i.e., same Job Scheduling API for on-premise and Windows Azure clusters)
Building HPC and Parallel Applications in the Cloud• Initializing HPC SQL Azure DB from HN in
Azure
All items considered general availability, unless noted.*Denotes Spring in the US, 1st half of 2012
Windows HPC Pack 2008 R2 Service Pack 4
Who would
use it?
Do it yourself clusterPower user
In house developer
Packaged softwareLeverage additional compute resource in Windows Azure
Software as a ServiceVendor pays for the compute resources
Vendor bills end user directly
9(-5) Tips for Best Practices
Hide cloud from your users by providing a transparent experienceMinimize data transfer between on-premises and the cloudCPU intensive jobs are best suited for pay as you go modelBe creative and focus on what you can do on Azure today
The Matrix can not be told, try it yourself!
Subscribe to my blog http://blogs.msdn.com/hpctrekkerRead Application Models and Data ConsiderationsDownload SP4 from download.microsoft.comGet a free Trial of Azure for 30 days Download and try out the Azure Burst SamplesImplement your own solution and let us know!
Thank You!
Track Resources
Meetwindowsazure.com
@WindowsAzure @ms_teched
DOWNLOAD Windows Azure
Windowsazure.com/teched
Hands-On Labs
Resources
Connect. Share. Discuss.
http://northamerica.msteched.com
Learning
Microsoft Certification & Training Resources
www.microsoft.com/learning
TechNet
Resources for IT Professionals
http://microsoft.com/technet
Resources for Developers
http://microsoft.com/msdn
Complete an evaluation on CommNet and enter to win!
Please Complete an Evaluation Your feedback is important!
Multipleways to Evaluate Sessions
Scan the Tagto evaluate thissession now on myTechEd Mobile
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to
be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS
PRESENTATION.
Track Resources
My Blog: http://blogs.msdn.com/hpctrekker
The Azure Burst Samples and White paper
Windows HPC ResourceKit
Windows HPC Server TechCenter