developing high performing parallel application services on windows azure wen-ming ye sr. technical...

Developing High Performing Parallel Application Services on Windows AzureWen-ming YeSr. Technical EvangelistMicrosoft Corporation

AZR303

Session Goals and Agenda

Understand Compute Intensive Workload in Windows AzureLearn Build, Deploy, and Run Services using the HPC SchedulerGet ideas for teaching your old CI apps some new Cloud tricks!

What’s the Problem?

Some apps run too slowly on a single computer

Splitting them up into chunks, then running those chunks in parallel on multiple computers can speed them up

Doing this effectively requires creating a compute cluster, with:

Tools to create and manage the clusterA batch scheduler for running apps on the cluster

How Does Azure Extend High Performance Computing to the Masses?

Example Workloads That Need a Cluster

Digital media rendering and encodingFinancial and insurance risk calculationEngineering modeling and simulationComputational life sciencesEarth sciencesData analytics

video

High resolution Computing

Earthquake and Weather Simulations

demo

A 20,000 x 20,000 pixel Tobacco Mosaic Virus render

Embarrassingly Parallel AppsAn illustration

1. Submit work

3. Get results

2. (Optionally) Interact with application

Compute Node

Application Logic

Compute Node

Application Logic

Compute Node

Application Logic

Compute Node Compute Node Compute Node

Compute Node Compute Node Compute Node

Application Logic

Application Logic

Application Logic

Application Logic

Application Logic

Application Logic

Compute Node

Application Logic

Compute Node

Application Logic

Compute Node

Application Logic

Compute Node Compute Node Compute Node

Compute Node Compute Node Compute Node

Application Logic

Application Logic

Application Logic

Application Logic

Application Logic

Application Logic

Tightly Coupled App (MPI)An illustration

1. Submit work

2. Get results

IO Intensive ApplicationsAn illustration

Compute Node

Application

Compute Node

Application

Compute Node

Application

Compute Node Compute Node Compute Node

Compute Node Compute Node Compute Node

Application Application Application

Application Application Application

1. Submit work

2. Get results

Compute Nodes

HPC Server Makes This Easy – offloading long running jobs

Logic

Logic

Logic

Windows HPC Server 2008 R2 SP3

Cloud - Applications in AzureJune 2012 Release

Application in hosted entirely in Azure No on-premise Head Node

Application is accessed fromRich client on premisePortalWeb Application

Multiple business modelsPackaged application is sold to a customer Application is available as a service in the cloud

Scheduler and Runtimes Supported in AzureWorkloads: Parametric Sweep, SOA, MPINo Head Node (Scheduler Service, no Admin Services)

Building a Scheduler Enabled Deployment

Azure SubscriptionYour applicationConstruct a base CS PackageCustomize and Deploy

Windows Azure Scheduler In Action

Application Front End (Web Page)

Application

Visual Studio Sample Solution – Easy first development experience for WAHS parallel applications in Azure

Deployment ApplicationAuto-create and initialize hosted service, storage, HPC database in same locationProgress bar and status

PSH Scripts available too

Example: Cloud Numerics…

demo

Deploy a cluster using Visual Studio

Scenario: Full desktop client offloading long running tasks to the Azure HPC Scheduler

Simplicity of Microsoft HPC Compute Intensive Paradigms Run on:

ClientClusterCloud

Transparently

demo

Rich client application connecting to windows Azure scheduler for offloading a fluid dynamics model

Openfoam compute offload

Moving files to and from Azure Nodes in a Windows HPC 2008 R2 cluster

Azure NodesWindows Azure Storage

3 Copy all packages from storage to Azure Nodes and extract files

Windows Azure

On-premises

hpcsync

HPC Head NodeComputer with

HPC client utilities installed

OR

Delete package from storage

hpcpack upload

2 Upload package to storage

hpcpack remove

Delete file from Azure Node

hpcfile get

hpcfile put

hpcfile del

Application files

hpcpack create

Zip package

Package your application files1

Upload or download a file or folder from a specific

Azure Node (up to 4 GB)

Scenario: More code to run: Moving code form Linux/Unix platforms

Develo

per

Tools

Compilers:• PGI • Intel• GCCBuild:• CMake• Visual StudioUnix Env:• Cygwin• SUA (win8)• MinGWPerf Tools• Vampir• TAU

Lib

rari

es

Solvers:• PETSc• Trilinos• FFTWIO:• HDF5MPI• NetCDF

Pop

ula

r H

PC

Ap

ps

ChemistryBio ScienceCFDStructureClimatePhysicsData MiningComputer Science

An

aly

sis

Tools

Rich client Parallel Visualization:• VisIt• ParaView

Python:• HTML5 front• Ipython• NumPY• SciPY

OSS Eco-system To create Scalable TC Azure services

Scenario: Building a Service for sharing simulation engine and data using the Azure HPC Scheduler

demo

Weather as a Service

Roadmap Spring 2012 Hybrid Clusters Across Enterprise and Cloud• Support for Client->Job Scheduler .NET API

tunneled over HTTP (i.e., same Job Scheduling API for on-premise and Windows Azure clusters)

Building HPC and Parallel Applications in the Cloud• Initializing HPC SQL Azure DB from HN in

Azure

All items considered general availability, unless noted.*Denotes Spring in the US, 1st half of 2012

Windows HPC Pack 2008 R2 Service Pack 4

Who would

use it?

Do it yourself clusterPower user

In house developer

Packaged softwareLeverage additional compute resource in Windows Azure

Software as a ServiceVendor pays for the compute resources

Vendor bills end user directly

9(-5) Tips for Best Practices

Hide cloud from your users by providing a transparent experienceMinimize data transfer between on-premises and the cloudCPU intensive jobs are best suited for pay as you go modelBe creative and focus on what you can do on Azure today

The Matrix can not be told, try it yourself!

Subscribe to my blog http://blogs.msdn.com/hpctrekkerRead Application Models and Data ConsiderationsDownload SP4 from download.microsoft.comGet a free Trial of Azure for 30 days Download and try out the Azure Burst SamplesImplement your own solution and let us know!

Thank You!

Track Resources

Meetwindowsazure.com

@WindowsAzure @ms_teched

DOWNLOAD Windows Azure

Windowsazure.com/teched

Hands-On Labs

Resources

Connect. Share. Discuss.

http://northamerica.msteched.com

Learning

Microsoft Certification & Training Resources

www.microsoft.com/learning

TechNet

Resources for IT Professionals

http://microsoft.com/technet

Resources for Developers

http://microsoft.com/msdn

Complete an evaluation on CommNet and enter to win!

Please Complete an Evaluation Your feedback is important!

Multipleways to Evaluate Sessions

Scan the Tagto evaluate thissession now on myTechEd Mobile

© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to

be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS

PRESENTATION.

Track Resources

My Blog: http://blogs.msdn.com/hpctrekker

The Azure Burst Samples and White paper

Windows HPC ResourceKit

Windows HPC Server TechCenter