Sylvain GoyetteScenario Adoption PMMicrosoft Corporation

TOL-147T

Building high-quality Windows PCs using the Assessment and Deployment Kit

Agenda

• ADK introduction

• Assessment tools scenarios• Assessment investments

• Windows Assessment Console

• Windows Assessment Services

• Assessment execution engine

• Windows Performance Toolkit (WPA/WPR)

Assessment and Deployment Kit

Assessment and Deployment Kit

• Consolidate OPK toolkit and WAIK• Core deployment tools and assessment toolkit

• Create, assess, and maintain quality Windows images• How we measure the PC today lacks the needed customer

view• Processor speed, megahertz, gigabytes, etc.• How exactly does this relate to an individual’s experience?

• The ADK system assessment tools help engineers deliver high-quality, differentiated devices that measure system quality through a customer experience

ONE-STOP SHOP FOR ALL PARTNERS

System assessment basics

• System assessment is a process that uses the ADK tools to measure and analyze a PC

• Assessments are core functionality tests• Combinations of these tests provide additional measures of

the entire PC experience• Quality expectations are changing• Software + hardware + Windows = PC experience• The way we measure PC quality must also change

Benchmarking doesn’t go far enough

• Advantages• Accepted and easy to discuss by techies• Used as part of certification or comparison• Widely adopted, especially in the press

• Disadvantages• Provided with little context on how benchmarks are

created • A “black box” to users• Less useful in system engineering

• Memory footprint• First boot experience• Media streaming• WinSAT comprehensive• Battery life (and idle efficiency

analysis)• MiniFilter driver performance

impact (option for other assessments)

• Internet browsing workload for battery life assessment

• Windows Media Player performance and quality

System assessments

• CheckLogo and driver assessments

• File handling• Photo handling• Internet Explorer launch/tab

create• Hybrid boot• On/off assessments

(boot/shutdown/S3/S4)• Browser assessment• Media transcode performance• Metro performance

Assessment tenets

• System quality assessments should:• Focus on assessing the end user’s experience, not just

the underlying technologies• Assess system quality in a quantifiable way• Allow for system comparisons• Identify issues and provide remediation steps wherever

possible• Provide the ability to dig deeper using tools like Windows

Performance Analyzer • Make collaboration easier• Be widely available

Windows assessment tools

Evolution of tools

Velocity Test Suite

(VTS / VOS)

Velocity (VTS)

Fundamental Quality Tools

Suite(FQTS)

System Power and

Performance Tools Suite(SPPTS)

SPPTS(WLK 1.5)

FQTSv3

ADK assessment tools

Velocity 2.x (VOS) Velocit

y 3.x (VOS)

Velocity 3.x (VTS)

SPPTS(WLK 1.4)

Velocity 4.x

FQTS2.x

Overview of problems with existing tools

• Existing tools reuse solutions designed to assess quality of single computer

• Introduce significant manual work, which grows as number of computers increases:

• Computer preparation (image deployment and drivers installation)

• Assessment execution and monitoring progress• Collection of results from computers

ASSESSING QUALITY OF MULTIPLE COMPUTERS

Two ways of using assessments

SINGLE MACHINE AND BASIC ARCHITECTURE LAB SCENARIOS AND SCALABILITY

What You Get in the Windows ADKA

DK

Ass

ess

ment

and D

eplo

ym

ent

Kit

WACWindows Assessment

Console

Used to access, configure, create, save, package jobs to run on different machines. Good for running assessments on one to a few machines. Used to view results from assessment run

WPAWindows Performance

Analyzer

Remediation tool used for analysis of performance traces

WASWindows Assessment

Services

Used for larger environments with the potential of running assessments on hundreds of computer

Assessments

Scenario-based tests of system quality, delivering metrics, identifying issues, and providing remediation steps wherever possible

Three key concepts

1. Jobs run, not assessments • Jobs contain one or more assessments• Jobs contain the parameters that are used to configure individual

assessments (as well as a few job parameters)

2. All jobs run in administrator mode• It’s a requirement of the Assessment Execution Engine (AXE)

3. It’s the assessment that has the “power”• The GUI and platform provide a test harness to expose things the

assessment wants to expose—it’s the assessment that does (and should do) most of the heavy lifting, analysis, and subject matter expertise

Assessments, jobs, workloads, and the Windows Assessment Console

demo

Assessment infrastructure

WINDOWS ASSESSMENT SERVICE (WAS AND WASC) – LAB ENVIRONMENT

• Alignment with WAC on the four primary goals

• Extends concept to distributed environment with multiple assets (machines, images, assessments)

• Adds capability to manage assets, monitor progress on multiple jobs and assets, and facilitate centralized diagnosis

Manage

DiscoverMonitor

Diagnose

Compose Execute Interpret

Assessment Execution Engine

• Small set of binaries to support “platform”• XML-driven definition and configuration

• Assessment manifest defines the assessment, including parameters, metrics, thresholds, etc.

• Job manifest defines the job (which assessments to run and parameter values)

• Results file defines the output• Each “subsequent” XML contains the previous files

in their entirety• Ability to launch any code/script/process

AXE core components:• AXE.exe• AL.exe• Etc.

AXE core components:• AXE.exe• AL.exe• Etc.

Assessment resources:• asmt.exe• Assessmentmanifest.x

ml• Axecore.dll• Other resources

(including samples)

Assessment resources:• asmt.exe• Assessmentmanifest.x

ml• Axecore.dll• Other resources

(including samples)

Integration goals

• Modular design provides multiple ways to extend and varying levels of integration

• Provide both interactive and automated tools• APIs and schema will be made available publicly

• Both native and managed execution APIs for running assessments• Assessment, job, and results XML schema definitions• Advanced managed administrative APIs for creating XML

documents

We used these same APIs and schemas to produce the tools and assessments delivered in the kit.

Ways to integrate and extend

• Convert or build your own assessments• Automate execution in your existing infrastructure• Import the results• Create jobs and packages

Windows Assessment Services (WAS)

Windows Assessment Services

• Infrastructure to support lab environments

• Automation to support large scale deployment and assessment of many machines

• Results database allows for system comparison, analysis, and reporting

Automation for the lab environment

• Computer preparation• Using Windows Deployment Services (WDS) and customized WinPE

image

• Assessment execution• Using Windows Remote Management (WinRM) to copy assessment

execution engine + assessments to computers and run them

• Collection of results from computers• Using WinRM to copy results to lab execution server and

preprocess them

SUPPORTS MULTIPLE COMPUTER SCENARIOS

End-to-end workflow

Create lab

execution USB

thumb drive

Boot computer

s from USB and add them

to inventory

Add images to inventory

Create a new

project

Create a new job

with selected computers, images,

and assessme

nts

Run the job

View results

What WAS is not?

• Assessment execution in the cloud / over the Internet. It is designed to work within a single subnet.

• Windows Hardware Certification framework• Testing computers on manufacturing floor• Large scale testing (there is an limit of 300 computers)• Testing computers that are already shipped and are in use

by end users• Debugging / fixing issues that are found by assessments• Issue management

WAS results database

• Provide an optional platform for aggregated results collection based on the AXE assessment results schema

• Focus on results collection for trending, comparison, and business reporting needs across large amounts of results data

• By contrast, WAC is focused on engineering views of assessment results for immediate analysis and triaging purposes

Windows Performance Toolkit (WPT)

Benefits of WPT

• Enables follow up on issues raised via ADK performance assessments• Enables holistic performance analysis

• All processes/threads• All key system resources• User and kernel mode

• Provides a way to analyze what customer actually experiences• Catch the problem as it happens• Capture-Anywhere-Decode-Anywhere

• Very low overhead versus alternatives

• Region of interest profiling• You get both what and when

Windows Performance Recorder and Analyzer

• Tools for: • Trace capture (WPR)• Trace analysis (WPA)

• Based on ETW (event tracing for Windows)

• Used within Microsoft to build Windows

• Core code reused in-box

• Documentation on MSDN

• Part of ADK and SDK in Windows 8

High-level changes

• Windows Performance Analyzer replaces xperview • Windows Performance Recorder replaces xbootmgr• Also replaces xperf trace capture functionality

• xperf command line actions remain in WPT

xperf in Windows 7

Benefits of WPA in Windows 8

• Broaden the target audience to mainstream developers• Leverage familiar UI experiences• Enable quick ramp up• Reduce “noise” to focus on the “right” issues

WPA in Windows 8

Graph Explorer

Detailed Graph

Graph

Table

Legend

Gold Bar Blue Bar

Grouping Columns

Aggregating

Columns

Graphing Columns

Call to action

• Evaluate the ADK assessment tools (file bugs, send feedback) until Beta

• Work with your local contact to start planning your implementation

• Integrate ADK assessments into your test processes to optimize user experience

• Training and evangelization will be communicated

• POC for adoption: adkcontact@microsoft.com

REQUESTS AND TIMELINE

Further reading and documentation

Event Site:• http://channel9.msdn.com/Ev

ents

Dev Center:• Windows Performance Analysis

MSDN documentation:• WPA and WPR• Event Tracing for Windows• Assessment and Deployment Ki

t• Assessment Execution Engine

Videos• Channel9 – BUILD ADK Videos

Books:• Windows Internals by Mark

Russinovich, David Solomon, and Alex Ionescu

Reference:• ADK_TechnicalReference.chm in

the ADK

Related sessions from //build conferenceGo to - www.buildwindows.com/sessions and search for these numbered sessions.

• [HW-141T] Reducing the Memory Footprint of Drivers and Apps• [HW-147T] Building High Quality Windows PCs using the Assessment

and Deployment Kit• [HW-148P] Scaling and Extending Windows Assessments to Improve

System Quality• [HW-149P] Improving Battery Life and Energy Efficiency• [HW-459T] Delivering Great Performance and Battery Life• [HW-59T] Improving Performance with the Windows Performance

Toolkit• [HW-915P] Introduction to Assessments• [HW-916P] Introduction to the Boot Performance Assessment• [HW-917P] Introduction to the IE Launch Assessment• [HW-920P] Introduction to the Streaming Assessment

Related sessions from //build conference (cont.)Go to - www.buildwindows.com/sessions and search for these numbered sessions.

• [HW-921P] Introduction to ADK Custom Jobs• [HW-922P] Capturing and Analyzing Performance Traces• [HW-925P] Customizing WPA Trace Views• [HW-926P] Introduction to the New WPA User Interface• [HW-927P] Understanding Pool Usage using WPA• [HW-928P] Understanding Heap Data using WPA• [HW-975P] Customizing ADK Assessment Reports• [HW-976P] Packaging Assessments for Use on a Second Computer• [HW-977P] Understanding VirtualAlloc Usage using WPA

Thank You!

For questions, please visit me in the Speakers Connection area following this session.

© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to

be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.