how we test @ microsoft

Solving the Software Quality Puzzle www.psqtconference.com

How We Test @ MicrosoftHow We Test @ Microsoft

How Microsoft Tests Software Today How Microsoft Tests Software Today and Where We are Headed Nextand Where We are Headed Next

Ken Johnston, Director Test ExcellenceMicrosoft Engineering ExcellencePSQT North 2006

Microsoft Engineering Excellence


Making the hard callMaking the hard call

Sev 1 BugSev 1 BugCrashes setup and Crashes setup and leaves the machine in leaves the machine in an unknown statean unknown stateOn upgrade install it On upgrade install it corrupts customer datacorrupts customer data

Reported by Reported by customercustomerReported by Dev, Reported by Dev, Test and PMTest and PMLast bug left!Last bug left!

What’s a young test What’s a young test lead to do?lead to do?

Sasquatch



Global CompanyGlobal Company

Multinational workforceMultinational workforceWorldwide customers and Worldwide customers and

partnerspartners



Products in Different CategoriesProducts in Different Categories

260 Products 260 Products Released in last Released in last 12 Months12 Months

There is no 1 There is no 1

““Microsoft Way”Microsoft Way”



There is No One Way to ShipThere is No One Way to ShipMany different ship cyclesMany different ship cycles– Daily/WeeklyDaily/Weekly– MonthlyMonthly– AnnuallyAnnually– Every 2-3 yearsEvery 2-3 years

• More than one product has followed this pathMore than one product has followed this path• Multiple Slip ScheduleMultiple Slip Schedule



Many Development Methods UsedMany Development Methods Used

Spiral Iterative/WaterfallSpiral Iterative/Waterfall

Agile TechniquesAgile TechniquesTest Driven DevelopmentTest Driven DevelopmentSCRUMSCRUM

TSP/PSPTSP/PSPTeam Software Plan & Personal Software PlanTeam Software Plan & Personal Software Plan



What to do with 50 min left?What to do with 50 min left?Look at the people landscapeLook at the people landscape

Cover as many techniques and tools as I canCover as many techniques and tools as I can

Try to squeeze in a little on the future of Try to squeeze in a little on the future of testingtesting

At the end I will point you to additional At the end I will point you to additional resourcesresources

With any luck we’ll have some fun along the With any luck we’ll have some fun along the wayway



People, Process and ToolsPeople, Process and Tools

Testing @Testing @

MicrosoftMicrosoft

PeoplePeople ProcesProcesss

ToolsTools



How many Engineers are There?How many Engineers are There?

More than More than 31,00031,000 Engineers World Wide Engineers World Wide

More than More than 7,5007,500 Test Engineers Test Engineers

What is the dev to test ratio?What is the dev to test ratio?Counting full time and temp/vendor Test Engineers Counting full time and temp/vendor Test Engineers Microsoft has more than 1 Test Engineer for every Microsoft has more than 1 Test Engineer for every DeveloperDeveloper

Hired more than Hired more than 4,0004,000 Engineers FY06 Engineers FY06Hired more than Hired more than 1,000 1,000 SDETs in FY06SDETs in FY06

Too many open positionsToo many open positionsMore than More than 5,000 5,000 open Engineering positionsopen Engineering positionsMore than More than 1,0001,000 open Test Engineer positionsopen Test Engineer positions



SDET ProfileSDET Profile

Mostly Computer Science Graduates & Industry Mostly Computer Science Graduates & Industry HiresHiresAlsoAlso

Electrical EngineeringElectrical EngineeringMathMathPhysicsPhysicsOther EngineeringOther Engineering

We look forWe look forStrong Programming SkillsStrong Programming SkillsProblem Decomposition SkillsProblem Decomposition SkillsDrive for ResultsDrive for ResultsPassion for QualityPassion for Quality

Experience in Test is not the highest order bitExperience in Test is not the highest order bit



Test Techniques Taught in New Test Techniques Taught in New TesterTester

Functional testing Functional testing techniquestechniques

Boundary value analysisBoundary value analysisEquivalence class Equivalence class partitioningpartitioningCombinatorial analysisCombinatorial analysis

Structural testing Structural testing techniques and code techniques and code coverage analysiscoverage analysis

Decision/Branch Decision/Branch coveragecoverageCondition coverageCondition coverageBasis path coverageBasis path coverageCode coverage analysisCode coverage analysis

Integration and system Integration and system level testing level testing

Exploratory testingExploratory testingApplication compatibility Application compatibility testingtestingAccessibility testingAccessibility testingSecurity testingSecurity testingGlobalization testingGlobalization testing

DebuggingDebugging Inspections and ReviewInspections and Review Model Based TestingModel Based Testing



DevelopmentDevelopment

TestTest

The Feature Team TriadThe Feature Team Triad

Program Program

ManagementManagement

Simple DescriptionsSimple Descriptions

Program Managers (PMs) Program Managers (PMs) research customer needs, research customer needs, competitors and develop the competitors and develop the requirementsrequirements

Developers (SDEs)write the Developers (SDEs)write the product code to the product code to the functional requirementsfunctional requirements

Test Engineers (SDETs) Test Engineers (SDETs) ensures the product meets ensures the product meets the functional requirementsthe functional requirements




TestTest

The Quartet for Web ServicesThe Quartet for Web Services

Program Program


IT/OperationsIT/Operations

Operations is critical Operations is critical partner in migration to partner in migration to web servicesweb services

~4,500 IT/Ops Engineers~4,500 IT/Ops Engineers



Many More Engineering Many More Engineering DisciplinesDisciplines


TestTest

Program Program


IT/OperationsIT/Operations

IPEIPE UsabilityUsability

DesignDesignCreativeCreative

ContentContent ResearchResearchEngineeringEngineering




Focus on a Few Processes and Focus on a Few Processes and TechniquesTechniques

Testing @Testing @

MicrosoftMicrosoft


ToolsTools



Run test caseEnter bug

report

Modifydesign

No or refine

Designor code

Design issue

Close bug

Reactivatebug

Investigation approved

Sustained Engineering

Transition Checklist

Design Okay

Regressbug

Pass

Fail

Not approved

Proposed code fix

Proposed design fix

Code issue

Fix approved

ProductCode

Specify code fix

Fix codeResolve

fixed bugBuild

workflow

Triage

Test Execution

Status Report Closethe bug

Resolve "Won't Fix"

Bugs Drive WorkBugs Drive Work



Not all Bugs are Code Not all Bugs are Code DefectsDefects



Perform pre-build

(sync codeand tools)

Perform compile and

code analysis (PREfast)

Perform link and

binary analysis (PREfix)

Product Code

Conduct build verification

tests

Conduct check-in tests

(unit tests)

Conduct post-build (cleanup)

Clean CIT (image, setup,

test)

Private and/or Buddy Build

Self-Test Build

Self-Host Build

Create the checkpoint release (copy source and

components)

Create release build (copy source

and binaries)

The Daily BuildThe Daily Build

The Heartbeat of Microsoft R&D



Build Reporting on Every Build Reporting on Every BuildBuild



Too Many ToolsToo Many Tools

Testing @Testing @

MicrosoftMicrosoft


ToolsTools



Microsoft has a lot of ToolsMicrosoft has a lot of Tools



Active Customer Feedback



Play Balance - Play Balance -

01a_look_start

01a_move_start

01a_shield_start

01a_tram_start

01b_1st_start

01b_1st_wv4

01b_1st_wv5_end

01b_atr2_start

01b_atr2_mid

01b_bay1_start

01b_bay1_mid

01b_bay2_start

01b_bay2_mid

01b_arm_start

01b_atr1_start

01b_trm1_start

01b_dck_start

01b_elv_start

01b_lvr_start

01b_gun_start

e1_rooftop_defense

e2_hunter_intro

e3_highway_underpass

e4_sniper_alley

e5_neighborhood

e6_hotel_entrance

e7_hotel_hall

e8_hotel_exit

e9_beach_infantry

e10_beach_fort

e11_tunnel_entrance

e12_tunnel_blockades

e13_tunnel_convoy

e14_bridge_ascent

e15_bridge_summit

e16_bridge_descent

e17_flooded_tunnel

e18_park

enc_ledge_lift

enc_ledge_lower

enc_high_hall_a

enc_high_hall_b

enc_room_a

enc_jail_ini

enc_jail_cell_a

enc_jail_cell_b

enc_jail_down_lift

enc_corridors_b

enc_tower_a_exterior

enc_gardens_a

enc_mid_tower

e4_generator_room

e2_scarab_cove

e19_park

e20_road_skirmish

e21_hospital_seige

e22_scarab_intro

e23_scarab_boarding

04a_1_landing_zone

04a_2_recycling_center

04a_3_hangar

04a_4_first_halls

04a_5_underhangar

04a_6_second_halls

04a_7_bottling_plant

04a_8_bottling_back

04a_9_dogfight

04b_1_entry

04b_2_disposal_chamber

04b_3_silo_start

04b_4_silo_mid

04b_5_silo_end

04b_6_hall_to_lab

04b_7_flood_lab_start

04b_8_flood_lab_mid_01

04b_9_flood_lab_mid_02

04b_10_flood_lab_end

04b_11_wraparound

04b_12_control_room_01

04b_13_control_room_02

04b_14_cable_room

04b_15_control_return

04b_16_power_core

04b_17_dogfight

04b_18_bottling_return_01

04b_19_bottling_return_02

04b_20_halls_to_under_01



04b_23_underhangar_return

04b_24_boss_fight_01



enc_maus_grand

enc_maus_bridge

enc_mausoleum

enc_maus_midpoint

e1_stealthy_insertion

e2_cylinder_room

e3_ramp_room0

e4_step_room

e5_ramp_room1

e6_ghost_ledge

e7_kidney0

e8_kidney1

e9_ski_run





05a_1_landing_zone

05a_2_overlook

05a_3_bridge

05a_4_winding_path

05a_5_temple_entrance

05a_6_tunnel

05a_7_old_temple

05a_8_old_temple_courtyard

05a_9_grotto

05a_10_tower_approach

05a_11_tower1_interior

05b_1_tower1

05b_2_bridges

05b_3_tower2

05b_4_central_platform

05b_5_gondola_01

05b_6_tower3

05b_7_tunnel1

05b_8_sunken_chamber

05b_9_tunnel2

05b_10_island_interior

05b_11_island_gully

05b_12_island_exterior

05b_13_gondola_02

05b_14_temple_exterior

05b_15_temple_halls

05b_16_temple_center

05b_17_temple_retreat

enc_insertion

enc_hall_a

enc_cond_a_sec_a

enc_cond_a_sec_b

enc_hall_b

enc_plug_launch

enc_plug_across

enc_plug_landing

e5_bridge

e6_jail

e7_scarab_ledge

e8_scarab_cove

e9_scarab_canyon

e11_scarab_boardings

e12_antechamber

e13_control_room

enc_gardens_b

enc_tower_b_exterior

e10_brute_camp

phantom_ledge

e3_ramp_room

enc_plug_landing_bk

enc_plug_landing_lower

enc_hall_c_sec_a

enc_hall_c_sec_b

enc_hall_c_sec_c

enc_ledge_a_sec_a

enc_ledge_a_sec_b

enc_cond_b_sec_a

enc_cond_b_sec_b

enc_cond_b_sec_c

enc_ledge_b

enc_ledge_c

enc_qz_initial_camp

enc_covenant_defense

enc_cov_charge

enc_vehicle_int

enc_vehicle_int_bk

enc_qz_ext_a_dam

enc_qz_ext_a

enc_ext_a_fact_ent

enc_crashed_factory_a

enc_crashed_factory_ext

enc_crashed_factory_b

enc_ext_b_fact_exit

enc_qz_ext_b

enc_qz_ext_b_bk

enc_qz_ext_b_exit

enc_e21

enc_e22

enc_e23

enc_e24

enc_e25

enc_council_wave_1

enc_council_pedestal

enc_council_wave_2

enc_jail_cell_a

enc_jail_cell_b

enc_council_ini

enc_grand_a

enc_ledge_ini

turbolift_base

plasma_chandelier

riptide_start

riptide_ambush

space_cave1

boobag_garden

space_cave2

hallway_attack

inner_sanctum

council_chamber

last_room

e1_wraith_canyon



Halo 2 Playtest DataHalo 2 Playtest Data

0

2

4

6

8

10

12

enc_

cov_

char

ge

enc_

veh

icle

_in

t

enc_

veh

icle

_in

t_b

k

enc_

qz_

ext_

a_d

am

enc_

qz_

ext_

a

enc_

ext_

a_fa

ct_e

nt

enc_

cras

hed

_fac

tory

_a

enc_

cras

hed

_fac

tory

_ext

enc_

cras

hed

_fac

tory

_b

enc_

ext_

b_f

act_

exit

enc_

qz_

ext_

b

enc_

qz_

ext_

b_b

k

enc_

qz_

ext_

b_e

xit

enc_

e21

enc_

e23

enc_

e25

Median

Mean

“unknown” deaths

≈ 8+ hours in…



Software Quality Metrics (SQM)Software Quality Metrics (SQM)

Software Quality Metrics / Software Quantitative Software Quality Metrics / Software Quantitative MechanicsMechanics

Client-side tool which measures statistical usage Client-side tool which measures statistical usage datadata

Server-side collection, reporting and management Server-side collection, reporting and management tooltool

Shared SAN Datawarehouse

Client SQMIntegrator

Front EndCluster

ReportingCluster

Dedicated Processing

DedicatedDatawarehouse

Shared Processing



O11 Beta2

O11 Beta 2TR

O11 RTM Early

O11 RTM All

O11 SP1 All

M1 Dogfood

M2 Dogfood

SQM in Office 2003 and Office SQM in Office 2003 and Office 1212

Mean

Tim

e T

o C

rash

(hours) 0

100

200

300

400

500

600

700

800

900

One Note OutlookApplication

Word



Windows Error Reporting (a.k.a. Windows Error Reporting (a.k.a. Watson)Watson)

Real-world data about user painReal-world data about user pain– Millions of hits every dayMillions of hits every day

Walt Mossberg 9/30/99:Walt Mossberg 9/30/99: “ “Dear Diary: I’m tired of the way Windows crashes and freezes…”Dear Diary: I’m tired of the way Windows crashes and freezes…”



Continuous Feedback ProcessContinuous Feedback Process



Tests and Test Tools

Bug ManagementBug Management

Test Case ManagementTest Case Management

Automation FocusAutomation Focus

Static Code AnalysisStatic Code Analysis

Dynamic AnalysisDynamic Analysis

Unit TestsUnit Tests

Code CoverageCode Coverage

Performance/LoadPerformance/Load

Web Load TestingWeb Load Testing

Model Based TestingModel Based Testing

Threat ModelingThreat Modeling

Fuzz TestingFuzz Testing



For Bugs We Use Product For Bugs We Use Product StudioStudio



Test Case ManagementTest Case Management

Test Case ManagementManual and Automated in one system

Test Case ManagementManual and Automated in one system



Microsoft has an Automation Microsoft has an Automation FocusFocus

Windows Vista + Office 12 > 200 MLOCWindows Vista + Office 12 > 200 MLOC

Most products target ~70% code coverage before Most products target ~70% code coverage before shipship

Some as high as 80% on unit testsSome as high as 80% on unit tests

Microsoft has >110,000 Test MachinesMicrosoft has >110,000 Test MachinesFY07 lab budgets 100 million+FY07 lab budgets 100 million+

Windows Vista has over 6.7 Million automated test Windows Vista has over 6.7 Million automated test jobsjobs

Office 12 has over 1 Million automated test casesOffice 12 has over 1 Million automated test cases

Why Automate so Much?Why Automate so Much?Many products must be supported for 10 yearsMany products must be supported for 10 yearsCompatibility regression passesCompatibility regression passesDaily build process increases reuseDaily build process increases reuse



Every SDET Learns to Every SDET Learns to useuseDebuggersDebuggers



Static Code AnalysisStatic Code Analysis

Static Code AnalyzerStatic Code Analyzer



Dynamic Analysis – Dynamic Analysis – AppVerifier and TruScanAppVerifier and TruScanMemory Leak CheckerMemory Leak Checker

Unutilized Variable CheckerUnutilized Variable Checker

Race Condition CheckerRace Condition Checker

Handle leak checkerHandle leak checker

Buffer Overrun checkerBuffer Overrun checker

Dangling Pointer checkerDangling Pointer checker



Unit TestingUnit Testing

Integrated Unit TestingIntegrated Unit Testing



Code CoverageCode Coverage

Code CoverageUnit, BVT, Suite, All

Code CoverageUnit, BVT, Suite, All



Performance measures of test and Systems Under Test

Alerts and warnings on Systems Under Test

Load GenerationLoad Generation



Timeline of memory consumption

Suspect functions, drillable to code

Code PerformanceCode Performance



View of content as rendered

Content validation

http request & response

Performance breakdown

Data substitution

Web TestingWeb Testing



Model Based TestingModel Based Testing

Significant growth in Significant growth in use of Model Based use of Model Based TestingTesting

CoursesCoursesLecturesLecturesE-LearningE-LearningActive communityActive community

MDE – Model Design MDE – Model Design EnvironmentEnvironment

Spec ExplorerSpec Explorerhttp://research.microsoft.com/SpecExplorer/Built on Spec Sharp Built on Spec Sharp (extension to C#)(extension to C#)http://research.microsoft.com/specsharp



Threat Modeling & Fuzz TestingThreat Modeling & Fuzz Testing

Threat Modeling - analyze the design for vulnerabilityThreat Modeling - analyze the design for vulnerabilityModel data flows Model data flows (How to Write Secure Code)(How to Write Secure Code)

SS - Spoofing Identity- Spoofing IdentityTT - Tampering with Data- Tampering with DataRR - Repudiation- RepudiationII - Information Disclosure- Information DisclosureDD - Denial of Service- Denial of ServiceEE - Elevation of Privilege- Elevation of Privilege

Fuzz testing is a method of finding software security holes by Fuzz testing is a method of finding software security holes by feeding purposely invalid and ill-formed data as input to program feeding purposely invalid and ill-formed data as input to program interfaces. Inputs include:interfaces. Inputs include:

FilesFilesNetwork portsNetwork portsAPIsAPIs



A Few Other QA Processes



Global Software CompanyGlobal Software Company

Ship Products World WideShip Products World WideLocalize products in >50 LanguagesLocalize products in >50 LanguagesFocus of IPE (formerly Localization) disciplineFocus of IPE (formerly Localization) discipline

Globalization/Global SufficiencyGlobalization/Global SufficiencyLocalizationLocalizationMarketizationMarketizationGeopolitical Geopolitical



LocStudio – the Tool for IPELocStudio – the Tool for IPE



Who’s Behind the One-way Who’s Behind the One-way Mirror?Mirror?

What do we get from Usability?What do we get from Usability?

provides perspective onto the provides perspective onto the

total situationtotal situation

reveals nuances of behavior, reveals nuances of behavior,

motivation, and emotion motivation, and emotion

enables comparison and enables comparison and

contrasting of attitudes, actions, contrasting of attitudes, actions,

and feelingsand feelings

uncovers latent needs users uncovers latent needs users

can’t articulatecan’t articulate

fosters user inspired empathy, fosters user inspired empathy,

advocacy, and creativityadvocacy, and creativity



Usability Labs Without WallsUsability Labs Without Walls

Virtual Research LabVirtual Research Lab



Where are We Headed Next?



Metrics then, now, and…Metrics then, now, and…

Product Size 1995 Office 95

MLOC – ~4MMLOC – ~4MTC – UnkTC – UnkBugs – 268KBugs – 268K

Windows 95

MLOC – ~7MMLOC – ~7M

TC TC –– Unk UnkBugs – >5*25KBugs – >5*25K

Data Sources 2006

Product Studio Test Case P/FLarge Checkin Suites

Automated per build BVTsAutomated test case results

Automatic bug filing

Stress, Scale, Perf, Icecap

PSSGranular call coding

Customer impact and cost

Customer feedbacksystems

Watson, SQM

Static AnalysisPreFix, PreFast, FXCop

Dynamic AnalysisApp Verifier, Driver Verifier

Magellancode coverage,

code churn, code complexity

Product Size 2006 Office 12

Blocks – 19.1MBlocks – 19.1MTC – ~1MTC – ~1M

VistaBlocks – 34.7MBlocks – 34.7MWTT Jobs – 6.7MWTT Jobs – 6.7M

Fast forward 11 years

Data Sources 1995

RAIDTest Case

Simple DB

Word Docs

Excel Files

Stress, Scale, Perf

PSS



Metrics next…Metrics next…Move from Quality Control to Move from Quality Control to Quality AssuranceQuality Assurance

QC is fixing the bugs we findQC is fixing the bugs we findQA is setting the goals for QA is setting the goals for key indicators that we know key indicators that we know will ensure high qualitywill ensure high quality

Data warehouse projects in Data warehouse projects in many product teamsmany product teams

PHI, Team Stats, MSN QA PHI, Team Stats, MSN QA Dashboard, MSN QOS, Dashboard, MSN QOS, Exchange, Office, Exchange, Office, Cross-correlation of data across Cross-correlation of data across large numbers of productslarge numbers of productsAuto-analysis of dataAuto-analysis of data



Testability – SOCK Testability – SOCK SimplicitySimplicity & Consistency & Consistency

Reducing cost of testing at each phase of SEARCHReducing cost of testing at each phase of SEARCHLess to test, more quickly we can test.Less to test, more quickly we can test.DesignDesignSimilar parts behave in a similar mannerSimilar parts behave in a similar manner

Design & Test Patterns

Standards, e.g. GUI, Design Guidelines

Naming conventions

ObservabilityObservabilityExposing state (aka Visibility or Transparency)Exposing state (aka Visibility or Transparency)

ControlControlExercising every nook & cranny as efficiently as possibleExercising every nook & cranny as efficiently as possible

Knowledge of expected results (aka Documentation)Knowledge of expected results (aka Documentation)Is observed behavior correct?Is observed behavior correct?



Getting Back into Code ChurnGetting Back into Code ChurnRecent research has proven Code Recent research has proven Code ChurnChurn

Windows Research showed correlation to Windows Research showed correlation to post ship defect densitypost ship defect density

Code Churn can be pulled fromCode Churn can be pulled fromSource Depot (source control)Source Depot (source control)Magellan (code coverage tool)Magellan (code coverage tool)

Not used to stop shipNot used to stop shipGood tool for risk indicationGood tool for risk indication



Tempest: Consolidated Complexity Tempest: Consolidated Complexity MetricsMetricsMeasures failure proneness based on statistical analysis of Measures failure proneness based on statistical analysis of previous failuresprevious failures

Leverages size, complexity, and coupling metrics to model Leverages size, complexity, and coupling metrics to model failuresfailures

Gives confidence bounds for code metrics Gives confidence bounds for code metrics collected through the binary and pdbcollected through the binary and pdb

Scalable and runs on the desktopScalable and runs on the desktopComplete Vista build can be Complete Vista build can be analyzed in 1.25 hours.analyzed in 1.25 hours.

Internal tool only right nowInternal tool only right now



Auto AnalysisAuto Analysis

110,000 Test 110,000 Test MachinesMachines

Millions of Millions of Automated Test Automated Test CasesCases

>500 Builds a day>500 Builds a dayThe volume of defects keeps increasingThe volume of defects keeps increasing



ResourcesResourcesAbout Testers (SDETs) at MicrosoftAbout Testers (SDETs) at Microsoft

Recruiting - Recruiting - http://www.http://www.microsoftmicrosoft.com/college/ft_.com/college/ft_softdesengtestsoftdesengtest..mspxmspx

http://members.http://members.microsoftmicrosoft.com/careers/.com/careers/careerpathcareerpath/technical//technical/softwaretestingsoftwaretesting..mspxmspx

Public Research Papers - Public Research Papers - http://research.http://research.microsoftmicrosoft.com.com

Some of the Tools from this presentationSome of the Tools from this presentation

•• Visual Studio - Visual Studio - http://http://msdnmsdn..microsoftmicrosoft.com/.com/vstudiovstudio//teamsystemteamsystem/tester//tester/

http://msdn2.http://msdn2.microsoftmicrosoft.com/en-us/default..com/en-us/default.aspxaspx

AppVerifier AppVerifier –– http://www.http://www.microsoftmicrosoft.com/windows/.com/windows/appexperienceappexperience

FXCOP – FXCOP – http://www.http://www.gotdotnetgotdotnet.com/team/.com/team/fxcopfxcop//

Blogs and Community ResourcesBlogs and Community Resources

Wiki content and video profiles - Wiki content and video profiles - http://channel9.http://channel9.msdnmsdn.com/.com/Bug Triage Video - Bug Triage Video - http://channel9.msdn.com/ShowPost.aspx?PostID=26792#26792 http://channel9.msdn.com/ShowPost.aspx?PostID=26792#26792



Ken Johnston, Director Test ExcellenceMicrosoft Engineering [email protected]



ResourcesResources

Research Papers (Nachi Nagappan)Research Papers (Nachi Nagappan)http://teamv2/sites/tvm http://teamv2/sites/tvm

Consolidated reportingConsolidated reporting

http://teamstatshttp://teamstats

http://phihttp://phi

http://office/c16/metricshttp://office/c16/metrics

This slide deck and related resources:This slide deck and related resources:http://eetwcforumhttp://eetwcforum



FxCop ResourcesFxCop Resourceshttp://www.gotdotnet.com/team/fxcop/http://www.gotdotnet.com/team/fxcop/

FxCop download site/bulletin boardFxCop download site/bulletin boardFxCop docs and rule topicsFxCop docs and rule topicsSupportSupport

[email protected]@Microsoft.com

AppVerifier ResourcesAppVerifier ResourcesDownload Site: Download Site: http://www.microsoft.com/windows/appexperiencehttp://www.microsoft.com/windows/appexperience

Newsgroup:Newsgroup:microsoft.public.win32.programmer.toolsmicrosoft.public.win32.programmer.tools http://msdn.microsoft.com/newsgroups/managedhttp://msdn.microsoft.com/newsgroups/managed



General Security ResourcesGeneral Security ResourcesGeneral General

http://www.microsoft.com/security http://www.microsoft.com/security

XP SP2 Resources for the IT Professional XP SP2 Resources for the IT Professional http://www.microsoft.com/technet/winxpsp2 http://www.microsoft.com/technet/winxpsp2

Security Guidance Center Security Guidance Center http://www.microsoft.com/security/guidance http://www.microsoft.com/security/guidance

Tools Tools http://www.microsoft.com/technet/Security/tools http://www.microsoft.com/technet/Security/tools

How Microsoft IT Secures Microsoft How Microsoft IT Secures Microsoft http://www.microsoft.com/technet/itsolutions/msit http://www.microsoft.com/technet/itsolutions/msit

E-Learning Clinics E-Learning Clinics https://www.microsoftelearning.com/security https://www.microsoftelearning.com/security

Events and Webcasts Events and Webcasts http://www.microsoft.com/seminar/events/security.mspx http://www.microsoft.com/seminar/events/security.mspx



Additional DocumentationAdditional Documentation

Writing Secure Code: Writing Secure Code: http://www.microsoft.com/MSPress/books/5957.asp http://www.microsoft.com/MSPress/books/5957.asp

Thread Modeling: Thread Modeling: http://www.microsoft.com/MSPress/books/6892.asphttp://www.microsoft.com/MSPress/books/6892.asp

Threat Modeling OnlineThreat Modeling Onlinehttp://msdn.microsoft.com/security/securecode/http://msdn.microsoft.com/security/securecode/

threatmodeling/default.aspxthreatmodeling/default.aspx Compiler Security Checks In Depth Compiler Security Checks In Depth http://msdn.microsoft.com/library/default.asp?url=/library/en-http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dv_vstechart/html/vctchcompilersecuritychecksindepth.aspus/dv_vstechart/html/vctchcompilersecuritychecksindepth.asp

how we test @ microsoft

Documents