running a successful beta test program
DESCRIPTION
Running a Successful Beta Test Program. Adam Long (CTO) Darren Gage (QA Mgr) Sam Soubra (Dev Mgr) September 2010. Agenda. About Us Running a Beta Test Program Runtime Intelligence Service Overview Code Demo Questions. About our Company. About our Software. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/1.jpg)
Running a SuccessfulBeta Test Program
Adam Long (CTO)Darren Gage (QA Mgr) Sam Soubra (Dev Mgr)
September 2010
![Page 2: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/2.jpg)
Agenda
• About Us• Running a Beta Test Program• Runtime Intelligence Service– Overview– Code Demo
• Questions
![Page 3: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/3.jpg)
About our Company
Established in 1995
Independent Software Vendor
based in Melbourne, Australia
90% of revenue from exports
> 400,000 customers in 150 countries
![Page 4: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/4.jpg)
About our Software
• Qualitative Data Analysis– Unstructured Data
• Documents• Audio / Video• Images• Spreadsheets
• What MS Excel does for structured numerical data, we do for unstructured non-numerical data
![Page 5: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/5.jpg)
About our Software
• NVivo 9– .NET 3.5
• Win Forms• WPF• WCF
– SQL Server 2008 R2 Express
• NVivo Server 9– Silverlight 3– SQL Server 2008 R2
Standard
![Page 6: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/6.jpg)
Beta Test
What• Beta testing is conducted by
a sample of customers to identify flaws in feature complete software
• Generally occurs after alpha testing (unit & system)
Why• Validate objectives
– Market– Features– Quality
• Improve outcomes– Collaboration / Innovation
• Assist marketing– Quotes / Case Studies
• Plan future direction– Opportunities
![Page 7: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/7.jpg)
7 Steps of a Beta Test Program
Plan
2 weeks
Promote
4 weeks
Select
2 weeks
Collect
4 weeks
Analyse
2 weeks
Prioritize
1 week
Recognize
1 week
![Page 8: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/8.jpg)
1. Plan
• Scope– Goals & Success Criteria
• Approach– Open vs. Closed
• Responsibilities– Management, Communications, Defects, Suggestions
• Schedule– Balance between having enough features available to
test versus enough time to incorporate feedback
![Page 9: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/9.jpg)
2. Promote
• Internal– Sales, Marketing, Training
• External– Existing Customers– Potential Customers
• Website & Social Media announcements
• Customer Newsletter• Email Select Customers
• Best Beta Testers– Interested– Independent– Vocal
• Avoid pressure to include– Sales Leads– Evaluators
![Page 10: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/10.jpg)
3. Select
• Online Application– Outline Program Expectations– 20 Questions
• Technical Environment• Market Segment• Research Background• Competitor Software• Motivation
• Result– 523 Applicants– 78 Testers Selected
• Selection Process– Top Previous Beta Testers– Top Forum Posters– Subject Matter Experts– Random Applicants
• Simulation Program to Optimize Spread of Answers to Questions
• Obtain Signed Agreement
![Page 11: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/11.jpg)
4. Collect
• Online Webcasts (Citrix GoToWebinar)– 5 recorded videos & 88 feedback forms
• Online Forums (IP.Board)– 412 suggestion posts & 335 defect posts
• Online Survey (Survey Monkey)– 58 survey responses
• Application Analytics (RIS)– 2,448 hours of usage statistics
![Page 12: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/12.jpg)
5. Analyse
• Identify– Defects– Suggestions
• Understand– Most liked features– Most disliked features– Most important
improvements needed
• Dogfooding– Used NVivo Beta to
analyse feedback• Videos• Forums• Surveys
– In conjunction with• Application Analytics
![Page 13: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/13.jpg)
6. Prioritize
• Prioritize Suggestions– 1 Must Address for RTM– 2 Preferably Address for RTM– 3 Consider for Service Pack– 4 Defer for Future Release– 5 Rejected
• Raise Defects in Bug Tracking
![Page 14: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/14.jpg)
7. Recognize
• Active Participants (85%)– Activated the software & provided some feedback– Average of 37 hours testing over 4 weeks– Receive free copy of software
• Top 20 Participants (25%)– Also receive Amazon voucher
![Page 15: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/15.jpg)
Beta Test Tips
• Allocate sufficient time & resources• Observe usage patterns using Application
Analytics and encourage testers to explore specific features
• Reprioritize effort based on final feedback• Hold weekly status meetings• Communicate outcomes to internal
stakeholders
![Page 16: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/16.jpg)
Runtime Intelligence – What is it?
• The purpose of runtime intelligence as a component of SDLC management is to support better development, support and R&D investment decision making.
• Leverages technologies, managed services and practices for the collection, integration, analysis, and presentation of application usage levels, patterns and practices
![Page 17: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/17.jpg)
Runtime Intelligence – Why use it?Gauge
Effectiveness of the beta
test
Collection Phase
Feature usage
Analysis Phase
Platform usage
Beta tester effectiveness
![Page 18: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/18.jpg)
Runtime Intelligence – Why use it?
Improve Decision Making
Usage context while Analysing
feedback
Prioritization PhaseAnalysis
Phase
Usage context while Prioritising
feedback
Understand Quality
![Page 19: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/19.jpg)
Runtime Intelligence – Tool Choice
• PreEmptive RIS– Features matched
those in competitor’s offerings
– We had used other PreEmptive tools (Dotfuscator)
– Integrated into Visual Studio 2010
• Other tools out there;– EQATEC Analytics– Gibraltar Software
![Page 20: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/20.jpg)
Runtime Intelligence – Architecture
11
2
2
3
3
Instrumented App is run
Usage data sent to Data Repository
Dashboard and Reports
![Page 21: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/21.jpg)
Runtime Intelligence – Implementation
• Credentials, Entry point, Endpoint & Security
• Features & Custom data
• System and Performance
• Opt-in policy
Configure analytics
• Inject Instrumentation
• Package and Deploy
Application distribution
• Beta Mgmt., UX program, Support
• Business Intelligence & ROI
• Operations Mgmt.
Runtime Intelligence Services
![Page 22: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/22.jpg)
Runtime Intelligence – Services
Preemptive Cloud
Yours
Aggregate
Validate
ManagePublish
Runtime Intelligence endpointRuntime Intelligence repository
Aggregate, validate, manage, & publishManaged or on-premise
For internal dev. and multi-tenant constituencies
Runtime IntelligenceReports & Dashboards
Owner, application, feature, user, & customRESTful API and export
IDE, CRM, ERP, & IT Operations
![Page 23: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/23.jpg)
Runtime Intelligence – Reports and Dashboard
• PreEmptives RIS Portal
![Page 24: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/24.jpg)
Runtime Intelligence – Export
• Data is rich
• You will need to spend time to mine it!
![Page 25: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/25.jpg)
Runtime Intelligence – Export
Circle Graph Dendrogram Horizontal Dendrogram Vertical Mds 2d Mds 3d
Total 36 65 52 82 92
5
15
25
35
45
55
65
75
85
95
Similarity Diagram Type
# us
es o
f fea
ture
s
![Page 26: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/26.jpg)
Runtime Intelligence – Export
Windows 7 Enter-
prise
Windows 7 Home
Premium
Windows 7 Profes-
sional
Windows 7 Ultimate
Windows XP Home Edition
Windows XP Pro-fessional
Windows(R) XP Pro-fessional x64 Edi-
tion
Windows Vista
Business
Windows Vista Home
Premium
Windows Vista
Starter
Windows Vista Ul-timate
Not De-fined
Total 6 9 14 13 1 27 1 8 9 1 5 1
2.5
7.5
12.5
17.5
22.5
27.5
OS Edition
# w
orks
tatio
ns
![Page 27: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/27.jpg)
Runtime Intelligence – Where to next?
• Move into Production
• Improve Development Decision Making for future releases
Development• P
erformance tuning
• Usability
QA• R
isk based testing
Documentation• M
aterial coverage
• Help usage
Application Usage
![Page 28: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/28.jpg)
Code Demo
Dotfuscator by PreEmptive Solutions• Runtime Intelligence defines several message
types:– Application and Session Start – Application and Session Stop – Feature – Performance Probe – System Profile – Tamper Detected
![Page 29: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/29.jpg)
Runtime Intelligence Portal’s dashboards
To have your application send these messages, you must:
• Be a Runtime Intelligence Service subscriber• Activate Runtime Intelligence from within
Dotfuscator• Attribute your application with Runtime Intelligence
attributes, including Setup and Teardown• Run your application through Dotfuscator with the
Send Analytics Messages option turned on
![Page 30: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/30.jpg)
Business Attribute
![Page 31: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/31.jpg)
Application Attribute
![Page 32: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/32.jpg)
Setup Attribute
![Page 33: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/33.jpg)
Teardown Attribute
![Page 34: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/34.jpg)
Feature Attribute
![Page 35: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/35.jpg)
Sample Application
![Page 36: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/36.jpg)
Sample Code
![Page 37: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/37.jpg)
ILDASM Before
![Page 38: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/38.jpg)
ILDASM After
![Page 39: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/39.jpg)
Analytical Data
![Page 40: Running a Successful Beta Test Program](https://reader036.vdocuments.site/reader036/viewer/2022062517/568130d6550346895d96eb6a/html5/thumbnails/40.jpg)
Questions?