benchmarkxprt benchmarks presentation to ebl-wg

2012HDXPRT Development Community

BenchmarkXPRT benchmarks presentation to EBL-WG

May 2013

2013

2012HDXPRT Development CommunityRoadmap for presentation

• Introduction to Principled Technologies, Inc.• The BenchmarkXPRT Development

Community• HDXPRT 2012• TouchXPRT 2013 • WebXPRT 2013 • MobileXPRT (final name under discussion)• Questions and answers

2BenchmarkXPRT Development Community | www.BenchmarkXPRT.com

HDXPRT Development Community

3

About Principled Technologies (PT)

• BenchmarkXPRT Community Administrator• Leading provider of fact-based marketing and technology

assessment services• Founded by Mark Van Name and Bill Catchings

‒ Over 25 years experience working together on technology assessment

‒ As journalists, published over a thousand articles on a wide array of technology subjects

‒ Created the Ziff-Davis Benchmark Operation (ZDBOp), which developed Winstone, 3D WinBench, WebBench, and many other benchmarks

‒ Co-founded and/or led several technology assessment firms

BenchmarkXPRT Development Community | www.BenchmarkXPRT.com


4

Why more benchmarks?

• The BenchmarkXPRT benchmarks share common defining characteristics:‒ Easy-to-understand results‒ Relatable scenarios

• Community model‒ Keeps us in touch with what people do in the real world‒ Why not open source? We need to control derivative

works to avoid benchmark manipulation.



5

The BenchmarkXPRT Development Community

• The BenchmarkXPRT Development Community is a membership-based forum where interested parties can contribute to the ongoing evolution of the XPRT family of benchmarks. Members play a key role in ensuring that these benchmarks incorporate the latest in user-experience benchmarking and meet the needs and interests of a range of individuals and organizations.‒ This is an evolution of the HDXPRT development community,

created in November 2010. All current members are grandfathered into the new community.

‒ Additional new benchmarks made the old HDXPRT-centric name obsolete.



6

BenchmarkXPRT source code

• Our goals for publishing the source code:‒ Increase transparency in the benchmarking process‒ Stimulate knowledge transfer to the benchmark community

• Enable innovation and accelerate improvements to benchmarking process• Provide meaningful results to consumers and OEMs

‒ Opportunity for interested community members to participate in the development process• Guide the design specification• Participate in actual improvement of the code base for all the XPRT benchmarks• Give a grounded basis for understanding the actual behavior of the benchmarks on

target systems

• Ground rules:‒ Source code is available only to BenchmarkXPRT community members.‒ Members must agree to a license agreement that prevents them from releasing

the code to the public.



7

BenchmarkXPRT resources

• BenchmarkXPRT Web site‒ News ‒ White papers‒ Member forum‒ Blog‒ Online database of results‒ Benchmark-specific Web pages for HDXPRT, TouchXPRT, and WebXPRT

• Videos• Twitter• Facebook • Presence on SlideShare, YouTube, and LinkedIn



8

What’s HDXPRT?



9

What’s HDXPRT? (cont.)

• HDXPRT, the High Definition Experience & Performance Ratings Test, evaluates the capabilities of PCs in consumer digital media usages.

• HDXPRT was the first XPRT benchmark. HDXPRT 2012 is the latest version.

• HDXPRT uses real, commercially available applications. • Runs on Windows® 7, 64-bit and Windows 8, 64-bit only.• Includes tests for popular consumer usage scenarios, providing both

performance scores for each scenario and an overall performance score.• Minimum system requirements are an Intel dual-core 2.0GHz processor

or equivalent with 2 GB of RAM, 40 GB of free disk space and 1,024 x 768, 24-bit color video display settings.

• Minimum supported operating system is Microsoft® Windows 7 64-bit edition (Language: US English).



10

What’s in HDXPRT 2012?

• HDXPRT 2012 has five use-case categories:‒ Media Organizer ‒ Media Creator ‒ Photo Blogger ‒ Video Producer ‒ Music Maker



11

What’s in HDXPRT 2012? (cont.)

• HDXPRT 2012 has nine applications: ‒ Adobe® Photoshop® Elements 10‒ Adobe Premiere® Elements 10‒ Apple® iTunes® 10.5.2.11‒ Audacity® 2.0‒ CyberLink MediaEspresso 6.5‒ CyberLink PowerDirector 10‒ GIMP 2.6.11‒ HandBrake 0.9.5‒ HDRsoft Photomatix Pro 4.1.4



12

HDXPRT 2012 scoring

• Scoring includes runtimes for individual usage models, as well as an overall score.

• The overall score is the geometric mean of the ratios between test system scores and scores of a calibration system.

• The Calibration system uses an Intel® Pentium® Processor E6800 processor with 4 GB of 800 MHz DDR2 RAM and a Seagate® Barracuda 1 TB 7200 RPM SATA drive.

• To see how HDXPRT 2012 compares a sample of Intel i3, i5, and i7 processors, check out the HDXPRT 2012 scaling paper at http://www.principledtechnologies.com/hdxprt/HDXPRT2012_Scaling_0912.pdf

• Learn more at HDXPRT.com. BenchmarkXPRT Development Community | www.BenchmarkXPRT.com

http://www.principledtechnologies.com/hdxprt/HDXPRT2012_Scaling_0912.pdf




13

HDXPRT 2012 scoring



14

What’s next for HDXPRT?

• We’re working on HDXPRT 2013. ‒ The most significant feedback we’ve received is that HDXPRT

2012 is too big and takes too long to run. ‒ We are working to make HDXPRT 2013 small enough to

download and shorten the running time. • The scenarios in HDXPRT 2013 are Media Creator, Video Producer,

and Music Maker.• The 6 applications in HDXPRT 2013 are Apple® iTunes® 11.0.2,

Adobe Photoshop® Elements 11, Audacity® 2.0.3, CyberLink MediaEspresso 6.5, CyberLink PowerDirector 11, HandBrake/X264 0.9.8.

‒ We believe we can reduce the size and running time without hurting the value of the benchmark.



15

What’s TouchXPRT?



16

What’s TouchXPRT? (cont.)

• TouchXPRT provides a fair and consistent basis for evaluating and comparing the capabilities of both Windows 8 and Windows RT devices.

• TouchXPRT supports both Intel-based and ARM-based devices.

• Unlike HDXPRT, which uses commercially available applications, the TouchXPRT scenarios are native custom code.

• TouchXPRT runs in the Windows 8 Modern UI (Metro) as a native app.



17


• TouchXPRT 2013 comprises five scenarios: ‒ Beautify Photo Album‒ Prepare Photos for Sharing ‒ Convert Videos for Sharing ‒ Export Podcast to MP3‒ Create Slideshow from Photos

• Since the release of the community preview in October 2012, TouchXPRT 2013 CP1 And TouchXPRT 2013 have been used to review and characterize numerous Windows RT and Windows 8 touch-enabled devices. ‒ First appearance was Anandtech’s Surface review, October 23 http://

www.anandtech.com/show/6385/microsoft-surface-review/10


http://www.anandtech.com/show/6385/microsoft-surface-review/10

http://www.anandtech.com/show/6385/microsoft-surface-review/10


18


• The source code for TouchXPRT 2013 is available to the community. ‒ We encourage the community to critique the scenarios and

scoring, and to make suggestions for improvement.‒ In the future, we hope that community members will contribute

scenarios to TouchXPRT.

• Development of future versions of TouchXPRT is ongoing. If you have feedback on TouchXPRT 2013 and suggestions for the next version, please let us know.



19

TouchXPRT Scoring

• For each scenario, TouchXPRT gives two scores: the elapsed time in seconds and the rate. The rate allows for “bigger is better” presentations.‒ For the Beautify Photo Album, Prepare Photos for Sharing, and

Create Slideshow from Photos scenarios, the rate is given as photos/sec.

‒ For the Convert Videos for Sharing and Export Podcast to MP3 scenarios, the rate is ratio of the length of the clip divided by the time required for the conversion. For example, an 85.2-second clip converted in 33.9 seconds is presented as 2.5X.

• Although the community preview did not provide an overall score, the released version of TouchXPRT 2013 does include an overall score.



20

TouchXPRT Scoring



21

What’s WebXPRT?



22

What’s WebXPRT? (cont.)

• WebXPRT is a processor-, OS-, and browser-agnostic benchmark that allows you to compare the performance of almost any Web-enabled device. We have successfully run this community preview on devices as diverse as the Kindle® Fire, Android® phones, iPad®, Macs®, and Windows PCs.

• WebXPRT uses standard HTML5 and JavaScript to implement representative Web-user scenarios. This reliance on standards allows it to run on a wide variety of devices.

• Unlike some other benchmarks in this class, WebXPRT uses scenarios that real people can relate to. WebXPRT 2013 comprises four separate workloads: ‒ Photo Effects‒ Face Detect‒ Stocks Dashboard ‒ Offline Notes

• Since the release of the community preview in December 2012, WebXPRT 2013 CP1 And WebXPRT 2013 have been used to review and characterize numerous web-enabled devices. ‒ WebXPRT’s first appearance in a review was in Anandtech’s Acer W510 review on

December 20. http://www.anandtech.com/show/6522/the-clover-trail-atom-z2760-review-acers-w510-tested BenchmarkXPRT Development Community | www.BenchmarkXPRT.com

http://www.anandtech.com/show/6522/the-clover-trail-atom-z2760-review-acers-w510-tested





23

What’s WebXPRT? (cont.)

• In addition to the scores, WebXPRT also gives a report on the HTML 5 capabilities of your device.

• Because it runs in the browser and runs on such a wide variety of devices, WebXPRT is limited in the amount of other information it can automatically capture about the test device.

• Because WebXPRT is based on PT Web servers, some information about the tests, such as the IP address, will be exposed to PT servers.‒ WebXPRT does not keep any personally identifying information. You

can see the list of fields WebXPRT collects at http://www.principledtechnologies.com/benchmarkxprt/webxprt/2013/data-collected


http://www.principledtechnologies.com/benchmarkxprt/webxprt/2013/data-collected




24

WebXPRT scoring

• Each scenario reports its elapsed time in milliseconds and a 95 percent confidence interval.

• The rollup score is the geometric mean of the ratios between test system scores and scores of a calibration system.

• The WebXPRT 2013 calibration system is an iPad 3 running iOS 6.

• The score calculation excludes outliers. For a full explanation of how WebXPRT 2013 determines outliers and calculates its results, see http://principledtechnologies.com/benchmarkxprt/whitepapers/2013/WebXPRT-2013_calculation.pdf.


http://principledtechnologies.com/benchmarkxprt/whitepapers/2013/WebXPRT-2013_calculation.pdf

http://principledtechnologies.com/benchmarkxprt/whitepapers/2013/WebXPRT-2013_calculation.pdf


25

WebXPRT scoring (cont.)

• Because this is a Web-based test and there may be some run-to-run variation, WebXPRT 2013 repeats each scenario seven times.

• To capture any variation, WebXPRT reports the 95 percent confidence interval for the run. Using an iPad 4 scoring 180 +/- 2 as an example, the odds that the score for any single iteration would be between 178 and 182 are 95 percent.

• We have successfully run WebXPRT 2013 on a wide variety of devices.



26

WebXPRT scoring (cont.)



27

What’s MobileXPRT



28

What’s MobileXPRT? (cont.)


• Formerly known as PhoneXPRT• A new benchmark for evaluating Android-based

devices (Android 4.1 and up)• Contains a mix of user experience and performance

tests using custom code rather than applications• Currently testing preliminary versions• More details over the next few weeks


29

What’s next?

• Continue the development of the existing benchmarks by adding new tests and features

• Increase community involvement• Look for other places where there’s need• Let us know what you’d like to see!



30

What we would like you to do

• Join the community (if not already a member). • Encourage others to join.• Take advantage of the design documents, betas, and

Community Previews.• Send us your suggestions for improvements or new

functionality as soon as possible.• Collaborate with us as we continue development of

the 2013 and 2014 releases of the benchmarks.• Participate in the community and spread the word!



31

Thanks to all Community members for your participation in the BenchmarkXPRT development process!

We invite all non-members to join the community to help with the ongoing evolution of the XPRT benchmarks.Go to www.benchmarkxprt.com to register.


http://www.hdxprt.com/


32

Q & A


benchmarkxprt benchmarks presentation to ebl-wg

Technology