measuring interactive performance with vncplay

Download Measuring Interactive Performance with  VNCplay

Post on 30-Jan-2016

47 views

Category:

Documents

0 download

Embed Size (px)

DESCRIPTION

Measuring Interactive Performance with VNCplay. Nickolai Zeldovich, Ramesh Chandra Stanford University. Measuring performance. Scientific computing or server workloads: How long does it take for my program to run? Measure total running time. Measuring performance. - PowerPoint PPT Presentation

TRANSCRIPT

  • Measuring InteractivePerformance with VNCplayNickolai Zeldovich, Ramesh ChandraStanford University

  • Measuring performanceScientific computing or server workloads:How long does it take for my program to run?

    Measure total running time

  • Measuring performanceUsers of interactive applications:I asked for something, when will I get it?

    But we still measure total runtimeWinBench runs Word, PowerPoint, etc as fast as possible, and measures total runtimeWe are measuring the wrong thing

  • Case in pointUse Microsoft PowerPoint for 5 minutes

    Two machines: 2.0 GHz and 300 MHzExpect to see different performance!

    Two performance metrics:Total runtime: from start to finishResponse time: from click to output on screen

  • Performance MetricsTotal RuntimeResponse Time

  • Users pause a lot. . .0 sec0.1 sec0.2 sec5 sec. . .Response time is noise in total runtime

  • OutlineOverview of VNCplayCurrent approachesHow VNCplay worksEvaluationQuantitative resultsQualitative experienceSummary

  • Overview of VNCplay

    To compare interactive performance, we need reproducible workloadsRecord interactive user sessionReplay session in different environments

    Compare response time between replays

  • Current approaches

    Display protocols: X11, VNC, WindowsKey strokes, mouse clicksScreen updates

  • Current approachesScreen updates. . .0 sec1 sec5 sec8 secTimeInputEventRecorderTools like Xnee, Visual Test, AutoIt, rfbplaymacro

  • Current approachesScreen updates. . .0 sec1 sec5 sec8 secTimeInputEventReplayer

  • What if system is slow?0 sec0.5 sec1.0 secNeed feedback of when the system responds1.5 secTime

  • What should happen?0 sec0.5 sec1.0 secNeed feedback of when the system responds1.5 secTime

  • Workarounds?Current tools allow user to insert delay statements to compensate for slowness

    Creates unrealistic user behavior

    Tedious and error-prone

  • VNCplay approachBig problem is mouse clicks

    Take a screen snapshot at each mouse click during recording

    Wait for the same thing during replay

  • VNCplay approach0 sec1 sec5 sec8 secTimeInputEventScreen updatesRecorder

  • VNCplay approachScreen updates0 sec1 sec5 sec8 secTimeInputEventReplayer

  • VNCplay detailsOnly snapshot the screen around the cursorReduces trace file sizeIgnores irrelevant things (e.g. system clock)

    Other tricks described in the paper

  • Analyzing ReplaysTimeResponse TimeSystem 1:System 2:Match

  • EvaluationOpenOffice on Linux

    Scaled CPU frequency from 2.0 GHz down to 300 MHz

    Compare total running time and interactive response time

  • Total Runtime

  • Median Response Time

  • Other experimentsPaper has details on more experiments

    Microsoft Windows, Linux

    Evaluate interactive performance of different disk IO schedulers

  • Experience using VNCplayPlatform independenceMeasured interactive performance of PowerPoint, Word, OpenOffice

    Reliable replayReplayed 5-minute PowerPoint session on a slow system, taking over an hour works OK!

  • CaveatsNear-perfect reproducibility needed

    VMware: start the same VM every time

    In Linux, reset user account (dotfiles)

  • CaveatsKeyboard input can be a problemUse mouse instead of keyboard shortcutsClick before and after you type

    Couple of tries to get a reliable sessionTooltips, pop-ups get in the way

  • Using VNCplayvncplay record server:5901 trace.vnc

    vncplay play server1:5901 trace.vnc out1.rfbvncplay play server2:5901 trace.vnc out2.rfb

    vncplay analyze out1.rfb out2.rfb > analyze.out

    vncanalyze median analyze.outvncanalyze cdf analyze.out > cdf.plotUse gnuplot, Excel to plot cdf.plot

  • Other usesAutomated testingMake sure your application behaves the same way with a new release of Linux / Windows

    GUI ``automationAutomate GUI-intensive tasks that have no scriptable interface

  • Future directionsIntegrate some GUI toolkit knowledge for smarter matching (e.g. ignore tooltips)

    Better keyboard input handlingMachine learningDetails in paper

    If you want to help, let us know!

  • In conclusion Need to measure interactive performanceRuntime benchmarks are not appropriate

    VNCplayPortable, reliable session replayResponse time measurement

  • Questions?

    http://suif.stanford.edu/vncplay/

    MotivationToday performance benchmarks are largely focused on throughput, runtimeNot appropriate for interactive performance measurementMotivationToday performance benchmarks are largely focused on throughput, runtimeNot appropriate for interactive performance measurementCase in point: PowerPoint workload, runs in the same amount of time on 2GHz machine & 600MHzBut user experience is significantly worse!Need tools that will measure interactive response

    Explain what total runtime and response time is.

    Can you tell from the left graph which is 300 MHz and which is 2.0 GHz? Probably not. Clearly this metric is not suited for evaluating such workloads!

    We need some metric that will let us evaluate performance that the user observes. For example, median response time.

    So whats going on in the previous example?What happens is that users tend to pause a lot when working with a computer.Here we have an example timeline: user clicks on a menu at time 0, and then 100 msec later the menu appears.The user thinks for a while, and clicks his mouse again at 5 seconds.

    What might happen on a slow system is that the menu might take an extra 100 msec to appear twice as slow!However, this is very small compared to the users delay of 5 seconds, so this ends up being noise in the total runtime.

    *** THIS ARGUES THAT WE NEED A TOOL TO MEASURE INTERACTIVE PERFORMANCE TOTAL RUNTIME IS NOT APPROPRIATE ***In VNCplay, we recognize that in order to compare interactive performance, we need to be able to have the same workload run on the systems we want to compare.So we first focus on being able to record and reliably replay an interactive session.

    Once weve replayed the same session in different environments, we can compare the results to see what the interactive performance is like.

    If the system responds slower, the mouse click goes to the desktop, or another application definitely not what the user intended.Derails the entire session trace.

    If the system responds slower, the mouse click goes to the desktop, or another application definitely not what the user intended.Derails the entire session trace.

    Run workload on one computerRun the same workload on another computer. Here the menu appears later.Compare the two workloads to find matching input events (e.g. mouse clicks) and matching screen updatesFor each matching screen update, find the closest preceding input event in all experiments, and assume that this input event caused the screen updateTake this to be the response time.Because we focus on reliable delivery of mouse click events, keyboard input is still a problem

    ConclusionResponse time measurement much better for evaluating interactive systemsRe-iterate experimental result: same runtime, different response timesTool for portable, reliable session recording and replayInteractive performance analysisAvailability: http://suif.stanford.edu/vncplay/