launching successful applications

Successful Applications

A systematic framework for launching successful applications

Chris K. ChewEnterprise Infrastructure TeamPearson eCollege

[email protected]

@chrischew

mailto:[email protected]



Launching Successful Applications

1. What To Count– Key Performance Indicators– KPI’s in software– Choosing your indicators

2. How To Count– Finding the right places to measure– Frameworks that can help

3. Count Out Load– Transparency builds goodwill– Advertising success creates momentum

4. When To Evolve Your Measurements


What To CountKey Performance Indicators

1

Presentation Title runs here l 00/00/00

Key Performance Indicators

KPI’s are a small number of measurable objectives that will add the most value to the organization for the foreseeable future.



• Targeted towards your strategic goals

• Must be objectively knowable

• Each measurement must clearly add value to the organization

• Articulate which scenarios are included in each measurement

• The fewer the better



• Quantitative: Something that is a number

• Practical: Something that ties into existing business processes

• Directional: Specifying whether something is improving, trending

• Actionable: Identifying a problem that is within the organization's ability to change

• Comparative: Comparing yourself with a "peer universe", e.g. industry. Also referred to as “financial”



Etsy’s Indicators

• Quantitative: How many items were sold last month?

• Practical: How many new members signed up?

• Directional: Comparison from January 2011 and February 2010

• Actionable: Number of complaints or returned items?

• Comparative: What are ebay’s trends for the same month?


Key Performance Indicators In Software

KPI’s are a small number of measurable objectives that will add the most value to the organization your application for the foreseeable future.



• Quantitative: Measure the occurrences and durations of activities and transactions

• Practical: Measuring strategically-targeted usage patterns

• Directional: Did recent changes help or hurt performance?

• Actionable: Any error should be an actionable indicator

• Comparative: Compare your application to similar applications in your installation



• Functional: Measuring the frequency and duration which features are used.

– Examples:

○Minutes in class○ Click-throughs○ SSO hand-offs○ Page views



• Non-Functional: Measuring the transactional throughput of an application.

– Examples:

○ Connections per server○ Response times○ Page load times



• Transactional: Measuring the frequency and duration which business transactions are executed.

– Examples:

○Messages○ Enrollments processed○ Courses viewed○ Thread responses



How To Identify Your KPI’s

• What are the nearest-term goals of your application?

• How will you know when you have reached your goals?



Example: Prospero Message Bus

Strategic Goal: To provide an always-on message bus that guarantees delivery with very low message publication latency.

What To Measure:• Availability• Message loss• Delivery latency



Example: Prospero Message Bus

Key Performance Indicators:• Messages produced (per second and day)• Number of times it takes longer than 500ms to publish a message• Number of delivery problems• Production-to-Delivery latency• Linux “load factor” of servers



Example: Anton.io Browser-Channel Test

Strategic Goal: To feel comfortable recommending the use of web sockets and flash sockets for realistic event-driven web UI.

What To Measure:• Connection reliability per transport• Connection density and server behavior of Node.js



Example: Anton.io Browser-Channel Test

Key Performance Indicators• Connections succeeded and failed• Transport used (web sockets vs. flash vs. XHR polling)• Connection time in milliseconds


How To CountCollecting and presenting your application’s statistics

2


How To Count

The Golden Rules Of Audit Logging

• Log transactions, not errors and debug statements

• Find the right places to place the hooks

• Record the transaction as concisely as possible

• Leverage existing frameworks


How To Count





• Leveraging existing frameworks


How To Count



– Error logs are for identifying code and system problems

– Debug statements are for development and last-resort debugging

– Audit logs need to be noiseless and easily parsed


How To Count







http://assets.en.oreilly.com/1/event/7/Improving%20Netflix%20Performance%20Presentation.pdf




How To Count

Finding The Right Places To Measure

...is all about knowing what pieces are included in the measurement


How To Count







How To Count

Anatomy Of An Audit Log Statement

• Items:– Timestamp– Action– “key=value” pairs supplying context

• Format:– TIMESTAMP action=ACTION [key1=value1 ...]– Exclude anything the is deducible or implicit, e.g. server name – $64k Question: Can you parse it with a regular expression?


How To Count

Anatomy Of An Audit Log Statement - Prospero Example

2011-04-11T16:13:53Z action=SUBSCRIPTION-CANCELLED sub=e0490bb8-1eb7-4221-a6f2-4cb1bc9c76d4

2011-04-11T16:15:56Z action=MESSAGE msg=c14fd86f-f448-4aa6-b662-87c7f27895fc principal=ONE length=11 realm=default content_type=text/plain tag=MessageType:messageType1 tag=SubSystem:default tag=System:default tag=ClientString:default tag=Client:default

2011-04-11T16:15:56Z action=ATTEMPTING-DELIVERY sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce

2011-04-11T16:15:56Z action=DELIVERED sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce del_ref=12345 duration_seconds=0 latency_seconds=1 ramp_churn=0

NOTE: This is word-wrapped...always do one statement per line for easier parsing!


How To Count







How To Count

Leverage Existing Frameworks

• Log files work well:

– Rotated with logrotate or CHOMP

– Many search utilities (grep, sed, awk)

– Easily processed by any language

– log4j and log4net write asynchronously

– They can always be latter parsed and inserted into a database


How To Count

Leverage Existing Frameworks

– Try our audit logging library available on GitHub at○ https://github.com/PearsonLearningStudio/audit-logging

– Java

– Coming soon:○ .NET○Node.js○ Erlang

https://github.com/PearsonLearningStudio/audit-logging

https://github.com/PearsonLearningStudio/audit-logging


Count Out LoudTransparency and advertisement

3


Count Out Loud, Be Transparent

• Being transparent with successes and failures builds goodwill towards your application.

– Users are more likely to trust your application

– Users will be more patient when problems occur


Count Out Loud, Advertise Your Results

• Advertising your results will help keep users aware of your application after the excitement of the initial launch dies down.

– Product folks are more likely to prioritize improvements to the application

– Potential users of the application will feel more comfortable adopting it

– The data you share may help your peers


When To Evolve Your MeasurementsMetrics change as goals change

4


When To Evolve Your Measurements

• Goals will change either because strategies change or because the goal is reached and new ones must be set.

• KPI’s should be adjusted at the same time when goals change


When To Evolve Your Measurements

• Goals will change either because strategies change or because the goal is reached and new ones must be set.

• KPI’s should be adjusted at the same time when goals change

• But it is also important to listen for negative feedback because you might be measuring the wrong things.


Wrong KPI?

• Inadvertently incentivize undesirable behavior

• Start measuring the negative feedback to determine if the problem is real

• Either replace the old KPI with the new one or add it to the squad


Buffer Bloat: Measuring The Wrong Thing

• Bufferbloat is the existence of excessively large (bloated) buffers in systems, particularly network communication systems




• Some experts believe that bufferbloat is the cause of much of the poor performance and human pain experienced using today’s Internet




• Some experts believe that bufferbloat is the cause of much of the poor performance and human pain experienced using today’s Internet

• If true, this is a quintessential example of picking the wrong industry KPI– Networks should probably use “Round Trip Latency” instead of

“Dropped Packets” because excessive work to not drop packets negates the effectiveness of TCP/IP



• See http://www.bufferbloat.net for details

http://www.bufferbloat.net/




• See http://www.bufferbloat.net for details

• Wager: Phở is on me if Buffer Bloat is not part of the general networking conversation by the end of 2012.




Conclusion

5



1. What To Count– Key Performance Indicators in software– Choosing your indicators






4. Evolve Your Measurements– When goals change– Negative feedback

Thank you

Chris K. ChewEnterprise Infrastructure Team

[email protected]

@chrischew



launching successful applications

Technology

similar applications

bettersuccessful applications

successful applications1

key performance indicatorskpis

hurt performance

number practical

software quantitative

software transactional