launching successful applications

Successful Applications

A systematic framework for launching successful applications

Chris K. ChewEnterprise Infrastructure TeamPearson eCollege

chrisch@ecollege.com

@chrischew

Launching Successful Applications

1. What To Count– Key Performance Indicators– KPI’s in software– Choosing your indicators

2. How To Count– Finding the right places to measure– Frameworks that can help

3. Count Out Load– Transparency builds goodwill– Advertising success creates momentum

4. When To Evolve Your Measurements

What To CountKey Performance Indicators

Presentation Title runs here l 00/00/00

Key Performance Indicators

KPI’s are a small number of measurable objectives that will add the most value to the organization for the foreseeable future.

• Targeted towards your strategic goals

• Must be objectively knowable

• Each measurement must clearly add value to the organization

• Articulate which scenarios are included in each measurement

• The fewer the better

• Quantitative: Something that is a number

• Practical: Something that ties into existing business processes

• Directional: Specifying whether something is improving, trending

• Actionable: Identifying a problem that is within the organization's ability to change

• Comparative: Comparing yourself with a "peer universe", e.g. industry. Also referred to as “financial”

Etsy’s Indicators

• Quantitative: How many items were sold last month?

• Practical: How many new members signed up?

• Directional: Comparison from January 2011 and February 2010

• Actionable: Number of complaints or returned items?

• Comparative: What are ebay’s trends for the same month?

Key Performance Indicators In Software

KPI’s are a small number of measurable objectives that will add the most value to the organization your application for the foreseeable future.

• Quantitative: Measure the occurrences and durations of activities and transactions

• Practical: Measuring strategically-targeted usage patterns

• Directional: Did recent changes help or hurt performance?

• Actionable: Any error should be an actionable indicator

• Comparative: Compare your application to similar applications in your installation

• Functional: Measuring the frequency and duration which features are used.

– Examples:

○Minutes in class○ Click-throughs○ SSO hand-offs○ Page views

• Non-Functional: Measuring the transactional throughput of an application.

– Examples:

○ Connections per server○ Response times○ Page load times

• Transactional: Measuring the frequency and duration which business transactions are executed.

– Examples:

○Messages○ Enrollments processed○ Courses viewed○ Thread responses

How To Identify Your KPI’s

• What are the nearest-term goals of your application?

• How will you know when you have reached your goals?

Example: Prospero Message Bus

Strategic Goal: To provide an always-on message bus that guarantees delivery with very low message publication latency.

What To Measure:• Availability• Message loss• Delivery latency

Example: Prospero Message Bus

Key Performance Indicators:• Messages produced (per second and day)• Number of times it takes longer than 500ms to publish a message• Number of delivery problems• Production-to-Delivery latency• Linux “load factor” of servers

Example: Anton.io Browser-Channel Test

Strategic Goal: To feel comfortable recommending the use of web sockets and flash sockets for realistic event-driven web UI.

What To Measure:• Connection reliability per transport• Connection density and server behavior of Node.js

Example: Anton.io Browser-Channel Test

Key Performance Indicators• Connections succeeded and failed• Transport used (web sockets vs. flash vs. XHR polling)• Connection time in milliseconds

How To CountCollecting and presenting your application’s statistics

How To Count

The Golden Rules Of Audit Logging

• Log transactions, not errors and debug statements

• Find the right places to place the hooks

• Record the transaction as concisely as possible

• Leverage existing frameworks

How To Count

• Leveraging existing frameworks

How To Count

– Error logs are for identifying code and system problems

– Debug statements are for development and last-resort debugging

– Audit logs need to be noiseless and easily parsed

How To Count

http://assets.en.oreilly.com/1/event/7/Improving%20Netflix%20Performance%20Presentation.pdf

How To Count

Finding The Right Places To Measure

...is all about knowing what pieces are included in the measurement

How To Count

Anatomy Of An Audit Log Statement

• Items:– Timestamp– Action– “key=value” pairs supplying context

• Format:– TIMESTAMP action=ACTION [key1=value1 ...]– Exclude anything the is deducible or implicit, e.g. server name – $64k Question: Can you parse it with a regular expression?

How To Count

Anatomy Of An Audit Log Statement - Prospero Example

2011-04-11T16:13:53Z action=SUBSCRIPTION-CANCELLED sub=e0490bb8-1eb7-4221-a6f2-4cb1bc9c76d4

2011-04-11T16:15:56Z action=MESSAGE msg=c14fd86f-f448-4aa6-b662-87c7f27895fc principal=ONE length=11 realm=default content_type=text/plain tag=MessageType:messageType1 tag=SubSystem:default tag=System:default tag=ClientString:default tag=Client:default

2011-04-11T16:15:56Z action=ATTEMPTING-DELIVERY sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce

2011-04-11T16:15:56Z action=DELIVERED sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce del_ref=12345 duration_seconds=0 latency_seconds=1 ramp_churn=0

NOTE: This is word-wrapped...always do one statement per line for easier parsing!

How To Count

Leverage Existing Frameworks

• Log files work well:

– Rotated with logrotate or CHOMP

– Many search utilities (grep, sed, awk)

– Easily processed by any language

– log4j and log4net write asynchronously

– They can always be latter parsed and inserted into a database

How To Count

Leverage Existing Frameworks

– Try our audit logging library available on GitHub at○ https://github.com/PearsonLearningStudio/audit-logging

– Java

– Coming soon:○ .NET○Node.js○ Erlang

Count Out LoudTransparency and advertisement

Count Out Loud, Be Transparent

• Being transparent with successes and failures builds goodwill towards your application.

– Users are more likely to trust your application

– Users will be more patient when problems occur

Count Out Loud, Advertise Your Results

• Advertising your results will help keep users aware of your application after the excitement of the initial launch dies down.

– Product folks are more likely to prioritize improvements to the application

– Potential users of the application will feel more comfortable adopting it

– The data you share may help your peers

When To Evolve Your MeasurementsMetrics change as goals change

When To Evolve Your Measurements

• Goals will change either because strategies change or because the goal is reached and new ones must be set.

• KPI’s should be adjusted at the same time when goals change

When To Evolve Your Measurements

• Goals will change either because strategies change or because the goal is reached and new ones must be set.

• KPI’s should be adjusted at the same time when goals change

• But it is also important to listen for negative feedback because you might be measuring the wrong things.

Wrong KPI?

• Inadvertently incentivize undesirable behavior

• Start measuring the negative feedback to determine if the problem is real

• Either replace the old KPI with the new one or add it to the squad

Buffer Bloat: Measuring The Wrong Thing

• Bufferbloat is the existence of excessively large (bloated) buffers in systems, particularly network communication systems

• Some experts believe that bufferbloat is the cause of much of the poor performance and human pain experienced using today’s Internet

• If true, this is a quintessential example of picking the wrong industry KPI– Networks should probably use “Round Trip Latency” instead of

“Dropped Packets” because excessive work to not drop packets negates the effectiveness of TCP/IP

• See http://www.bufferbloat.net for details

• Wager: Phở is on me if Buffer Bloat is not part of the general networking conversation by the end of 2012.

Conclusion

1. What To Count– Key Performance Indicators in software– Choosing your indicators

4. Evolve Your Measurements– When goals change– Negative feedback

Thank you

Chris K. ChewEnterprise Infrastructure Team

chrisch@ecollege.com

@chrischew

launching successful applications

similar applications

bettersuccessful applications

successful applications1

key performance indicatorskpis

hurt performance

number practical

software quantitative

software transactional

Technology

launching a successful nursing home dialysis program ·...

launching applications with docker, coreos, kubernetes and...

writing successful grant applications

8 strategies for launching a successful blog

successful job applications

ask alkarmi ama secrets of launching and scaling successful...

launching a successful digital platform€¦ · launching a...

crafting and launching successful beacon apps - 11 keys to...

amanda sibley - 50 actionable tips for launching a...

making successful applications

launching a successful online business

sunrise prioritizes new applications after launching ... ·...

planning and launching a successful reflections program kima...

launching a successful online business and ec projects...

webinar wednesday: launching & managing a successful...

preparing successful federal grant applications

7 secrets to launching a successful beverage brand

chapter 16 launching a successful online business

lda foundation ngo in delhi launching successful campaigns

chapter 16 launching a successful online business and ec...