application testing

Application TestingApplication TestingReggie Niccolo SantosReggie Niccolo SantosUP ITDCUP ITDC

OutlineOutline

The Testing PhaseThe Testing Phase

Testing your Web AppsTesting your Web Apps

Testing Strategies and Tactics for Mobile Testing Strategies and Tactics for Mobile ApplicationsApplications

ReferencesReferences


In many software engineering methodologies, a In many software engineering methodologies, a separateseparate phase performed by a phase performed by a differentdifferent team after the implementation is completedteam after the implementation is completed


Regression TestRegression Test

Internal TestingInternal Testing

Unit TestingUnit Testing

Application TestingApplication Testing

Stress TestingStress Testing


Form a Form a suitesuite of programs that test one or more of programs that test one or more featuresfeatures of the system of the system

A valid regression test generates A valid regression test generates verified resultsverified results

““gold standardgold standard””

Validity driven by the rValidity driven by the requirement documentequirement document

In practice, the In practice, the implementation teamimplementation team is is responsible for validity interpretationresponsible for validity interpretation


As development continues, more tests are added, As development continues, more tests are added, while old tests may remain validwhile old tests may remain valid

Because of new development, an old test may no Because of new development, an old test may no longer be validlonger be valid

If this is the case, the old test results are If this is the case, the old test results are alteredaltered in the “gold standard” to in the “gold standard” to matchmatch the current the current expectationsexpectations

The test suite is run generating new resultsThe test suite is run generating new results


Make sure all Make sure all internal, non-customer-internal, non-customer-visible componentsvisible components work well work well

Deals with Deals with low-levellow-level implementation implementation

Each Each functionfunction or or componentcomponent is tested is tested

Accomplished by the Accomplished by the implementation implementation teamsteams


Clear-box/white-box testingClear-box/white-box testing

All details are All details are visiblevisible to the test to the test

Internal limitsInternal limits are tested here are tested here


Make sure all Make sure all customer-visible componentscustomer-visible components work wellwork well

Deals with testing a Deals with testing a unit as a wholeunit as a whole

Test the Test the interaction of many functionsinteraction of many functions but but confineconfine the test within the test within one unit one unit

Supporting test code, sometimes called Supporting test code, sometimes called scaffoldingscaffolding, may be necessary to support an , may be necessary to support an individual testindividual test


Driven by the Driven by the architecture and architecture and implementation teamsimplementation teams

Black-box testingBlack-box testing because only the because only the details details of the interface are visibleof the interface are visible to the test to the test

Limits that are Limits that are globalglobal to a unit are tested here to a unit are tested here


Make sure the application can complete Make sure the application can complete all all scenariosscenarios

Driven by scenarios from the Driven by scenarios from the analysis teamanalysis team

Application limits and featuresApplication limits and features are tested are tested herehere


Must Must successfully execute all scenariossuccessfully execute all scenarios before before it is ready for general customer availabilityit is ready for general customer availability

Represents the Represents the bulkbulk of the testing done by the of the testing done by the industryindustry

Unlike the internal and unit testing, which are Unlike the internal and unit testing, which are programmed, these tests are usually programmed, these tests are usually driven by driven by scriptsscripts that run the system with a collection of that run the system with a collection of parameters and collect resultsparameters and collect results


Run the application in an environment that is Run the application in an environment that is more stressfulmore stressful than the target environment than the target environment

Deals with the Deals with the qualityquality of the application in the of the application in the environmentenvironment

Requires a joint effort from Requires a joint effort from all teamsall teams


Test environment established with Test environment established with many testing many testing stationsstations

At each station, a At each station, a script is executingscript is executing the system the system

These scripts are usually based on the regression These scripts are usually based on the regression suitesuite

More and more stations are added, all More and more stations are added, all simultaneously hammering on the system, until simultaneously hammering on the system, until the system the system breaksbreaks


The system is repaired and the stress test is The system is repaired and the stress test is repeated until a level of stress is reached that repeated until a level of stress is reached that is is higher than expectedhigher than expected to be present at a to be present at a customer sitecustomer site

Race conditionsRace conditions and and memory leaksmemory leaks are are often found under stress testingoften found under stress testing

Race ConditionRace Condition

Conflict between Conflict between at least twoat least two tests tests

Each test works correctly when done in Each test works correctly when done in isolationisolation

When the two tests are run in parallel, one or When the two tests are run in parallel, one or both of the tests both of the tests failfail

Usually due to an Usually due to an incorrectly managed incorrectly managed locklock

Memory LeakMemory Leak

Happens when a test Happens when a test leaves allocated leaves allocated memory behindmemory behind and does not correctly return and does not correctly return the memory to the memory allocation schemethe memory to the memory allocation scheme

The test seems to run correctly, but after being The test seems to run correctly, but after being exercised several times, exercised several times, available memory is available memory is reducedreduced until the system fails until the system fails


1.1. ObjectivesObjectives

Establish your testing objectives Establish your testing objectives up frontup front and and make sure that they are make sure that they are measurablemeasurable

Make sure your objectives are Make sure your objectives are prioritizedprioritized

Ask yourself questions like: Ask yourself questions like:

1.1. ““What is most important: What is most important: minimal defectsminimal defects or or time-to-markettime-to-market?” ?”


2.2. Process and ReportingProcess and Reporting

Make sure that everyone on your testing Make sure that everyone on your testing team knows his or her team knows his or her rolerole

WhoWho should report should report whatwhat to to whomwhom and and whenwhen??

Define your Define your testing processtesting process



Guide questions: Guide questions:

HowHow will issues be reported? will issues be reported?

WhoWho will assign issues? will assign issues?

HowHow will issues be categorized? will issues be categorized?




WhoWho needs needs whatwhat report and report and whenwhen do they need it? do they need it?

2.2. Are team meetings scheduled Are team meetings scheduled in advancein advance or or scheduled scheduled as neededas needed? ?

Organize your team in a way that Organize your team in a way that supportssupports your testing your testing objectives and takes into account the objectives and takes into account the individualindividual personalities on your teampersonalities on your team


3.3. Tracking ResultsTracking Results

You will want a way to easily You will want a way to easily store, organize store, organize and distribute informationand distribute information about bugs, about bugs, issues, and defects to the appropriate technical issues, and defects to the appropriate technical team membersteam members

You will also need a way to You will also need a way to keep management keep management informedinformed of the status of your testing efforts of the status of your testing efforts

e.g. e.g. AdminiTrack.comAdminiTrack.com

http://AdminiTrack.com/


4.4. Test EnvironmentTest Environment

Separate from your Separate from your developmentdevelopment and and productionproduction environment environment

Includes a Includes a separate web server, database separate web server, database server, and application serverserver, and application server if if applicableapplicable


4.4. Test EnvironmentTest Environment

Create an explicitly defined procedure for Create an explicitly defined procedure for moving codemoving code to and from your test to and from your test environment and make sure the procedure is environment and make sure the procedure is followedfollowed

Work with your development team to make Work with your development team to make sure each new version of source code to be sure each new version of source code to be tested is tested is uniquely identifieduniquely identified


5.5. Usability TestingUsability Testing

Looking at aspects of your web application that affect Looking at aspects of your web application that affect the the user’s experienceuser’s experience


How How easyeasy is it to navigate through your web is it to navigate through your web application? application?

Is it Is it obviousobvious to the user which actions are available to the user which actions are available to him or her? to him or her?


5.5. Usability TestingUsability Testing


Is the look-and-feel of your web application Is the look-and-feel of your web application consistentconsistent from page to page, including font from page to page, including font sizes and colorssizes and colors

Design for Design for accessibilityaccessibility especially if it is a legal especially if it is a legal requirementrequirement

http://www.w3.org/TR/AERThttp://www.w3.org/TR/AERT

http://www.w3.org/TR/AERT


6.6. Unit TestingUnit Testing

Focused on verifying Focused on verifying smallsmall portions of portions of functionalityfunctionality

e.g. range checking e.g. range checking


7.7. Verifying the HTMLVerifying the HTML

World Wide Web ConsortiumWorld Wide Web Consortium's free HTML 's free HTML Validation ServiceValidation Service

http://validator.w3.orghttp://validator.w3.org

Net MechanicNet Mechanic

http://www.netmechanic.comhttp://www.netmechanic.com

http://www.netmechanic.com/



Make sure that your syntax is correctMake sure that your syntax is correct

Opening and closing tags matchOpening and closing tags match



Verify how your pages Verify how your pages looklook in in different different browsers, at different screen resolutionsbrowsers, at different screen resolutions, , and on and on different operating systemsdifferent operating systems

Create a Create a profileprofile of your target audience and of your target audience and make some decision on what browsers you will make some decision on what browsers you will support, on which operating systems, and at what support, on which operating systems, and at what screen resolutionsscreen resolutions


8.8. Load TestingLoad Testing

You want to simulate how users will use your You want to simulate how users will use your web application in the web application in the real worldreal world

The The earlierearlier you perform the load testing the you perform the load testing the betterbetter

Performance tuningPerformance tuning should be tightly should be tightly integrated with the design of your applicationintegrated with the design of your application


8.8. Load TestingLoad Testing

Try to make sure that all of your pages load Try to make sure that all of your pages load in in 15 seconds or less15 seconds or less

Depends on your Depends on your particular applicationparticular application and the expectations of the people using itand the expectations of the people using it


9.9. User Acceptance Testing (UAT)User Acceptance Testing (UAT)

You are making sure your web application You are making sure your web application fitsfits the use which it was intended the use which it was intended

One way is to set up a One way is to set up a beta testbeta test


10.10. Testing SecurityTesting Security

You need to test how You need to test how securesecure your web your web application is from both application is from both externalexternal and and internalinternal threats threats

Should be planned for and verified by Should be planned for and verified by quality security specialistsquality security specialists



Additional resources: Additional resources:

CERT Coordination CenterCERT Coordination Center

http://www.cert.org/http://www.cert.org/

Computer Security Resource CenterComputer Security Resource Center

http://csrc.nist.gov/http://csrc.nist.gov/

http://csrc.nist.gov/



Make sure to also perform Make sure to also perform ongoing security ongoing security auditsaudits to ensure your web application to ensure your web application remains secure over time as people and remains secure over time as people and technology changetechnology change

Testing Strategy and Testing Strategy and Tactics for Mobile Tactics for Mobile ApplicationsApplications1.1. Goal is not to find errors but to understand the Goal is not to find errors but to understand the

qualityquality of your offering of your offering

Does it Does it workwork? ?

Does it function Does it function as expectedas expected? ?

Will it meet the Will it meet the needsneeds of your users, so that of your users, so that they come back again and again? they come back again and again?

Testing Strategy and Testing Strategy and Tactics for Mobile Tactics for Mobile ApplicationsApplications

2.2. Mobile Testing for Native ApplicationsMobile Testing for Native Applications

The app download is often controlled by the The app download is often controlled by the gate-keeping app storegate-keeping app store, with mechanisms in , with mechanisms in place to charge potential consumersplace to charge potential consumers

Testing Strategy and Testing Strategy and Tactics for Mobile Tactics for Mobile ApplicationsApplications2.2. Mobile Testing for Native ApplicationsMobile Testing for Native Applications

Needs to ascertain that the app can be Needs to ascertain that the app can be successfully downloaded successfully downloaded to the device, to the device, executed executed on the device and on the device and interactinteract with the with the supporting back content infrastructuresupporting back content infrastructure

When updates are made, you need to be sure When updates are made, you need to be sure that the application can be that the application can be pushed out topushed out to and and acceptedaccepted by the end user by the end user

Testing Strategy and Testing Strategy and Tactics for Mobile Tactics for Mobile ApplicationsApplications2.2. Mobile Testing for Native ApplicationsMobile Testing for Native Applications

Tied to the hip with the Tied to the hip with the hardwarehardware and and operating operating systemssystems for which they are written for which they are written

Test on the Test on the physical devicesphysical devices supported by supported by your applicationyour application

Ensure Ensure backward compatibilitybackward compatibility with each older with each older generation of the device you are expected to generation of the device you are expected to supportsupport


2.2. Mobile Testing for Native ApplicationsMobile Testing for Native Applications

Ensure that when issues are found with your Ensure that when issues are found with your native apps, they can be native apps, they can be quickly capturedquickly captured and and sharedshared with others with others


3.3. Mobile Testing Challenges for Web Mobile Testing Challenges for Web ApplicationsApplications

Understand the Understand the globalglobal dynamic dynamic


4.4. Devices: The Biggest Mobile Testing ChallengeDevices: The Biggest Mobile Testing Challenge

There are There are potentially tens of thousands of potentially tens of thousands of different client devicesdifferent client devices that could be used to that could be used to access your mobile app or website, and they access your mobile app or website, and they must therefore all be considered when testing must therefore all be considered when testing your mobile applicationsyour mobile applications


4.4. Devices: The Biggest Mobile Testing ChallengeDevices: The Biggest Mobile Testing Challenge

Can be Can be reducedreduced to an extent, but each time to an extent, but each time you reduce the number of device types that you you reduce the number of device types that you test against, you are taking a chance that your test against, you are taking a chance that your application application might not workmight not work on a device, on a device, locking out a number of potential consumerslocking out a number of potential consumers

Testing Strategy and Testing Strategy and Tactics for Mobile Tactics for Mobile ApplicationsApplications4.4. Devices: The Biggest Mobile Testing ChallengeDevices: The Biggest Mobile Testing Challenge

3 options: 3 options:

Test exclusively using Test exclusively using realreal devices devices

Test exclusively with Test exclusively with emulatedemulated devices devices

Use a Use a combinationcombination of each of each



Have the advantage of having all the Have the advantage of having all the limitationslimitations and and quirksquirks present in the actual present in the actual client hardware and firmware combination in client hardware and firmware combination in the hands of your target consumersthe hands of your target consumers

Can be Can be expensiveexpensive



Can be Can be disorganizeddisorganized and and labor-intensivelabor-intensive if the testing environment is not conducive to if the testing environment is not conducive to creating, collecting and reproducing results in creating, collecting and reproducing results in a consistent mannera consistent manner


Test exclusively using Test exclusively using emulatedemulated devices devices

Relatively Relatively easiereasier to manage to manage

You can switch device types by simply You can switch device types by simply loading a new device profileloading a new device profile

4.4. Run on more Run on more powerful PCspowerful PCs and and serversservers



Were designed for Were designed for testingtesting in mind in mind

Typically Typically fully instrumentedfully instrumented to capture to capture detailed diagnostics about the protocols that detailed diagnostics about the protocols that go back and forth between client and server at go back and forth between client and server at the various levels of the application stackthe various levels of the application stack



Cost effectiveCost effective because a single platform with because a single platform with frequent updates of device profiles can be used to frequent updates of device profiles can be used to test every device on the market both today and test every device on the market both today and tomorrowtomorrow

Big disadvantage is that they lack the Big disadvantage is that they lack the quirks, quirks, faultsfaults and and characteristicscharacteristics that only the real that only the real device can providedevice can provide



May not give the May not give the pixel-perfect accurate rendering pixel-perfect accurate rendering that you are assured to have with a real device solutionthat you are assured to have with a real device solution

Processing power of a local PC can Processing power of a local PC can hidehide any issues that any issues that you may have with the you may have with the responsivenessresponsiveness of your web of your web applicationapplication

Not sensitiveNot sensitive to the to the ambient conditionsambient conditions that can that can impact the behaviour of the deviceimpact the behaviour of the device


Use a Use a combinationcombination of each of each

Start testing in an emulated environment to take Start testing in an emulated environment to take advantage of the advantage of the speedspeed and and device diversity device diversity that an that an emulator can provideemulator can provide

Add real devicesAdd real devices into your test plan later in the into your test plan later in the development cycle so you can validate the applications development cycle so you can validate the applications are functioning as expected and certify that all are functioning as expected and certify that all development requirements and objectives have been development requirements and objectives have been metmet

Testing Strategy and Testing Strategy and Tactics for Mobile Tactics for Mobile ApplicationsApplications5.5. Network: A Regional ChallengeNetwork: A Regional Challenge

Each mobile operator may Each mobile operator may support multiple network support multiple network technologiestechnologies including LTE, CDMA, GSM, and some including LTE, CDMA, GSM, and some use less common or local networking standards such as use less common or local networking standards such as iDEN, FOMA, and TD-SCDMAiDEN, FOMA, and TD-SCDMA

Each network has a unique combination of network Each network has a unique combination of network infrastructure that infrastructure that tunnels the packet-based tunnels the packet-based protocolsprotocols used by mobile networks into TCP-IP used by mobile networks into TCP-IP protocols used by the mobile Webprotocols used by the mobile Web


When implemented, can When implemented, can restrictrestrict the flow of the flow of information that travels between your server information that travels between your server and the test clientand the test client

Some limit the sites that can be accessed via a Some limit the sites that can be accessed via a phone to only those approved by the operator phone to only those approved by the operator (“(“walled gardenwalled garden”)”)


Others might use “Others might use “transcodingtranscoding” in an attempt to ” in an attempt to scale down fixed Web content to better fit onto scale down fixed Web content to better fit onto mobile phones, thus expanding the number of mobile phones, thus expanding the number of mobile sites that can be seenmobile sites that can be seen

Some Some strip vital informationstrip vital information from the HTTP from the HTTP headers that your application might depend on to headers that your application might depend on to provide functionality or to provide device adaptationprovide functionality or to provide device adaptation


To fully test the full network stack on a To fully test the full network stack on a particular operator’s network infrastructure, particular operator’s network infrastructure, you must be you must be connected to the target connected to the target networknetwork

Traveling to every network operator that you Traveling to every network operator that you need to support can be need to support can be very expensivevery expensive


We can We can bypassbypass the lower layers of the network the lower layers of the network and simply test over the Internet or LANand simply test over the Internet or LAN

You use You use TCP/IPTCP/IP to connect directly to the to connect directly to the server and you server and you ignoreignore the GPRS tunnelling the GPRS tunnelling systems used by network operatorssystems used by network operators


You will need to use a You will need to use a device simulatordevice simulator to perform to perform the bypassthe bypass

Advantage is that you will Advantage is that you will not need to usenot need to use and and thus thus pay for airtimepay for airtime

Disadvantages are that is that we often Disadvantages are that is that we often cannot cannot emulateemulate the effects and timing of the network and the the effects and timing of the network and the various network elements such as proxiesvarious network elements such as proxies


We can use the We can use the real networkreal network by using either by using either phones or modemsphones or modems

Many device emulators Many device emulators support modemssupport modems that that allow you to use your emulated devices on the allow you to use your emulated devices on the local networklocal network

But again, there is the cost of But again, there is the cost of travelingtraveling into into range of the networkrange of the network


One piece of useful test equipment is a One piece of useful test equipment is a real device in real device in the cloudthe cloud

Consists of a Consists of a physical handsetphysical handset mounted in a remote mounted in a remote box with a box with a remote control unitremote control unit and a and a remote remote antennaantenna

The remote control unit is physically connected to the The remote control unit is physically connected to the device’s screen and keypad control circuitsdevice’s screen and keypad control circuits and is and is capable of pressing keys and collecting screen imagescapable of pressing keys and collecting screen images


Often have the ability to Often have the ability to record a testrecord a test for for subsequent replaysubsequent replay

Reduce the Reduce the cost of travelcost of travel to foreign networks to foreign networks

Most of the companies that make this type of Most of the companies that make this type of equipment offer the ability to “equipment offer the ability to “rentrent” testing time on ” testing time on a resource that is shared with others and is a resource that is shared with others and is managed for youmanaged for you

Testing Strategy and Testing Strategy and Tactics for Mobile Tactics for Mobile ApplicationsApplications6.6. Scripting: The Repeatability ChallengeScripting: The Repeatability Challenge

The method that is actually used to The method that is actually used to execute the execute the test scripttest script

Script execution can either be Script execution can either be manualmanual or or automatedautomated

You either write down the scripts in a document or a You either write down the scripts in a document or a spreadsheet, which is then used by a test engineer spreadsheet, which is then used by a test engineer who who manually enters keystrokesmanually enters keystrokes, or you , or you run run automated scriptsautomated scripts that in turn evoke the keystrokes that in turn evoke the keystrokes and record the resultsand record the results

Testing Strategy and Testing Strategy and Tactics for Mobile Tactics for Mobile ApplicationsApplications6.6. Scripting: The Repeatability ChallengeScripting: The Repeatability Challenge

Most emulators are capable of automating test Most emulators are capable of automating test execution using a execution using a higher-level, abstract higher-level, abstract scripting languagescripting language that is not device that is not device dependentdependent

Many automated scripting tools have a special Many automated scripting tools have a special ability to “ability to “spiderspider” or “” or “crawlcrawl” a mobile Web ” a mobile Web sitesite


7.7. SummarySummary

Invest in a Invest in a device emulatordevice emulator

Take advantage ofTake advantage of remote real devices in remote real devices in the cloudthe cloud

AutomateAutomate wherever possible wherever possible

SummarySummary

7.7. SummarySummary

Invest in a Invest in a device emulatordevice emulator

Take advantage ofTake advantage of remote real devices in remote real devices in the cloudthe cloud

AutomateAutomate wherever possible wherever possible

SummarySummary







SummarySummary


ObjectivesObjectives

Process and ReportingProcess and Reporting

Tracking ResultsTracking Results

Test EnvironmentTest Environment

Usability TestingUsability Testing

SummarySummary



Verifying the HTMLVerifying the HTML

Load TestingLoad Testing

User Acceptance Test (UAT)User Acceptance Test (UAT)

Testing SecurityTesting Security

SummarySummary

Testing Strategies and Tactics for Mobile Testing Strategies and Tactics for Mobile ApplicationsApplications




http://infolab.stanford.edu/~burback/watersluice/node19.htmlhttp://infolab.stanford.edu/~burback/watersluice/node19.html

10 Tips for Testing Apps for the Real World10 Tips for Testing Apps for the Real World

http://www.techrepublic.com/blog/10-things/10-tips-for-testing-apps-for-the-real-world/#http://www.techrepublic.com/blog/10-things/10-tips-for-testing-apps-for-the-real-world/#


http://www.adminitrack.com/articles/TestingYourWebApps.aspxhttp://www.adminitrack.com/articles/TestingYourWebApps.aspx

Web UsabilityWeb Usability

"Don't Make Me Think! A Common Sense Approach to Web Usability" by Steve Krug and Roger "Don't Make Me Think! A Common Sense Approach to Web Usability" by Steve Krug and Roger BlackBlack

Testing Strategies and Tactics for Mobile ApplicationsTesting Strategies and Tactics for Mobile Applications

http://www.keynote.com/mobile-app-testing.htmlhttp://www.keynote.com/mobile-app-testing.html

http://infolab.stanford.edu/~burback/watersluice/node19.html

http://www.techrepublic.com/blog/10-things/10-tips-for-testing-apps-for-the-real-world/

http://www.adminitrack.com/articles/TestingYourWebApps.aspx

application testing

Software

testing adeals

test withinthe test

test limits

test thetest

web apps testing strategies

calledsupporting test

old test results areif

new resultsthe test