ipc2008 slide qa in depth best practises

Quality Assurance in PHP Projects

In-Depth Best-Practises

Max Horváth & Wojciech Duda, studiVZ Ltd. 2

Who are we?

•  Max Horváth

•  Actively involved in web development for the past 10 years.

•  Quality Manager at studiVZ Ltd.


Who are we?

•  Wojciech Duda

•  Involved in software testing for the past 6 years.

•  QA Engineer at studiVZ Ltd.


Who are YOU?

•  What is your experience with: •  QA in PHP projects?

•  Metrics of software?

•  Testing of software?


Schedule


Schedule / Lots of coffee

•  10:45am – 11:15am Coffee Break

•  12:15am – 01:30pm Lunch

•  03:15pm – 03:45pm Coffee Break


Schedule (Part I)

•  09:30am – 10:00am Intro / Tutorial

•  10:45am – 11:15am Coffee Break

•  11:15am – 12:15am Tutorial

•  12:15am – 01:30pm Lunch


Schedule (Part II)

•  01:30pm – 02:30pm Tutorial

•  03:15pm – 03:45pm Coffee Break

•  03:45pm – 05:00pm Q&A


Objectives of this workshop

•  To introduce you to the quality assurance process and key quality assurance activities



•  To introduce you to the quality assurance process and key quality assurance activities

•  To explain the role of standards in quality assurance



•  To explain the concept of a software metric, predictor metrics and control metrics



•  To explain the concept of a software metric, predictor metrics and control metrics

•  To explain how measurement may be used in assessing software quality


Topics covered

•  Quality assurance and standards


Topics covered


•  Quality planning


Topics covered



•  Quality control


Topics covered




•  Software measurement and metrics


What is Quality Assurance?

•  QA is part of all good production processes.




•  Quality Assurance (QA) is part of the process that ensures:




•  Quality Assurance (QA) is part of the process that ensures: •  quality in work





•  activities are being performed effectively





•  activities are being performed effectively

•  the product meets requirements



•  Involves defining appropriate quality standards and procedures and ensuring that these are followed



•  Involves defining appropriate quality standards and procedures and ensuring that these are followed

•  Should aim to develop a ‘quality culture’ where quality is seen as everyone’s responsibility


What is quality?

•  Quality, simplistically, means that a product should meet its specification


What is unique about Software QA?



!=


It is not equal

•  Software QA has some differences:


It is not equal

•  Software QA has some differences: •  Functions, benefits and production costs are not as

easily measured as a physical product.


It is not equal



•  When a software product is delivered, it is not necessarily complete ... it lives, it evolves, it gets updated.


It is not equal




•  Finished is a term applied for the purpose of releasing it for use.


It is not equal




•  Finished is a term applied for the purpose of releasing it for use.

•  Software specifications are usually incomplete and often inconsistent


The challenge

•  Processes and methods to manage, monitor, and measure quality in web applications can be as fluid and elusive as the defects they are meant to keep in check.


The challenge

•  Processes and methods to manage, monitor, and measure quality in web applications can be as fluid and elusive as the defects they are meant to keep in check.

•  For processes and methods to be most effective, you need to implement both standards and company guidelines.


Quality Assurance is not limited to Quality Control

•  Testing is an essential part of QA




•  But QA tests the entire process:




•  But QA tests the entire process: •  Your development process / efficiency





•  Functional accuracy of your software releases





•  Functional accuracy of your software releases

•  Ideally preventing future software defects


The benefits of QA

•  Improved client satisfaction:


The benefits of QA

•  Improved client satisfaction: •  profitable relationships


The benefits of QA


•  good testimonials


The benefits of QA


•  good testimonials

•  waves of referrals


The benefits of QA

•  Reduced cost of development:


The benefits of QA

•  Reduced cost of development: •  fewer defects


The benefits of QA


•  streamlined and simplified stages of development


The benefits of QA



•  retesting runs smoothly


The benefits of QA




•  deployment on time


The benefits of QA




•  deployment on time

•  on budget


The benefits of QA

•  Reduced cost of maintenance:


The benefits of QA

•  Reduced cost of maintenance: •  good development is less troublesome to support


The benefits of QA

•  Reduced cost of maintenance: •  good development is less troublesome to support

•  support is costly


Quality Assurance activities

•  Quality management



•  Quality management •  Establish organizational procedures and standards for

quality




quality





quality

•  Quality planning •  Select applicable procedures and standards for a

particular project and modify these as required




quality







quality



•  Quality control •  Ensure that procedures and standards are followed by

the software development team



•  Important:



•  Important: •  Quality management should be separate from project

management to ensure independence


Quality assurance and standards

•  Standards are the key to effective quality management



•  Standards are the key to effective quality management

•  They may be international, national, organizational or project standards



•  Product standards define characteristics that all components should exhibit e.g. a common programming style



•  Product standards define characteristics that all components should exhibit e.g. a common programming style

•  Process standards define how the software process should be enacted


Importance of standards

•  Encapsulation of best practice - avoids repetition of past mistakes




•  Framework for quality assurance process - it involves checking standard compliance




•  Framework for quality assurance process - it involves checking standard compliance

•  Provide continuity - new staff can understand the organization by understand the standards applied


Problems with standards

•  Not seen as relevant and up-to-date by software engineers




•  Involve too much bureaucratic form filling




•  Involve too much bureaucratic form filling

•  Unsupported by software tools so tedious manual work is involved to maintain standards


Testing in PHP Projects

•  Testing is one of the most important tasks of QA.




•  We are going to talk about these testing techniques and tools:




•  We are going to talk about these testing techniques and tools: •  Unit Tests (PHPUnit)





•  Continuous Integration (phpUnderControl)






•  Software Metrics (PHPUnit / PHP_CodeSniffer)






•  Software Metrics (PHPUnit / PHP_CodeSniffer)

•  Acceptance Tests (PHPUnit / Selenium)


Testing Types

•  Unit Tests


Testing Types

•  Unit Tests •  Executable code fragments test the correctness of

parts – units – of the software (system under test).


Testing Types



•  System / Integration Tests


Testing Types



•  System / Integration Tests •  Conducted on a complete, integrated system to

evaluate the system's compliance with ist specified requirements.


Testing Types

•  Non-Functional Tests


Testing Types

•  Non-Functional Tests •  Test of performance, stability or security of

application.


Testing Types


application.

•  Acceptance Tests


Testing Types


application.

•  Acceptance Tests •  Ensure that the code does what the customer wants.


Testing Tools

•  To make testing viable, good tool support is needed.


Testing Tools

•  To make testing viable, good tool support is needed.

•  This is where a testing framework such as PHPUnit comes into play.


Testing Tools

•  Unit Tests •  PHPUnit

•  PHPT

•  SimpleTest


Testing Tools

•  Acceptance Tests •  Selenium Suite

•  Selenium Remote Control •  PHPUnit to talk Selenese to Selenium RC


Testing Tools

•  Non-Functional Tests •  Performance Tests

•  ab •  Apache JMeter


Testing Tools

•  Non-Functional Tests •  Performance Tests

•  ab •  Apache Jmeter

•  Security Tests •  Chorizo •  Selenium and Chorizo


PHPUnit

•  PHPUnit is a member of the xUnit family of testing frameworks (SUnit, jUnit, pyUnit...).


PHPUnit

•  PHPUnit is a member of the xUnit family of testing frameworks (SUnit, jUnit, pyUnit...).

•  Written for PHP 5.


PHPUnit

•  The xUnit frameworks provide a way to automate unit tests.


PHPUnit

•  Make it easy to test, and you'll do it more often.


PHPUnit

•  Make it easy to test, and you'll do it more often.

•  If you use version control, test before every check-in.


PHPUnit

•  The PHPUnit Pocket Guide


PHPUnit


•  Tells you everything you need to know when writing PHPUnit test cases.


PHPUnit


•  Tells you everything you need to know when writing PHPUnit test cases.

•  Buy the book, or read the HTML version at: •  http://www.phpunit.de/pocket_guide/index.en.php


PHPUnit

Let's look at a test case!


PHPUnit <?php require_once 'PHPUnit/Framework.php';

class ArrayTest extends PHPUnit_Framework_TestCase { protected $fixture;

protected function setUp() { // Create the Array fixture. $this->fixture = array(); }

public function testNewArrayIsEmpty() { // Assert that the size of the Array fixture is 0. $this->assertEquals(0, sizeof($this->fixture)); }

public function testArrayContainsAnElement() { // Add an element to the Array fixture. $this->fixture[] = 'Element';

// Assert that the size of the Array fixture is 1. $this->assertEquals(1, sizeof($this->fixture)); } } ?>


PHPUnit

•  Run a PHPUnit test case with the phpunit command.


PHPUnit


phpunit UnitTest UnitTest.php


PHPUnit


phpunit UnitTest UnitTest.php

•  Tells PHPUnit to look for a class UnitTest in the UnitTest.php file (filename is optional).


PHPUnit

•  PHPUnit can output test logs to a variety of formats.


PHPUnit

•  PHPUnit can output test logs to a variety of formats.

•  This allows results to be used for Continuous Integration.


Continuous Integration

•  Continuous integration is a way to automatically run your test suite regularly.




•  Once a day …





•  Once an hour …





•  Once an hour …

•  Every time someone checks in code …


What is integration?

•  Integrating a change (code, database structure …) into the source code repository without breaking the project.


When is integration becoming important?

•  For any change that might break functionality.




•  More than one developer and/or multiple components.




•  More than one developer and/or multiple components.

•  More developers, more components = more complexity.



•  It is a software engineering practice.




•  It describes how to integrate changes frequently.




•  It describes how to integrate changes frequently.

•  It stabilizes the product through self-testing code.



•  It uses rapid feedback to solve integration issues fast and putting the pain upfront the development cycle. (Remember: software defects are more expensive the later they are fixed.)



•  It uses rapid feedback to solve integration issues fast and putting the pain upfront the development cycle. (Remember: software defects are more expensive the later they are fixed.)

•  Goal: produce as many working builds as possible.



•  Without unit tests Continuous Integration does not help a lot.




•  Start using PHPUnit or extend your test suite:




•  Start using PHPUnit or extend your test suite: •  Each identified bug deserves a unit test to prevent

regression.





regression.

•  Develop unit tests for each new feature.





regression.

•  Develop unit tests for each new feature.

•  You don’t need full code coverage. Find the right level!


How to get started with Continuous Integration

•  Keep everything under version control.



•  Keep everything under version control. •  Automation does not work if pieces are missing.




•  Clone your production environment:




•  Clone your production environment: •  At least versions of main components should match.




•  Clone your production environment: •  At least versions of main components should match.

•  Automate the deployment of that environment.



•  Continuous Integration is more than setting up a CI server.




•  You have to live Continuous Integration.





•  Keep in mind: CI is a customized , tailored process:





•  Keep in mind: CI is a customized , tailored process: •  CI in company A is not necessarily the same as in

company B.


Benefits of Continuous Integration

•  Enforce your build policies.




•  Inform developers of build statuses.





•  Make builds and reports available to everyone.





•  Make builds and reports available to everyone.

•  Automate deployments to various testing environments.


Your team and Continuous Integration

•  The team should always see the monitor.




•  Transparency is important – Developers need to see status and bugs.




•  Transparency is important – Developers need to see status and bugs.

•  Define some punishment for breaking the build.


CruiseControl / phpUnderControl

•  The mother of all CI servers.




•  phpUnderControl features:




•  phpUnderControl features: •  Support for different code metric views.





•  It includes, but is not limited to, plugins for email notification, Ant, Phing, and various source control tools.





•  It includes, but is not limited to, plugins for email notification, Ant, Phing, and various source control tools.

•  A web interface is provided to view the details of the current and previous builds.



•  phpUnderControl is customization of CruiseControl that caters to the needs of PHP projects:



•  phpUnderControl is customization of CruiseControl that caters to the needs of PHP projects: •  PHPUnit




•  PHPDocumentor





•  PHP_CodeSniffer






•  PHP_Depend






•  PHP_Depend •  PHP_CompatInfo






•  PHP_Depend •  PHP_CompatInfo

•  … your own tools


Code metrics

•  A software metric is a measured value for a quality of software.


Code metrics

•  A software metric is a measure of some property of a piece of software or is specifications.

•  You cannot control what you cannot measure.


Measuring code quality

•  Lines of code.




•  Text-based metric for code size





•  Several definitions:





•  Several definitions: •  Lines of Code (LOC)






•  Comment Lines of Code (CLOC)







•  Non-Comment Lines of Code (NCLOC)







•  Non-Comment Lines of Code (NCLOC)

•  Executable Lines of Code (ELOC)




•  Ratios are of interest:




•  Ratios are of interest: •  I.e. ratio of CLOC / (E)LOC



•  Code coverage




•  Tells how many lines of code are covered by your unit tests.




•  Tells how many lines of code are covered by your unit tests.

•  100% Code coverage is a required criteria, but it doesn’t tell if your tests do test your code sufficient enough.




•  Which statements, branches, and paths are executed when the tests run?




•  Which statements, branches, and paths are executed when the tests run? •  C0-Coverage: Statement Coverage





•  C1-Coverage: Branch Coverage





•  C1-Coverage: Branch Coverage

•  C2-Coverage: Path Coverage



•  Cyclomatic Complexity




•  Metric for the complexity of a code unit





•  Counting the branching points:





•  Counting the branching points: •  if, for, foreach, while, case, catch, &&, ||, ternary

operator (?:)




•  Interpretation:




•  Interpretation: •  Higher complexity leads to more errors




•  Interpretation: •  Higher complexity leads to more errors

•  Higher complexity makes testing harder


Analyze your code quality

•  Use the following tools:



•  Use the following tools: •  PHPUnit




•  PhpDocumentor •  Generate API documentation





•  PHP_CodeSniffer •  Coding Style •  Correctness of API Documentation






•  Php_Depend •  Analyze code dependencies






•  Php_Depend •  Analyze code dependencies

•  Zend Code Analyzer •  Unreachable Code •  Unused parameters and variables


Collect code analysis

•  Either collect all analysis results in your CI server or build a tool to let your developers see a wrap-up of all analyzer results.




•  Categorize code analysis results in different levels.




•  Categorize code analysis results in different levels.

•  Higher rated issues need to be fixed first.


Continual metrics

•  Metric values are not definite values.


Continual metrics


•  Wire metrics and analysis into continuous integration.


Continual metrics



•  Discuss on iteration boundaries.


Continual metrics



•  Discuss on iteration boundaries.

•  It is important to note that variations of metrics need more attention than its values.


Issue tracking

•  Issues and tasks need to be:


Issue tracking

•  Issues and tasks need to be: •  organized


Issue tracking


•  allocated


Issue tracking


•  allocated

•  scheduled


Issue tracking


•  allocated

•  scheduled

•  tracked


Issue tracking


•  allocated

•  scheduled

•  tracked

•  That’s where an issue tracker comes into play.


Issue tracking

•  An Issue tracker is a software application that is designed to help quality assurance and programmers keep track of reported software bugs in their work.


Issue tracking

•  An Issue tracker is a software application that is designed to help quality assurance and programmers keep track of reported software bugs in their work.

•  Issues may include the time a bug was reported, its severity, the erroneous program behavior, and details on how to reproduce the bug; as well as any programmers who may be working on fixing it.


Issue tracking

•  Typical issue tracking systems support the concept of the life cycle for a bug/feature which is tracked through the status assigned to the issue.


Issue tracking

•  Typical issue tracking systems support the concept of the life cycle for a bug/feature which is tracked through the status assigned to the issue.

•  You can also assign the expected time needed to fix a bug so you can plan how many issues may be solved in one iteration.


Issue tracking

•  If supported by your issue tracker and version control system use a hook script to let developers mark commits to an active issue.


Issue tracking

•  If supported by your issue tracker and version control system use a hook script to let developers mark commits to an active issue.

•  If you require an issue ID in the commit message developers are forced to use the issue tracking software.


Pair Programming

•  Helps write better code by dividing responsibility between two people.


Pair Programming


•  Observer – strategy


Pair Programming


•  Observer – strategy •  Improvements


Pair Programming



•  Future problems


Pair Programming




•  Driver – tactics


Pair Programming




•  Driver – tactics •  Completing the task


Pair Programming

•  Observer functions as safety net and guide.


Pair Programming

•  Observer functions as safety net and guide.

•  Observer and Driver can exchange roles.


Pair Programming

•  Benefits


Pair Programming

•  Benefits •  Design quality


Pair Programming


•  Reduced cost due to fewer defects


Pair Programming



•  Learning & Training


Pair Programming




•  Shared knowledge of product


Pair Programming




•  Shared knowledge of product

•  Improved discipline and time management


Pair Review/Buddy System

•  Review your code/tests regularly with different partners in short meetings.




•  Solve difficult problems.





•  Exchange best practices, ideas, elegant solutions.





•  Exchange best practices, ideas, elegant solutions.

•  Just spread knowledge.


Code Reviews: Too time consuming?

•  Code Reviews may seem to be to be too time consuming when time pressure increases.




•  Without specific approaches to manage the reviews they might be in fact too time consuming.




•  Without specific approaches to manage the reviews they might be in fact too time consuming.

•  These issues are simply problems with some review processes.


Code Reviews: Your return on investment

•  Regular peer review is a proven process with demonstrable ROI.




•  The benefits vary from team to team but commonly include:




•  The benefits vary from team to team but commonly include: •  Identifying bugs and defects early.





•  Sharing expertise and encouraging knowledge transfer.






•  Encouraging adherence to internal standards and style conventions.






•  Encouraging adherence to internal standards and style conventions.

•  Identifying individual strengths and weaknesses.


Code Reviews: Processes are your friend!

•  Defining Code Review Iterations is as important as defining how to do them!


Code Reviews: Processes are your friend!

•  Defining Code Review Iterations is as important as defining how to do them!

•  Don‘t skip any Code Reviews as changes to your source code are more expensive the later you do them!


Code Reviews: What are the best tools?

•  A human being!




•  There is pair programming – why not make it pair reviewing?




•  There is pair programming – why not make it pair reviewing?

•  An initial Code Review is faster and more reliable to be done, if both Reviewer and Developer can communicate directly with each other.


Code Reviews: Web-based tool: Crucible

•  Crucible is a web based Code Review tool by Atlassian Software Systems Pty Ltd.




•  Commercial tool – but available free of charge for Open Source projects.




•  Commercial tool – but available free of charge for Open Source projects.

•  Crucible's mission is to streamline the process aspects so development teams can access the benefits.



•  This is how it looks like:



•  You create a new review from your version control system:



•  Your review is open:



•  Both reviewer and developer can communicate about the code:



•  Commits can either be done after committing to a version control system or using pre-commit diff files.


Testing your web application

•  Now that we covered the code analysis what about the product you are creating?



•  Now that we covered the code analysis what about the product you are creating?

•  We want to have automated tests for our web application that do regression testing, compare results and check whether our application runs flawlessly in all of our supported browsers.



•  Challenges:



•  Challenges: •  Modern GUI app testing (AJAX & Co.)




•  OS / browser / platform




•  OS / browser / platform

•  JavaScript engines / versions / levels



•  There is the Selenium product family:




•  Selenium Core





•  Selenium IDE





•  Selenium IDE

•  Selenium RC



•  Selenium Core:



•  Selenium Core: •  JavaScript engine




•  runs in browser





•  controls web application in other frame





•  controls web application in other frame

•  exposes its API



•  Selenium IDE:



•  Selenium IDE: •  Firefox plugin




•  directly communicates with Selenium Core in the browser





•  Records / replays scenarios





•  Records / replays scenarios

•  great to get started (interactively)



•  Selenium RC:



•  Selenium RC: •  remote control




•  server component and proxy





•  allows test automation in various ways





•  allows test automation in various ways

•  has a capability called Selenium Grid, to drive multiple instances of the Selenium RC server


Advantages of Selenium

•  Tests run directly in browser




•  Implemented entirely using browser technologies:




•  Implemented entirely using browser technologies: •  JavaScript





•  DHTML





•  DHTML

•  Frames



•  Selenium interacts with the DOM served to the browser



•  Selenium interacts with the DOM served to the browser

•  Selenium is agnostic of server side technology


Concepts of Selenium

•  Element Locators:



•  Element Locators: •  specify HTML elements




•  Patterns:




•  Patterns: •  for pattern matching values





•  Actions:





•  Actions: •  manipulate application state



•  Accessors:



•  Accessors: •  store results in variables




•  Assertions:




•  Assertions: •  verify that the application is in a certain state


Useful Selenium Tools

•  For Firefox!



•  For Firefox! •  Web Developer Toolbar




•  Firebug




•  Firebug

•  XPather




•  Firebug

•  XPather

•  Xpath Checker


Maintaining Selenium Tests

•  Test code is just like production code.




•  Needs refactoring, abstraction.




•  Needs refactoring, abstraction.

•  Try to maintain meaning – in test names, variable names, comments, and test suite organization.


Test Fragility

•  Tests are fragile.


Test Fragility


•  They break when your code changes.


Test Fragility


•  They break when your code changes.

•  Stabilize them!


Test Fragility

•  Minimize influences:


Test Fragility

•  Minimize influences: •  Use Mock Objects


Test Fragility


•  Recreate dependencies – SetUp


Test Fragility



•  Clean up your test environment – TearDown


Test Fragility



•  Clean up your test environment – TearDown

•  Use non-fragile locators


Mock Objects

•  Let Unit Tests test the code not the dependencies:


Mock Objects

•  Let Unit Tests test the code not the dependencies: •  Simulate complex objects


Mock Objects


•  Simulate database connections


Mock Objects



•  Simulate uncontrollable variables – time, temperature etc.


Mock Objects



•  Simulate uncontrollable variables – time, temperature etc.

•  Simulate objects that do not exist yet


Test SetUp

•  Use the SetUp method to recreate testable states for functional tests:


Test SetUp

•  Use the SetUp method to recreate testable states for functional tests: •  Use methods that are unit-tested


Test SetUp


•  Fill database with necessary data


Test SetUp


•  Fill database with necessary data

•  Tests can be run on additional fresh test environments.


Test TearDown

•  Use the TearDown method to clean up your test environment:


Test TearDown

•  Use the TearDown method to clean up your test environment: •  Remove no longer necessary database entries


Test TearDown


•  Clean caches, no longer needed log files


Test TearDown



•  Reduces uncontrollable influences on tests.


Test TearDown



•  Reduces uncontrollable influences on tests.

•  Keeps test execution quick.


Non-Fragile Locators

•  When using Selenium, use non-fragile locators:



•  When using Selenium, use non-fragile locators: •  Keep locators independent from DOM




•  Use ids or names





•  Use wildcards where applicable






•  Dom-Locators (JavaScript expressions)






•  Dom-Locators (JavaScript expressions)

•  Page elements can be moved around, but your tests will still work.


Release process

•  Make sure you have a standard procedure for releasing / launching new versions of your product.


Release process


•  Assign responsibility or have a dedicated release management.


Release process


•  Assign responsibility or have a dedicated release management.

•  Use the tools to the fullest.


Release process: Versioning

•  Use the versioning system – choose a stable revision and branch.



•  Use the versioning system – choose a stable revision and branch.

•  Development can continue without breaking the stable branch.



•  If you find defects before next major version you can release stable versions by fixing them in the stable branch.



•  If you find defects before next major version you can release stable versions by fixing them in the stable branch.

•  Reintegrate fixes for bugs into the trunk so they are do not resurface.


Release process: Delta Builder

•  Tool to collect data about stable branch.




•  List changes to the codebase since last release.




•  List changes to the codebase since last release.

•  Have developers review their changes.



•  Highlight important pieces of your product – configuration files, most used libraries etc.



•  Highlight important pieces of your product – configuration files, most used libraries etc.

•  Match versioning data with issue tracker – have all completed tasks been really checked into the versioning system?


Release process: Tests

•  Dedicate time to test the stable branch before release.



•  Dedicate time to test the stable branch before release.

•  Run all your tests as first step – last remaining defects found can be fixed before release.



•  Make sure all your tests pass.



•  Make sure all your tests pass.

•  A manual test can help you assure stability – find defects not identifiable by tests.


Release process: Checklists

•  Minimize human error




•  Collect all tasks for the release




•  Collect all tasks for the release •  Running tests





•  Building dependencies






•  Deploying all components of your product






•  Deploying all components of your product

•  Communicating – notify all those that need to know!


Checklists in general

•  Once again - Minimize human error!




•  All repetitive but complicated tasks:




•  All repetitive but complicated tasks: •  Creating the skeleton of a new module





•  Manually check the functionality of a module






•  Set up a new system ready to be shipped






•  Set up a new system ready to be shipped

•  …


Questions?


Thank You!

ipc2008 slide qa in depth best practises

Education