probabilistic stochastic test data
TRANSCRIPT
-
8/14/2019 Probabilistic Stochastic Test Data
1/21
Bj RollisonTest Architect
Microsoft
http://www.TestingMentor.comhttp://blogs.msdn.com/imtesty
-
8/14/2019 Probabilistic Stochastic Test Data
2/21
The test data dilemma
Benefits and drawbacks of static test data
Trips and traps of random data generation
Effective data decomposition
Effective sampling techniques
-
8/14/2019 Probabilistic Stochastic Test Data
3/21
Large number of input variablesVirtually infinite permutations of variables
Impractical to test all permutationsExample:NetBIOS name 15 alphanumeric characters
ASCII only chars, 82 allowable characters(0x20 \ * + = | : ; ? < > , ) invalid*Total number of possible input tests equals
(82 15 + 82 14 + 82 13 + 82 1)51,586,566,049,662,994,687,009,994,574
Thats a RBN (really big number)!
-
8/14/2019 Probabilistic Stochastic Test Data
4/21
Static test data Random test data
Tester generatedExperienceSystem knowledgeLimited populationSpecialized knowledge
Computer generated
Increases breadthEliminates human biasNot representativeViolates constraintsNot reproducible
Customer dataDomain expertise
Test data (tribal knowledge)Historical failure indicators
Disadvantages
Limited in scopeDiminishing effectivenessOutdatedMisused
-
8/14/2019 Probabilistic Stochastic Test Data
5/21
Static dataCustomer data
Domain/business expertiseTest dataLibrary of historical failure indicators
Generally limited in scopeLoses effectiveness for multiple iterations
Random dataTester generated data
Experience, intuition
Limited input population, keyboard mappingComputer generated data
-
8/14/2019 Probabilistic Stochastic Test Data
6/21
Not representative does not look realIf the data is representativeof the total population thenany permutation of theelements is allowable.
Violates constraintsData decomposition(equivalence class subsets)
Deterministic algorithmsNot reproducible .
Seeded random generation
-
8/14/2019 Probabilistic Stochastic Test Data
7/21
-
8/14/2019 Probabilistic Stochastic Test Data
8/21
Pseudo-random number generatorsProvides a sequence of numbers that meetcertain statistical requirements for randomnessElements chosen with equal probability from afinite set
Most use a date/time seed by defaultBut, must be able to pass a parameterizedconstructor as a seed value for repeatability
Not perfect, but reasonably random for practicalpurposeslets see!
-
8/14/2019 Probabilistic Stochastic Test Data
9/21
Define representative data sets (valid and invalid)Example Credit card numbers
341846580149320Card length (BIN + digits)between 14 and 19 depending on card type
Bank IdentificationNumber (BIN) between 1 and 4digits dependingon card type
Checksum Luhn (Mod 10) algorithm
-
8/14/2019 Probabilistic Stochastic Test Data
10/21
Equivalence class partitioning decomposes datainto discrete valid and invalid class subsets
Card type Valid Class subsets Invalid Class subsets
AmericanExpress BIN 34, 37Length 15 digitsChecksum Mod 10
Unassigned BINsLength = 14 digitsFail Checksum
Maestro BIN 5020, 5038,6034, 6759Length 16, 18Checksum Mod 10
Unassigned BINsLength = 19 digitsLength == 17 digitsFail Checksum
Input variable Valid input Invalid input
-
8/14/2019 Probabilistic Stochastic Test Data
11/21
348702004783719
Random credit card number
RandomBIN
RandomLength
RandomNumbers
Pseudorandom
generator
Seedvalue
Seedvalue
-
8/14/2019 Probabilistic Stochastic Test Data
12/21
One random generator and seed per test run!Dynamic seed
Seed variable must be preserved in test logfor repeatability!User seed
Tester provides seed value for repeatability
private int seedVal = 0;
public int SeedValue{
get { seedVal = GenerateSeed(); }set { seed = value; }
}
-
8/14/2019 Probabilistic Stochastic Test Data
13/21
Assigned BINs ensures the data looks realThe Mod10 check ensures the data feels real
Result is representative of real data!
GetCardNumberGet BINGet CardLengthAssign BIN to cardNumber;
Generate a new random object;for (cardNumberLength < CardLength)
Generate a random number 0 9Append it to the cardNumber
if Not_Valid_Card_Numberwhile Not_Valid_Card_Numberincrement last number by 1
return cardNumber;
Deterministic
algorithm togenerate a validrandom creditcard
-
8/14/2019 Probabilistic Stochastic Test Data
14/21
JCB Type 1
BIN = 35 Len = 16JCB Type 2
BIN = 1800, 2131, Len = 15
-
8/14/2019 Probabilistic Stochastic Test Data
15/21
Modeltestdata
Generatetest data
Applytestdata
Verifyresults
Decompose the
data set for eachparameter usingequivalence classpartitioning
Generate valid
and invalid testdata adhering toparameter properties,business rules, andtest hypothesis
Apply the test
data to theapplicationunder test
Verify the actual
results againstthe expectedresults oracle!
-
8/14/2019 Probabilistic Stochastic Test Data
16/21
RobusttestingMulti-languageinputtesting
String lengthfixed or variable
Seed value
Custom range forgreater controlUnicode
languagefamilies
Assigned codepoints
Reservedcharacters
Unicode surrogate
pairs
-
8/14/2019 Probabilistic Stochastic Test Data
17/21
1000 Unicode charactersfrom the sample population
-
8/14/2019 Probabilistic Stochastic Test Data
18/21
Character corruption anddata loss
135 characters (bytes)obvious data loss
-
8/14/2019 Probabilistic Stochastic Test Data
19/21
Static test data wears out!Recklessly generated random test data that is notrepeatable or not representative may find defects,or may throw a lot of false negativesProbabilistic stochastic test data
Modeled representation of the populationStatistically unbiasedTests robustnessIncreases breadth of data coverage
Increased value in using both!
-
8/14/2019 Probabilistic Stochastic Test Data
20/21
http://[email protected]
http://hwtams.com
-
8/14/2019 Probabilistic Stochastic Test Data
21/21
Practice .NET Testing with IR DataBj Rollisonhttp://www.stpmag.com/issues/stp-2007-06.pdf
Automatic test data generation for path testingusing a new stochastic algorithmBruno T. de Abreu, Eliane Martins, Fabiano L. de Sousahttp://www.sbbd-sbes2005.ufu.br/arquivos/16-%209523.pdf
Data Generation Techniques for AutomatedSoftware Robustness TestingMatthew Schmid & Frank Hill
http://www.cigital.com/papers/download/ictcsfinal.pdfToolshttp://www.TestingMentor.com