Randomness and testing Analysis, Lecture 12 Claire Le Goues February 24, (c) 2015 C. Le Goues Learning goals Define random testing, describe its benefits and drawbacks, contrast with partition testing, and enumerate an example tool. Describe fuzz testing, including its technical distinction from regular random testing, and the defects its particularly well suited to find. Explain how randomness can help with testing for robustness, usability, integration, and performance testing. Define mutation testing and explain why its useful. 2(c) 2015 C. Le Goues Switch statements and V(G) a more complicated question than youd think. Short answer: # of cases. Long answer: counting switch statements in terms of actual number of edges can lead to misleadingly high complexity numbers. McCabe suggested just ignoring them. 3(c) 2015 C. Le Goues Random testing Testing: verb The more or less thorough execution of the software with the purpose of finding bugs before the software is released for use and to establish that the software performs as expected. Random: adjective 1.proceeding, made, or occurring without definite aim, reason, or pattern: the random selection of numbers. 2.Statistics. of or characterizing a process of selection in which each item of a set has an equal probability of being chosen , Dictionary.com (c) 2015 C. Le Goues In a nutshell Select inputs independently at random from the programs input domain: Identify the input domain of the program. Map random numbers to that input domain. Select inputs from the input domain according to some probability distribution. Determine if the program achieves the appropriate outputs on those inputs. Random testing can provide probabilistic guarantees about the likely faultiness of the program. E.g., Random testing using ~23,000 inputs without failure (N = 23, 000) establishes that the program will not fail more than one time in 10,000 (F = 10 4 ), with a confidence of 90% (C = 0.9). 5(c) 2015 C. Le Goues Random testing 6 The systematic variation of values through the input space with the purpose of identifying abnormal output patterns. When such patterns are identified a root cause analysis is conducted to identify the source of the problem. In this case the state 3 outputs seem to be missing When Only Random Testing Will Do Dick Hamlet, 2006 (c) 2015 C. Le Goues Why use random testing? Its cheap, assuming you solve the oracle problem. You may need more tests to find the same number of faults, but generating tests is very easy. Can calculate the reliability of an application using established probability theory (next slide). Can augment release criteria. E.g., no random failures for 3 weeks prior to a release under a given random testing protocol. Good when: Lack of domain knowledge makes it difficult or meaningless to partition the input space into equivalence classes. When the amount of state information is important. When large volume of data are necessary, such as for load testing, stress testing, robustness testing, or reliability calculations. Useful complement to partition testing. 7(c) 2015 C. Le Goues When to use Random Testing Lack of domain knowledge makes it difficult or meaningless to partition input into equivalence classes. The amount of state information is important. When large volume of data are necessary: Load testing Stress testing Robustness testing Reliability calculations To complement partition testing 8(c) 2015 C. Le Goues Mathematical reliability Assume program P has a constant failure rate of. The probability that P will fail a given test is ; that it will succeed,. On N independent tests: probability of universal success: probability of at least one failure: is the confidence probability that a failure will occur no more often than once in runs. Solve for, which is also the mean time to failure: The number of tests required to attain confidence in this MTTF: 9(c) 2015 C. Le Goues Challenges Selecting or sampling from the input space. Uniform distribution: uniform random selection Equispaced: unsampled gaps are the same size Operational profile (only makes sense at the system level) Proportional sampling: sample according to subdomain distribution (partition the input space) Adaptive sampling: take the pattern of previously-identified failure-causing inputs into consideration in sample strategy. The oracle problem: a test case is an input, an expected output, and a mechanism for determining if the observed output is consistent with the expect output. Root cause analysis: how to debug using a randomly generated input 10(c) 2015 C. Le Goues 1.if((((l_421 || (safe_lshift_func_uint8_t_u_u (l_42 1, 0xABE574F6L))) && 2.(func_77(func_38((l_424 >= l_425), g_394, g_30.f0), func_8(l_408, g_345[2], 3.g_7, (*g_165), l_421), (*l_400), func_8(((*g_349) != (*g_349)), (l_426 != 4.(*l_400)), (safe_lshift_func_int16_t_s_u((**g_349), 0xD5C55EF8L)), 0x0B1F0B62L, 5.g_95), (safe_add_func_uint32_t_u_u((*g_165),l_431)))^ 6.((safe_rshift_func_uint8_t_u_s(((*g_165)>=(**g_349)),(safe_mul_func_int8_t_s_s 7.((*g_165), l_421))))