dissertation committee: efficient sampling of sat and smtrtd/talks/phd_talk.pdf · efficient...
TRANSCRIPT
Efficient Sampling of SAT and SMT Solutions for Testing and VerificationRafael Tupynambá DutraEECS DepartmentUC Berkeley
Dissertation committee:Koushik Sen, Jonathan Bachrach, Sanjit Seshia, Theodore SlamanMay 10, 2019 https://events.berkeley.edu/index.php/calendar/sn/eecs.html?event_ID=125634(Last modified September 2019)
A constraint solver can generate one solution:
Problem definition
Input: SAT (Boolean Satisfiability) or SMT (Satisfiability Modulo Theories)
2
mem[0] mem[1]
σ0
∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])
1 0 0 0 1 0 0 0
mem ∈ Array(BV[4], BV[4])
Goal: Quickly generate lots of solutions that satisfy the constraint
Problem definition
3
mem[0] mem[1]
σ0
σ1
σ2
σ3
σ4
σ5
1 0 0 0 1 0 0 0
0 0 0 0 1 0 1 0
1 1 0 0 1 0 0 1
1 0 0 0 0 1 0 0
0 1 0 0 1 0 1 1
0 0 0 0 0 1 1 0
∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])
mem ∈ Array(BV[4], BV[4])
Input: SAT (Boolean Satisfiability) or SMT (Satisfiability Modulo Theories)
● Symbolic execution
Motivation
● Thoroughly exercising some target functionality● Constrained-Random Verification
4
int4 x, y, z, w;int4 mem[4] = {x, y, z, w};for (int4 i = 0; i < 4; ++i) {
mem[mem[i]] *= -1;}
i < 4
mem[0] < 0∨ mem[0] ≥ 4
Applications
● Testing○ Uniform Sampling of SAT Solutions for Configurable Systems: Are We There Yet? [ICST 2019]
● Synthesis○ Bug Synthesis: Challenging Bug-Finding Tools with Deep Faults [ESEC/FSE 2018]
● Weighted Sampling○ Current work
5
Example
void main(int4 x, int4 y) {if ((x & y) == 4) {
if (x < y) {interesting(x, y);
}}
}
6
void main(int4 x, int4 y) {if ((x & y) == 4) {
if (x < y) {interesting(x, y);
}}
}
Example: SMT formula
7
∧ ((x & y) == 4)∧ (x < y)
Path Constraint
void main(int4 x, int4 y) {if ((x & y) == 4) {
if (x < y) {interesting(x, y);
}}
}Bit-blast
Example
8
∧ ((x & y) == 4)∧ (x < y)
Path Constraint
x3 x2 x1 x0
int4 x;
y3 y2 y1 y0
int4 y;
void main(int4 x, int4 y) {if ((x & y) == 4) {
if (x < y) {interesting(x, y);
}}
}Bit-blast
Example: SAT formula
∧(¬x0 ∨ ¬y0)∧ (¬x1 ∨ ¬y1)∧ x2∧ y2∧ (¬x3 ∨ ¬y3)∧ ( ∨ (x3 ∧ ¬y3)∧ ( ∨ (x3=y3 ∧ ¬x2 ∧ y2)∧ ( ∨ (x3=y3 ∧ x2=y2 ∧ ¬x1 ∧ y1)∧ ( ∨ (x3=y3 ∧ x2=y2 ∧ x1=y1 ∧ ¬x0 ∧ y0)∧ )
9
∧ ((x & y) == 4)∧ (x < y)
Path Constraint
Boolean formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
x3 x2 x1 x0
int4 x;
y3 y2 y1 y0
int4 y;
Example: SAT formula
∧(¬x0 ∨ ¬y0)∧ (¬x1 ∨ ¬y1)∧ x2∧ y2∧ (¬x3 ∨ ¬y3)∧ ( ∨ (x3 ∧ ¬y3)∧ ( ∨ (x3=y3 ∧ ¬x2 ∧ y2)∧ ( ∨ (x3=y3 ∧ x2=y2 ∧ ¬x1 ∧ y1)∧ ( ∨ (x3=y3 ∧ x2=y2 ∧ x1=y1 ∧ ¬x0 ∧ y0)∧ )
10
Boolean formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
SMT: Satisfiability Modulo Theories
11
SMT formula φ
∧(mem[0] ≥ 0 ∧ mem[0] < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wheremem’ = store(mem, mem[0], -1 * mem[mem[0]])
mem ∈ Array(BV[4], BV[4])
mem ∈ Array(BV[4], BV[4])
SMT: Satisfiability Modulo Theories
12
SMT formula φBit-vector
∧(mem[0] ≥ 0 ∧ mem[0] < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wheremem’ = store(mem, mem[0], -1 * mem[mem[0]])
SMT: Satisfiability Modulo Theories
13
SMT formula φBit-vector
Array
mem ∈ Array(BV[4], BV[4])
∧(mem[0] ≥ 0 ∧ mem[0] < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wheremem’ = store(mem, mem[0], -1 * mem[mem[0]])
State of the art
● Markov Chain Monte Carlo (MCMC)○ Works for linear constraints and can generate biased solutions
● Constraint solver heuristics○ Can be expensive, requiring one solver call per solution
● Universal hashing○ Expensive, but can guarantee uniform sampling
14
QuickSampler
15
Rafael Dutra, Kevin Laeufer, Jonathan Bachrach, and Koushik Sen. 2018. Efficient Sampling of SAT Solutions for Testing. In ICSE’18.
QuickSampler
Our goals:
● Samples should satisfy the formula >50% of the times
● Generate samples >100x faster than other techniques
● Sampling should be close to uniform
Our approach:
● Compute patterns of bit flips which preserve satisfiability
● Combine those bit flip patterns to generate lots of samples
16
QuickSampler
Our goals:
● Samples should satisfy the formula >50% of the times
● Generate samples >100x faster than other techniques
● Sampling should be close to uniform
Our approach:
● Compute patterns of bit flips which preserve satisfiability
● Combine those bit flip patterns to generate lots of samples
17
QuickSampler Algorithm
● Start by generating a random assignment σ’
Random assignment
σ′
18
σ
QuickSampler Algorithm
● Start by generating a random assignment σ’
● Use MAX-SAT to find the closest solution σ to σ’
Random assignment
Base solution
σ′
19
σ1
σ0
σ
QuickSampler Algorithm
● Start by generating a random assignment σ’
● Use MAX-SAT to find the closest solution σ to σ’
● From σ, use MAX-SAT to find the closest solutions that flip some bits (such as σ0 and σ1)
Random assignment
Base solution
Closest solutions
σ′
20
σ1
σ01
σ0
σ
QuickSampler Algorithm
● Start by generating a random assignment σ’
● Use MAX-SAT to find the closest solution σ to σ’
● From σ, use MAX-SAT to find the closest solutions that flip some bits (such as σ0 and σ1)
● Combine those mutation to generate new samples (such as σ01)
Random assignment
Base solution
Closest solutions
Generated samples
σ′
21
22
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
23
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
24
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ 0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
MAX-SAT
25
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ 0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
σ0
MAX-SAT
26
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
σ0
MAX-SAT
27
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
σ0
MAX-SAT
28
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0 0 1 1 1 1 0 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
σ1
σ0
MAX-SAT
29
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0 0 1 1 1 1 0 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
σ1
σ0
MAX-SAT
30
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0 0 1 1 1 1 0 1 0 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
σ1
σ0
MAX-SAT
31
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
σ1
σ0
MAX-SAT
32
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
σ1
σ0
MAX-SAT
33
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
σ1 σ3
σ0
MAX-SAT
34
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
...
σ1 σ3
σ0
MAX-SAT
35
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0
...
σ1 σ3
δ31 0 0 0 1 0 0 0 δ1δ0
= σ ⊕ σ0 = σ ⊕ σ1 = σ ⊕ σ3
δ01
σ0
MAX-SAT
36
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0
...
σ1 σ3
δ31 0 0 0 1 0 0 0 δ1δ0
1 1 0 1 1 1 0 0
= δ0 ∨ δ1
σ01
δ01
σ0
MAX-SAT
37
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0
...
σ1 σ3
δ31 0 0 0 1 0 0 0 δ1δ0
1 1 0 1 1 1 0 0
1 1 1 1 0 0 1 0
= σ ⊕ δ01
σ01
δ01
σ0
MAX-SAT
38
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0
...
σ1 σ3
δ31 0 0 0 1 0 0 0 δ1δ0
1 1 0 1 1 1 0 0
1 1 1 1 0 0 1 0
= σ ⊕ δ01
Why does it work?● δ0 and δ1 are a minimal set of bits that can be
flipped and preserve the satisfiability of the formula● It’s likely that the formula has some clauses
establishing a relation between those bits● Those clauses will likely still be satisfied when
flipping both the bits in δ0 and δ1
σ01
δ01 δ03
σ0
MAX-SAT
39
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0
...
1 0 0 1 1 0 0 0
σ1 σ3
δ31 0 0 0 1 0 0 0 δ1δ0
1 1 0 1 1 1 0 0
1 1 1 1 0 0 1 0 = δ0 ∨ δ3
σ01
δ01 δ03
σ03
σ0
MAX-SAT
40
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0
...
1 0 0 1 1 0 0 0
1 0 1 1 0 1 1 0
σ1 σ3
δ31 0 0 0 1 0 0 0 δ1δ0
1 1 0 1 1 1 0 0
1 1 1 1 0 0 1 0
= σ ⊕ δ03
σ01
δ01 δ03 δ13
σ03
σ0
MAX-SAT
41
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0
...
1 0 0 1 1 0 0 0
1 0 1 1 0 1 1 0
σ1 σ3
δ31 0 0 0 1 0 0 0 δ1δ0
1 1 0 1 1 1 0 0
1 1 1 1 0 0 1 0
0 1 0 1 0 1 0 0
= δ1 ∨ δ3
σ01
δ01 δ03 δ13
σ03
σ0
MAX-SAT
42
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0
...
1 0 0 1 1 0 0 0
1 0 1 1 0 1 1 0
σ1 σ3
δ31 0 0 0 1 0 0 0 δ1δ0
1 1 0 1 1 1 0 0
1 1 1 1 0 0 1 0
0 1 0 1 0 1 0 0
= δ1 ∨ δ3
σ01
δ01 δ03 δ013
σ03
σ0
MAX-SAT
43
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0
...
1 0 0 1 1 0 0 0
1 0 1 1 0 1 1 0
σ1 σ3
δ31 0 0 0 1 0 0 0 δ1δ0
1 1 0 1 1 1 0 0
1 1 1 1 0 0 1 0
1 1 0 1 1 1 0 0
= δ0 ∨ δ1 ∨ δ3
σ01
δ01 δ03 δ013
σ03
σ0
MAX-SAT
44
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0
...
1 0 0 1 1 0 0 0
1 0 1 1 0 1 1 0
σ1 σ3
δ31 0 0 0 1 0 0 0 δ1δ0
1 1 0 1 1 1 0 0
1 1 1 1 0 0 1 0
1 1 0 1 1 1 0 0
= δ0 ∨ δ1 ∨ δ3
σ01
δ01 δ03
σ03
σ0
MAX-SAT
45
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0
...
1 0 0 1 1 0 0 0
1 0 1 1 0 1 1 0
σ1 σ3
δ31 0 0 0 1 0 0 0 δ1δ0
1 1 0 1 1 1 0 0
1 1 1 1 0 0 1 0
σ01
δ01 δ03
σ03
σ0
MAX-SAT
46
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0
...
1 0 0 1 1 0 0 0
1 0 1 1 0 1 1 0
σ1 σ3
δ31 0 0 0 1 0 0 0 δ1δ0
1 1 0 1 1 1 0 0
1 1 1 1 0 0 1 0
At most ngenerate atomic mutations
=O(n6)mutations: NO MAX-SAT ( ) n
6 samples by combining
MAX-SAT calls to
σ01
δ01 δ03
σ03
σ0
MAX-SAT
47
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0
...
1 0 0 1 1 0 0 0
1 0 1 1 0 1 1 0
σ1 σ3
δ31 0 0 0 1 0 0 0 δ1δ0
1 1 0 1 1 1 0 0
1 1 1 1 0 0 1 0
At most 50generate atomic mutations
15 890 700mutations: NO MAX-SAT
samples by combining
MAX-SAT calls to
σ01
δ01 δ03
σ03
σ0
MAX-SAT
48
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0
...
1 0 0 1 1 0 0 0
1 0 1 1 0 1 1 0
σ1 σ3
δ31 0 0 0 1 0 0 0 δ1δ0
1 1 0 1 1 1 0 0
1 1 1 1 0 0 1 0
σ01
δ01 δ03
σ03
σ0
MAX-SAT
49
Formula φ(x0,x1,x2,x3,y0,y1,y2,y3)
Solution σ
1 0 1 0 0 1 1 0UNSAT
0 1 1 1 1 0 1 0 0 0 1 1 1 1 1 0
0 0 1 0 1 1 1 0
0 0 1 0 1 1 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SAT
0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0
...
1 0 0 1 1 0 0 0
1 0 1 1 0 1 1 0
σ1 σ3
δ31 0 0 0 1 0 0 0 δ1δ0
1 1 0 1 1 1 0 0
1 1 1 1 0 0 1 0
Implementation
● Implemented in C++ using Z3 as the constraint solver● https://github.com/RafaelTupynamba/quicksampler
Optimizations:
● Eager generation of samples● Independent support● Unsatisfiable variables
50
Experiments
We evaluated QuickSampler on 163 industrial benchmarks.
51
Largest 6 Benchmarks # Variables # Clauses
tutorial3.sk_4_31 486193 2598178
diagStencilClean.sk_41_36 378131 2110471
karatsuba.sk_7_41 19594 82417
enqueueSeqSK.sk_10_42 16466 58515
20.sk_1_51 15475 60994
77.sk_3_44 14535 27573
Experiments
We compared QuickSampler against two state-of-the-art samplers:● UniGen2 [1]
○ Uses universal hashing to partition the solution space and produce provably uniform samples
● SearchTreeSampler [2]○ Generates pseudo-solutions: partial assignments that can be completed to full solutions
[1] Supratik Chakraborty, Daniel J Fremont, Kuldeep S Meel, Sanjit A Seshia, and Moshe Y Vardi. 2015. On Parallel Scalable Uniform SAT Witness Generation. In TACAS 2015.[2] Stefano Ermon, Carla P Gomes, and Bart Selman. 2012. Uniform solution sampling using a constraint solver as an oracle. In UAI 2012.
52
Experiments: Correctness
53
# of Atomic Mutations combined
Average Samples generated
% of Valid Samples
0 1 100%
1 32 100%
2 511 96%
3 5 619 93%
4 47 493 89%
5 346 367 82%
6 2 143 385 73%
Total 2 543 409 75%
● QuickSampler generates valid solutions
○ 102.5±0.8 times faster than SearchTreeSampler
○ 104.7±1.0 times faster than UniGen2● QuickSampler generates unique valid solutions
○ 102.3±0.7 times faster than SearchTreeSampler
○ 104.4±1.1 times faster than UniGen254
Experiments: Speed
55
Higher is betterNumber ofvalid solutionsper time
Experiments: Unique Solutions
56
Higher is betterNumber of uniquevalid solutionsper time
Experiments: Unique Solutions
57
Higher is better
Experiments: Uniformity
58
Challenges
Problems of encoding SMT into SAT
● Converting into SAT loses high-level SMT structure● SMT structure can be used for faster solving● High level structure can also help generate diverse solutions
● Diversity of samples: (x ≥ 4) ∨ φ(x, y, z)
59
SMTSampler
60
Rafael Dutra, Jonathan Bachrach and Koushik Sen. 2018. SMTSampler: Efficient Stimulus Generation from Complex SMT Constraints. In ICCAD’18.
SMTSampler
Our goals:
● Allow efficient sampling from large and complex SMT constraints
● Generate millions of unique solutions in minutes
● Achieve high coverage of the constraint space
Our approach:
● Extend QuickSampler technique to work over SMT formulas with:○ Bit-vectors○ Arrays○ Uninterpreted functions
61
SMTSampler
Our goals:
● Allow efficient sampling from large and complex SMT constraints
● Generate millions of unique solutions in minutes
● Achieve high coverage of the constraint space
Our approach:
● Extend QuickSampler technique to work over SMT formulas with:○ Bit-vectors○ Arrays○ Uninterpreted functions
62
Types of variables
63
Improvements over QuickSampler
● Extend the mutations to work over bit-vectors, arrays and uninterpreted functions
● Output only valid and unique solutions● Adaptive generation of solutions based on accuracy● Improved scalability for more complex formulas
64
65
Formula φ
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
66
Formula φx = mem[0]y = mem[1]
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
67
Solution σ 1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
68
Solution σ 1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SMT
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
69
Solution σ 1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SMT
MAX-SMT
Hard constraints● φ● x0 ≠ 1
Soft constraints● x1 = 0● x2 = 0● x3 = 0● y0 = 1● y1 = 0● y2 = 0● y3 = 0
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
SMTbit
70
Solution σ 1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SMT
MAX-SMT
Hard constraints● φ● x0 ≠ 1
Soft constraints● y = 1000
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
SMTbv
σ0
71
Solution σ
0 0 0 0 1 0 1 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SMT
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ0
72
Solution σ
0 0 0 0 1 0 1 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
MAX-SMT
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ0
73
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
σ1
MAX-SMT
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ0
74
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
σ1
MAX-SMT
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ0
75
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
σ1
MAX-SMT
MAX-SMT
1
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ0
76
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
σ1
MAX-SMT
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ0
77
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
σ1
MAX-SMT
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ0
78
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
σ1
MAX-SMT
MAX-SMT
1
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ0
79
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
σ1
MAX-SMT
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ0
80
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
σ1
MAX-SMT
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ0
81
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
σ1 σ4
MAX-SMT
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
...
σ0
82
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
σ1 σ4
MAX-SMT
MAX-SMT
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ0
83
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
...
σ1 σ4
δ41 0 0 0 0 0 1 0 δ1δ0
MAX-SMT
MAX-SMT
= σ ⊕ σ0 = σ ⊕ σ1 = σ ⊕ σ4
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
δ01
σ0
84
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
...
σ1 σ4
δ41 0 0 0 0 0 1 0 δ1δ0
1 1 0 0 0 0 1 1
MAX-SMT
MAX-SMT
= δ0 ∨ δ1
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ01
δ01
σ0
85
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
...
σ1 σ4
δ41 0 0 0 0 0 1 0 δ1δ0
1 1 0 0 0 0 1 1
0 1 0 0 1 0 1 1
MAX-SMT
MAX-SMT
= σ ⊕ δ01
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ01
δ01 δ04
σ0
86
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
...
1 0 0 0 1 1 1 0
σ1 σ4
δ41 0 0 0 0 0 1 0 δ1δ0
1 1 0 0 0 0 1 1
0 1 0 0 1 0 1 1
MAX-SMT
MAX-SMT
= δ0 ∨ δ4
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ01
δ01 δ04
σ04
σ0
87
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
...
1 0 0 0 1 1 1 0
0 0 0 0 0 1 1 0
σ1 σ4
δ41 0 0 0 0 0 1 0 δ1δ0
1 1 0 0 0 0 1 1
0 1 0 0 1 0 1 1
MAX-SMT
MAX-SMT
= σ ⊕ δ04
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ01
δ01 δ04 δ14
σ04
σ0
88
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
...
1 0 0 0 1 1 1 0
0 0 0 0 0 1 1 0
σ1 σ4
δ41 0 0 0 0 0 1 0 δ1δ0
1 1 0 0 0 0 1 1
0 1 0 0 1 0 1 1
0 1 0 0 1 1 0 1
MAX-SMT
MAX-SMT
= δ1 ∨ δ4
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ01
δ01 δ04 δ14
σ04
σ0
89
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0
...
1 0 0 0 1 1 1 0
0 0 0 0 0 1 1 0
σ1 σ4
δ41 0 0 0 0 0 1 0 δ1δ0
1 1 0 0 0 0 1 1
0 1 0 0 1 0 1 1
0 1 0 0 1 1 0 1
MAX-SMT
MAX-SMT
σ141 1 0 0 0 1 0 1
= σ ⊕ δ14
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ01 σ04
σ0
90
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
...
0 0 0 0 0 1 1 0
σ1 σ4
0 1 0 0 1 0 1 1
MAX-SMT
MAX-SMT
σ141 1 0 0 0 1 0 1
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ01 σ04
σ0
91
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
...
0 0 0 0 0 1 1 0
σ1 σ4
0 1 0 0 1 0 1 1
MAX-SMT
MAX-SMT
σ141 1 0 0 0 1 0 1
17 / 18 valid solutions
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ01 σ04
σ0
92
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
...
0 0 0 0 0 1 1 0
σ1 σ4
0 1 0 0 1 0 1 1
MAX-SMT
MAX-SMT
σ141 1 0 0 0 1 0 1
σ0140 1 0 0 0 1 1 1
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ01 σ04
σ0
93
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
...
0 0 0 0 0 1 1 0
σ1 σ4
0 1 0 0 1 0 1 1
MAX-SMT
MAX-SMT
σ141 1 0 0 0 1 0 1
σ0140 1 0 0 0 1 1 1 6 / 8 valid solutions
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ01 σ04
σ0
94
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
...
0 0 0 0 0 1 1 0
σ1 σ4
0 1 0 0 1 0 1 1
MAX-SMT
MAX-SMT
σ141 1 0 0 0 1 0 1
σ0140 1 0 0 0 1 1 1
0 new solutions
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ01 σ04
σ0
95
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
...
0 0 0 0 0 1 1 0
σ1 σ4
0 1 0 0 1 0 1 1
MAX-SMT
MAX-SMT
σ141 1 0 0 0 1 0 1
σ0140 1 0 0 0 1 1 1
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
σ01 σ04
σ0
96
Solution σ
0 0 0 0 1 0 1 0UNSAT
1 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0
1 0 0 0 1 0 0 0
1 0 1 0 1 0 0 0Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3
...
0 0 0 0 0 1 1 0
σ1 σ4
0 1 0 0 1 0 1 1
MAX-SMT
MAX-SMT
σ141 1 0 0 0 1 0 1
σ0140 1 0 0 0 1 1 1
Formula φx = mem[0]y = mem[1]
(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]])
Implementation
● Implemented in C++ using Z3 as the constraint solver● https://github.com/RafaelTupynamba/SMTsampler
97
Experiments on SMT-LIB
We evaluated SMTSampler on 213 industrial benchmarks from 22 classes.
98
Benchmark Class Average # Nodes Average # Bits
QF_AUFBV/ecc 291 2785
QF_ABV/bmc-arrays 855 53
QF_ABV/stp_samples 1139 192
QF_BV/bmc-bv-svcomp14 7518 7607
QF_BV/tacas07 8812 16620
QF_BV/sage/app8 978 1047
Experiments
We compared two approaches for SMT sampling against the SAT-based QuickSampler● SMTbit
○ One soft constraint per bit
● SMTbv○ One soft constraint per bit-vector
● QuickSampler○ Our prior work that generates solutions to SAT formulas (Boolean constraints)
99
Experiments: Unique Solutions
100
Higher is betterNumber of uniquesolutionsper time
Experiments: Unique Solutions
101
Higher is betterNumber of uniquesolutionsper time
Measuring Coverage of a Formula
● Look at values of internal nodes○ Check if each bit has received values 0 and 1
in the test cases
● Analogous to internal wires in a circuit
102
Experiments: Coverage
103
Higher is betterCoverageof the formula
Experiments: Coverage
104
Higher is betterCoverageof the formula
Results
● The SMT-based approaches outperformed the SAT-based approach both in terms of solutions generated and coverage of the formula
○ If considering total execution time, the difference is accentuated
● SMTbit generally performs better than SMTbv due to more fine-grain soft constraints
● But SMTbv is more robust for formulas where MAX-SMT queries are harder to solve
105
Challenges
● Coverage of the formula might still not be ideal even using the SMT-based approaches
● User might be interested in a specific notion of coverage for the produced solutions
106
GuidedSampler
107
Rafael Dutra, Jonathan Bachrach and Koushik Sen. 2019. GuidedSampler: Coverage-guided Sampling of SMT Solutions. In FMCAD’19.
Goal: Generate solutions to φ such that the predicates ψ1, ψ2, …, ψn are covered uniformly
Problem definition
108
mem[0] mem[1]
σ0
σ1
σ2
σ3
σ4
σ5
1 0 0 0 1 0 0 0
0 0 0 0 1 0 1 0
1 1 0 0 1 0 0 1
1 0 0 0 0 1 0 0
0 1 0 0 1 0 1 1
0 0 0 0 0 1 1 0
∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])
mem’[1] < 0
Input: SMT formula φ
Input: Coverage predicates ψ1, ψ2, …, ψn
mem’[1] ≥ 4 mem’[0] < 0
Goal: Generate solutions to φ such that the predicates ψ1, ψ2, …, ψn are covered uniformly
Problem definition
109
mem[0] mem[1] ψ1 ψ2 ψ3
σ0
σ1
σ2
σ3
σ4
σ5
1 0 0 0 1 0 0 0
0 0 0 0 1 0 1 0
1 1 0 0 1 0 0 1
1 0 0 0 0 1 0 0
0 1 0 0 1 0 1 1
0 0 0 0 0 1 1 0
∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4)∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4),wherex = mem[0],y = mem[1],mem’ = store(mem, mem[0], -1 * mem[mem[0]])
mem’[1] < 0
Input: SMT formula φ
0 1 0
1 0 0
0 0 1
1 1 1
0 1 1
1 1 0
Input: Coverage predicates ψ1, ψ2, …, ψn
mem’[1] ≥ 4 mem’[0] < 0
● Explore all states
Motivation
● User-specified coverage metrics
110
Coverage-guided Sampling
Our goals:
● Sample solutions from a formula φ, but have the distribution determined by the coverage predicates ψ1, ψ2, …, ψn
● Uniformly sample solutions from the different coverage classes
● Uniformly sample within each coverage class
Our approach extends SMTSampler by:
● Randomizing coverage class of initial base solution
● Flipping coverage predicates to compute neighboring solutions
● Discarding new solutions that repeat a previously seen coverage class
111
Modifications
Random assignment
σ′
112
σ
Modifications
● In the MAX-SMT query to generate σ, set coverage predicates to random values
Random assignment
Base solution
σ′
113
σ1
σ0
σ
Modifications
● In the MAX-SMT query to generate σ, set coverage predicates to random values
● Find neighboring solutions that flip coverage predicates
Random assignment
Base solution
Closest solutions
σ′
114
σ1
σ01
σ0
σ
Modifications
● In the MAX-SMT query to generate σ, set coverage predicates to random values
● Find neighboring solutions that flip coverage predicates
● Whenever generating a new sample, discard it if it’s from a repeated coverage class
Random assignment
Base solution
Closest solutions
Generated samples
σ′
115
Experiments: Unique Coverage Classes
116
Higher is better
GuidedSampler vs. SMTSampler
Number of uniquecoverage classesper time
Experiments: Unique Coverage Classes
117
Higher is better
S3 = GuidedSamplerS0 = SMTSamplerBS, BH: baselines
Number of uniquecoverage classesper time
Experiments: Uniformity over Coverage Classes
118
S3 = GuidedSamplerS0 = SMTSamplerBS, BH: baselines
→ GuidedSampler generated > 100 000 classes
Conclusion
● Generate solutions efficiently given a SAT or SMT formula
● Generate millions of solutions with tens of solver calls
● Achieve better coverage of the constraint space, even for user-defined coverage classes
119
σ01 = σ ⊕ δ01
δ01 = δ0 ∨ δ1
σ0
Solution σ
0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 1
1 0 0 0 1 0 0 0
0 1 0 0 0 0 0 1
σ1
1 0 0 0 0 0 1 0 δ1δ0
1 1 0 0 0 0 1 1
0 1 0 0 1 0 1 1
MAX-SMT
Acknowledgements
120
Acknowledgements
121
Acknowledgements
122