EE 244: Fundamental Algorithms for System

Modeling, Analysis, and Optimization

Fall 2016

Model Checking

Stavros Tripakis
University of California, Berkeley

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 1 / 68

Recall: the model-checking problems for LTL and

CTL
Given:
the implementation: a transition system (Kripke structure)
M = (AP, S, S0 , L, R)
the specification: a temporal logic (LTL or CTL) formula φ
check where M satisfies φ:
?
M |= φ

If φ is LTL: every execution trace of M must satisfy φ.

If φ is CTL: every initial state of M must satisfy φ.

For finite-state M , the question can be answered fully automatically!

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 2 / 68
ACM Turing Award for Model-Checking
Clarke, Emerson, and Sifakis won the ACM Turing Award in 2007,
for their role in developing Model-Checking into a highly
effective verification technology that is widely adopted in
the hardware and software industries.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 3 / 68

Simplest model-checking problem: checking

invariants
Suppose φ is of the form

Gψ or AGψ

where ψ is a propositional formula (boolean expression on atomic

propositions).
E.g.,
G(p ∨ q), G(p → q), ···

Then ψ is invariant: it must hold at all reachable states.

Examples:
“Whenever train is at intersection the gate must be lowered”
“If the autopilot is off then the pilot must not believe it is on”
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 4 / 68
Reachability Analysis and State-Space Exploration
Suppose we want to model-check an invariant, i.e., check whether
transition system (Kripke structure) M satisfies Gψ, for boolean
expression ψ.
Model checking such formulas is conceptually easy:
Explore (generate) all reachable states of M .
Check that every one of them satisfies ψ. (Is this easy? Why?)

This is called reachability analysis.

For finite-state systems, it can be done exhaustively and fully
automatically!
... at least in theory ... in practice, often state explosion ...

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 5 / 68

Recall: Transition System (Kripke Structure)

A tuple (P, S, S0 , L, R).

P : set of atomic propositions, e.g., P = {p, q}.

S: set of states, e.g., S = {s1 , s2 , s3 }.
S0 : set of initial states, could be more than one, in this example just one: S0 = {s1 }.
L : S → 2P : labeling function, e.g., L(s1 ) = {p, q}, L(s2 ) = {q}, ...
R ⊆ S × S: transition relation, e.g., R = {(s1 , s2 ), (s2 , s1 ), (s2 , s3 ), (s3 , s3 )}.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 6 / 68

Reachable States
Given transition system (P, S, S0 , L, R).

A state s ∈ S is called reachable if there exists a finite path (in the

transition system) reaching that state:
s0 −→ s1 −→ · · · −→ sk , such that k ≥ 0 and sk = s.
The path is formed by initial state s0 ∈ S0 , and transitions
(si , si+1 ) ∈ R, for i = 0, ..., k − 1.
Why would some states be unreachable?
E.g., a counter modulo 10, represented in 4 bits.
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 7 / 68

Caveat: Deadlocks
We have implicitly assumed that our system is deadlock-free.
Deadlock: a state with no successors:

s is a deadlock iff @s0 : s −→ s0

Are deadlocks problematic for checking invariants? Why?

Only infinite paths count for the verification of a property such as
Gp. If the system deadlocks after every time it violates p, then,
formally speaking, it satisfies Gp!

How can we check that a given system is deadlock-free?

Use reachability analysis!

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 8 / 68

Reachability analysis: summary

Generate all reachable states ...

... while at the same time checking that each of them is “OK”,
i.e.,
I it is not a deadlock state
I it does not violate an invariant
I ...

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 9 / 68

Reachability Algorithms

Enumerative (also called “explicit state”).

I These are basically search algorithms on directed graphs.

Symbolic
I Bounded model-checking using SAT/SMT solvers.
I Symbolic reachability.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 10 / 68

An Enumerative Algorithm: Depth-First Search
Assume given: Kripke structure (P, S, S0 , L, R).
main:
1: V := ∅; /* V : set of visited states */
2: for all s ∈ S0 do
3: DFS(s);
4: end for

DFS(s):
1: check s; /* is s a deadlock? is given p ∈ L(s)? ... */
2: V := V ∪ {s};
3: for all s0 such that (s, s0 ) ∈ R do
4: if s0 6∈ V then
5: DFS(s0 ); /* recursive call */
6: end if
7: end for
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 11 / 68

An Enumerative Algorithm: Depth-First Search

Let’s simulate the algorithm on this graph.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 12 / 68

An Enumerative Algorithm: Depth-First Search
Quiz:
Does the algorithm terminate? Yes, if state space is finite.
Does it visit all reachable states? Yes: if s is reachable, then either
s ∈ S0 , or s is the immediate successor of some s0 , which is itself
reachable. In the first case, s is inserted into V because of the main
loop. In the second case, assuming (by induction) that s0 is inserted
to V , s will also be inserted to V by loop in lines 3-6.
Does it visit any unreachable states? No: following the “inverse”
of the argument above, if s is inserted into V , either this is done
because of the main loop, or because of the loop in lines 3-6. In the
first case, s must be in S0 , so it’s an initial state, so it’s reachable.
In the second case, s must be successor of some s0 , which by
induction must be itself in V , therefore reachable.
What is the complexity of the algorithm? O(n + m) where n is
number of nodes/states and m is number of edges/transitions in the
graph. Every node and edge are visited at most once.
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 13 / 68

Other enumerative algorithms

Every search algorithm on finite graphs can be used for

reachability analysis:
DFS: depth-first search
BFS: breadth-first search
Best-first search:
I every state is assigned a “value” (using some heuristic value
function, e.g., how “close” we are likely to be to the goal – in
our case a “bad” state) and then next state to explore is the
one with the highest value.
A*: classic search technique in artificial intelligence.
...

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 14 / 68

Other enumerative algorithms
Every search algorithm on finite graphs can be used for reachability
analysis: DFS, BFS, A*, ...

Most of these have been tried by researchers in verification.

Basic complexity is the same for all: need to store all reachable
states
I in the “worst case” from the algorithmic point of view
I but in fact “best case” from the verification point of view, since
we are trying to prove that our system is correct! ⇒ all
reachable states must be correct
State explosion: the number of reachable states is too large

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 15 / 68

State explosion
How many states does a chip with 100 flip-flops have?
I 2100 (potentially reachable) states.
I That is 1267650600228229401496703205376 states.
I Even if each state costs 1 bit to store, this still makes
2100−60−3 = 237 = 137, 438, 953, 472 exabytes ...
1
I Even if only 32 states are reachable, this still makes
2100−5 = 295 states.
How many states does a piece of concurrent software have?
Assume n asynchronous processes (e.g., threads), with k states
each.
I k n (potentially reachable) states.
What if the processes also communicate with queues? Consider
a single queue of size ` (i.e., can hold ` messages), and m
possible types of messages.
I m` (potentially reachable) states for just one queue.
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 16 / 68
The real complexity of reachability

Searching a graph is linear in the size of the graph, which appears to

be a very nice worst-case complexity ...

... until we realize that the size of the graph is exponential in the
number of state variables, processes, etc.

This is not just a practical observation. There is theoretical

complexity results about this, e.g., checking intersection emptiness of
a set of DFA is PSPACE-complete.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 17 / 68

Enumerative methods to remedy state explosion

Bit-state hashing: instead of storing the entire state vector, just store 1
bit per state: its hash value [Holzmann, 1998].
I Do you see a problem with this method?
I Incomplete: two states may hash to the same value ⇒ only
one will be visited ⇒ some reachable states may be missed!
I And as we saw, even 1 bit per state may be too much already.
Partial-order reduction: in asynchronous concurrent systems, transitions
of different processes are often independent ⇒ no need to explore all
interleavings [Valmari, 1990, Godefroid and Wolper, 1991].
Symmetry reduction: many state spaces are symmetric ⇒ equivalence
relation on states ⇒ suffices to explore just one state per equivalence
class [Ip and Dill, 1996, Clarke et al., 1998, Sistla and Godefroid, 2004].
...
All these help, but don’t eliminate the state-explosion problem.
Note: above references are representative, there is a lot more work on these
topics.
In-depth discussion: Computer-Aided Verification course by Sanjit Seshia.
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 18 / 68
 SYMBOLIC METHODS

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 19 / 68

Symbolic Methods: Why?

Motivation: attack the state explosion problem.

A seminal paper: Symbolic model checking: 1020 states and

beyond. [Burch et al., 1990].
1020 is less than 267 , so still not quite enough for modern circuits.
Nevertheless: a great leap forward at that time.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 20 / 68

Symbolic Representation of State Spaces

Key idea:
Instead of reasoning about individual states, reason about
sets of states.

How do we represent a set of states?

Symbolic representation:
Set = predicate.
Set of states = predicate on state variables.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 21 / 68

Symbolic Representation of Sets of States

Examples:

1 Assume 3 state variables, p, q, r, of type boolean.

S1 : p ∨ q = {pqr, pqr, pqr, pqr, pqr, pqr}

2 Assume 3 state variables, x, i, b, of types real, integer, boolean.

S2 : x > 0 ∧ (b → i ≥ 0)

How many states are in S2 ?

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 22 / 68

Symbolic Representation of Transition Relations

Key idea:
Use a predicate on two copies of the state variables:
unprimed (current state) + primed (next state).

If ~x is the vector of state variables, then the transition relation R is a

predicate on ~x and ~x0 :
R(~x, ~x0 )
e.g., for three state variables, x, i, b:

R(x, i, b, x0 , i0 , b0 )

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 23 / 68

Symbolic Representation of Transition Relations

Examples:

1 Assume one state variable, p, of type boolean.

R1 : (p → ¬p0 ) ∧ (¬p → p0 )

Which transition relation does this represent? Is it a relation or a

function (deterministic)?

2 Assume one state variable, n, of type integer.

R2 : n0 = n + 1 ∨ n0 = n

Which transition relation does this represent? Is it a relation or a

function (deterministic)?

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 24 / 68

Symbolic Representation of Kripke Structures
Kripke structure:
(P, S, S0 , L, R)

Symbolic representation:

(P, Init, Trans)

where
P = {x1 , x2 , ..., xn }: set of (boolean) state variables, also taken
to be the atomic propositions.1
Predicate Init(~x) on vector ~x = (x1 , ..., xn ) represents the set
S0 of initial states.
Predicate Trans(~x, ~x0 ) represents the transition relation R.
Basis of the language of NuSMV.
1
this is done for simplicity, the two could be separated
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 25 / 68

Example: NuSMV model

MODULE inverter(input)
VAR
output : boolean;
INIT
output = FALSE
TRANS
next(output) = !input | next(output) = output

What is the Kripke structure defined by this NuSMV program?

What about P and L?

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 26 / 68

Example: Kripke Structure

Represent this symbolically.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 27 / 68

SYMBOLIC REACHABILITY ANALYSIS

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 28 / 68

Recall: Symbolic Representation of Kripke
Structures

(P, Init, Trans)

where
P = {x1 , x2 , ..., xn }: set of boolean state variables, also taken
to be the atomic propositions.
Predicate Init(~x) on vector ~x = (x1 , ..., xn ) represents the set
S0 of initial states.
Predicate Trans(~x, ~x0 ) represents the transition relation R.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 29 / 68

Recall: Symbolic Representation

Set of states = predicate φ(~x) on vector of state variables ~x.

E.g.:
I Init(x, y, z) : x ∧ ¬y
I Bad (x1 , x2 ) : x1 = crit ∧ x2 = crit

Transition relation = predicate Trans(~x, ~x0 ) on state variables

and next-state variables. E.g.:
I Trans(x, y, x0 , y 0 ) : x0 = x + 1 ∧ (y 0 = 0 ∨ y 0 = 1)

How do we perform set-theoretic operations with predicates?

I Union of two sets represented by φ1 and φ2 : φ1 ∨ φ2 .
I Intersection of two sets represented by φ1 and φ2 : φ1 ∧ φ2 .
I Complement of a set represented by φ: ¬φ.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 30 / 68

Symbolic Reachability Analysis

Main idea:
Start with set of initial states S0 .
Compute S1 := S0 ∪ {all 1-step successors of S0 }.
Compute S2 := S1 ∪ {all 1-step successors of S1 }.
...
Until Sk+1 = Sk .
Sk contains all reachable states.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 31 / 68

Computing Successors Symbolically

Given a set of states represented as a predicate φ(~x).

We want to compute a new predicate φ0 , representing the set of all

1-step successors of states in φ(~x).

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 32 / 68

Predicate Transformer
Successors can be computed by a predicate transformer :

succ φ(~x) := ∃~x : φ(~x) ∧ Trans(~x, ~x0 ) [~x0 ; ~x]

I ∃~x : φ(~x) ∧ Trans(~x, ~x0 ): successors of states in φ

I [~x0 ; ~x]: renames variables so that resulting predicate is over
current state variables

Example:
φ = 0≤x≤5
Trans = x ≤ x0 ≤ x + 1
succ(φ) = (∃x : 0 ≤ x ≤ 5 ∧ x ≤ x0 ≤ x + 1)[x0 ; x]
= (∃x : 0 ≤ x ≤ 5 ∧ 0 ≤ x0 ≤ 5 + 1)[x0 ; x]
= (0 ≤ x0 ≤ 6)[x0 ; x]
= 0≤x≤6
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 33 / 68

Predicate Transformer

succ φ(~x) := ∃~x : φ(~x) ∧ Trans(~x, ~x0 ) [~x0 ; ~x]

How to do quantifier elimination automatically?

In the case of propositional logic, quantifier elimination is simple.
Suppose x is a boolean variable:

∃x : φ ⇔ φ[x ; 0] ∨ φ[x ; 1]

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 34 / 68

Predicate Transformer: Another Example

succ(p ∧ q) = (∃p, q : p ∧ q ∧ Trans)[p0 ; p, q 0 ; q]

= (∃p, q : p ∧ q ∧ p0 ∧ q 0 )[p0 ; p, q 0 ; q]
= (p0 ∧ q 0 )[p0 ; p, q 0 ; q]
= p∧q

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 35 / 68

Symbolic Reachability Analysis Algorithm

1: Reachable := Init;
2: terminate := false;
3: repeat
4: tmp := Reachable ∨ succ(Reachable);
5: if tmp ⇔ Reachable then
6: terminate := true;
7: else
8: Reachable := tmp;
9: end if
10: until terminate
11: return Reachable;

Does the algorithm terminate? Why?

Quiz: modify the algorithm to make it check reachability of a set of
bad states characterized by predicate Bad .
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 36 / 68
Symbolic Reachability: checking for Bad states
1: Reachable := Init;
2: terminate := false;
3: error := false;
4: repeat
5: tmp := Reachable ∨ succ(Reachable);
6: if tmp ⇔ Reachable then
7: terminate := true;
8: else
9: Reachable := tmp;
10: end if
11: if SAT(Reachable ∧ Bad ) then
12: error := true;
13: end if
14: until terminate or error
15: return (Reachable,error);
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 37 / 68

Symbolic Reachability: Example

Let’s check this system symbolically!

We want to check that all reachable states satisfy p ∨ q.
In temporal logic parlance:

CTL: AG(p ∨ q)
LTL: G(p ∨ q)

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 38 / 68

Symbolic Model-Checking: Implementation
For finite-state systems, boolean variables can be used to encode
state.
All predicates then become boolean expressions.
Efficient data structures for boolean expressions:
I BDDs (Binary Decision Diagrams) [Bryant, 1992] (paper
available in bcourses - follow link from lectures web page)
Efficient algorithms for implementing logical operations
(conjunction, disjunction, satisfiability check, ...) on BDDs.
Note: logical operations correspond to set-theoretic operations:
I Conjunction: intersection
I Disjunction: union
I Satisfiability check: emptiness check
I ...

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 39 / 68

Example: BDD

Can you guess which boolean expression this BDD represents?

x4 x3 (x2 + x2 x1 ) + x3 (x2 x1 + x2 ) + x4 x2 x1

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 40 / 68

 BDDs

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 41 / 68

Binary decision trees

Binary decision tree:

A tree representing all possible variable assignments, and
corresponding truth values of a boolean expression.
For n variables, the tree has 1 + 2 + 22 + · · · + 2n = 2n+1 − 1
nodes (including the leaves).

Let’s draw the binary decision tree for

(z1 ∧ z3 ) ∨ (z2 ∧ z3 )

(assuming the order of variables z1 , z2 , z3 ).

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 42 / 68

 From binary decision trees to BDDs
Main idea: make the representation compact (i.e., smaller) by
eliminating redundant nodes.

If two subtrees (including leaves) T1 and T2 are identical then

keep only T1 . All incoming links to T2 are redirected to T1 .
If both the true-branch and the false-branch of a node v lead to
the same node v 0 , then node v is redundant: v can be removed,
with its incoming links being redirected to v 0 .
394 Computation Tree Log

The result is a reduced ordered binary decision diagram

(ROBDD).
It is a DAG: directed acyclic graph.
We often use BDD to mean ROBDD.
Let’s try this on the following formulas: z1

Computation
(z1 z∧2 zTree Logic
a + b, and 3 ) ∨ (z2 ∧ z3 ) z2
394 Computation Tree Logic
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 43 / 68
z3 z3 z3 z3

1 0 1 0 1 0 0 0
From binary decision trees to BDDs
z1
z1
z2 z2 z1

z2 z2
z2 z2
z3 1: z3 z3 z3
z3
z3 z3 z3 z3
1 0 1 0 1 0 0 0
1 0
1 0 1 0 1 0 0 0

z1 z1

z1 z2
z2 z2
2: 3:
z2 z2
z3 z3

z3
1 0 1 0

1 0
z1
Figure taken from [Baier and Katoen, 2008].
Figure 6.22: Binaryz1decision diagrams for f = (z1 ∧ z3 ) ∨ (z2 ∧ z3 ).
z2
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 44 / 68
z2
BDDs: a canonical representation of boolean
functions
ROBDDs are a canonical representation of boolean functions.
This means that two boolean functions (or expressions) f1 and f2 are
equivalent iff their corresponding ROBDDs (for the same variable
ordering) are identical.

Is this an important property? What is an example where it is useful?

Recall the symbolic reachability algorithm stopping criterion:

tmp ⇔ Reachable

If B and B 0 are the BDDs representing tmp and Reachable,

respectively, then tmp ⇔ Reachable holds iff B and B 0 are identical.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 45 / 68

The bad news: variable ordering matters greatly

BDD size depends on variable ordering
I For the same boolean function, different variable orderings may
result BDDs which are very different in size.
I For example, consider the function

(x1 ∧ y1 ) ∨ (x2 ∧ y2 ) ∨ (x3 ∧ y3 )

and the two orderings:

x1 , y1 , x2 , y2 , x3 , y3

and
x1 , x2 , x3 , y1 , y2 , y3
Some BDDs have exponential size no matter which ordering we
pick.
Deciding whether a given order is optimal is NP-hard.
Land of heuristics ...
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 46 / 68
Operations on BDDs

We want to compute set-theoretic, or equivalently, logical, operations

on BDDs:

Check for emptiness / satisfiability.

Check for universality / validity.
Intersection / conjunction.
Union / disjunction.
Complementation / negation.

Which of these operations are easy to perform on ROBDDs?

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 47 / 68

Operations on BDDs

Check for emptiness / satisfiability.

I Check whether the BDD is the leaf 0. If yes ⇒ empty / unsat.

Check for universality / validity.

I Check whether the BDD is the leaf 1. If yes ⇒ valid.

Complementation / negation.
I Replace the leaf 0 with 1, and 1 with 0.

We next look at conjunction and disjunction.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 48 / 68

Shannon expansion
Let f be a boolean expression and x be a boolean variable.
Recall:
f [x ; 0]
denotes the new formula f 0 obtained by replacing any occurrence of
x in f by 0.
Similarly for f [x ; 1].

f [x ; 1] and f [x ; 0] are called the (positive and negative)

cofactors of f , and are denoted fx and fx .

Then
f ⇔ x · fx + x · fx
| {z }
this is called the Shannon expansion of f
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 49 / 68

Shannon expansion

f ⇔ x · fx + x · fx

This is the essence of binary decision trees and BDDs: if f is the

root, then
fx is the sub-tree rooted at the 0-branch (“false”-branch) child
of f
fx is the sub-tree rooted at the 1-branch (“true”-branch) child
of f

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 50 / 68

Recursive application of boolean operations based
on Shannon expansion
Suppose  is some boolean operation (e.g., conjunction or
disjunction).
Let f and g be two boolean expressions, and x be a boolean variable
(usually f and g refer to x, but they don’t have to).
Then
f g ⇔ x · (fx  gx ) + x · (fx  gx )
For instance, if  is conjunction:

f ·g ⇔ x · fx · gx + x · fx · gx

This leads to the apply function.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 51 / 68

The apply function

Takes as input:
I A boolean operation  (e.g., conjunction or disjunction).
I Two BDDs Bf and Bg (with the same variable ordering)
representing two boolean functions f and g.

Computes as output:
I A BDD B representing f  g:

B = apply(, Bf , Bg ) such that B ⇔ Bf g

Operates recursively based on Shannon expansion.

Resulting BDD may not be reduced, so needs to be generally
reduced afterwards.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 52 / 68

The apply function
We are computing apply(, Bf , Bg ). Let vf and vg be the root nodes of
Bf and Bg respectively.

There are the following cases to consider:

1 Both vf and vg are leaves (i.e., 0 or 1). Then, apply returns the leaf
BDD with truth value vf  vg .
2 Both vf and vg are internal x-nodes, i.e., corresponding to variable
x. Then, let Bfx , Bgx be the positive sub-BDDs (i.e., positive
cofactors, i.e., BDDs rooted at the true-branch children) of vf and
vg , respectively; and similarly with Bfx , Bgx . Then:
1 Recursively compute BDD Bx := apply(, Bfx , Bgx ).
2 Recursively compute BDD Bx := apply(, Bfx , Bgx ).
3 Create and return a new BDD with root x and Bx as positive
sub-BDD and Bx as negative sub-BDD.
The justification for this comes directly from
f g ⇔ x · (fx  gx ) + x · (fx  gx )
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 53 / 68

The apply function (continued)

3 vf is an internal x-node, but vg is either a leaf (0 or 1) or an internal
y-node, with y > x, i.e., variable y is after x in the ordering (y is
lower in the tree). Then we know, since Bf and Bg must follow the
same variable ordering, that Bg is independent from x at this point
in the tree. So we proceed as follows:
1 Recursively compute BDD Bx := apply(, Bfx , Bg ).
2 Recursively compute BDD Bx := apply(, Bfx , Bg ).
3 Create and return a new BDD with root x and Bx as positive
sub-BDD and Bx as negative sub-BDD.
Do you see room for optimization here?
E.g., when  is + and vg is 0 or 1. If 0, return vf . If 1, return 1.
4 Symmetric to case 3 above, but with vg being higher in the tree than
vf instead of lower.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 54 / 68

The apply function: example
Let’s try apply(+) on the two BDDs below:
6.2 Algorithms for reduced OBDDs 375

R1 x1 S1 x1

R2 x2

x3 R3 x3 S2
+

R4 x4 S3 x4

R5 R6 S4 S5
0 1 0 1

Figure 6.15. An example of two arguments for a call apply (+, Bf , Bg ).

Figure taken from [Huth and Ryan, 2004].

of BTripakis
Stavros f and(UC
BgBerkeley)
downwards
to construct nodes
EE 244, Fall 2016 of the OBDD Bf op g . Let
Model rf be
Checking 55 / 68
the root node of Bf and rg the root node of Bg .

1. If both rf and rg are terminal nodes with labels lf and lg , respectively (recall
Existential quantifier elimination
that terminal labels are either 0 or 1), then we compute the value lf op lg and
let the resulting OBDD be B0 if that value is 0 and B1 otherwise.
2. In the remaining cases, at least one of the root nodes is a non-terminal. Suppose
Recallthatthat if x is a boolean variable then:
both root nodes are xi -nodes. Then we create an xi -node n with a dashed
line to apply (op, lo(rf ), lo(rg )) and a solid line to apply (op, hi(rf ), hi(rg )), i.e.
∃xapply
we call ⇔ f [x
: f recursively on ; 0] ∨ off [x
the basis ; 1] ⇔ fx ∨ fx
(6.2).
3. If rf is an xi -node, but rg is a terminal node or an xj -node with j > i,
Let Bthen we know that there is no x -node in Bg because
f be the BDD for f . Howi to compute
the twofor
the BDD OBDDs
∃x : have
f?
a compatible ordering of boolean variables. Thus, g is independent of xi
We know (g ≡ g[0/x
howi ]to≡ g[1/x i ]). Therefore,
compute we create
disjunction of an xi -node
BDDs n with aItdashed
already. sufficesline to
to apply (op, lo(rf ), rg ) and a solid line to apply (op, hi(rf ), rg ).
be 4.able
Thetocase
compute
in which rsubstitutions like f [x ; 0].
g is a non-terminal, but rf is a terminal or an xj -node with
j > i, is handled symmetrically to case 3.
This is simple:
The result of this procedure might not be reduced; therefore apply finishes
For every x-node v in Bf , eliminate v and redirect all incoming
by calling the function reduce on the OBDD it constructed. An example of
links to the 0-child of v.
apply (where op is +) can be seen in Figures 6.15–6.17. Figure 6.16 shows
the(Ifrecursive
we wanted f [x
descent ; 1]structure
control instead,of we would
apply redirect
and Figure them
6.17 showstothe
the
final result.of
1-child this example, the result of apply (+, Bf , Bg ) is Bf .
In v.)
Figure 6.16 shows that numerous calls to apply occur several times with
We
the same must then reduce
arguments. Efficiencythecould
resulting BDD.
be gained if these were evaluated only

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 56 / 68

Putting it all together
Recall: Symbolic Reachability Analysis Algorithm
1: Reachable := Init;
2: terminate := false;
3: repeat
4: tmp := Reachable ∨ succ(Reachable);
5: if tmp ⇔ Reachable then
6: terminate := true;
7: else
8: Reachable := tmp;
9: end if
10: until terminate
11: return Reachable;
where
x0 ) [~


0
succ φ(~
x) := ∃~
x : φ(~
x) ∧ Trans(~
x, ~ x ;~
x]
We have all the ingredients to implement this algorithm using BDDs:
Init, Reachable, tmp are each represented as a BDD on state variables ~x.
Trans is represented as another BDD on ~x, ~x0 .
We know how to compute ∧, ∨, ∃ on BDDs.
Renaming variables [~x0 ; ~x] is straightforward also.
We know how to check ⇔ on BDDs.
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 57 / 68

FINITE-HORIZON REACHABILITY
(a.k.a. BOUNDED MODEL-CHECKING)

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 58 / 68

Bounded reachability
Question:

Can a “bad” state be reached in up to n steps (transitions)?

i.e., given a transition system (P, S, S0 , L, R) and a set of states

Bad ⊆ S, does there exist a path
s0 −→ s1 −→ · · · −→ sk
in the transition system such that s0 ∈ S0 and sk ∈ Bad , and k ≤ n.

Key idea:

Reduce the above question to a SAT (satisfiability) problem.

SAT problem NP-complete for propositional logic.

In practice, today’s SAT solvers can handle formulas with thousands
of variables (or more!): see [Malik and Zhang, 2009].
BMC (bounded model-checking)
Stavros Tripakis (UC Berkeley)
has emerged thanks
EE 244, Fall 2016
to the
Model Checking 59 / 68
advances in SAT solver technology.

Bounded reachability
Suppose I have predicates Init(~x), Trans(~x, ~x0 ), and Bad (~x).
How to use them for bounded reachability?

Bad state reachable in 0 steps iff

SAT Init(~x) ∧ Bad (~x)

Bad state reachable in 1 step iff

SAT Init(~x0 ) ∧ Trans(~x0 , ~x1 ) ∧ Bad (~x1 )

...
Bad state reachable in n steps iff


SAT Init(~x0 )∧Trans(~x0 , ~x1 )∧· · ·∧Trans(~xn−1 , ~xn )∧Bad (~xn )

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 60 / 68

Bounded reachability algorithm – outer loop

1: for all k = 0, 1, ..., n do

2: φ := Init(~x0 )∧Trans(~x0 , ~x1 )∧· · ·∧Trans(~xk−1 , ~xk )∧Bad (~xk );
3: if SAT(φ) then
4: print “Bad state reachable in k steps”;
5: output solution as counter-example;
6: end if
7: end for
8: print “Bad state unreachable up to n steps”;

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 61 / 68

Bounded reachability: soundness and completeness

1: for all k = 0, 1, ..., n do
2: φ := Init(~x0 ) ∧ Trans(~x0 , ~x1 ) ∧ · · · ∧ Trans(~xk−1 , ~xk ) ∧ Bad (~xk );
3: if SAT(φ) then
4: print “Bad state reachable in k steps”;
5: output solution as counter-example;
6: end if
7: end for
8: print “Bad state unreachable up to n steps”;

BMC algorithm is sound in the following sense:

if algorithm reports “reachable” then indeed a bad state is reachable
if algorithm reports “unreachable up to n steps” then there is no
path of length ≤ n that reaches a bad state.

Can we make BMC complete?

It should report unreachable iff there are no reachable bad states
(w.r.t. any bound).
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 62 / 68
Is this even possible in general? For finite-state systems? Yes!
Complete BMC: “brute-force” threshold
1: for all k = 0, 1, ..., n do
2: φ := Init(~x0 ) ∧ Trans(~x0 , ~x1 ) ∧ · · · ∧ Trans(~xk−1 , ~xk ) ∧ Bad (~xk );
3: if SAT(φ) then
4: print “Bad state reachable in k steps”;
5: output solution as counter-example;
6: end if
7: end for
8: print “Bad state unreachable up to n steps”;

A finite-state transition system is essentially a finite graph.

How can we turn BMC into a complete method for finite-state
systems?
If we know |S| (the number of all possible states) then we can set
n := |S|.
Because no acyclic path can have length greater than |S|, and we
only care about acyclic paths.
Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 100 Model Checking
But: with 100 boolean variables, |S| = 2 , so this isn’t practical 63.../ 68
(formulas become too big).

Complete BMC: a better threshold

Reachability diameter: number of steps that it takes to reach any

reachable state.

d := min{i | ∀s ∈ Reach : ∃ path s0 , s1 , ..., sj : j ≤ i∧s0 ∈ S0 ∧sj = s}

where Reach is the set of reachable states.

d is generally a much better threshold than |S|. Why?

d ≤ |Reach| ≤ |S|.

Problem: we don’t know |Reach|, therefore how to compute d?

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 64 / 68

Complete BMC: the Completeness Threshold
Recurrence diameter : length of the longest cycle-free path.

r := max{i | ∃ path s0 , s1 , ..., si : s0 ∈ S0 ∧∀0 ≤ j < k ≤ i : sj 6= sk }

Claim: d ≤ r. Why?
⇒ using r instead of d is safe. Why?
Can we compute r? How?
Use a SAT solver!

r := max{i | SAT Init(~x0 ) ∧ Trans(~x0 , ~x1 ) ∧ · · · ∧ Trans(~xi−1 , ~xi )
i−1
^ i
^ 
∧ ~xj 6= ~xk }
j=0 k=j+1

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 65 / 68

Bibliography I
Baier, C. and Katoen, J.-P. (2008).
Principles of Model Checking.
MIT Press.
Biere, A., Cimatti, A., Clarke, E. M., Strichman, O., and Zhu, Y. (2003).
Bounded model checking.
Advances in Computers, 58:117–148.

Bryant, R. E. (1992).
Symbolic boolean manipulation with ordered binary-decision diagrams.
ACM Comput. Surv., 24(3):293–318.

Burch, J., Clarke, E., Dill, D., Hwang, L., and McMillan, K. (1990).
Symbolic model checking: 1020 states and beyond.
In 5th LICS, pages 428–439. IEEE.

Clarke, E., Emerson, E., Jha, S., and Sistla, A. (1998).

Symmetry reductions in model checking.
In CAV’98, pages 147–158. Springer.

Clarke, E., Grumberg, O., and Peled, D. (2000).

Model Checking.
MIT Press.
Courcoubetis, C., Vardi, M., Wolper, P., and Yannakakis, M. (1992).
Memory efficient algorithms for the verification of temporal properties.
Formal Methods in System Design, 1:275–288.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 66 / 68

Bibliography II
Godefroid, P. and Wolper, P. (1991).
Using partial orders for the efficient verification of deadlock freedom and safety properties.
In 4th CAV.
Holzmann, G. (1998).
An analysis of bitstate hashing.
In Formal Methods in System Design, pages 301–314. Chapman & Hall.

Huth, M. and Ryan, M. (2004).

Logic in Computer Science: Modelling and Reasoning about Systems.
Cambridge University Press.

Ip, C. and Dill, D. (1996).

Better verification through symmetry.
Formal Methods in System Design, 9(1-2):41–75.

Latvala, T., Biere, A., Heljanko, K., and Junttila, T. (2004).

Simple Bounded LTL Model Checking.
In Formal Methods in Computer-Aided Design, volume 3312 of LNCS, pages 186–200. Springer.

Malik, S. and Zhang, L. (2009).

Boolean satisfiability: From theoretical hardness to practical success.
Communications of the ACM, 52(8):76–82.

Sistla, A. P. and Godefroid, P. (2004).

Symmetry and reduced symmetry in model checking.
ACM Trans. Program. Lang. Syst., 26(4):702–734.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 67 / 68

Bibliography III

Valmari, A. (1990).
Stubborn sets for reduced state space generation.
LNCS 483.

Stavros Tripakis (UC Berkeley) EE 244, Fall 2016 Model Checking 68 / 68