Design Verification

Mike Butts
Synopsys
Prof. Kurt Keutzer
Dr. Serdar Tasiran
EECS
UC Berkeley
Mike Butts
1

Design Process

Design : specify and enter the

design intent

Verify:
verify the Implement:
correctness of refine the
design and design
implementation through all
phases

Kurt Keutzer 2

1
 Design Verification
HDL
specification
RTL manual
Synthesis design

netlist Is the
Library/
module logic design
generators optimization consistent
with the original
netlist specification?

physical Is what I think I want

design what I really want?
layout

Kurt Keutzer 3

Implementation Verification
HDL

RTL manual
Synthesis design

netlist a 0

1
d
q
Is the
b

Library/
logic
s clk
implementation
module
generators optimization consistent
with the original
netlist
a

b
0

1
d
q design intent?
s clk

physical Is what I
design implemented
what I
layout
wanted?
Kurt Keutzer 4

2
 Manufacture Verification (Test)
HDL Is the
manufactured
RTL manual circuit
Synthesis design
consistent
netlist a 0 d
with the
implemented
q
b 1

Library/
module logic
s clk
design?
generators optimization
a

b
0

1
d
q
Did they
netlist s clk
build
what I
physical wanted?
design

layout

Kurt Keutzer 5

Design Verification
HDL
specification
RTL manual
Synthesis design

netlist Is the
Library/
module logic design
generators optimization consistent
with the original
netlist specification?

physical Is what I think I want

design what I really want?
layout

Kurt Keutzer 6

3
Verification is an Industry-Wide Issue

Intel: Processor project verification:

“Billions of generated vectors”
“Our VHDL regression tests take 27 days to run. ”
Sun: Sparc project verification:
Test suite ~1500 tests > 1 billion random simulation cycles
“A server ranch ~1200 SPARC CPUs”
Bull: Simulation including PwrPC 604
“Our simulations run at between 1-20 CPS.”
“We need 100-1000 cps.”
Cyrix : An x86 related project
“We need 50x Chronologic performance today.”
“170 CPUs running simulations continuously”
Kodak: “hundreds of 3-4 hour RTL functional simulations”
Xerox: “Simulation runtime occupies ~3 weeks of a design cycle”
Ross: 125 Million Vector Regression tests

Design
Design Teams
Teams are
are Desperate
Desperate for
for Faster
Faster Simulation
Simulation
Kurt Keutzer 7

Verification Gap

10,000,000 Logic Tr./Chip 100,000,000

Tr./S.M.
Logic Transistors per Chip

.10µ 1,000,000 10,000,000

Trans./Staff - Mo.

100,000 58%/Yr. compound 1,000,000

Productivity

Complexity growth rate

.35µ 10,000 100,000
Verification Gap
(K)

1,000 10,000
x
100 x x 1,000
x x x
x
2.5µ 10 21%/Yr. compound 100
Productivity growth rate
1 10
1991

1999
2001
2003

2007
1987
1989

1993
1995
1997

2005

2009
1983
1985
1981

Source:
SEMATECH

Kurt Keutzer 8

4
Why the Gap?

logic_transistors lines_in_design bugs

X X
chip logic_transistors line_of_design

bugs
=
chip

Kurt Keutzer 9

Filling in Reasonable Numbers

logic_transistors lines_of_design bugs

X X
chip logic_transistors lines_of_design

10,000,000 trs 1 1
X X
chip 10 10,000

100 bugs
=
chip
Kurt Keutzer 10

5
Raising the Level of Abstraction

logic_transistors lines_of_design bugs

X X
chip logic_transistors lines_of_design

10,000,000 trs 1 1
X X
chip 100 10,000

10 bugs
= this year!!
chip
Kurt Keutzer 11

Moore’s Law Implies More Bugs

logic_transistors lines_of_design bugs

X X
chip logic_transistors lines_of_design

100,000,000 trs 1 1
X X
chip 100 10,000

100 bugs
= within 5 years!!
chip
Kurt Keutzer 12

6
The Verification Bottleneck

Verification problem grows even faster due to the

combination of increased gate count and increased vector count

10,000x more Vectors

Required to Validate

100 x 10,000 =
10B 2002 1 million times
more Simulation Load
100M 1996
1M
1990

100k 1M 10M
100x Gate Count

Kurt Keutzer 13

Time to boot VxWorks M. Butts - Synopsys

1 million instructions, assume 2 million cycles

Today’s verification choices:

50M cps: 40 msec Actual system HW

5M cps: 400 msec Logic emulator 1 (QT Mercury)
500K cps: 4 sec Cycle-based gate accelerator 1 (QT CoBALT )
50K cps: 40 sec Hybrid emulator/simulator 2 (Axis)
5K cps: 7 min Event-driven gate accelerator 2 (Ikos NSIM)
500 cps: 1.1 hr
50 cps: 11 hr CPU and logic in HDL simulator 3 (VCS)
5 cps: 4.6 days

1: assumes CPU chip 2: assumes RTL CPU 3: assumes HDL CPU

Kurt Keutzer 14

7
Aspects of Design Verification

Event Driven
Specification
Event-driven Simulation – Interactive Phase
Validation
– High flexibility
Functional
– Quick turnaround time
Verification
(interactive) – Good debug capabilities
Cycle -based simulation
Cycle-base
Functional simulation – Regression Phase
Verification – Highest performance
(regressions) – Highest capacity
Emulation and Acceleration
In-System Emulation – In-System Verification
Verification
– Highest performance
– Highest Capacity
Implementation – Real system environment
Equivalence Checking
Verification

Kurt Keutzer 15

Approaches to Design Verification

Software Simulation
– Application of simulation stimulus to model of circuit
Hardware Accelerated Simulation
– Use of special purpose hardware to accelerate
simulation of circuit
Emulation
– Emulate actual circuit behavior - e.g. using FPGA’s
Rapid prototyping
– Create a prototype of actual hardware
Formal verification
– Model checking - verify properties relative to model
– Theorem proving - prove theorems regarding
properties of a model

Kurt Keutzer 16

8
Simulation: The Current Picture

Simulation Simulation
Monitors
driver engine

SHORTCOMINGS:
• Hard to generate high quality input stimuli

– A lot of user effort

– No formal way to identify unexercised aspects
• No good measure of comprehensiveness of validation

– Low bug detection rate is the main criterion

• Only means that current method of stimulus
generation is not achieving more.

Kurt Keutzer 17

Simulation

Simulation Drivers
Simulation Monitors
engine
driver

Symbolic
simulation

Diagnosis of
Vector Coverage
Input stimuli consistent with circuit generation
unverified
portions
analysis

interface must be generated

Environment of circuit must be represented faithfully
Tests can be generated
– pre-run (faster, hard to use/maintain)
– on-the-fly (better quality: can react to circuit state)
Environment and input generation programs written in
– HDL or C, C++, or
– Object-oriented simulation environment
• VERA, Verisity
Sometimes verification environment and test suite come with
product, e.g. PCI implementations, bridges, etc.

Kurt Keutzer 18

9
 Simulation

Simulators
Simulation Monitors
engine
driver

Symbolic
simulation

EVENT DRIVEN Vector

generation
Diagnosis of
unverified
Coverage
analysis
portions

• VCS
• Affirma
• Verilog-XL, ...
CYCLE-BASED
• Cyclone VHDL
• Cobra, ...
HYBRID
• VSS

Kurt Keutzer 19

Simulation

Monitors
Simulation Monitors
engine
driver

Symbolic
simulation

Reference models (e.g. ISA model) Vector

generation
Diagnosis of
unverified
Coverage
analysis
portions

Temporal and snapshot “checkers”

Can be written in C, C++, HDLs, and
VERA and Verisity: A lot of flexibility
Assertions and monitors can be automatically
generated: 0-in’s checkers
Protocol specification can be given as
a set of monitors
a set of temporal logic formulas
(recent GSRC work)

Kurt Keutzer 20

10
Types of software simulators

Circuit simulation
– Spice, Advice, Hspice
– Timemill + Ace, ADM
Event-driven gate/RTL/Behavioral simulation
– Verilog - VCS, NC-Verilog, Turbo-Verilog, Verilog-XL
– VHDL - VSS, MTI, Leapfrog
Cycle-based gate/RTL/Behavioral simulation
– Verilog - Frontline, Speedsim
– VHDL - Cyclone
Domain-specific simulation
– SPW, COSSAP
Architecture-specific simulation

Kurt Keutzer 21

Event-driven simulation

Key elements:
– Circuit models and libraries
• cells
• interconnect
– Event-wheel
• Maintains schedules of events
• Enables sub-cycle timing
Advantages
– Timing accuracy
– Handles asynchronous
Disadvantage - performance and data management

Kurt Keutzer 22

11
Event versus cycle-based simulation

D Q
data D Q
clock QN Combo
Logic clock QN
D Q

clock QN

clock

data Event-Driven Simulator:

• Simulates Function
• Tracks event activities and timing

data Cycle Based Simulator:

• Simulates Function
• Accurate at Clock boundaries

Kurt Keutzer 23

Approaches to Design Verification

Software Simulation
– Application of simulation stimulus to model of circuit
Hardware Accelerated Simulation
– Use of special purpose hardware to accelerate
simulation of circuit
Emulation
– Emulate actual circuit behavior - e.g. using FPGA’s
Rapid prototyping
– Create a prototype of actual hardware
Formal verification
– Model checking - verify properties relative to model
– Theorem proving - prove theorems regarding
properties of a model

Kurt Keutzer 24

12
 Gate-level Event-driven Sim Acceleration

HW implementation of gate-level event-driven

algorithm
– Full timing, many states
Event Scheduler
– Exploits low-level parallelism (pipelining)
Primitive Evaluators
Design partitioned for high-level parallelism
Event Detector
– Limited: irregular topology, event
Netlist Fanout

Event Interconnect
distribution
– Much work in the 1980’s: order 10X, not
100X
Performance
Event Scheduler

Primitive Evaluators – 5G/eval * 100 MHz * 10 procs @ Max. 5B

eps
Event Detector

Netlist Fanout
– “7-25X HDL simulator”, “500 to 5K cps”
(NSIM)
Usability
– Easy to use, quick compilation
M. Butts - Synopsys – Full timing and states

Kurt Keutzer 25

Gate-level Event-driven Simulation Accelerator

Just one: Ikos NSIM

– 4-input table primitives, RTL synthesis front -ends
– 8 to 64 processors, 0.5M to 15M gates
Value
– Much faster than unaccelerated simulators
– Not quite fast enough to run much code on the design
Competition
– Modern compiled or cycle-based SW on standard
multi-processor platforms
– Gate-level event-driven HW accelerator usually isn’t
enough better
• Today’s GP multiprocessors exploit low and high-
level parallelism
Conclusion: Limited future

M. Butts - Synopsys
Kurt Keutzer 26

13
Gate-level Cycle-based Acceleration

Levelized compiled simulation in massively parallel hardware form

– All gates evaluate every cycle
– No run-time data dependencies, so processors and IPC network
are scheduled at compile time
Severe design constraints
– No asynchronous feedback, latches, etc.
– No timing: multiple related clock domains only by LCD slowdown
– Commonly OK for microprocessors, much less so in general
Compilation
– Given design constraints, relatively easy to use
– Fast: 2M gates per hour (CoBALT)
History
– IBM: Yorktown Simulation Engine, ET3 / Quickturn CoBALT
– Arkos

M. Butts - Synopsys
Kurt Keutzer 27

Gate-level Cycle-based Accelerator

Just one: Cadence/Quickturn CoBALT
IBM Poughkeepsie ET3 technology

– 500 MHz custom chip, compiler core

– Up to 20M ASIC gates:
• 128 3-input prims / processor
(CE, new CL: 2.5X)
• 64 processors per chip
• 64 chips per board, 8 boards
Performance
– 32 trillion gate evaluations /sec (max)
(2 gate equivalents / processor cycle *
64 processors/chip * 64 chips/board *
8 boards * 500 MHz)
– 10K to 500K cps in actual practice
Usage
– Vector memories or in-circuit cable
– PCI link to workstation simulator

M. Butts - Synopsys
Kurt Keutzer 28

14
Gate-level Cycle-based Acceleration
Much faster than SW or event-driven accelerator
: Runs actual code and data, in actual target systems
Harder to use than SW or event-driven accelerator, but easier than emulator
Severe restrictions on design style
- Purely synchronous design OK, else No.
Expensive, complex, proprietary HW, SW
- Custom chips, interconnect, PCBs, connectors, chassis,
instrumentation
- Compiler is substantial effort to develop & maintain
Isolated from simulation, separate environment, proprietary simulator
Conclusion:
– Good solution for large fully synchronous projects that can afford
it
– Not a mainstream technology

M. Butts - Synopsys
Kurt Keutzer 29

Approaches to Design Verification

Software Simulation
– Application of simulation stimulus to model of circuit
Hardware Accelerated Simulation
– Use of special purpose hardware to accelerate
simulation of circuit
Emulation
– Emulate actual circuit behavior - e.g. using FPGA’s
Rapid prototyping
– Create a prototype of actual hardware
Formal verification
– Model checking - verify properties relative to model
– Theorem proving - prove theorems regarding
properties of a model

Kurt Keutzer 30

15
FPGAs as logic evaluators

Today: 2 trillion gate evaluations per second per FPGA (200K gates, 10M cps)
– Growing with Moore’s Law as designs do
– $1.5B industry behind it (XLNX+ALTR+ACTL)
Potent tool for logic verification and validation
How best to put the FPGA to use?

M. Butts - Synopsys
Kurt Keutzer 31

Logic Emulation M. Butts - Synopsys

Ultra-large “FPGA”
Live hardware, gate-for-gate.
Entire design or major module is
flattened, and compiled at once into
multi-FPGA form.
Logically static circuit-switched
interconnect.
In-circuit or vector-driven
Regular clock rate, > 1M cps.

Kurt Keutzer 32

16
Verification using Emulation

RTL or Gate
design System Hardware
– Customized parallel
processor system for
emulating logic
Emulation Box – In-circuit target interface
Compiler
Software Compiler
– Mapping RTL & Gate
designs to emulator
Mapper Runtime Software
– C-API
– Open SW architecture for
SBUS i/f tight integration
– Flexible modes of stimulus

uP

In-circuit Target Board

Kurt Keutzer 33

Logic Emulation HW M. Butts - Synopsys

Tens to hundreds of large FPGAs

Interconnect, either:
Control
Network
Computer – Programmable crossbars
(QT), or
Logic Board Logic Board
FPGA FPGA FPGA FPGA
– Nearest-neighbor with time-
multiplexing (Ikos).
XBar XBar XBar XBar
SRAMs for modeling memory
CPUs for behavioral simulation &
testbenches (QT Mercury)
XBar XBar Dedicated logic analyzer / pattern
generator for visibility & vectors
Logic Analyzer
Pattern Generator In-Circuit Cable In-circuit cable plugs into target

Kurt Keutzer 34

17
Logic Emulation SW M. Butts - Synopsys

Entire design is flattened

Input – Emulation-specific HDL synthesis
Design
Clock tree timing analysis
Design Reader – To avoid functional errors when
gated clocks get split across
HDL Synthesis
D
FPGAs
Tech Mapper A Multi-level, multi-way partitioning
T
Clock Analysis
A – NP-hard, very compute intensive
B
System Partitioner A System placement (Ikos only)
S
System Placer (if needed) E Place & route for every FPGA
– Can be run in parallel
System Router
– Interdependent due to
interconnect
FPGA Place&Route
FPGA
FPGAPlace&Route
Place&Route Design database system
Needs to be automatic and totally successful
Binary Chip Programming Files

Kurt Keutzer 35

Recent Logic Emulators M. Butts - Synopsys

Cadence/Quickturn Mercury
– 10M gates, XC4000XL, 2-level time-muxed partial crossbar
interconnect
– Each board has a PowerPC: Hybrid emulator + Verilog HW
accelerator
Ikos VirtuaLogic
– 5M gates, XC4000XL, nearest-neighbor time-multiplexed
interconnect
– Virtual wires compiler analyzes clock trees to synchronize time-
muxing
Mentor/Meta Celaro
– Custom FPGAs, not available in US
Axis Xcite
– FPGAs on PCI cards emulate HDL structures
– Tightly coupled to proprietary Verilog simulator
Simutech RAVE
– FPGA cards connected to time-multiplexed bus interconnect
Kurt Keutzer 36

18
Emulation + Accelerated Simulation

M. Butts - Synopsys
Event Backplane

Logic Board Logic Board

CPU+ CPU+
FPGA FPGA FPGA FPGA
Mem Mem

XBar XBar XBar XBar

XBar XBar

QT Mercury SimServer
Bauer, Bershteyn, Kaplan, Vyedin. A Reconfigurable Logic Machine for
Fast Event-Driven Simulation, Proc. 35th DAC, 1998.
– Multiprocessing HW-accelerated Verilog simulator + emulator
– Automatic HDL partitioning: synthesizable modules to emulator,
behavioral modules to PowerPC CPUs (up to 10)
– Accelerated time wheel, event detection in emulator FPGAs

Kurt Keutzer 37

Characteristics of Logic Emulation

Maximum Validation, fastest runtime speed

: Runs actual code and data, in actual target systems
No restrictions on design style
- Gated clocks split across FPGAs may cause correctable functional errors
Expensive, complex, proprietary HW, SW
- Interconnect, PCBs, connectors, instrumentation; big FPGA tech. lag
- Compiler is hard to develop & maintain, user must be full-time expert
Inflexible

- Interconnect architecture makes FPGAs interdependent - changes often

cause long recompile
Isolated from simulation or integrated with proprietary simulator
It's HW speed, but not design speed; target HW slowdown required
FPGA logic capacity tracks Moore's Law...
But interconnect capacity does not track Moore's Law.

M. Butts - Synopsys
Kurt Keutzer 38

19
 The Emulation Interconnect Problem
M. Butts - Synopsys
10000000 100000

1000000
FPGA capacity 10000 FPGA capacity is emulation usage:
Pins needed 8 gates / 4-LUT+FF, 75% packing.
100000
Pins needed is for emulation usage:
1000 p = 2.75g 0.58
10000 Package Pins pins Package pins are Xilinx FPGA IOBs
gates
(1991- 2000, extrapolated afterwards).
1000 100
1990 1995 2000 2005 2010

Rent's Rule (p = Kg r) applies to partitioned designs.

FPGA logic capacity: 2X / 1.5 yr (Moore's Law)
FPGA pins needed by emulator: 2X / 2.5 yr (Moore + Rent)
Package pins: 2X / 4 yr - Can't keep up.
Vendors are time -multiplexing pins more and more to compensate.

– But that’s only a linear effect; it does not change the doubling time.

Kurt Keutzer 39

Emulation Conclusions M. Butts - Synopsys

Market is flat at $100M/year

Expensive HW, SW, cost of sales
– High-end supercomputer-like business
Current competition
– Simulation farms have similar $/cycle/sec for regression
vector sets
– FPGA-based rapid prototyping for validation, SW execution

Good solution for large projects that can afford it

Ultimately the basic concept is limited by IC packaging

Kurt Keutzer 40

20
 Approaches to Design Verification
Software Simulation
– Application of simulation stimulus to model of circuit
Hardware Accelerated Simulation
– Use of special purpose hardware to accelerate
simulation of circuit
Emulation
– Emulate actual circuit behavior - e.g. using FPGA’s
Rapid prototyping
– Create a prototype of actual hardware
Formal verification
– Model checking - verify properties relative to model
– Theorem proving - prove theorems regarding
properties of a model

Kurt Keutzer 41

Rapid System Prototyping Environment

u Need low-
low- cost, instrument-
instrument- like system prototyping environment
u Must be well-
well- integrated into overall component-
component - based flow
Aptix System Explorer™ Aptix System Explorer™
MP3C or MP4
Development Software
Sun, HP

Debug Environment

Ethernet

Kurt Keutzer 42

21
Rapid Prototyping of ASICs and SoCs

Target-specific tools
– ASIC/core+FPGA:
Philips/VLSI Velocity, ARM
($5K)
– FPGA+RAM: Altera/ARC
“SoC” board (100KG, $5K)
GP tool
– Aptix : daughtercards, prog.
breadboard, > $100K

Rapid Prototyping Characteristics

: Real HW running at MHz,

low cost HW
- Isolated from simulation,
throwaway effort

M. Butts - Synopsys
Kurt Keutzer 43

Rapid Prototyping of ASICs and SoCs

Rapid prototyping is a rapidly growing verification technology
Do-it-Yourself: Get some FPGAs, build a prototype board
– Synopsys FPGA Compiler II accepts dc_shell scripts
– Synplicity /VeriBest Certify “RTL Prototyping”
• Automatic partitioning of ASIC, synthesis into FPGAs, automatic PCB netlist
• “Gary Smith of Dataquest said the tool very likely will pose a threat to Aptix
and may even ruffle feathers at Quickturn and Mentor.” -- EE Times 10/4/99

Kurt Keutzer
M. Butts - Synopsys 44

22
 Approaches to Design Verification
Software Simulation
– Application of simulation stimulus to model of circuit
Hardware Accelerated Simulation
– Use of special purpose hardware to accelerate
simulation of circuit
Emulation
– Emulate actual circuit behavior - e.g. using FPGA’s
Rapid prototyping
– Create a prototype of actual hardware
Formal verification
– Model checking - verify properties relative to model
– Theorem proving - prove theorems regarding
properties of a model

Kurt Keutzer 45

How to make it smarter: Intelligent Simulation

Simulation Simulation
Monitors
driver engine

al
ntion Symbolic
nve simulation
Co el
v
No

Diagnosis of
Vector Coverage
unverified
generation analysis
portions

Kurt Keutzer 46

23
 How to make it smarter: Intelligent Simulation

CLOSED FEEDBACK LOOP

Simulation Simulation
Monitors
driver engine

nal Symbolic
v entio
n simulation
Co el
v
No

Diagnosis of
Vector Coverage
unverified
generation analysis
portions

Kurt Keutzer 47

Simulation

Symbolic Simulation
Simulation Monitors
engine
driver

Symbolic
simulation

IDEA: One symbolic run covers many Vector

Diagnosis of
unverified
Coverage
generation analysis
runs with concrete values. portions

Some inputs driven with symbols instead of concrete values

•2(# symbols) equivalent binary coverage

Kurt Keutzer 48

24
 Simulation

Symbolic Simulation
Simulation Monitors
engine
driver

Symbolic
simulation

INNOLOGIC: Vector
Diagnosis of
Coverage
unverified
generation analysis
BDD-based symbolic Verilog simulators portions

l ESP-XV: For processor and networking applications

l ESP-CV: For memory verification and sequential

equivalence checking

l Monitors can have symbolic expressions

l Can symbolize time, e.g., event occurring after time T, 10 < T < 20.

l If bug is found, computes actual values exercising it

l Current “sweet-spots” of technology

– Memory verification: CAMs, caches, register files

– Unit level RTL functional verification: DMA, PCI,100-1000K
gate blocks
– Data movement, datapath
Kurt Keutzer 49

Simulation

Symbolic Simulation
Simulation Monitors
engine
driver

Symbolic
simulation

Diagnosis of
Vector Coverage
INNOLOGIC: Limitations generation
unverified
portions
analysis

l Capacity limits:
– ~ 1 million gate equivalents
– # of symbols - design dependent.
• < 50 in worst cases (multipliers)
• several thousand in the best cases
(memory, data movement).
• When out of memory, turn symbols into binary
values - coverage lost but simulation
completes.
l Roughly 10 times slower than Verilog-XL
l Can’t use in conjunction with Vera or Verisity currently.
è Definitely worth a shot: Extra cost of symbols offset
quickly, doesn’t require major change in framework.
è Full benefits of technology have not been realized yet.
Kurt Keutzer 50

25
 Simulation

Coverage Analysis
Simulation Monitors
engine
driver

Symbolic
simulation

Why? Vector
generation
Diagnosis of
unverified
Coverage
analysis
portions

• To quantify comprehensiveness
of validation effort
– Tells us when not to stop
– Even with completely formal methods, verification
is only as complete as the set of properties checked
• To identify aspects of design not adequately exercised
– Guides test/simulation vector generation
• Coordinate and compare verification efforts
– Different sets of simulation runs
– Different methods: Model checking, symbolic
simulation, ...

Kurt Keutzer 51

Status of Design Verification

Software Simulation
– Too slow
– Moving to higher levels is helping – but not enough
Hardware Accelerated Simulation
– Too expensive
Emulation
– Even more expensive
Rapid prototyping
– Too ad hoc
Formal verification
– Not robust enough
Intelligent Software Simulation
– Symbolic simulation – not robust enough
– Coverage metrics – useful, but not useful enough
– Automatic vector generation – not robust enough

Kurt Keutzer 52

26