RISC-V Summit 2019
Open Source RISC-V Processor
Verification Platform
Tao Liu, Richard Ho, Udi Jonnalagadda
Agenda
● RISCV-DV: Open source RISC-V instruction generator
● YAML based end-to-end verification flow
● Case study : ibex core verification
● Roadmap
RISCV-DV
SV/UVM based open source RISC-V instruction generator
https://github.com/google/riscv-dv
Google Cloud Platform Confidential & Proprietary 3
RISCV-DV
Generate a complete random RISC-V assembly program for processor verification
Random instructions
Page tables
PT 0 PT 2
PT 1 PT 3
PT 4
Trap handlers RISCV-DV
Data and stack sections
Data page 0
….
Data page n
Stack section
RISCV-DV new features
Initial release @01/2019 New features
ISA: RV32IMC, RV64IMC + ISA: F/D/A extensions
M/S/U privileged mode + Debug mode support
Virtual address translation + Functional coverage model
Regression suite + Hint/illegal instruction testing
Simulator: vcs, incisive + Handshake mechanism
ISS: spike + Vectored interrupt handling
Simple simulation flow + Bare program mode
+ Simulator: Questa, Metrics
+ ISS: ovpsim, whisper, sail-riscv
+ YAML based end-to-end flow
+ Improve performance
+ New regression suite
...
RISCV-DV new features
Illegal/HINT instruction support Configurable memory regions
mem_0
HINT Section mem_0
ILLEGAL Section mem_1 link
mem_1
HINT
Illegal instruction handler Section mem_2
mem_2
Type of illegal instructions:
Reserved opcode Use cases:
Unsupported instruction Access various internal memories
CSR access to unsupported CSR PMP verification
Operation in the wrong privileged mode Virtual address translation verification
...
Debug mode support
Debug mode test scenarios
ebreak handler
Normal program
... > ebreak with dcsr.ebreakx enable/disable
ebreak
... > Enter debug mode through external debug request
External
debug mret
> Access non debug mode CSR in debug mode
request
> ebreak in debug ROM
ebreak Debug ROM
> Exception in debug ROM
dpc check
> Single step debug mode
randomize dcsr
> ….
?
save GPR
...
ebreak How to check the processor’s behavior if
... ISS cannot be put to debug mode the
same time as RTL design?
Restore GPR
dret
Handshake mechanism
Handshake packet definition CORE_STATUS : 0
TEST_RESULT: 1 Testbench Normal program Debug ROM
24 bits 8 bits
WRITE_GPR : 2
debug re
data cmd WRITE_CSR : 3 quest
data Variable length
data packets
data CORE_STATUS: IN_DEBUG_MODE
check
WRITE_CSR: dpc
Use cases:
Debug mode testing WRITE_CSR: dcsr
Interrupt testing dret
Memory access fault testing
Self-checked tests
...
Functional coverage flow
ISS log Independent of test
ISS
ISS log
log log2csv
spike log generation flow
Fast coverage closure
RISC-V ISA without RTL sim. No
ISS log ISS
ISS log
log SV functional RTL tracer needed
ISS
ISSlog
loglog log2csv Tracelog
ISS CSV
ovpsim coverage model
Flexible to support
different ISS
ISS
ISSlog
log log2csv
ISS loglog
whisper
Functional coverage model
● Cover all possible operands of each instruction
● Hazard conditions
● Corner cases like overflow, underflow, divide by zero
● Aligned/unaligned load/store
● Positive/negative immediate value
● Forward/backward branches, branch hit history
● Hint instruction
● Illegal instruction
● All opcode
● Access to all implemented privileged CSR
● Exception and interrupt
Generator performance improvement
test_0, batch 1, 20 iterations
35s
test_0, batch 2, 20 iterations
test_0, 100 iterations
test_0, batch 3, 20 iterations
test_1, 10 iterations
test_0, batch 4, 20 iterations
test_2, 5 iterations
… ... test_0, batch 5, 20 iterations
7s
Test_n, x iterations
Average test generation time (s) Run large regression in parallel with small batches
YAML based end-to-end verification flow
An ecosystem for RISC-V processor verification
Google Cloud Platform Confidential & Proprietary 12
YAML based end-to-end verification flow
iss.yaml
simulator.yaml
ISS sim log log2csv csv
RISCV-DV RISC-V test
Compile test
source compiler RISC-V ELF compare
generator
code (gcc, llvm)
RTL sim log log2csv csv
Generate
RISC-V Assembly test
Assembly test
assembly Assembly test simulator.yaml
Assembly test
tests
testlist.yaml Add new RTL simulator, ISS, test to
the flow simply by modifying the
YAML configure file
YAML based end-to-end verification flow
ISS.yaml testlist.yaml
- iss: spike - import: <riscv_dv_root>/yaml/base_testlist.yaml
path_var: SPIKE_PATH
cmd: >
<path_var>/spike --isa=<variant> -l <elf> ... - test: riscv_machine_mode_rand_test
description: >
- iss: ovpsim Machine mode random instruction test
path_var: OVPSIM_PATH
iterations: 2
cmd: >
gen_test: riscv_instr_base_test
<path_var>/riscvOVPsim.exe
--controlfile <cfg_path>/riscvOVPsim.ic gen_opts: >
… +instr_cnt=10000
+num_of_sub_program=5
- iss: whisper
+boot_mode=m
path_var: WHISPER_ISS
rtl_test: core_base_test
cmd: >
<path_var> <elf> --log --xlen <xlen> --isa <variant> ...
YAML based end-to-end verification flow
# Run the generator with specific RTL simulator
python3 run.py --simulator vcs
# Run the generator with two ISS and cross compare the result
python3 run.py --iss ovpsim,spike
# Run the generator with RV64IMC variant
python3 run.py --target rv64imc
# Run a test with 10 iterations in verbose mode
python3 run.py --test_name riscv_rand_instr_test -i 10 -v
An ecosystem for RISC-V processor
verification
RTL simulator ISS Pre-defined targets
SPIKE RV32I
RV32IMC
RV64IMC
RV64GC
Sail-RISCV ...
Case study: Ibex core verification
Source code: https://github.com/lowRISC/ibex/tree/master/dv/uvm
Ibex features
RV32IMC + Debug mode +
User mode + PMP + ….
Case study : Ibex core verification
Categories of found bugs
Roadmap
Complete RISC-V verification platform
Complete ISA support Functional coverage model Reference testbench
Improve test library for Extend the current model to UVM environment framework
supported ISA extensions. support more ISA extensions, Test/sequence infrastructure
Vector extension
privileged spec coverage etc. Interface agent library
Bitmanip extension
Predefined targets for open
source RISC-V processors
Machine Learning for Verification Research Platform
Environment
Action
testlist.yaml Improve the generator controllability
Instruction Enhance coverage model to provide better
generator
differentiation of coverage results
ISS sim
Provide standard API to interact with ML models
RL Agent
Coverage model Benchmark platform for different ML algorithms
State/Reward
Unified configuration
riscv-config
Instruction Compliance Coverage
ISS ...
generator test model
A single configuration format for all flows
Backup Slides
Generate program header Generate data/stack section
Privileged mode setup Generate page tables
Page table randomization Generate intr/trap handler
Generator flow
Initialization routine Test completion section
Generate main/sub programs Call stack randomization
Branch target assignment Apply directed instructions
Benchmark flow
Processor candidates
Pulpino RI5CY:
4 stages, RV32-IMC, DSP extension
Pulpino Ariane :
6-stage, RV64-IMC, single issue,
in-order, support M/S/U privileged levels
Merlin:
Open Source RV32I[C] CPU
ISS simulator
Spike
Benchmark metrics
Bug hunting capability, test coverage
Flow integration effort, performance
Ariane core architecture
Bugs found
LSU
wr rd
Fence
Issue
Load SSTATUS mxr ... ALU
Store
...
Load MSTATUS mxr ...
Load Multiplier
Cache line access racing privileged CSR access FENCE operation failure
PT 0 PT 2
Load Branch Add ... Mult MULHSU
PT 1 PT 3
PT 4
page fault handling Incorrect branch execution ALU corner case bug
Handshake packet format
IN_DEBUG_MODE 0 CORE_STATUS
TEST_PASS 1 TEST_RESULT
MSTATUS 2
WRITE_GPR: MSTATUS
0x0000_1234
