2018 19th International Workshop on Microprocessor and SOC Test, Security and Verification (MTV)

Different Reference Models for UVM Environment

to Speed Up the Verification Time
Amr Moursi∗ , Romaisaa Samhoud∗ , Yaseen Kamal† , Mazen Magdy† , Sameh El-Ashry‡ ,
Ahmed Shalaby§
∗ Mentor,
a Siemens Business, Cairo, Egypt,
† Department of Communications and Electronics, Faculty of Engineering, Alexandria University, Alexandria, Egypt,
‡ Department of Computer and Systems Engineering, Ain Shams University, Cairo, Egypt,
§ Department of Computer Science, Faculty of Computers and Informatics, Benha University, Egypt.
Email: amr moursi@mentor.com, romaisaa samhoud@mentor.com, yassink1992@ieee.org,
es-Mazen.Magdy1217@alexu.edu.eg, samehelashry@ieee.org, ahmed.shalaby@ejust.edu.eg.

Abstract—With increasing digital systems complexity intro- proaches, AVM, OVM, and UVM have been introduced by
duced by the sophisticated architectures, design verification Accellera to unify engineer’s style of verification [2].
becomes challenging and crucial. Verification is required to UVM is the most recent verification methodology in the
provide enough confidence in the design before proceeding with
further expensive design stages, thus reliable reference models are industry that is fully automated and based on Object-Oriented
needed. Developing an efficient reference model is not an easy Programming (OOP) to be eligible for the reusability of
task and may cause delays in the verification process increasing the test bench. This makes it flexible to validate different
the time-to-market. This work elaborates the advantages of designs in addition to its ability to verify complex architecture
using high-level language reference models over the conventional accurately. UVM proposes a firm and productive approach
SystemVerilog ones in digital design verification. Comparison
between Python, C/C++, SystemC, Matlab, and conventional Sys- to build and reuse verification environments and components,
temVerilog reference models is explained from the perspectives making the verification environment flow consistent. It can
of the run-time, memory consumption, complexity of implemen- also be easily interpreted by any verification engineer and
tation and UVM-reference model interface. A separate Universal makes most of the environment plug and play for similar
Verification Methodology (UVM) environment is implemented for designs.
each reference model.
Since the verification is all about checking the correctness
Keywords—DPI, Python, Reference Model, SystemC, UVM, of the design, one of the most critical parts of any verification
Verification. environment is the expected output calculation. UVM output
is calculated by the reference model, which is normally
I. I NTRODUCTION implemented in SystemVerilog.
The problem is that when designing a UVM environment for
Verification of digital systems has been very challenging for complex designs as the wireless baseband digital systems. The
any design process since it requires verifying complex arith- implementation of the reference model would be a bottleneck
metic algorithms such as Fast Fourier Transform (FFT) [1]. since it is complicated. The reference model also must be
In Order to overcome the overhead that may be caused by coded with a high degree of accuracy which might be very
the verification process, reliable, fast and reusable verification difficult to achieve using Hardware Description Languages
techniques should be used. The used methods for the digital (HDL) especially with bulky designs that involve sophisticated
system verification have undergone many improvements in algorithms. Hence, the implementation of the design would
the past few years. It started with a simple testbench till not be coded in a straightforward readable manner using
the standardized techniques such as Advanced Verification this approach. Also the Design Under Test (DUT) is already
Methodology (AVM), Open Verification Methodology (OVM) written in HDL, so a high-level language is more convenient
and Universal Verification Methodology (UVM). to be used in a wide variety of applications. For instance,
Conventional testbenches are based on ”Directed Testing” the communication and signal processing communities utilize
by predefined stimuli in order to verify the expected behavior. Matlab for prototyping and delivering the abstraction models
However, it is insufficient and can not cover all the corner for the designs. While the video coding and image processing
cases. The basic solution to this problem is increasing the utilize C/C++. For neural network and machine learning devel-
number of test vectors. But it would take a huge effort with a opers prefer python. On the other hand, high-level synthesis
possibility of human errors and may consume a lot of time, yet and ASIC designers use SystemC [3]. So in order to unify
good coverage would not be achieved. Constrained Random the utilization of UVM for all communities and speed-up the
Test (CRT) is used to add constraints to the generated random time for verification development, different interfaces for all
stimuli with unexpected combinations that can be generated utilized languages are provided, which saves a lot of time in
during runtime. In order to standardize the verification ap- verification time-cycle.

2332-5674/18/$31.00 ©2018 IEEE 67

DOI 10.1109/MTV.2018.00023
 Figure 1: PHY transmitter

In this paper, a comparison between various high-level lan-

guage reference models is presented against the conventional
UVM environment. The implementation schemes, advantages, Figure 2: Proposed UVM environment reference models
and disadvantages are discussed for each of them, the perfor-
mance of interfaces is evaluated in terms of memory utilization
and simulation time to provide the infoormation with the
to Analog Converter (DAC) of 8 bits is the most appropriate
gain/loss with respect to SystemVerilog.
practical choice to achieve the required specification, therefore
The paper is organized as follows, in section II a case
the maximum number of bits for each of the I & Q arrays is
study design “IEEE 802.15.4 PHY Transmitter” is interpreted.
32,770 bits for each which may be a limiting factor in the
Section III gives an overview of the proposed UVM envi-
reference models interfacing based on Equation 1.
ronment. Section IV presents a detailed description of UVM
environments including a C reference model with DPI con-
nection, Matlab reference model using DPI and socket based ((P ayload length × 8)/4 × 32 × 8 + 4)/2 (1)
connections, Python reference model connected via DPI link
and a SystemC model using the latest UVM Connect (UVMC)
III. P ROPOSED UVM E NVIRONMENT
library. Section V illustrates the achieved performance results
for each of the reference models and the advantages and
UVM blocks collaborate to verify the functionality of the
disadvantages in terms of design complexity, interfacing and
design, as shown in Fig. 2. Control signals are driven into
data types compatibility point of views in addition to the
DUT to maintain the transmission process and randomize
memory consumption. Finally, the conclusion is summarized
the payload length with some constraints according to the
in section VI.
standard. Data has to be randomized between 0 to 127 bytes.
II. CASE STUDY Data is randomized according to the random payload length
generated, the sequence generates multiple transactions while
In order to make a practical reliable comparison, IEEE the sequencer pushes the transactions into the driver. The
802.15.4 PHY transmitter has been used as a DUT as illus- driver pulls the randomized data accompanied with the payload
trated in Fig. 1. The design has been simulated using Matlab length of the transaction then sends them to the DUT, it also
before hardware implementation to ensure that it is functioning drives the control signals. It sets the start signal high to start
properly and complying with the standard specifications. the transmission of data through the DUT and waits for the
As specified in the standard, the payload length for the done signal to be asserted indicating that the data has been
packet can range from 1 to 128 bytes of data. The payload transmitted successfully. Finally, the Driver is blocked waiting
accompanied with its length are received from the Medium for the next packet.
Access Control (MAC) layer after start signal is asserted. “monitor dut” is a passive block where it collects the output
Then they are concatenated to the PHY headers to form the signals (I-phase & Q-phase) from DUT and sends them to
Physical Protocol Data Unit (PPDU) and be divided into four scoreboard through the analysis port ”mon ap before” to be
bits data symbols. Each symbol is mapped into a 32-chip PN compared with signals from the reference model. It waits until
sequence [4]. the valid signal is high to start collecting data and finishes
The 32-chip sequences representing each symbol are mod- when the done signal is raised indicating that the packet is
ulated using O-QPSK with half-sine pulse shaping. Even- transmitted.
indexed chips are modulated to the In-phase (I-phase) carrier
“monitor ref” extracts the payload length and data gener-
and odd-indexed chips are modulated to the Quadrature-phase
ated through the interface and send them to a reference model,
(Q-phase) carrier. In order to form the offset between I-phase
giving the exact ideal output which is sent to Scoreboard via
and Q-phase chip modulation, the Q-phase chips are delayed
”mon ap after”. It synchronizes both monitors together and
by a chip time (Tc) as 4 bits with respect to the I-phase
checks the correctness of the DUT output by comparing the
chips where Tc is the inverse of the chip rate. Based on
actual output of DUT from “monitor dut” and the expected
Matlab simulation results, it is proven that using a Digital
output from “monitor ref”.

68
 Figure 3: C++ reference model Figure 4: Matlab reference model

IV. P ROPOSED R EFERENCE M ODELS compilation command. Questa Sim invokes the GCC passing
Each reference model is implemented and tested in a the needed compilation flags and include files [7].
standalone UVM environment. The algorithm of calculating
B. Matlab Model using “qmw” library
the expected output is the same in all the models as well as
UVM environment blocks, the only difference is the way the When it comes to designing a certain Digital Signal Process-
model is implemented and connected in order to guarantee a ing (DSP) algorithm, the most preferable choice is Matlab as it
fair comparison between each. contains all of the needed functions which verifies the design
architecture before proceeding to any further time-consuming
A. C/C++ Model using DPI connections implementation stages. Consequently, this initial simulation
The C/C++ is considered the most used programming lan- output would be the reference for all of the next stages.
guage overtime for many complex applications in embedded Therefore integrating Matlab implementation with UVM as
systems. Consequently, one might think C is the best can- a reference model adds more confidence to the verification
didate for developing a high-level language reference model. environment. Also, it will save the implementation time for
Thankfully with Direct-Programming Interface (DPI), using a any other reference models since it was already implemented
SystemVerilog-C connection has never been easier and we no in the earlier simulation stages. Furthermore, Matlab offers a
longer need to go deep with the VPI and PLI details [5]. wide variety of reliable tool boxes for different applications
SystemVerilog provides DPI that allows importing foreign with clear documentation that serves many areas.
language subroutine such as C functions. SystemVerilog tasks However, Matlab has a major drawback. It is not an open
and functions can also be exported to a foreign language. source and requires a license that may be expensive and
DPI libraries act as an interface for high-level language. It needs installations. When it comes to portability a Matlab
consists of two separate layers: SystemVerilog layer and a based UVM environment will not be the best solution, and
foreign language layer, the two layers are completely isolated since UVM is all about reusability some will find that Matlab
from each other [6]. reference model may not cope with this important feature.
A conventional UVM environment is designed but instead In order to have the ability to connect Matlab reference
of embedding the reference model into the “monitor ref” it model to UVM environment, there are two approaches. First
is implemented externally as a normal C function “tx ref”. via C based DPI interface. Second by using SystemC with
On SystemVerilog layer, this function is imported and de- TLM 2.0 as presented in [6]. Finally, the method used here
clared inside it. After the packet is generated, memory is as shown in Fig. 4 is a Questa Sim dedicated library to
allocated for the random packet along with the arrays that make the two programs communicate directly using socket-
hold the predicted output as a SystemVerilog open array since based interface without the need to implement interconnecting
C language lacks the concept of parameterized arguments, mechanisms. The used library “qmw” is precompiled and
on the other hand open arrays offer querying and access does not need any further implementation code. It simply
functions. Then the random packet is generated as stimulus initiates a bridge connection between SystemVerilog DPI and
and received at the “monitor ref”. It is sent to the C-model Matlab’s API [8]. On the Matlab side, a server is started with
by handle “svOpenArrayHandle” accompanied with the output a specific port that enables the data communication from and
arrays handle. In the C layer, the received packet can be to Matlab. This is done with the built-in Matlab’s function
accessed easily using a pointer to the open array handle. “hdldaemon” where it enables data transmission either through
Then, the computation of the expected output is calculated “I- shared memory or a socket port [9]. Using a port is better
phase and Q-phase components” that are saved in the arrays since it enables having Matlab installed on another machine
previously allocated using the array handles in SystemVerilog than the one used to run UVM environment on so it provides
side. After the calculation, “monitor ref” sends the output to more flexibility.
the scoreboard for comparison and repeat the process for other A TCL layer is implemented that allows sending any
packets as shown in Fig. 3. signals either at runtime or at post-simulation to Matlab. TCL
The environment execution is nearly the same where all commands can be executed using “tclHdlSim” to provide the
the Verilog files and C files are compiled using the normal interaction with QuestaSim. On QuestaSim side the design is

69
 Figure 6: SystemC reference model

necessary steps are made for the purpose of getting the

packet ready to be received in the reference model as a list,
Figure 5: Python reference model “PyList SET ITEM” is used in order to convert the array
which returns “PyObject *list”. It worth mentioning that all
variables and functions needed to be accessed in C are an
compiled along with the library SystemVerilog file which set extension of PyObject. After that “Py BuildValue” is executed
up the needed functions and DPI interfaces, after which UVM to prepare the arguments that would be passed to Python’s
environment is simulated normally with passing the ”qmw” function, then it is called using “PyObject CallObject” passing
library (.so file) to the vsim command. After the transaction to it the built arguments “generated packet”. Python script is
is received from the stimulus inside the “monitor ref”, it executed returning two output arrays which are sent to the c
calls DPI functions “qmw sendIntVector” in order to send the code as a PyObject. “PyList GET ITEM” is used as a reverse
transaction to Matlab reference model, and “qmw sendCmd” for “PyList SET ITEM”. The resulted arrays are copied into
to send Matlab commands needed to be executed after the the SV open arrays using the handles passed by SystemVerilog
expected output is calculated. The transaction is sent back as discussed earlier [13]. Finally, results are compared in the
to the “monitor ref” using “qmw getIntVector” imported DPI SystemVerilog side.
function and the normal UVM comparisons take place.
C. Python Model using two Step C based Link D. SystemC using UVMC
Recently, Python is being heavily used in many applications SystemC is a set of C++ classes and macros with signals,
as it provides different capabilities needed, with the help of events and synchronization primitives which enables mimick-
a wide variety of third-party libraries available as NumPy, ing the behavior of HDL in the terms of concurrency, but with
SciPy, Matplotlib and scikit which is used for data mining system or architectural level modeling instead of a register
and data analysis in machine learning. Python has different transfer one, SystemC is a promising language when it comes
functionalities which make it suitable for developing many to writing the reference model with high abstraction level [14].
DSP applications such as the library ”PyLTEs” presented SystemC connection with UVM environment is a bit tricky,
by [10]. Also, it is a general purpose language that facilitates but Mentor provides UVMC library that enables the duplex
the programming. Python is a free language which is a connection between UVM and SystemC [15]. However the
great advantage over Matlab. Additionally, it dominates the earlier versions of this library “do pack and do unpack” func-
digital verification field by implementing parts of verification tions implemented as methods of transaction class but it has
environment [11] or even the whole environment [12]. major problems such as the payload length being transmitted
The connection between Python reference model and UVM cannot exceed 4KBytes [16]. Another method is field macros
environment is achieved using a DPI bridge connection as conversion functions implemented via “uvm field” macros
interpreted in Fig. 5. On SystemVerilog side, the C-function is which are an alternative to implement a UVM transaction’s
imported which encapsulates the interface between the refer- pack and unpack methods indirectly [17] along with a DPI-
ence model and the “monitor ref” to support extending Python based connection [18].
with C [13]. Python API (Application Programming Interface) The methodology used here is based on UVMC 2.3.1.
is embedded in a C source file by including the header UVMC is a library based on transaction level modeling TLM
”Python.h”. The generated packet accompanied by its length 1.0 and TLM 2.0, it allows objects passing between SystemC
is sent using the DPI as mentioned in the C-model imple- and SystemVerilog UVM environment components. It also has
mentation. At the C side, Python interpreter is initialized then access and control to the UVM environment through the com-
Python script is imported using “PyImport ImportModule”, mand API where it can wait for transactions, raise and drop ob-
afterwards the reference model function is called in the C jections, set report filters and more. The latest version is used
code. to overcome the transaction size limitation. Fig. 6 shows the
One important challenge of this reference model is the flow of the environment till the packet is received at the “mon-
data types compatibility where C deals with arrays while itor ref”. “uvm tlm generic payload” class is instantiated as
Python deals with lists as an equivalent data type. Some “gp” used to send and receive the data between SystemVerilog

70
 700
6
Simulation time in Sec[Log Scale]

Matlab
600
5 SystemC

Memory Consumption in Mbytes

500
SystemVerilog
4
Python 400
3 C
300
2
200
1
100
0
100 1000 10000 100000 0
Number of packets Matlab SystemC SystemVerilog Python C

Figure 7: Simulation time versus packets number Figure 8: Memory consumption versus packets number

and SystemC through memory mapped schemes. The bus

Size (VSZ). However, Matlab consumes a little more as the
operation type is then set to write using “set command”
calculation requires launching a parallel instance of Matlab
method with passing “UVM TLM WRITE COMMAND” to
itself.
the “m command” variable in order to write the transaction
produced by the sequence on the bus. The data length is Regarding the complexity of the interfacing between each
then set using “set data length” method, the transaction is reference model and UVM environment, the C reference
sent to SystemC after setting both address and data using model is the easiest one to be linked. DPI has helped to
“set address” and “ set data” methods. Initiator “monitor ref” connect the reference model using only a simple DPI export
allocates storage for the transaction, the generated packet is function and a system call to the required function passing
then passed through the socket using the TLM 2.0 blocking the necessary data handles to it. Python Interfacing is not
transport method “b transport” which is implemented inside simple since there is no direct interface between SystemVer-
the Target SystemC reference model . Blocking transport ilog and Python. As a result, an extra C based connection
method passes the packet data by reference, it has a timing is implemented, so a bunch of C code is written for the C-
argument which indicates the start and end of the transac- Python connection. For Matlab reference model, connection
tion. On the other side, SystemC contains a virtual function is smooth since the “qmw” library provided by Mentor is
that gets the “generic payload” class handle from which a already doing the necessary setup for Matlab-SystemVerilog
pointer to the sent packet can be retrieved then the process- connection, although there are a bit complex procedures done
ing starts. After the expected output is calculated another under the hood. Interfacing SystemC with a UVM environment
“TLM WRITE COMMAND” is initiated from SystemC side is easy task due to the availability of UVMC library. One of the
then the output is sent back to the UVM environment for main benefits is that it is based on TLM 2.0 which makes the
comparison with the DUT output. interfacing with UVM environment simple since UVM deals
UVMC version 2.3.1 adds support for “fast packer with the data as transactions already.
converters” which improve the performance and support Concerning the complexity of the reference model imple-
for TLM generic payload with no fixed limitations on mentation, Python is the easiest to design since it is a general
data payload sizes. The “fast packer converter” class purpose language with a lot of useful data types as dictionaries
“uvmc tlm gp coverter” is declared to be used in the connect and all the required arithmetic functions are ready as “NumPy”
phase of the agent, while on SystemC side it is specified library. Matlab has been easy to design since it already has all
in the top class when registering the ports used for data the functions needed for any design algorithm. C design is a
transfer. Both ports are registered with the same lookup string, little bit more complicated as it does not have some data types
during the elaboration phase UVMC will connect the two ports for instance, dictionaries so a hash table has to be coded in
(SystemVerilog and SystemC ports) together. order to implement this functionality. SystemVerilog has been
the most complex to design where all of the function has to
V. R ESULTS be manually implemented, no wide range of data types and
Table I shows a comparison between the implemented no ready libraries to include.
reference models. From the performance perspective, all the As for the data types compatibility between each of the high
reference models nearly have the same performance except level language reference models and SystemVerilog. C and
Python, as shown in Fig. 7. SystemVerilog has on average Matlab are capable of sending and receiving massive arrays
the best performance while Python is the worst as it involves easily since DPI sends and receives them using handles. On
multi-stage interfacing with system Verilog. When it comes to the other hand Python, it is a little more complex where it
the memory consumption, Fig. 8 shows that all the reference lacks the array as data type. Consequently, the C array has
models consumed nearly an equal amount of Virtual Memory to be converted to List “PyObject” and vice versa. SystemC

71
 Table I: Reference models comparison
SystemVerilog C Matlab Python SystemC
Performance Fast Fast Fast Slow Fast
Memory Average Average High Average Average
Interfacing Easiest Easy Moderate Complex Moderate
Reference Complex Moderate Easy Easiest Moderate
Model
Data Types High Low Low Low Medium
Compatibility

offers many hardware data types which are considered very [3] M. Smirnov and A. Takach, “A systemc superset for high-level syn-
close to those used in HDL languages. It also has methods thesis,” in 2009 Forum on Specification Design Languages (FDL), Sept
2009, pp. 1–6.
and threads equivalent to functions and tasks that make the [4] IEEE LAN/MAN Standards Committee and others, IEEE Standard for
reference model really simulates the behavior of the DUT but Local and metropolitan area networks, IEEE Std 802.15.4, IEEE.
with a syntax close to C++ and with all of the benefits of a [5] SystemVerilog DPI tutorial. Doulos. [Online]. Available:
https://www.doulos.com/knowhow/sysverilog/tutorial/dpi/
high-level language. [6] IEEE Computer Society and the IEEE Standards Association Corporate
Advisory Group, IEEE Standard for SystemVerilog Unified Hardware
VI. C ONCLUSION Design, Specification, and Verification Language, IEEE.
In this paper, we have presented various methods to im- [7] Questa SIM User’s Manual, Mentor, a Siemens Business.
[8] Matlab/integration. Mentor, a Siemens Business. [Online]. Available:
plement a high-level reference model for UVM environment. https://verificationacademy.com/cookbook/matlab/integration
The implementation methodology is listed for each model, [9] hdldaemon. The MathWorks Inc. [Online]. Available:
The PHY transmitter is used as a case study to achieve a https://www.mathworks.com/help/hdlverifier/ref/hdldaemon.html
[10] M. Slabicki and K. Grochla, “PyLTEs—Python LTE evaluation frame-
practical comparison between them. Consequently, the advan- work for quick and reliable network optimization,” in Telecommunica-
tages and disadvantages of each are revealed. This allows the tions and Signal Processing (TSP), 2016 39th International Conference
verification engineers to choose which one suits their needs. on. IEEE, 2016, pp. 64–67.
[11] M. Cieplucha and W. A. Pleskacz, “New constrained random and
The study has evaluated the reference models according to metric-driven verification methodology using Python,” in Design and
their time performance, memory consumption, the interfacing Verification Conference (DvCon US), 2017.
easiness, the simplicity of the model implementation and the [12] PotentialVentures. (2014) Cocotb (coroutine based cosim-
ulation testbench environment). [Online]. Available:
available data types. C/C++ was the easiest to connect with https://cocotb.readthedocs.io/en/latest/introduction.html
UVM, Matlab and Python reference models were effortless to [13] (2011, May) Embedding Python in another applica-
implement however, Python was complicated in the connection tion. Python Software Foundation. [Online]. Available:
https://docs.python.org/2/extending/embedding.html
as it required building a library from scratch to integrate it [14] IEEE Computer Society, IEEE Standard for Standard SystemC Language
with UVM environment, moreover, it achieved relatively poor Reference Manual, IEEE.
performance. SystemC is rich with different data types as well [15] UVM-Connect and TLM-2. 0 primer, Mentor, a Siemens Business.
[16] N. C. S. Campos, H. A. Monteiro, A. V. Brito, A. M. N. Lima, E. U. K.
as its ability to mimic the RTL behavior but on the other side, it Melcher, and M. R. A. Morais, “A framework for design and validation
needed a workaround to overcome its limitation of the payload of face detection systems,” in 2017 CHILEAN Conference on Electrical,
length. Electronics Engineering, Information and Communication Technologies
(CHILECON), Oct 2017, pp. 1–7.
R EFERENCES [17] Accellera Organization, Universal Verification Methodology (UVM) 1.0
Class Reference, IEEE.
[1] Y. Lee, V. Koo, and Y. Chan, “Design and development of FPGA- [18] Y. Zhu, T. Li, J. Guo, H. Zhou, and F. Fu, “A novel low-cost interface
based FFT co-processor for synthetic aperture radar (SAR),” in Progress design for SystemC and SystemVerilog Co-simulation,” in ASIC (ASI-
in Electromagnetics Research Symposium-Fall (PIERS-FALL), 2017. CON), 2015 IEEE 11th International Conference on. IEEE, 2015, pp.
IEEE, 2017, pp. 1760–1766. 1–4.
[2] UVM 1.2 User Guide and Reference Manual.
http://www.accellera.org/activities/vip, 2015.

72