Writing Application Programs for

RDMA using OFA Software

Presented in two parts

Open Fabrics Alliance

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 1
Instructor team

Rupert Dance – [email protected]

Bob Russell – [email protected]

Paul Grun – [email protected]

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 2

Useful references

• InfiniBand Architecture – www.infinibandta.org

– IB specifications – free download

• Introduction to InfiniBand™ for End Users

– http://members.infinibandta.org/kwspub/Intro_to_IB_for_End_Users.pdf
– free download, no registration required

• iWARP RFCs – www.ietf.org/rfc.html

– RFC 5040 – A Remote Direct Memory Access Protocol Specification (RDMA)
– RFC 5041 – Direct Data Placement over Reliable Transports (DDP)
– RFC 5044 – Marker PDU Aligned Framing for TCP Specification (MPA)
– others
• Mellanox User’s Manual
– http://www.mellanox.com/related-
docs/prod_software/RDMA_Aware_Programming_user_manual.pdf

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 3

 A two part course

“The purpose of this course is to provide students with knowledge and

experience in writing application programs using RDMA.”

Part One: Introduction to RDMA Part Two: Programming with RDMA

Programming with RDMA is

different from sockets How to write a user application to
programming (for good reason) take full advantage of RDMA

This part gives you the background

needed to understand Part Two

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 4

Agenda – Introduction to RDMA

• Some key terminology

• Introduction – I/O architecture
• A Messaging Service
• Address translation and network operations
• RDMA Architecture
• Verbs Introduction
• The OFED Stack
• Introduction to wire protocols

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 5

 Useful terminology

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 6

 RDMA – Remote Direct Memory Access

virtual space

O App App O
S S

Remote Direct Memory Access

-Remote: communication at a distance
-Direct: does not require a ‘higher authority’ for each access
-Memory Access: virtual-to-virtual transfers…even across a network

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 7

 Channel Adapter - CA

App

App
CA

CA
O O
S S

Channel Adapter (CA)

An I/O device that allows an application to conduct RDMA operations directly

We will use the expression Channel Adapter, or CA, consistently.

A channel adapter is comprised of both hardware and software.

There is wide latitude in CA design and in h/w – s/w partitioning

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 8
Three types of Channel Adapters (CAs)

RNIC HCA ‘NIC’

iWARP InfiniBand RoCE

Generically, all three are called Channel Adapters (CA).

We will discuss some of the differences between them later.

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 9

 Verbs

App

App
CA

CA
O O
S S

Verbs
An API used by an application to control and conduct an RDMA operation.

(It has another meaning too, which will be described later.)

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 10

Requester, Responder

WRITE
requester responder
READ

initiates and controls responds to requests

an RDMA operation

- A request flows from the requester to the responder

- A requester may request a READ or a WRITE operation
- A WRITE operation transfers data from the requester to the responder
- A READ operation transfers data from the responder to the requester

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 11

 Introduction

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 12

Why RDMA?

Lots of reasons, and many value propositions:

1. low latency
– stack bypass, copy avoidance
2. reduces CPU utilization
3. reduces memory b/w bottlenecks
4. high bandwidth utilization
5. …

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 13

Why RDMA?

Lots of reasons, and many value propositions:

1. low latency
– stack bypass, copy avoidance
2. reduces CPU utilization
3. reduces memory b/w bottlenecks
4. high bandwidth utilization
5. …
Achieving these value propositions is at
the heart of the differences between
RDMA programming and conventional
sockets programming
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 14
The RDMA value proposition derives from the way
it treats I/O

To understand the differences between application

programming for RDMA and conventional I/O
programming, we begin by looking at the I/O
model

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 15

Quick Review of I/O – h/w model

proc … proc
- A compute platform consists of
I/O bus - a CPU/memory complex and
- an I/O subsystem

The I/O subsystem is a shared system resource

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 16

Quick Review of I/O – s/w model

App
App
App
O - I/O resources are owned by the OS
S - It provides I/O services to the supported
applications
I/O

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 17

 I/O Services

App
App
App
App
App
App
O O
S S

HBA NIC NIC HBA

- Typically, applications demand three types of I/O - storage, networking and IPC
- Usually, this means different wires, and different I/O protocols…

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 18

 A closer look at the I/O subsystem

App
App
App
App
App
App
O O
S S

HBA NIC NIC HBA

- Typically, applications demand three types of I/O - storage, networking and IPC
- Usually, this means different wires, and different I/O protocols…

…and the OS is usually involved in each of them

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 19

A closer look at the I/O subsystem

App
App
App
App
App
App
O O
S S

HBA NIC NIC HBA

The OS is actually part of the I/O subsystem

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 20

A closer look at the I/O subsystem

To achieve its value

App
App
App
App
App
App
O proposition,
O RDMA seeks to
S eliminate the
S OS as part of the
I/O subsystem
HBA NIC NIC HBA

The OS is actually part of the I/O subsystem

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 21

I/O Services

• Storage I/O
– block storage , e.g. SCSI (SAS, SATA, FC…)
• typically a kernel app doesn’t impact user apps
– file or object I/O, e.g. NFS, Lustre, NFS
• Sockets-based I/O
– both IP networking and IPC
– both are based on sockets

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 22

Some basic requirements

An I/O system that is not based on the OS must:

• provide all classes of I/O service to applications

• provide the same, or better, protection and

isolation services

• deliver improved I/O ‘performance’

– however you measure ‘performance’

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 23

The challenge: RDMA-based I/O

App
App
App
Applications exist in virtual memory, but…
O
S

I/O I/O devices exist in physical memory

The OS bridges the gap between applications residing in virtual

memory space and I/O devices residing in physical memory space

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 24

The challenge: RDMA-based I/O

App
App
App
Applications exist in virtual memory, but…
O
S

I/O I/O devices exist in physical memory

The OS bridges the gap between applications residing in virtual

memory space and I/O devices residing in physical memory space

If the goal is to eliminate the OS as part of the I/O

subsystem, we have to deal with this challenge

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 25

A different I/O concept: Channel I/O

App
App
App

App
App

App
O O
S S

- An I/O channel is a conduit between applications

- The OS establishes the channel, thus a channel is isolated and protected
- But the OS is not itself part of the channel!

Channel I/O allows applications to communicate very efficiently

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 26

Physically

App
App
App

App
App
App
O O
S S
channel i/f channel i/f

CA CA

-Physically, a channel is created between Channel Adapters

- An application accesses the channel via a thin layer of software
- This thin software layer is the channel interface
- Important Observation: The channel interface runs in user space
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 27
Channel I/O characteristics

1. Provides direct communication from application

to application
– Remember, applications exist in virtual space
2. Provides an app or process with direct access
to the channel – no need for OS intervention
3. Provides an appropriate set of communication
services to the application

The fundamental differences between programming for RDMA

vs sockets programming are rooted in these characteristics

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 28

 I/O, Message Passing

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 29

Message passing

- ‘Message Passing’ is a very common programming model

- Useful for communicating between processes or threads

App
messages
App

Inspiration for a new I/O model?

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 30

A message passing service

App App

Messaging Service

- There are lots of ways to implement message passing

- Consider a black box, and call it the messaging service
- The application doesn’t need to know how messages are passed…
…but it will help if you do
- The application only needs to know what the service does and how to use it

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 31

 Message passing over a distance

App App
compute compute
platform* platform*

Messaging Service

- Gets more interesting when the two processes reside in disjoint physical spaces
- This requires a networking construct to transport the messages

* ‘compute platform’ implies a single physical address space

under control of a single entity such as an OS (or a hypervisor)
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 32
Channel I/O defined

• Channel I/O is a communications paradigm that uses

RDMA mechanisms to pass messages directly between
processes
• These processes reside in disjoint virtual address
spaces which are located in disjoint physical address
spaces.
• Therefore, using an I/O channel places a slight burden
on the application programmer.
• This is why we are here today.

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 33

 Channel I/O

Channel I/O is a message passing paradigm

App App
end payload header

message

- The I/O protocol is carried within the message payload

e.g. SCSI, IPC, MPI…
-The message itself is agnostic to the payload
a payload is a payload is a payload…
- This means that messages can be transmitted and received simply and efficiently

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 34

 Storage I/O message passing

- An I/O protocol can be conducted using message passing

- I/O commands, data, and responses are passed as messages

storage client storage application

App App

message message
buffer buffer

SCSI command msg

Data msgs
SCSI response msg
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 35
Sockets based IPC (TCP)

An IPC application creates a

queue of messages to be sent
IPC message
IPC message The queued messages are
IPC message copied to a socket buffer…
IPC message

…and transmitted as a stream of

bytes
…message
message message
message message, byte byte … byte byte byte…
message…

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 36

Sockets based IPC (TCP)

app app
send user recv recv

send kernel anon copy

bfr switch bfr
switch

•bytes are received in an anonymous

buffer.
•application is notified
•application allocates receive buffer
•bytes are copied

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 37

Sockets based IPC

IPC is naturally message-oriented

Unfortunately….

-sockets is stream-oriented: It transfers IPC messages as a stream of bytes

-sockets is synchronous. The application waits for buffer copies on both ends

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 38

Message oriented IPC

As before, an IPC application

creates a queue of messages to
be sent
IPC message
IPC message
IPC message
IPC message The queued messages are
transmitted, as messages, not bytes

message message…message message…

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 39

Message oriented IPC

Messages are delivered directly to the application…

-no anonymous buffer
-no intermediate copy
IPC message
IPC message
IPC message
IPC message

message message…message message…

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 40

IPC message passing

IPC is naturally message-oriented,

Channel I/O is message-oriented

-an application posts an IPC message

-the channel delivers the message directly to the other end

Messages can range in size from very small up to very, very large.
The channel handles the transmission and delivery of the whole message.

(Don’t confuse a ‘message; with a ‘packet’...)

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 41

Remote Direct Memory Access

App App

platform platform

- RDMA is a mechanism for delivering messages over a distance

- It has some characteristics of a network

RDMA is the underlying mechanism beneath channel I/O

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 42

 Address translation and network
operations

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 43

Addressing

App App

I/O I/O

Remember this?
- Applications exist in virtual space
- Device adapters, (NICs, HBAs…) exist in physical space
- Each compute platform occupies a disjoint physical address space

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 45

 Three address translations

App App

v-2-p p-2-v

I/O I/O

p-2-p
Therefore, three address translations are required end-to-end:
- a virtual-to-physical address translation
- a physical-to-physical ‘translation’, and
- a physical-to-virtual translation
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 46
 Sockets-based address translation

App App

buf buf

virtual virtual
OS n n OS
buf i i buf
c c
physical physical

- The OS performs the v-p and p-v translations

- This ensures isolation and protection of each application’s virtual memory
space

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 47

Channel-based address translation

App App
channel adapters contain
buf address translation tables. buf

C C
A A
virtual virtual
physical
space space

-V-P, P-V translations performed by the channel hardware

- The channel provides protection and isolation services

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 48

Channel-based address translation

App App

buf buf

C C
A A

OS OS

- Translation and protection tables in the CA are created by the OS

- The OS supervises the channel
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 49
Channel-based address translation

App App

buf buf

C C
A A

The RDMA messaging service is built on a networking model to allow

communication between disjoint physical address spaces

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 50

 Channel-based address translation

App App

buf buf

C C
A A

An I/O Channel connects virtual address spaces. The virtual address spaces
can exist in disjoint physical address spaces.

Remote Direct Memory Access means that an application can directly read
and write remote virtual memory

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 51

 Inside the messaging service
RDMA architecture

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 52

 The RDMA messaging service

App App
Application layer

Session layer

Transport layer
RDMA Messaging Service
Network layer

Link layer
The messaging service contains a full
network stack. Phy layer

The question is: how does the application

OSI-like reference
access the network stack??
model
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 53
How TCP sockets does it

skts app app

application user send rcv

socket kernel send

copy rcv copy
buffers

Transport TCP

Network
- Sending app uses the sockets API to post a message to the
Link OS’ I/O service
- Using the I/O subsystem, the OS delivers bytes to the
receiver.
Phy - The OS uses the sockets API to copy a stream of bytes to
the receiver
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 54
RDMA architecture – 3 components

Application
s/w xport 1. Software Transport Interface
interface
Session
RDMA
2. RDMA Protocols
protocols
Transport
Transport
Network 3. Network Transport Service

Link
Let’s agree not to be too literal about the functional
Phy
definitions of each layer.

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 55

Software transport interface

Application
s/w xport • Virtual channel adapter architecture
interface
Session
RDMA
protocols
Transport
Transport
Network

Link
- a queued interface - ordered operations
Phy - queues are mapped into application virtual space
- a message oriented architecture

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 56

 A queued interface

App

s/w xport
interface
QP CQs
RDMA
protocols

Transport -The s/w transport interface contains a series of queues

- QPs are queues of work to be executed
-CQs are queues of work completions
-A CQ can be associated with more than one QP

Think of the work queues as being the endpoints of the channel

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 57
A queued interface

app pulls
applications put App completions off
requests on the queue
the CQ

QP CQs

channel adapter pulls

CA puts
requests off bottom of CA completions on
the queue
the CQ

Work queues drive the channel interface, CQs signal completed work

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 58

 Virtual adapter architecture

App

- Work queues are mapped into

the app’s virtual address space
QP CQs
- This gives the app direct
access to the channel

- Creating a work queue virtual space

requires a kernel call CA

This simple observation accounts for much of the RDMA value proposition:

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 59

 Work queues, queue pairs

App App

s/w xport Work queues always come in

interface pairs, hence the concept of a
RDMA QP Queue Pair – a QP
protocols
Each QP is associated with exactly
Transport one application
send
queue

receive
queue

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 60

 Work requests

App

WR
s/w xport WR A Work Request (WR) is a data structure that
interface WR describes a piece of work to be completed:
WR -a message to be sent,
RDMA WR -a message to be received…
protocols
An application posts a WR to a queue.
Transport
QPs CQs

You will learn a great deal about these data structures in the next part of the course.

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 61

 Architecturally…

App
Once posted to a work queue, a WR becomes
WR an element of that queue, called a Work Queue
s/w xport WR Element (WQE – pronounced wookie)
interface WR
WR Similarly, the elements on a completion queue
RDMA WR are called CQEs – pronounced cookie
protocols

Transport
QPs CQs

OFED deals strictly in Work Requests…you won’t find references to WQEs in the
code. (But you will find them in the architecture, if you go looking).

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 62

 An asynchronous interface

App App

s/w xport
interface
RDMA QP CQ

protocols

Transport
- App posts a WR to the queue and returns immediately.
It does not wait for a buffer copy.

- The channel notifies the app when a WR is complete by

posting a response to the Completion Queue (CQ)

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 63

 Ordering

App App

s/w xport
interface
QPs QPs QPs QPs
RDMA
protocols

Transport

-Ordering is guaranteed for all WRs submitted to a given send queue

-Ordering is guaranteed to all WRs submitted to a given receive queue
-There are no ordering guarantees between send and receive queues
-There are no ordering guarantees between QPs

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 64

 Connecting QPs

App App

s/w xport
interface
RDMA QPs QPs
protocols

Transport

- For connected services, a connection exists between a pair of QPs

- Thus a message on a send queue can only be delivered to the other
end of the connection
- Conversely, no other application can send a message using this
connection.
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 65
Using the S/W transport interface

• A user creates a Work Request to send or receive a message

• To prepare the hardware to receive a message, the user posts a

Receive WR to the receive queue.

•To send a message the user posts Send WR to the send queue.

You will learn a great deal more about these verbs during the next section.

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 66

Summarizing the s/w transport interface

- Consists of a series of work

queues and completion queues
Application
s/w xport - An application uses the channel
interface by posting work requests (WR) to
Session the queues, and receiving
RDMA
protocols completions through the CQs
Transport
Transport - Work Requests (WRs) are data
Network structures that succinctly describe
the work to be accomplished.
Link
- Think of WRs as commands from
Phy the application to the channel
hardware.

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 67

RDMA Architecture - RDMA protocols

Application
s/w xport
interface
Session
RDMA
• Memory-to-memory transfer protocols
protocols
Transport
Transport
Network

Link

Phy

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 68

RDMA operations

A choice of message passing operations

– SEND/RECEIVE – a ‘channel semantic‘
– RDMA READ or RDMA WRITE – a ‘memory semantic’

App App

RDMA Messaging Service

-SEND/RECEIVE
-RDMA Read/RDMA Write

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 69

 SEND/RECEIVE
Requester Responder
SEND WR RECEIVE WR

QP QP
CA

Responder posts a WR to the RECEIVE queue

Requester posts a WR to the SEND queue

Every SEND/RECEIVE:
-consumes a WQE from the Requester’s SEND queue,
-consumes a WQE from the Responder’s RECEIVE queue.

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 70

 SEND/RECEIVE

send send receive receive

buffers queue send WQE queue buffers
-src bfr
-… receive WQE
-…
-dst bfr
-…

CA CA

-Send WQE defines source buffer -Receive WQE defines destination buffer
-Receive buffers must be pre-posted

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 71

SEND/RECEIVE

• SEND/RECEIVE is a double ended operation – it involves both the

Requester and the Responder
– Requester = SEND side, Responder = RECEIVE side

• A SEND operation consumes a WQE on both the Requester (SEND) side

and on the Responder (RECEIVE) side.

• Destination buffer address is controlled by the Responder side

• RECEIVE WQE must be posted before the SEND WQE

– failure to do so risks a race condition and a failed operation

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 72

 RDMA WRITE, RDMA READ
Requester Responder
RDMA WRITE WR
RDMA READ WR

QP QP
CA

Requester posts a WR to the SEND queue (yes, the SEND queue)

Responder does nothing.

Every RDMA WRITE or RDMA READ operation:

-consumes a WQE from the Requester’s SEND queue
-NO WQE is consumed from the responder side; CA handles the request directly.
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 73
 Identifying the remote buffer

Requester Responder
buffers
App App

buffer -VA
descriptor -R-KEY

- Responder’s buffer is described by a Virtual Address (VA) and an R-KEY

- Responder “passes control of the buffer” to the Requester, before the
operation begins
- Requester owns the buffer until it returns control to the Responder
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 74
Passing the buffer descriptor

Requester Responder
buffers
App App

Responder uses e.g. a SEND operation to buffer -VA

transmit the buffer descriptor to the Requester. descriptor -R-KEY

Once buffer control has been passed, the

Responder should not access that memory until
the Requester notifies the Responder that the
operation is complete, and returns buffer control
to the responder.

The means for passing control of the buffer back and forth is not specified in the
RDMA architecture; it is defined by the upper layer protocol

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 75

 RDMA WRITE

source send destination

RDMA WQE
buffers queue buffers
-src bfr
-dst VA
-dst R-KEY

VA & R-KEY are passed back to

responder CA in the request packet
header
CA CA

- RDMA WQE defines BOTH the source and destination buffers

- Responder CA resolves the destination buffer VA into a physical address
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 76
 RDMA READ

destination send source

RDMA WQE
buffers queue buffers
-dst bfr
-src VA
-src R-KEY

CA CA

-RDMA READ is exactly the same, except for data transfer direction

(and Yes, the RDMA READ REQUEST is posted to the SEND queue)

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 77

RDMA WRITE, RDMA READ

• A memory semantic – the Responder exposes its buffers directly to the

Requester

• A single ended operation – reads and writes are opaque to the responder
while they are happening

• The requester accesses the responder’s memory with the help of the
responder side channel hardware

• It takes a little more setup to get started, but…

…is a very efficient way to transfer blocks of data

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 78

An unfortunate bit of confusion

• There are two types of RDMA operations

• SEND/RECEIVE & RDMA READ/RDMA WRITE

• Both are lumped under the rubric ‘RDMA’

• ‘RDMA’ usually refers to the generic notion of memory-to-memory transfers

– including SEND/RECEIVE and RDMA READ/WRITE

• ‘RDMA READ’ or ‘RDMA WRITE’ refers to the specific operation

• Usually clear from the context

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 79

A bit more confusion to clear up

• A SEND/RECEIVE writes data to the responder

• The requester SENDS, the responder RECEIVES

• An RDMA WRITE also writes data to the responder

• An RDMA READ reads data from the responder

• So there is no symmetry between

SEND/RECEIVE and RDMA READ/WRITE

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 80

RDMA Operations - summary

Two semantics are supported

1. SEND/RECEIVE (‘channel semantic’)

• requester and responder each control its own buffer resources
• requester sends a message, responder receives a message

2. RDMA READ and RDMA WRITE (‘memory semantic’)

• requester controls both ends of the operation
• requester reads and writes remote memory directly

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 81

 The concise ordering rules

Operations on the SEND queue are transmitted on the wire in order.

Operations at the RESPONDER side are executed in the order received.

A SEND or RDMA WRITE may be executed before an RDMA READ!

Operations on the SEND queue are completed in the order in which they were
posted.

3 SEND responder
2 RDMA RD
1 SEND
SEND 3 RDMA RD SEND 1

??? ACK 1 READ DATA ACK 3

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 82

Transport service

Application
s/w xport
interface
Session
RDMA
protocols
Transport
Transport
Network •Transport service

Link

Phy

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 83

 Transport service

App
The transport layer provides (at least) two services:
s/w xport
Reliable, Connection-oriented (RC). TCP is an example
interface
of a reliable connected service.
RDMA
protocols Unreliable Datagram (UD). UDP is an example of an
unreliable datagram service.
Transport
How the service is provided is a function of the
underlying wire protocol.

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 84

 Technical definition of ‘Reliable’

“Data is delivered to a receiver exactly once and in order, in the absence

of errors”

“Exactly once”: the responder is notified only

once that a message has been delivered.

“In order”: the transport will execute each WQE

in order, and adhere to certain rules controlling receive
queue
the order in which operations are completed.

“In the absence of errors” means that the CA

transport will either recover from an error
condition or report it. It won’t lie about it. completion
queue responder

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 85

‘Delivered’

For a message-oriented transport, a message is considered to have

been delivered when all packets have been placed in the designated
receive buffer in the correct order, and the receiving application has
been notified (if so configured).

There are no guarantees about

receive
the state of the receive buffer until queue
the CA has delivered the
message.
SEND CA
It is a bad idea to use a value in
completion
the message buffer as a flag or queue responder
signal.

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 86

‘Reliable’ vs ‘Lossless’

One small, but significant clarification.

A ‘lossless wire’ (i.e. one that does not drop packets) is NOT the same thing as a
reliable transport.

Losslessness refers to the likelihood that a packet might be dropped in transit.

A reliable transport is capable of detecting and recovering from a dropped

packet. Or at least reporting it. A reliable transport implements a complex
protocol for this very purpose.

The InfiniBand reliability protocol is implemented by the InfiniBand Transport.

Reliability for iWARP is implemented by TCP. Both are reliability protocols.

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 87

 So what’s a ‘connection’?

The concept of a connection is fundamental to reliable connected service

A connection is created between two QPs when they have exchanged a set of
attributes, and have agreed that they are connected.

connection
App between QPs
App
QP QP

src ID : src QPn dst ID : dst QPn

The connection ID is stored by each QP in its local context. The hardware

validates every inbound packet against its local context.

This guarantees isolation and protection of the channel between the two
applications.

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 88

Reliable, Connection-Oriented service

• a connection exists between endpoints

• the connection ID is used to validate inbound packets

• an ACK/NAK protocol is used to detect missing or damaged

packets. Missing or out of order packets are re-sent.
Unrecoverable packets are reported to the application.

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 89

Unreliable Datagram (UD) service

• Unconnected - No connection exists between endpoints (e.g. QPs)

• Unreliable – unacknowledged service. Damaged packets are

dropped
– ‘Fire and forget’

• Protection mechanisms are provided to ensure a UD packet is

delivered correctly to its intended destination
– Q-KEYs are provided for this purpose. More on these later.

• A message is limited to a single packet

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 90

Unreliable Datagram (UD) service

App

App App

App

UD service allows an application to communicate with several other

applications using a single QP.
Dropped or lost packets are gone for good.
Each message is limited to a single packet – no segmentation and re-assembly

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 91

Multicast

App
switch

App App

App

Multicast is implemented using UD service

Each packet contains a multicast identifier
Packets (messages) are replicated in the switch

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 92

Summarizing RDMA architecture

• The heart of RDMA is housed in the session and

transport layers
– software transport i/f + RDMA protocols + transport
• A queued, asynchronous interface is provided
• Two RDMA operations are supported*
– RDMA READ/WRITE and SEND/RECEIVE
• Both Reliable Connected (RC) and Unreliable Datagram
(UD) services are provided.

* Actually there are a couple more operations, but not being covered here.
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 93
 Verbs introduction

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 94

 Verbs API

App App

QPs CQs QPs CQs

RDMA Messaging Service

The Queues (work queues plus completion queues) represent the inputs and
outputs to the channel.

The verbs API is the programmatic interface to the channel

Thus the Verbs plus the Queues together represent the Channel Interface

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 95

 For example…

WR
App WR
WR
WR
WR

s/w xport Verbs An application uses the POST SEND REQUEST

interface API verb to submit a SEND WR to the send queue.
RDMA
protocols
QPs CQs
Transport
Interesting note: POST SEND REQUEST is
used to post both SEND and RDMA WR/RD
requests to the send queue, but with a different
set of attributes
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 96
A simple view of using an I/O channel

1. Open an HCA
2. Create a Protection Domain (what the heck is that?)
3. Create a Queue Pair
4. Create (a) Completion Queue(s)
5. Register memory regions (coming up shortly)
6. Post work requests
7. Wait for completions

Verbs is the mechanism by which all this is done.

Part 2 of this course will take us through the verbs in some detail.

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 97

Categories of verbs

• Here is a view of the categories of verbs.

• We will go through the categories in detail in the next portion of this course

Transfer Posting verbs how to begin a message transfer

Transfer Completion verbs waiting for a transfer to end

Memory Registration verbs address translation and control

Connection Management verbs where a msg is sent

Miscellaneous verbs as it says…miscellaneous

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 98

Verbs - a semantic description

• The concept of a ‘Verb’ was described in the InfiniBand Architecture

Specification, circa 1999

• As specified, the Verbs are a semantic description of the required behavior

of the messaging service, but with no programmatic (e.g. OS specific)
details.

• Thus, verbs as specified are OS independent

• Unfortunately, “Verbs” are not an API (believe it or not).

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 99

Verbs – an API

• To prevent fracturing of the InfiniBand market the OpenFabrics Alliance (nee

OpenIB) was formed to create a common implementation of the Verbs
specification

• The resulting (OS-dependent) APIs are also called ‘Verbs’

• Therefore, “Verbs” are an API (fortunately)

You will hear people talk about Verbs, sometimes in reference to the abstract
semantic description and sometimes in reference to the API.
It is rarely confusing.
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 100
Transport independence

• Soon, others realized that there can be different implementations of an RDMA

messaging service

• The RDMA Consortium spiffed up the Verbs spec while defining iWARP

• The IB verbs spec was updated to incorporate those improvements, thus

preserving consistency between IB and iWARP

• Nowadays, three different ‘wire protocols’ are supported

• InfiniBand, iWARP and RoCE. Each looks different on the wire

• All three present the same interface (APIs) and semantic behavior

The RDMA messaging service is accessed using OFA software,

regardless of the underlying transport or wire protocol
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 101
 Isolation and protection
mechanisms

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 102

Basic theory

To deliver performance and scalability benefits, RDMA provides a user

application with direct access to system-level resources

This direct access creates a requirement for protection and isolation

mechanisms

Basic protection requirements:

1. Allow direct user access to the Channel Adapter
2. Protect the user’s virtual memory space
3. Create isolated and protected channels between applications

In this section, we will discuss:

1. Memory Registration: R-Keys, L-Keys
2. Protection domains for connected service
3. Protection domains for unconnected service: Q-Keys

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 103

Memory Registration

To provide direct user access to a Channel Adapter, Queue Pairs and

Completion Queues are mapped directly into the application’s virtual
address space

The mapping is done via a memory registration process

This means the application can manipulate the channel directly - no need
for a context switch to a privileged entity

Memory registration is a privileged operation and involves the OS

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 104

 Memory regions

verb API

- A memory region is created when an app

registers virtual memory with the CA O
App
S
- A memory region is a chunk of virtually
contiguous memory -R-KEY
-L-KEY
- Once registered, the CA returns an R-KEY and
an L-Key for each memory region

Basic idea: The L-Key and R-Key are used memory

CA
by the channel to execute virtual-to-physical region
and physical-to-virtual address translations

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 105

 Memory regions: L-Key, R-Key usage

Requester Responder
- VA
App App
- R-KEY

L-KEY
- L-KEY is passed to the local CA as part of a WR
- R-KEY controls remote buffer access (RDMA ops)

- VA memory
CA CA
- R-KEY region

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 106

End-to-end protection and isolation

For Connected services:

A connection is between QPs

app app
QP QP

How to create end-to-end

protection and isolation?

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 107

 Protection Domains

For Connected services:

A connection is between QPs

app app
QP QP

PD PD

A Protection Domain (PD) creates an association between an application and a QP

This completes the end-to-end ‘chain of protection’

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 108

Protection Domains

- A Protection Domain (PD) is a container that

QP
associates one or more QPs with a memory
region
mem
app QP
region
- It controls which QPs can access that memory
QP region

- A PD binds together:
- an application
- one or more memory regions, and
- one or more QPs

- An application establishes a PD through the use

of a verb

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 109

Protection Domains

QP

mem
QP
region R-KEY 1
app QP
R-KEY 2
mem
region

- A Protection Domain (PD) can contain multple

memory regions, identified by different R-KEYs
and L-KEYs

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 110

Protection Domains

- An app can create multiple PDs

QP
mem
region - A memory region can only be bound to one PD
QP
app - But, a given piece of memory can be registered
mem twice, and thus appear in two different protection
QP
region domains

- A QP can only bind to a single memory region

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 111

Protection Domains

QP QP app
mem
app QP The PD ensures that only QPs that are
region
registered to that virtual memory can access it.
QP
For connected service, the combination of two
PDs plus the connection between QPs creates
a protected, isolated channel between
application virtual memory

QP QP app
mem
app
region
QP

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 112

Protection Domains - Summary

• An app (or process) may communicate with several other apps or

processes
• Protection domains allow the app to control access to its memory regions by
controlling which QPs can access a memory region
• In order:
– an application allocates one or more protection domains
– QPs are allocated to a single PD
– memory regions are registered to a single PD
• An app may allocate multiple PDs
• A memory region is associated with only one PD
• A QP is associated with only one PD
• A given range of physical memory may be registered more than once

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 113

Channel protection & isolation

• Connected and unconnected services use different mechanisms to isolate

and protect applications and the channel between them

• Connected service:
– the receiver validates incoming packets based on the connection ID
– that creates a protected QP-to-QP connection…
– …but it doesn’t create end-to-end protection!
– we’ll need something more to complete the chain of protection

• Unconnected (datagram) service

– There is no connection ID, so how does the receiver validate an inbound packet?

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 114

End-to-end protection and isolation

For Unconnected services: There is no connection

between QPs!

app
QP

app app
QP
QP

app
QP

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 115

End-to-end protection and isolation

For Unconnected services:

-Q-Key is used to validate inbound packets (instead of connection ID)

Q-Keys

app
QP

app app
QP
QP

PD
PD

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 116

Summing up isolation and protection

• An application registers memory with the CA.

• Fast Memory Registration (FMR) – not covered in this section
• Memory windows – not used currently
• Memory registration returns access keys
• L-KEY – controls access to a memory region by the LOCAL user
• R-KEY – controls access to a memory region from a remote
• Protection domains allow an application to control who has access to its
memory regions
• It controls access on a per-QP basis
• Inbound packets are validated by
• Connection ID for connection oriented service
• Q-KEY for datagram services

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 117

 OFED stack

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 118

 Quick recap

App Application

VERBS
S/W transport i/f
Session
RDMA protocols
Transport
RDMA messaging Transport
service Network
Link
Phy

- An application accesses the RDMA messaging service by using VERBS

- The messaging service includes a complete network stack
- The principal unique elements of the stack are in the session and transport layers

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 119

 A high level look at OFED

Consumer user applications kernel apps

Layer

The mid-layer (not shown) contains:

- fabric management
- subnet management ULPs
- connection management
These are all part of system administration

user verbs kernel verbs

H/W driver H/W driver H/W driver

Provider
Layer hardware hardware hardware

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 120

A high level look at OFED

Consumer user applications kernel apps

Application
Layer

Session
Transport ULPs
Network
Link
Phy user verbs kernel verbs

H/W driver H/W driver H/W driver

Provider
Layer hardware hardware hardware

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 121

 A high level look at OFED

user applications kernel apps

It is not always convenient or
practical to (re-)write an
application to the verbs API.

ULPs
VNIC SDP SRP

IPoIB iSER RDS

user verbs kernel verbs
cluster NFS
file systems RDMA

Upper Layer Protocols (ULPs) H/W driver H/W driver H/W driver
allow a standard application to hardware hardware hardware
execute over an RDMA network
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 122
 Main focus of this course…

Consumer user applications kernel apps

Layer

Now that you know about the

provider layer and what it does,
the rest of the course is ULPs
focused on the user verbs

user verbs kernel verbs

H/W driver H/W driver H/W driver

Provider
Layer hardware hardware hardware

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 123

And in particular…

user applications kernel apps

…it is about coding user

space applications directly
to the user verbs API
ULPs

user verbs kernel verbs

you already know what H/W driver H/W driver H/W driver
the black box can do. hardware hardware hardware

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 124

 OFED – the details
SA Subnet
Application IP Based Sockets
Various
Block
Clustered
Access to Administrator
Level App Based Storage File
Diag Open MPIs DB Access MAD Management
Access Access Access Systems
Tools SM Datagram

User Level SMA Subnet Manager

UDAPL Agent
MAD API
User OpenFabrics User Level Verbs & CMA / API PMA Performance
Manager Agent
APIs
User Space SDP Lib IPoIB IP over InfiniBand

SDP Sockets Direct

Upper Kernel Space Protocol

Layer VNIC IPoIB SDP SRP iSER RDS

NFS-RDMA Cluster SRP SCSI RDMA
RPC File Sys Protocol (Initiator)
Protocol
iSER iSCSI RDMA
Protocol (Initiator)
Connection Manager
Abstraction (CMA) RDS Reliable Datagram
Kernel bypass

Kernel bypass
Mid-Layer Service
SA Connection Connection
MAD SMA VNIC
Client Virtual NIC
Manager Manager
UDAPL User Direct Access
OpenFabrics Kernel Level Verbs / API Programming Lib
HCA Host Channel
Adapter
Provider Hardware Hardware Specific
R-NIC RDMA NIC
Specific Driver Driver
Common Apps &
Key Access
Hardware InfiniBand HCA iWARP R-NIC InfiniBand Methods
for using
iWARP OF Stack
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc.
© 2011 OpenFabrics Alliance, Inc.
08/23/2011 125
 Introduction to wire protocols

InfiniBand, iWARP, RoCE

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 126

 Multi apps, one channel interface

storage file native

skts apps
apps systems apps

SDP, IPoIB SRP, iSER

Verbs

RDMA messaging service

Channel
Interface

The channel interface provides a consistent programmatic interface

to the message passing service
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 127
 Multi apps, one CI, three wires

storage file native

skts apps
apps systems apps

SDP, IPoIB SRP, iSER

Verbs

InfiniBand iWARP RoCE

Channel
Interface

IB IP Enet
network network fabric
www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 128
Wire level protocols

App App App

verbs

RDMAP
DDP
MPA Xport i/f Xport i/f
RNIC TCP HCA IB Xport ‘NIC’ IB Xport
IP IB N/W IB N/W
Enet link IB Link Enet link
iWARP InfiniBand RoCE

CAUTION! Don’t read the hardware/software partitioning too literally!

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 129
Supported transport services

Service Type iWARP IB RoCE

reliable connected TCP RC RC
unreliable connected n/a UC UC
reliable datagram n/a RD RD not supported
unreliable datagram UDP UD UD multicast

connected service – a connection exists between QPs

datagram service – no connection exists between QPs

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 130

 Same behaviors, different wire level
protocols

Application App App App

RDMAP
Session s/w transport interface
DDP
RDMA protocols
MPA Xport i/f Xport i/f
Transport Transport
TCP IB Xport IB Xport
Network IP IB N/W IB N/W
Link Enet link IB Link Enet link
Phy Enet phy IB phy Enet phy
ref model iWARP InfiniBand RoCE
(OSI, sort of…)

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 131

Summarizing wire protocols

• OFED software supports different types of

applications
• The channel interface abstracts the underlying
wire and presents an RDMA messaging service
to the application
• OFED currently supports three different wire
protocols, running on two different types of wires
– InfiniBand, Ethernet

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 132

 Break for lunch, then…
…onward to Part 2:

RDMA Programming with OFA

Software using the OFA verbs API

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 133

 Backup

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 134

A list of the verbs

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 135

How TCP sockets does it

app
user app
send
copy
send kernel anon
bfr switch bfr
switch

•bytes are received in an

anonymous kernel buffer.
•application is notified
•application allocates receive
buffer
•bytes are copied

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 136

 Virtual-to-virtual transfers

App

App
O O
S S

An I/O Channel connects virtual address spaces. The virtual address spaces
can exist in disjoint physical address spaces.

Remote Direct Memory Access means that an application can directly read
and write remote virtual memory

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 137

 Network addressing

App

App
O O
S S

CA CA

The RDMA messaging service is built on a networking model to allow

communication between disjoint physical address spaces

www.openfabrics.org © 2011 OpenFabrics Alliance, Inc. 08/23/2011 138