Communication Patterns for Embedded

System Design
Johan Philips
University of Leuven, Belgium
http://people.mech.kuleuven.be/~jphilips/

March 25, 2015

Communication Patterns for Embedded System Design

Johan Philips 1
March 25, 2015

Objectives of this talk

I discuss different communication methods and their design

implications
I give a brief introduction to computer networking: network
models, protocols and topologies
I define a set of communication patterns to address common
issues (ZeroMQ)
I apply lessons learned to currently ongoing EU-projects
SHERPA and Pick-n-Pack

We will NOT cover messaging formats and serialisers

Communication Patterns for Embedded System Design

Johan Philips 2
March 25, 2015

Communication Examples
Analog world
I telephone call

I conversation

I oral examination

I leave a note

I passing in team sports

I using the TV remote

I sending a letter

I filling in an evaluation report

I writing on blackboard

I handshaking

Communication Patterns for Embedded System Design

Johan Philips 3
March 25, 2015
 Communication Examples
Digital world
I sending an email

I writing on electronic blackboard

I writing data to file

I posting a message on Facebook

I chatting with your girlfriend

I publishing sensor data on data bus

I playing WoW with your friends

I clicking on a news item online

I opening a folder on your computer

I tweeting about your latest achievement

Communication Patterns for Embedded System Design

Johan Philips 4
March 25, 2015

Asynchronous vs synchronous
Asynchronous
I takes place outside of real time → time lag

I no need to wait for reply → flexible

I data is transmitted intermittently → notification required

I lack of immediacy → information can be out of date

Synchronous
I similar to a conversation → no time lag

I real time responses → involved parties need to be ready and

willing
I immediacy → no need to wait for reply

I more interactive than asynchronous communication

Communication Patterns for Embedded System Design

Johan Philips 5
March 25, 2015

Communication in software systems

Depending on the deployment of your system (of systems),

communication can occur
I within one process

I between processes on the same machine

I between processes on different machines in same network

I between processes on different machines in different networks

Communication Patterns for Embedded System Design

Johan Philips 6
March 25, 2015
Specific challenges in embedded systems

I Concurrency: how to interconnect many nodes?

I High performance: how to ensure communication does not

block computations?

I Small footprint: how to provide communication with a

minimal footprint (memory, CPU)?
→ We need design patterns for communication! But first some
more Computer Networking 101...

Communication Patterns for Embedded System Design

Johan Philips 7
March 25, 2015

The OSI model

Open Systems Interconnection model
I application layer: user-facing application

I presentation layer: crypto, conversion between

representations
I session layer: tie together multiple streams

I transport layer: multiplexing applications

I network layer: move packets to any other node in the

network
I data link layer: move frames to other node across link

I physical layer: move bits to other node across link

→ On which levels have we been focussing during this course?

Communication Patterns for Embedded System Design

Johan Philips 8
March 25, 2015

Internet (TCP/IP) model

Presentation and session layer mixed with application and
transport layer and no physical layer
I application layer
I provide user services
I formatting specifics and data presentation part of libraries and
APIs
I treat transport layer as black box which provides stable
network connection
I transport layer
I unconcerned with specifics of higher level protocols
I no examination of encapsulated traffic
I internet layer
I link layer
In embedded systems, lightweight IP (lwIP) is widely used open
source TCP/IP stack
Communication Patterns for Embedded System Design
Johan Philips 9
March 25, 2015
 IP stack connections
Network Topology

Host Host
Router Router
A B

Data Flow
Application process-to-process Application

Transport host-to-host Transport

Internet Internet Internet Internet

Link Link Link Link

Fiber,
Ethernet Satellite, Ethernet
etc.

”IP stack connections” by en:User:Kbrose - Prior Wikipedia artwork by en:User:Cburnett. Licensed under CC BY-SA

3.0 via Wikimedia Commons -

http://commons.wikimedia.org/wiki/File:IP stack connections.svg#/media/File:IP stack connections.svg

Communication Patterns for Embedded System Design

Johan Philips 10
March 25, 2015

Data encapsulation

Data Application

UDP UDP Transport

header data

IP
IP data Internet
header

Frame Frame
Frame data Link
header footer

”UDP encapsulation” by en:User:Cburnett original work, colorization by en:User:Kbrose - Original artwork by

en:User:Cburnett. Licensed under CC BY-SA 3.0 via Wikimedia Commons -

http://commons.wikimedia.org/wiki/File:UDP encapsulation.svg#/media/File:UDP encapsulation.svg

Communication Patterns for Embedded System Design

Johan Philips 11
March 25, 2015

Challenges to layered model

I What belongs to which layer?

I How do you divide services?
I Which abstractions do you make?
I What do you leave out?

Communication Patterns for Embedded System Design

Johan Philips 12
March 25, 2015
 Some common protocols

What do you need to communicate and who do you talk to?

I application layer: HTTP, SMTP, Telnet, ...

I presentation layer: MIME, ...
I session layer: RTP, RTCP, ...
I transport layer: TCP, UDP, ...
I network layer: IP, ICMP, ...
I data link layer: MAC, ARP, ...
I physical layer: Ethernet, CAN bus, GSM, IEEE 802.11,
Bluetooth, RS-232, ...

Communication Patterns for Embedded System Design

Johan Philips 13
March 25, 2015

OSI model usage in embedded systems

I Embedded systems typically operate on the lowest layers in

the OSI model

I Trend to more complex embedded systems and systems of

systems also require higher layers!

Communication Patterns for Embedded System Design

Johan Philips 14
March 25, 2015

Meanwhile, on the physical layer...

... different topologies and hardware architectures exist!

I based on size: PAN, LAN, MAN, WAN

I based on purpose: SAN, EPN, VPN

I based on topology: P2P, bus, star, ring, mesh, tree, cloud

→ We need communication patterns to address some physical

requirements and/or limitations we might have!

Communication Patterns for Embedded System Design

Johan Philips 15
March 25, 2015
 Intermezzo: programming models

Synchronous model
I single-threaded

I simple programming style

I each task is completely finished

before another starts

I particular order in which tasks are

executed → scheduling
I e.g. Arduino car

Figure from Dave Peticolas’ introduction on asynchronous programming

Communication Patterns for Embedded System Design

Johan Philips 16
March 25, 2015

Intermezzo: programming models (2)

Threaded model
I each task is executed in a separate thread of control

I OS manages threads

I tasks are implicitly scheduled by OS

I complex due to inter thread communication and coordination

I e.g. ODROID board

Figure from Dave Peticolas’ introduction on asynchronous programming

Communication Patterns for Embedded System Design

Johan Philips 17
March 25, 2015

Intermezzo: programming models (3)

Asynchronous model
I tasks are interleaved

I single thread of control

I explicitly schedule tasks

I performs well with large number of

tasks (less waiting time, no context

switches, ...)
I e.g. composition pattern

Figure from Dave Peticolas’ introduction on asynchronous programming

Communication Patterns for Embedded System Design

Johan Philips 18
March 25, 2015
 Intermezzo: programming models (4)

Event-driven programming
I no blocking for I/O
I program flow is determined by events
I callbacks to receive notifications
I very common in GUI frameworks
I Reactor pattern to separate event logic
from business logic

Communication Patterns for Embedded System Design

Johan Philips 19
March 25, 2015

Remember? Communication in an Activity

when triggered
do {
communicate() % get latest events
coordinate() % react to them
configure() % possibly requiring reconfiguration
schedule() % now do one’s Behaviour
coordinate() % execution could trigger new events
communicate() % that others might want to know about
log()
}

Communication Patterns for Embedded System Design

Johan Philips 20
March 25, 2015

Back to Communication: Sockets

I endpoint of inter-process communication flow
I follow the protocol specifications that define interfaces
I typically interface between application layer and transport
layer

Communication Patterns for Embedded System Design

Johan Philips 21
March 25, 2015
 ZeroMQ

I socket library, but much more...

I models for communication
I patterns addressing typical networking issues
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0

Communication Patterns for Embedded System Design

Johan Philips 22
March 25, 2015

ZeroMQ - zero-em-queue - MQ

I Connect your code in any language, on any platform.

I Support for many protocols: inproc, IPC, TCP, TIPC, multicast.
I Smart sockets like pub-sub, push-pull, and router-dealer.
I High-speed asynchronous I/O engines, in a tiny library.
I Backed by a large and active open source community.
I Build any architecture: centralised, distributed, small, or large.
I Free software with full commercial support.

adopted from zeromq.org, copyright (c) 2015 iMatix Corporation and licensed under cc-by-sa 3.0

Communication Patterns for Embedded System Design

Johan Philips 23
March 25, 2015

SHERPA project

I Search & rescue in alpine environments

I Hetereogeneity of actors: human,
ground vehicle and aerial robotic platforms
I Distributed cognitive world model
I QoS for network infrastructure
I http://sherpa-project.eu/

Communication Patterns for Embedded System Design

Johan Philips 24
March 25, 2015
 Pick-n-Pack project

I Flexible production line in food industry

I Line controller and several modules with
one or more devices
I Traceability of products, processes and
processors
I Communication and interaction between
all actors
I http://www.picknpack.eu/

Communication Patterns for Embedded System Design

Johan Philips 25
March 25, 2015

Communication challenges in these projects

I Uncertain communication network
I Dynamic configurations of multiple entities/robots
I Synchronisation of mission execution/production
I Bootstrapping and initialisation of coooperative robot
application
I Tracing of data and intermediate (mission) states
I Failure detection
I Highly configurable communication protocol and
infrastructure
I Deployment based communication and data optimisations
I ...

Communication Patterns for Embedded System Design

Johan Philips 26
March 25, 2015

REQ-REP

I Example: GUI sending request to

robot controller, HTTP request Client
I Typically used in synchronised REQ

Client/Server model
I Client sends request to Server and Hello World

(actively) waits for reply

I Server (actively) waits for request REP

from Client and sends reply when Server

request is received
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0

Communication Patterns for Embedded System Design

Johan Philips 27
March 25, 2015
 PUB-SUB

I Example: Sensor broadcasting data Publisher

measurements PUB

bind
I Typically used in asynchronised p2p updates
model
I Publisher write data to PUB socket updates updates updates

I Subscriber subscribes to SUB connect connect

socket and receives a notification if SUB

Subscriber
SUB

Subscriber
new data arrives
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0

Communication Patterns for Embedded System Design

Johan Philips 28
March 25, 2015

PUSH-PULL & PAIR-PAIR

Ventilator
Step 1
PUSH
PAIR

tasks
Ready!

PULL PULL PAIR

Worker Worker Step 2
PUSH PUSH PAIR

Ready!
results

PULL
PAIR

Sink Step 3

I Synchronise work flow between different computations within

one process (PAIR) or between processes (PUSH-PULL)
I PAIR: optimise by using shared memory
I PUSH-PULL: divide work (ventilator) load and collect results
(sink)
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0

Communication Patterns for Embedded System Design

Johan Philips 29
March 25, 2015

ROUTER-DEALER

Client
I Asynchronous request-reply
DEALER I N to N communication
I Router sockets adds addressee in
message header to route replies
correctly
ROUTER I Dealer sockets enforce explicit
Server
bookkeeping in application code
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0

Communication Patterns for Embedded System Design

Johan Philips 30
March 25, 2015
 Patterns on http://zguide.zeromq.org

I Espresso I Parallel pipeline w/

I Binary Star I Lazy Pirate kill signalling
I Last value caching I Parallel task I Load balancing
I Relay race distribution and I Majordomo
I Freelance collection I Paranoid Pirate
I Simple Pirate I Bridge I One-way data
I Node coordination I RPC and task distribution
I Suicidal snail distribution I Titanic
I Shared queue I Black box

Communication Patterns for Embedded System Design

Johan Philips 31
March 25, 2015

Patterns on http://zguide.zeromq.org

I Espresso I Parallel pipeline w/

I Binary Star I Lazy Pirate kill signalling
I Last value caching I Parallel task I Load balancing
I Relay race distribution and I Majordomo
I Freelance collection I Paranoid Pirate
I Simple Pirate I Bridge I One-way data
I Node coordination I RPC and task distribution
I Suicidal snail distribution I Titanic
I Shared queue I Black box

Communication Patterns for Embedded System Design

Johan Philips 32
March 25, 2015

Node coordination

Publisher

PUB REP
I Synchronise initialisation
(3) (2)
I Separate channel to
(1) bootstrap communication
I Solves late joiner problem
SUB REQ

Subscriber
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0

Communication Patterns for Embedded System Design

Johan Philips 33
March 25, 2015
 Paranoid pirate

Client Client
I Heartbeating between
Retry Retry peers: e.g. server and its
REQ REQ
workers
I (Basic) discovery of modules
ROUTER ready to provide particular
Queue

Heartbeat
service
ROUTER I Asynchronous
communication between
DEALER DEALER
requester (client), provider
Heartbeat Heartbeat (server) and executor
Worker Worker
(worker)
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0

Communication Patterns for Embedded System Design

Johan Philips 34
March 25, 2015

Espresso

Publisher Publisher I Plugin a proxy between

PUB-SUB
PUB PUB

connect connect connect

bind
I Log all or some
XSUB

Proxy
communication
XPUB (configurable)
bind
I Limit data transfer
connect connect

SUB SUB I Convert data formats

Subscriber Subscriber
between peers
images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0

Communication Patterns for Embedded System Design

Johan Philips 35
March 25, 2015

SHERPA relevant patterns

I Node coordination (Synchronised PUB-SUB & REQ-REP)

I initialisation between base stations and robots assigned to a
mission
I Paranoid Pirate (DEALER-ROUTER-ROUTER-DEALER)
I discovery of new UAVs by base station
I detecting out-of-range issues in large missions
I reliable & robust communication with heart beating
I Espresso and/or Last value caching
I local and/or remote data logging
I allows distributed world model sharing
I enables mission tracking

Communication Patterns for Embedded System Design

Johan Philips 36
March 25, 2015
 Pick-n-Pack relevant patterns

I Node coordination (Synchronised PUB-SUB & REQ-REP)

I initialisation of Line Controller and PnP Modules
I Paranoid Pirate (DEALER-ROUTER-ROUTER-DEALER)
I discovery of new Modules by Line Controller
I reliable & robust communication with heart beating
I failure detection
I Espresso (PUB-XSUB, XPUB-SUB & PAIR-PAIR)
I data logging in Modules
I enables traceability

Communication Patterns for Embedded System Design

Johan Philips 37
March 25, 2015

Real world (Embedded Systems) use cases

I Discovery - how do we detect peers in a network?
I Presence - how do we track dynamic nodes?
I Connectivity - how do we connect nodes?
I Point-to-point messaging - how do we send messages?
I Group messaging - how do we multicast messages?
I Testing and simulation - how do we simulate large
networks and test performance?
I Distributed logging - how do we log data in a cloud of
nodes?
I Content distribution - how do we send content around?
→ Centralise everything or create a distributed solution?
adopted from chapter 8 of the ZeroMQ guide

Communication Patterns for Embedded System Design

Johan Philips 38
March 25, 2015

Further Reading

Communication Patterns for Embedded System Design

Johan Philips 39
March 25, 2015
 RFC and STD
RFC → http://ietf.org/rfc
I Request For Comments by Internet Engineering Task Force

(IETF) and Internet Society

I Describes novel concepts submitted for peer review

I Some RFCs become standards

STD
I Special case of RFC or set of RFCs which has been turned

into an Internet Standard

I http://tools.ietf.org/html/rfc5000 contains (long) list of

STDs
All information is publicly available!

Communication Patterns for Embedded System Design

Johan Philips 40
March 25, 2015

IP

Internet Protocol
I Hourglass architecture of the Internet: IP works over many
network types with many (application) protocols on top
I Used by most computer networks today
I IPv4 (32-bit address) vs IPv6 (128-bit address) → allows
3.4x1038 addresses
I Does not provide much service → encapsulate another
protocol within IP

Communication Patterns for Embedded System Design

Johan Philips 41
March 25, 2015

UDP & TCP

User Datagram Protocol

I packet-switching
I adds multiplexing on top of IP
I unreliable: packets can be dropped, reordered, duplicated
Transmission Control Protocol
I provides illusion of reliable pipe between two processes
I handles congestion and flow control
I requires connection and handshaking

Communication Patterns for Embedded System Design

Johan Philips 42
March 25, 2015
 HTTP

HyperText Transfer Protocol

I Application layer protocol typically on top of TCP/IP
I Request / Reply communication in Client / Server model
I Used in the World Wide Web but not exclusively
I Many embedded devices have server running that supports
HTTP!

Communication Patterns for Embedded System Design

Johan Philips 43
March 25, 2015

ROUTER-DEALER extended example

Client Broker
statebe

connect bind

I Local request-reply flow between

broker and its clients and workers.
I Cloud request-reply flow between
broker and its peer brokers.
I State flow between broker and its
request request state
peer brokers.
connect connect

Worker Broker
statefe

images copyright (c) 2014 iMatix Corporation and licensed under cc-by-sa 3.0

Communication Patterns for Embedded System Design

Johan Philips 44
March 25, 2015

Why real world application need distributed

solutions?

I Dawn of Internet of Things (IoT) → exponential increase of

devices over time
I Using wires is so 20th century...
I Control shifts away from the center: star topology is being
replaced by the cloud of clouds (intercloud).
I Networks are increasingly collaborative, less controlled: BYOD
I The cost of connecting a new node must fall proportionally →
reduce configuration

Communication Patterns for Embedded System Design

Johan Philips 45
March 25, 2015
 ZRE and ZOCP
ZeroMQ Realtime Exchange Protocol (ZRE) -
http://rfc.zeromq.org/spec:36/ZRE
I Auto discovery of group of peers in a network

I Uses ZMTP, a transport layer protocol for exchanging

messages
I Goals: robust on poor networks, work without centralised

service
Z25 Orchestration Control Protocol (ZOCP) -
https://github.com/z25/pyZOCP
I Goal: control multiple computers in live performances → low

latency, reliability
I Zero configuration, open standards, KISS

I Declare capability models to allow data exchange

Communication Patterns for Embedded System Design

Johan Philips 46
March 25, 2015

Concluding remarks

I Communication is everywhere and at many levels!

I Existing standards in computer networks offer many
solutions at lower OSI layers
I Communication patterns are essential to model and design
applications
I Embedded systems will become (if not already) omnipresent
in our daily life
I Distributed solutions are required to tackle communication
challenges in system of systems

Communication Patterns for Embedded System Design

Johan Philips 47
March 25, 2015

Concluding remarks (cont’d)

I ZeroMQ offers many mechanisms (socket pairs) & policies

(patterns) and has a large community
I Existing & tested patterns can be adopted to robotics use
cases:
I for initialising multiple cooperative robots
I for logging & tracing data
I for setting up QoS monitoring for communication
I for detecting failures (in the network or of robots)
I Sockets promote flexibility by shielding off legacy and
facilitating integration
I Advanced protocols such a ZRE and ZOCP offer solutions to
many real world communication challenges.

Communication Patterns for Embedded System Design

Johan Philips 48
March 25, 2015