SpaceWire Users Guide
SpaceWire Users Guide
Steve Parkes
3
4
Copyright © 2012 STAR-Dundee Limited
All rights reserved. No part of this book may be reproduced in any form
without permission.
ISBN: 978-0-9573408-0-0
5
6
Table of Contents
GLOSSARY 10
1 INTRODUCTION 12
2 THE SPACEWIRE DATA-HANDLING NETWORK 13
2.1 The Rationale for and Brief History of SpaceWire 13
2.2 An Example SpaceWire Application 15
2.3 How SpaceWire Works 17
2.3.1 SpaceWire Links 17
2.3.2 SpaceWire Packets 18
2.4 SpaceWire Architectures 22
2.4.1 Point to Point Links 22
2.4.2 Fault Tolerant Links 24
2.4.3 Router Based Architecture 26
2.4.4 Instrument Data Concentrator 31
2.4.5 Bridge to Low Data Rate Sensor Bus 32
2.5 Example SpaceWire Mission Architectures 33
2.5.1 Missions Using SpaceWire 33
2.5.2 ESA ExoMars 36
2.5.3 NASA Lunar Reconnaissance Orbiter 38
2.5.4 BepiColombo 40
2.5.5 ASNARO 43
3 SPACEWIRE LINKS 45
3.1 Physical Level 46
3.1.1 Cables 46
3.1.2 Connectors 47
3.1.3 Cable Assemblies 47
3.1.4 Printed Circuit Board Tracks 49
3.2 Signal Level 49
3.2.1 Signal Level and Noise Margins 49
3.2.2 Data encoding 52
3.3 Character Level 53
3.3.1 SpaceWire Characters 53
3.3.2 Parity Coverage 55
7
3.3.3 Character Priority 56
3.3.4 Character Functions 56
3.3.5 Character Synchronisation 57
3.4 Exchange Level 58
3.4.1 SpaceWire Link Interface 58
3.4.2 Link Initialisation 60
3.4.3 Link Flow Control 65
3.4.4 Link Error Handling 67
3.4.5 Auto-Start 70
3.5 Packet Level 71
3.6 Link error Recovery 73
4 SPACEWIRE NETWORKS 77
4.1 SpaceWire Nodes 77
4.2 SpaceWire Routing Switch 77
4.3 Routing Tables 78
4.4 Group Adaptive Routing 81
4.5 Wormhole Routing 82
4.6 Header Deletion 84
4.7 Time-code Broadcast 87
4.8 Router Configuration 88
4.9 Packet Distribution 88
4.10 Example SpaceWire Router 88
4.10.1 SpW-10X Architecture 88
4.10.2 Watchdog Timers 91
4.10.3 Routing to a Not-Connected Port 91
4.10.4 Routing to a Non-Existent Port 91
4.10.5 Routing to a Busy Port 91
4.10.6 Start On Request, Disable On Silence 92
4.10.7 Tristate 95
4.10.8 Disable Transmit Clocks 95
4.10.9 Priority Packet Delivery 95
5 TIME-CODES 97
5.1 Time-code Structure 97
5.2 Time-code Interface 97
8
5.3 Time-counter 98
5.4 Time Master 99
5.5 Time-codes across a Link 99
5.6 Router Action on Receiving a Time-code 100
5.7 Time-code Distribution across a Network 100
5.8 Lost Time-Codes 103
5.9 Time-code Latency 105
5.10 Time-code Applications 106
5.10.1 Synchronisation 106
5.10.2 Time Distribution 106
5.10.3 Event Signalling Across A Point-To-Point Link 106
5.10.4 Multiple Time-codes 107
5.10.5 Interrupt scheme 107
6 SPACEWIRE PROTOCOLS 108
6.1 Protocol Identifier 108
6.2 Remote Memory Access Protocol 108
6.3 CCSDS Packet Transfer Protocol 109
REFERENCES 110
9
GLOSSARY
10
Router – A packet switch that forwards packets towards their
destination, selecting a link to forward the packet
through based on the destination address of the
packet
11
1 Introduction
The SpaceWire User’s Guide aims to introduce the reader to SpaceWire.
Section 3 goes into much more detail about SpaceWire links. It describes
each level of the SpaceWire standard in detail, explaining the way each
level operates and the reasons SpaceWire is designed the way it is.
12
2 The SpaceWire Data-Handling Network
SpaceWire is a data-handling network for use on-board spacecraft, which
connects together instruments, mass-memory, processors, downlink
telemetry, and other on-board sub-systems [1] [2] [3]. SpaceWire is simple
to implement and has some specific characteristics that help it support data-
handling applications in space: high-speed, low-power, simplicity, relatively
low implementation cost, and architectural flexibility making it ideal for many
space missions. SpaceWire provides high-speed (2 Mbits/s to 200 Mbits/s),
bi-directional, full-duplex data-links, which connect together SpaceWire
enabled equipment. Data-handling networks can be built to suit particular
applications using point-to-point data-links and routing switches.
Since the SpaceWire standard was published in January 2003, it has been
adopted by ESA, NASA, JAXA and RosCosmos for many missions and is
being widely used on scientific, Earth observation, commercial and other
spacecraft. High-profile missions using SpaceWire include: Gaia, ExoMars
rover, BepiColombo, James Webb Space Telescope, GOES-R, Lunar
Reconnaissance Orbiter and Astro-H.
Back in 1992, when work on what became SpaceWire started, there was
also substantial interest in high performance digital signal processing
systems which were beyond the capability of the single-chip devices
13
available at that time. The use of parallel processing was investigated and
this required some form of network to interconnect the individual processing
elements [4]. The Inmos Transputer [5], a microprocessor designed for
parallel processing was studied, and the serial communication links being
developed for the T9000 Transputer [6] were identified as being an
attractive solution for spacecraft on-board networking. This serial link
technology was subsequently published as IEEE 1355-1995 [7].
Several radiation tolerant devices were developed using the IEEE 1355-
1995 standard and it was used on some space missions. However, there
were many problems with this standard, which had to be resolved if this
technology was to continue to be used for ESA spacecraft. University of
Dundee received a contract from ESA [8] to examine and solve these
problems which eventually resulted in the SpaceWire standard.
14
2.2 An Example SpaceWire Application
SpaceWire is able to support many different payload processing
architectures using point-to-point links and SpaceWire routing switches. The
data-handling architecture can be constructed to suit the requirements of a
specific mission, rather than having to force the application onto a restricted
bus or network with restricted topology.
SpW
1 2 3 4
SpaceWire Router 1
5 6 7 8 9
SpW SpW SpW SpW
Data
Control
Mass Memory Module SpW Processor /
Processor
Compressor
SpW Payload
Platform
Downlink Telemetry
15
insufficient to handle the data-rate from this instrument then two or more
links may be used in parallel.
This Mass Memory Module can receive data from any of the instruments
either directly, as is the case for Instrument 1, or indirectly via Router 1.
Data stored in the Mass Memory Module can be sent to the Telemetry
Formatter/Encryption Module for sending to Earth, or it may first be sent to a
Data Processing or Data Compression Unit. This unit may return the
processed/compressed data to the Mass Memory Module or send it straight
to the Telemetry Module via Router 1.
16
With several instruments and the Data Processor/Compressor sending data
to the Mass Memory Module via Router 1, a single link from that Router to
the Mass Memory Module may be insufficient to handle all the data, so a
second link has been added to provide more bandwidth. In a SpaceWire
network links can be added to provide additional bandwidth or to add fault
tolerance to the system. In Figure 1 no redundancy has been included for
clarity. In a spaceflight application, an additional pair of routers would be
included with duplicate links to the modules to provide redundancy. It is
straightforward to support traditional cross-strapped, redundant modules
using SpaceWire.
17
Character synchronisation is performed once only, when the link is started.
If character synchronisation is ever lost it will be detected as a parity error
and the link restarted to recover character synchronisation.
SpaceWire provides a simple mechanism for starting a link, keeping the link
running, sending data over the link, ensuring that data is not sent if the
receiver at the other end of the link is not ready for it, and for recovering
from any errors on the link. All this is handled by the link state-machine in
the SpaceWire interface and is transparent to the user application.
The "Destination Address" is the first part of the packet to be sent and is a
list of data characters that represents either the identity of the destination
node or the path that the packet has to take through a SpaceWire network
to reach to the destination node. In the case of a point-to-point link directly
between two nodes (no routers in between) the destination address is not
necessary.
18
The “End of Packet” (EOP) is used to indicate the end of a packet. The data
character following an EOP is the start of the next packet. There is no limit
on the size of a SpaceWire packet.
As can be seen, the packet format for SpaceWire is very simple. It is,
however, also very powerful, allowing SpaceWire to be used to carry a
range of user protocols, with minimal overhead.
Each link interface may be considered as comprising an input port (the link
interface receiver) and an output port (the link interface transmitter). A
SpaceWire router transfers packets from the input port of the switch where
the packet arrives, to a particular output port determined by the packet
destination address. A router uses the leading data character of a packet
(one of the destination address characters) to determine the output port of
the router to which the packet is to be routed. If there are two input ports
waiting to use a particular output port when the previous packet has finished
being sent, an arbitration mechanism decides which input port is to be
served.
19
2.3.2.3 Packet Addressing
20
can have a maximum of 31 ports along with an internal configuration port,
each data character forming a path address is in the range 0 to 31.
Destination
<Cargo><EOP>
3 2
<3><Cargo><EOP>
2 SpaceWire 4 1 SpaceWire 3
Router 3 Router 2
1
4
<1,3><Cargo><EOP>
2
3
1 SpaceWire
Router 1 4
<2,1,3><Cargo><EOP>
Source
21
of logical addressing is illustrated in Figure 4. Logical addressing uses just a
single data character to identify the destination which is in the range 32 to
255 so that it does not get confused with path addresses.
Destination
“Dundee”
<44><Cargo><EOP>
44 (Dundee) exit 3
42 (Kirriemuir) exit 2
3 2
44 (Dundee) exit 1
53 (Aberdeen) exit 2
<44><Cargo><EOP>
2 SpaceWire 4 1 SpaceWire 3
Router 3 Router 2
1 4
<44><Cargo><EOP>
2
44 (Dundee) exit 2
65 (Edinburgh) exit 3
3
1 SpaceWire
Router 1 4
<44><Cargo><EOP>
Source
22
SpaceWire
Instrument Memory
The instrument can send SpaceWire packets containing the instrument data
directly to the memory over the SpaceWire link. The data can be packaged
into SpaceWire packets with a size appropriate to the application. For
example if the instrument is some form of push broom imager it may be
appropriate to send one line of the image at a time e.g. 34 KByte packets. If
the instrument is a camera a complete image could be transferred in one
packet e.g. 2 MBytes.
The instrument can simply send data to the memory once it has collected
the data or the memory could send a command to the instrument to ask for
the next set of data. SpaceWire includes low-level flow control so that if the
memory is not ready the instrument is unable to send data until the memory
becomes ready. This means that if the memory is not able to accept data
from the camera as soon as it is ready, the camera will have to buffer the
data.
Simplicity
23
This latter point serves to highlight another capability of SpaceWire. A
SpaceWire interface can initialise very rapidly (20 µs). If an instrument
provides data in bursts or occasionally on demand, then it is possible to turn
off the link when it is not being used. If one end of the link is set to auto-start
mode it will start up as soon as some traffic appears on the link. So, for
example, if the instrument is to send data to the memory occasionally when
it has detected some event, the SpaceWire interface on the memory may be
put into auto-start mode. The SpaceWire interface in the memory will stop
and wait listening for any traffic on its input. The instrument can then switch
off its SpaceWire interface. When an event occurs the SpaceWire interface
in the instrument is enabled and started, the initialisation traffic on the link is
detected by the SpaceWire interface in the memory unit causing its link to
start and a connection to be made. The instrument can then transfer its
data. It takes just 20 µs to achieve this connection.
The type of application where the single point to point link is used is for the
direct connection of an instrument to memory where no fault tolerance is
required i.e. it is acceptable that if the SpaceWire link fails the instrument is
lost.
The avoidance of possible single point failures is important for most space
missions especially for mission critical services. SpaceWire provides a
simple means of adding fault tolerance into a system where it is required.
SpaceWire is fairly robust due to the good EMC properties of LVDS and the
cable screening used. Rarely are errors seen on a link unless they are
injected. If a transient error does occur then the SpaceWire link immediately
disconnects itself electrically and goes through the re-initialisation process.
In 20 µs the link is up and running again. The packet that was in the process
of being transferred is truncated and terminated by a special Error End of
Packet (EEP) character to indicate that it was terminated prematurely. The
next packet to be sent will be delivered successfully provided that the fault
24
was temporary. If the packet that was terminated by the fault was important
then it is up to the user application to detect the fact that it was not delivered
properly and to resend the information. Protocols for providing this type of
service which run over SpaceWire have been and are being developed by
the SpaceWire working group.
If the fault on a SpaceWire link is permanent, for example the wires may
have become disconnected or a SpaceWire interface may have stopped
working, then recovery requires a second, redundant SpaceWire link. This is
illustrated in Figure 6.
Prime
Instrument Memory
Redundant
If the prime SpaceWire link stops working then the redundant link has to be
started and data sent over this link. Simple logic in the instrument can
provide this functionality.
Prime
Instrument Memory
Prime Prime
Redundant
SpaceWire
Redundant
Instrument Memory
Redundant Redundant Redundant
Figure 7 Cross-Strapping
25
one prime and one redundant. Similarly there are two memory units. During
normal operation the redundant instrument and memory units are switched
off. The prime instrument sends data to the prime memory. If the prime
instrument fails then it is switched off and the redundant instrument is
switched on. The prime memory then receives instrument data via its
redundant SpaceWire interface. This classical cross-strapping is readily
supported by SpaceWire – additional links are simply put where redundancy
is required.
Simplicity
Fault tolerant
It is important that a failure on the prime link does not propagate and cause
a failure on the redundant link.
26
Instrument
Memory
1
Router
Instrument
Processor
2
Versatile architecture
27
The possible blocking of a router can occur if, for example, the two
instruments both decide to send data to the memory unit at the same time.
Since the router only has one link to the memory unit, only one instrument
can send a packet down this link at a time. If the packet from instrument 1
gets there first it will be sent, but the packet from instrument 2 will be
blocked until the packet from instrument 1 has been sent. If the packet from
instrument 1 is large then instrument 2 may have to wait for a long time.
This is illustrated in Figure 9.
Instrument 1 2
Memory
1
Router
Instrument 3 4
Processor
2
There are several ways to avoid this situation. If the required total data rate
from the two instruments is larger than can be provided by a single
SpaceWire link then a second link is required between the router and
memory, possibly using group adaptive routing, which will provide graceful
degradation in the event of a failure of one link. If the total data rate from the
28
two instruments is less than the data rate of a single SpaceWire link then
the router can act as a concentrator permitting a packet from say instrument
1 first, followed by a packet from instrument 2. In this case the amount of
data buffering in the instruments will depend upon the size of the packets
being sent. Hence, it makes sense to use short packets rather than long
ones to reduce the amount of buffer memory required. Another alternative is
for the memory (or processor) to control when an instrument is allowed to
send data. For example, the memory could request data from instrument 1
and once this has been received it could request data from instrument 2.
There are many possibilities: SpaceWire is flexible enough to support the
requirements of different types of mission.
It should be noted that since the router uses wormhole routing and does not
support packet buffering in the router, the speed of all the SpaceWire links
attached to a router should normally all be set to the same rate.
Figure 10 shows a prime and redundant pair of routers which have been
included in prime and redundant data handling units together with the mass
memory and processing units. This provides router redundancy while
reducing the lengths (and hence mass) of the SpaceWire links.
29
Prime
Instrument 1 Memory
Prime Router
Processor
Instrument 1
Redundant
Instrument 2
Redundant
Prime
Memory
Instrument 2 Router
Redundant Processor
30
2.4.4 Instrument Data Concentrator
It has already been mentioned that a SpaceWire router can act as a data
concentrator. Another example of this is shown in Figure 11.
Instrument 1 4 Memory
1
Router
High Rate
2 3 Processor
Instrument Prime
2 1
Instrument Memory
3 6 Router
Instrument Router Processor
4 5
Redundant
Instrument 4
5
Note: numbers by router ports are example port numbers
31
The disadvantages of the particular architecture shown in Figure 11 are:
Some sensors, like thermocouples, are very low data rate and a low data
rate sensor bus (e.g. CAN bus) may be more suitable for gathering data
from them. The data gathered may still be sent to the data-handling system
via a SpaceWire link if a bridge between the low data rate sensor bus and
SpaceWire is used. This is illustrated in Figure 12.
Instrument
1 Memory
High Rate Router
Processor
Instrument Prime
2
Instrument Memory
Router
3 Router
Instrument Processor
4
Redundant
Low Data Rate Bus
RTC
Sensor Sensor Sensor
A B C
32
The Remote Terminal Computer (RTC) provides a bridge between the
SpaceWire network and the low data rate sensor bus. The RTC gathers
data from the low data rate sensors, puts it in a SpaceWire packet and
sends it to the data handling system. Commands for the low data rate
sensors (or actuators) can be passed from the data-handling processor to
the RTC for distribution to the low data rate sensors.
Multiple low data rate sensors attached to low speed bus - sensor
data packed and sent to data-handling unit over SpaceWire
The applications for this type of architecture are data handling systems that
include several low data rate sensors or that require to support legacy
devices.
Other units like the downlink telemetry and uplink telecommand system can
be readily included in a SpaceWire system. SpaceWire architectures can be
adapted to the mission requirements including links to serve one or more
instruments, routers to provide architectural flexibility, and interfaces to
electronic ground support equipment.
33
GAIA a very high resolution star-mapper [16];
LCROSS the mission that was deliberately crashed into the south
pole of the moon and discovered ice there [25];
34
Magnetospheric Multi-Scale mission which is a multi-satellite
mission that will explore the Earth’s magnetosphere [27];
35
RosCosmos the Space Agency of the Russian Federation have recently
approved SpaceWire for use on their spacecraft, regarding SpaceWire as a
key technology for their future space missions [36].
36
The SpaceWire data-handling architecture used on ExoMars is illustrated in
Figure 14. It follows closely the example architecture of Figure 1.
MAST Electronics
SpW SpW SpW SpW
SpW
SpaceWire Router
MASS IMAGE
PROCESSOR
MEMORY PROCESSOR
ExoMars makes extensive use of the RMAP protocol [14] for passing data
from cameras, to mass memory and to/from the processor and image
processing chip.
37
2.5.3 NASA Lunar Reconnaissance Orbiter
SpW
Command and Data
Mass Memory SpaceWire Router
Handling Computer
SpW SpW
Ka-COMMS S-COMMS
38
SpaceWire is used to connect the LRO Cameras (Narrow angle Cameras,
NAC1 and NAC2, and Wide Angle Camera, WAC), and Mini-RF radar
instrument, to the Command and Data-handling (C&DH) system. SpaceWire
is also used to pass data from the Command and Data-handling system to
the Ka and S-Band communications systems. Information from the Lyman-
Alpha Mapping Project (LAMP) instrument is passed into an I/O board in the
C&DH system (HK/IO) and then sent over SpaceWire to the C&DH
computer/mass memory. The C&DH computer includes a 4-port SpaceWire
router for handling the SpaceWire communications.
An example image from LRO is given in Figure 17. It shows the Apollo 12
landing site. The Apollo 12 (Intrepid) descent stage is clearly visible in the
image as is the earlier Surveyor 3 spacecraft. The black lines radiating out
from Intrepid and pointed to by arrows in other parts of this image are the
footsteps of the Apollo 12 astronauts. This image travelled over SpaceWire
twice: from the NAC to the mass memory and from the mass memory to the
downlink communication system.
39
2.5.4 BepiColombo
40
The SpaceWire data-handling architecture used on MPO is shown in Figure
19.
Payload 1
Payload 2
Payload 3
Payload 4
Payload 5
Payload 6
Payload 7
Payload 8
Payload 9
SpW SpW SpW
SpW
SpW SpaceWire Router SpaceWire Router SpaceWire Router
On-Board Computer
SpaceWire Router
OBC Interface
SpW
EGSE Interface SpW SpaceWire Router
SpW SpW
X-Band Ka-Band
Telemetry Telemetry
Format Format
Generator Generator
MPO uses SpaceWire for the data-handling from its nine science payloads.
These payload instruments are connected to the on-board mass memory
unit via point-to-point links and three SpaceWire routers that are integrated
within the mass memory unit. These routers are also connected to the on-
board computer and to each other in a daisy chain. This enables the on-
board computer to access each of the routers and payload instruments for
configuration and control purposes. Another router is used to connect the
mass memory unit to the X-band and Ka-band downlink telemetry format
generators, which format the payload data before sending it back to Earth.
This SpaceWire router is also attached to the on-board computer to allow
the router and downlink telemetry format generators to be configured and
controlled. SpaceWire links are also used to connect the on-board computer
to the spacecraft platform and electronic ground support equipment (EGSE).
41
The SpaceWire routing devices used are the ESA SpW-10X routers (Atmel
AT7910E).
Figure 19 has been simplified for clarity. For redundancy purposes the data-
handling units are duplicated with cross-strapping between units to enhance
overall reliability and avoid possible single-point failures.
42
Mission Data SpW MEA1
Processor MEA2
Data MIA
Processing
MSA
Unit 1
Spacecraft
System HEP-electron
HEP-ion
Data
ENA
Processing
Unit 2
MDM
MSASI
MEA Mercury Electron Analyser
MIA Mercury Ion Analyser
MSA Mercury Mass Spectrometer Analyser
HEP High Energy Particle Instrument MGF-out
ENA Energetic Neutrals Analyser
MDM Mercury Dust Monitor MGF-in
MSASI Mercury Sodium Atmosphere Spectral Investigation
MGF Magnetic Field Investigation PWI
PWI Plasma Wave investigation
MAST/WPA Mast and Wire Probe Antenna Control MAST/WPT-E
2.5.5 ASNARO
43
Flyer) with funding from the Japanese Ministry of Economy, Trade and
Industry. The overall objective of the ASNARO project is to develop a next-
generation high-performance mini-satellite bus system based on open
architecture techniques and manufacturing methodologies to drastically
reduce the cost and the development period with adoption of up-to-date
electronics technologies [34].
S-Band
Data-Handling
Telemetry-
Computer
Telecommand
SpW SpW
SpaceWire Router
SpW SpW
ASNARO uses SpaceWire for platform and attitude and orbit control
(AOCS) as well as for payload data-handling. The payload control computer
is attached to the payload using SpaceWire. The platform electronics
including the data-handling computer, payload computer, attitude and orbit
control computer, heater control, GPS receiver, power control and S-Band
telecommand-telemetry unit, are interconnected using a SpaceWire router.
A separate SpaceWire network connects the AOCS sensors and actuators
to the AOCS computer.
44
3 SpaceWire Links
In this section the operation of a SpaceWire link is considered in more
detail. SpaceWire is based on two previous standards IEEE 1355-1995 [2]
and ANSI/TIA/EIA-644 [3].
45
3.1 Physical Level
The physical level of the SpaceWire standard covers cables, connectors,
cable assemblies and printed circuit board tracks. SpaceWire was
developed to meet the EMC specifications of typical spacecraft.
3.1.1 Cables
The SpaceWire cable comprises four twisted pair wires with a separate
shield around each twisted pair and an overall shield. To achieve a high
data signalling rate with SpaceWire over distances up to 10 m a cable with
the following characteristics is used:
Low cross-talk.
Conductor 28 AWG
(7 x 36 AWG)
Insulating layer
Filler
Twisted pair
(100 ohm differential impedance)
46
One of the drawbacks of SpaceWire is the mass of the cable which is
around 87 g/m. A new form of SpaceWire cable is currently being developed
by ESA which is of significantly lower mass. This cable only uses one level
of shielding.
SpaceWire is able to operate at 200 Mbits/s over 10m cable. For longer
distances it is possible to increase the wire gauge of the conductors to
reduce cable attenuation.
3.1.2 Connectors
1 2 3 4 5
6 7 8 9
47
Din+ Dout+
1 9
Din- Dout-
6 5
Sin+ Sout+
2 8
Sin- Sout-
7 4
GROUND GROUND
3 3
Sout+ Sin+
8 2
Sout - Sin-
4 7
Dout+ Din+
9 1
Dout- Din-
5 6
This arrangement is far from ideal and in future it is recommended that all
shields are terminated to the backshell and pin 3 at both ends of the
connector.
48
3.1.4 Printed Circuit Board Tracks
SpaceWire can also be run over printed circuit boards (PCBs) including
backplanes. Because of the high-speed of SpaceWire signals which have a
bandwidth of over 1 GHz (at 200 Mbps the signal frequency is 100 MHz and
that of the signal edges around ten times this, giving 1 GHz) care has to be
taken with the PCB layout. The following guidelines should be adhered to:
No right-angle turns
Data and strobe signal pairs to be of the same length (< 5 mm)
49
Voltage Across 100 ohm Termination Resistor Receiver Input Thresholds
0 1 0
Vin-
+250 to +400 mV
typical +100 mV typical
1.2V typical O V (differential) Transition
(O V differential) Region
-250 to -400 mV -100 mV typical
typical
Vin+ [Vin+ - Vin-]
A typical LVDS driver and receiver are shown in Figure 28, connected by a
media (cable or PCB traces) of 100 ohm differential impedance. The LVDS
driver uses current mode logic. A constant current source of around 3.5 mA
provides the current that flows out of the driver, along the transmission
medium, through the 100 ohm termination resistance and back to the driver
via the transmission medium. Two pairs of transistor switches in the driver
control the direction of the current flow through the termination resistor.
When the driver transistors marked “+” in Figure 28 are turned on and those
marked “-” are turned off, current flows as indicated by the arrows on the
diagram creating a positive voltage across the termination resistor. When
the two driver transistors, marked “-”, are turned on and those marked “+”
are turned off, current flows in the opposite direction producing a negative
voltage across the termination resistor. LVDS receivers are specified to
have high input impedance so that most of the current flows through the
termination resistor to generate around 350 mV with the nominal 3.5 mA
current source.
Vcc
~3.5mA
- +
50
LVDS has several features that make it very attractive for data signalling [6]:
Near constant total drive current (+3.5 mA for logic 1 and -3.5 mA
for logic 0) which decreases switching noise on power supplies.
The signal levels and noise margins for SpaceWire are derived from
ANSI/TIA/EIA-644 [3] which defines the driver output characteristics and the
receiver input characteristics. The eye diagram for SpaceWire signals sent
over 10 m of cable at 200 Mbits/s baud rate is shown in Figure 29.
51
Figure 29 SpaceWire Eye Diagram (200 MHz, 10 m cable)
Data 0 1 0 0 1 1 0 1 1 0
D
S
CLK
52
A SpaceWire link comprises two pairs of differential signals, one pair
transmitting the D and S signals in one direction and the other pair
transmitting D and S in the opposite direction. That is a total of eight wires
for each bi-directional link.
A bit trace from a SpaceWire link analyser is shown in Figure 31. The yellow
lines are the data and strobe signals (DIN and SIN). At each transition of the
strobe line a dotted white line has been drawn vertically from the strobe
signal over the data signal. The value of the data line between these vertical
lines or the transitions of the data line is the decoded data value, shown as
white 0 or 1 on the trace. Bit synchronisation on SpaceWire is therefore
quite straightforward, but determining the boundaries between SpaceWire
characters is more difficult in a bit trace.
SpaceWire has two types of characters: data and control characters which
are illustrated in
Figure 32.
53
Data characters which hold an eight-bit data value transmitted
least-significant bit first. Each data character contains a parity-bit, a
data-control flag and the eight-bits of data. The parity-bit covers
the previous eight-bits of data or two-bits of control code, the
current parity-bit and the current data-control flag. It is set to
produce odd parity so that the total number of 1’s in the field
covered is an odd number. The data-control flag is set to zero to
indicate that the current character is a data character.
In addition to the data and control characters there are two control codes:
NULL and time-codes.
NULL is formed from ESC followed by the flow control token (FCT).
NULL is transmitted, whenever a link is not sending data or control
tokens, to keep the link active and to support link disconnect
detection.
54
Data Characters
P 0 X X X X X X X X
0 1 2 3 4 5 6 7
LSB MSB
Data-Control Flag
Parity Bit
Control Characters
P 1 0 0
FCT Flow Control Token
P 1 0 1
EOP Normal End of Packet
P 1 1 0
EEP Error End of Packet
P 1 1 1
ESC Escape
Control Codes
P 1 1 1 0 1 0 0 NULL
P 1 1 1 1 0 T T T T T T T T Time-code
0 1 2 3 4 5 6 7
LSB MSB
P 0 X X X X X X X X P 1 0 1 P 1 0 0
Parity Parity
Coverage Coverage
55
3.3.3 Character Priority
FCTs
N-Chars
Time-codes are highest priority because they are used to transfer time or
synchronisation information and have to be delivered with low jitter. FCTs
are higher priority than N-Chars, so that a large amount of data being sent in
one direction does not stop FCTs being sent, thus causing the other end of
the link to stop sending data because it has run out of flow control. NULLs
are the lowest priority as they are only sent to keep the link running and
synchronised when there is nothing else to send.
Characters in SpaceWire are used for three different functions: link control,
sending packets and sending time-codes.
The characters and control codes used for link control are the NULL and
FCT, which are known as L-Chars or Link characters. They are used in the
Exchange level and not passed up to any higher level.
The characters and control codes used for sending packets are the data
characters and end of packet markers (EOP and EEP), which are known as
N-Chars or Normal characters. These characters are passed up to the
packet layer.
The time-codes are used for sending time and synchronization information
and are passed to the time-code handler of a node or router.
56
packet. A received character should have its parity checked before it is
acted upon.
One end of the link (bottom trace, End B) starts to send Nulls, a specific
sequence of 8 data bits. The other end detects this sequence, synchronises
its receiver and starts to send Nulls back (top trace, End A). When End B
receives these Nulls it synchronises its receiver. Further handshaking is
done between the two ends of the link which is described later in section
3.4. The SpaceWire interfaces initially start operating at a line data
signalling rate of 10 Mbits/s. Once the connection has been made the two
ends of the link can switch to higher speed operation if required. This is
clearly visible in Figure 34.
Before a link sends its first Null both data and strobe lines are set low as
shown in Figure 35.
57
First NULL
0 1 1 1 0 1 0 0
Unfortunately the bit sequence for a Null is not unique in a SpaceWire bit
sequence so it cannot be reliably used for resynchronisation of the signals
without first stopping the SpaceWire link and then restarting and re-
synchronising it.
58
SpaceWire CODEC Interface
SpW SpW
Time-Code Packet Packet Time-Code Packet Level
TX TX RX RX
TX Flow RX
FIFO Control FIFO
Exchange Level
State
Machine
TX RX Character Level
Signal Level
Physical Lvel
The SpaceWire link can send and receive SpaceWire packets and time-
codes once it has been initialised and is running. As described in section
2.3.2 a SpaceWire packet comprises a destination address, cargo and end
of packet marker. To send a SpaceWire packet over a SpaceWire link it is
passed character by character to the transmit FIFO starting with the first
character of the destination address. SpaceWire packets are received into
the RX FIFO and can be read out by the application. To send a time-code, it
is presented to the SpaceWire interface and will be transmitted as soon as
the current character has finished being transmitted. When time-codes are
received they are made available via the time-code interface. Time-codes
need to be validated before they are used, see section 5.
Before a SpaceWire link can send and receive SpaceWire packets and
time-codes, it needs to be initialised. This is done under control of the link
59
state machine. This state machine also manages recovery from any errors
detected on the link, by re-initialising the link.
Link initialisation is necessary to get both ends of the link fully synchronised
and ready to receive data and EOP characters and time-codes. Bit
synchronisation is performed by decoding the data-strobe signals to
produce the bit clock and data stream. Character synchronisation is
performed once during link initialisation. Both ends of the link have to be
character synchronised or the link will automatically reset and attempt to
resynchronise.
Following reset the SpaceWire interface is held in the reset state until it is
instructed to start and attempt to make a connection with the SpaceWire
interface at the other end of the link. A connection is made following a
handshake that ensures both ends of the link are able to send and receive
characters successfully. Each end of the link sends NULLs, waits to receive
a NULL, then sends FCTs and waits to receive an FCT. Since a SpaceWire
interface is not allowed to send FCTs until it has received a NULL, receipt of
one or more NULLs followed by receipt of an FCT means that the other end
of the link has received NULLs successfully and that full connection has
been achieved. This exchange of Nulls and FCTs is known as the Null/FCT
handshake.
60
Reset
ErrorReset
Reset Tx
Link Disabled Reset Rx
After 6.4us
Run ErrorWait
Send FCTs/NChars/Nulls Reset Tx
Enable Rx Enable Rx
After
12.8us
gotFCT
Ready
Connecting
Reset Tx
Send FCTs/Nulls
Enable Rx
Enable Rx
On reset the SpaceWire interface state machine enters the ErrorReset state
where both the transmitter and receiver in the SpaceWire interface are
reset. The state machine remains in this state for 6.4 µs to ensure that the
interface is properly reset, and then moves to the ErrorWait state where it
keeps the transmitter reset but enables the receiver. The reason for this will
become clear later. After waiting 12.8 µs in the ErrorWait state the
SpaceWire interface moves to the Ready state, where it waits for a
command to start the link. When the SpaceWire interface is instructed to
start the link (by the local application logic using the SpaceWire interface) it
moves to the Started state where it begins to send out Nulls. If the other end
(End B) of the link is also sending Nulls then End A will receive the Null
(gotNull) and move to the Connecting state. In the Connecting state the
transmitter is allowed to send Nulls and FCTs. It will send out an initial burst
of FCTs (see flow control section later on) as part of the initialisation
handshake. The other end of the link (End B) will have received Nulls and
with therefore also be in the Connecting state sending out FCTs and Nulls.
The FCTs will be received at End A and the state machine will move to the
61
Run state. In the Run state both ends of the link have made a connection (or
are just about to) and the SpaceWire interface is ready to start transferring
data. The link will now remain in the Run state until one of the SpaceWire
interfaces is disabled by the local application logic asserting the Link
Disabled control bit.
If the other end of the link (End B) is not ready and does not send Nulls and
FCTs when expected, End A will wait in the Started or Connecting state for
12.8 µs and then give up waiting, move back to the ErrorReset state and try
again to make a connection. This is illustrated in Figure 38.
Reset
ErrorReset
Reset Tx
Link Disabled Reset Rx
After 6.4us
Run ErrorWait
Send FCTs/NChars/Nulls Reset Tx
Enable Rx Enable Rx
After
12.8us
gotFCT
After After
12.8 us 12.8 us
Ready
Connecting
Reset Tx
Send FCTs/Nulls
Enable Rx
Enable Rx
To avoid having to set both the Start Link bit in both ends of the link to start
a link, an Auto-Start mode is included in the SpaceWire state-machine. A
SpaceWire interface in the Auto-Start mode will automatically start up when
it receives a bit (gotBit) on its receiver. This functionality is achieved by
62
adding an extra condition on the transition between the Ready and Started
states. This addition is illustrated in Figure 39.
Reset
ErrorReset
Reset Tx
Link Disabled Reset Rx
After 6.4us
Run ErrorWait
Send FCTs/NChars/Nulls Reset Tx
Enable Rx Enable Rx
After
12.8us
gotFCT
After After
12.8 us 12.8 us
Ready
Connecting
Reset Tx
Send FCTs/Nulls
Enable Rx
Enable Rx
gotNull Started
Send Nulls Start Link OR (AutoStart AND GotBit)
Enable Rx
One end of the link (End B) has its AutoStart control bit set and has reset,
moved through ErrorReset and ErrorWait, and is now sitting in the Ready
state. The other end of the link (End A) has also reset and is waiting in the
Ready state. When End A gets the command to Start Link, it moves to the
Started state and begins sending Nulls. End B receives the start of the Null
and gotBit is asserted causing it to also move to the Started state. The
Null/FCT handshake now takes place with both ends moving through
Started and Connecting to Run.
When one end of the link is trying to make a connection, i.e. Start Link is
asserted, and the other end of the link (End B) is not ready to make a
connection for whatever reason, End A will send Nulls for 12.8 µs and then
go silent for 19.2 µs (6.4 µs + 12.8 µs) repeatedly. Observing bursts of Nulls
63
from one end of a link while nothing is being sent in the other direction is a
clear indication that one end is trying to send and the other end is not
responding. This could be because that end of the link does not have its
Auto-Start control bit asserted, or alternatively its Disable bit is asserted.
1. The link is disconnected causing both ends of the link to reset, moving
through the ErrorReset and ErrorWait states to the Ready state.
2. End B of the link has Start Link asserted and so moves to the Started
state and begins sending Nulls, which are received at End A initially
causing GOT-BIT to be asserted and then a series of Nulls to be
received.
3. End A has AutoStart asserted, so as soon as it has received the first bit
(GOT-BIT) it moves to the Started state and sends at least one Null,
which is received at End B.
4. When End A receives a Null, and after it has sent at least one Null in
response, it moves to the Connecting state and is able to send one or
more FCTs.
6. The FCT from End A is received at End B, so End B moves to the Run
state and switches to the required link operating speed (note the
change in character timing).
7. The FCT sent from End B is received at End A, so End A moves to the
Run state and switches to the required link operating speed. The link is
now running in both directions.
64
1
2
3
4
5 6
Note that during initialisation at least one Null must be sent before FCTs are
sent, although this is not clearly stated in the SpaceWire standard.
65
flow control token (FCT). The FCT is received at the other end of the link
(end B) enabling the transmitter at end B to send up to eight more N-Chars.
If there is more room in the receive FIFO, multiple FCTs can be sent, one
for every eight spaces in the receive buffer. Correspondingly, if multiple
FCTs are received then it means that there is a corresponding amount of
space available in the receive FIFO e.g. four FCTs means that there is room
for 32 N-Chars. Each FCT is exchanged in this way for 8 N-Chars. The
operation of flow control is illustrated in Figure 41.
The FCT sent by End B and received by End A (1), permits End A to send
eight more N-Chars (2). The FCT goes in the opposite direction to the data
that it is exchanged for. The SpaceWire link in Figure 41, is sending data in
both directions of the link so FCTs are also being sent in both directions.
66
continuous data transfer across a link. Smaller FIFOs may also be used, in
which case fewer than the maximum of seven FCTs will ever be
outstanding.
There are several types of error that can occur in a SpaceWire link:
Parity errors occurring within a data or control character are detected when
the next character is sent, since the parity bit for a data or control character
is contained in the next character. Once a parity error has been detected,
the link attempts to recover from the error.
67
starting to transmit. These periods of time are sufficient to ensure that the
receivers at both ends of the link are ready to receive characters before
either end starts transmission. The two ends of the link go through the
NULL/FCT handshake (see Figure 40) to re-establish a connection and
ensure proper character synchronisation.
68
Reset RxErr OR
gotFCT OR
RxErr OR ErrorReset gotN-Char OR
CreditError OR Reset Tx gotTime-Code
[Link Disabled] Reset Rx
After 6.4 µs
Run ErrorWait
Send Time-Codes/ Reset Tx
FCTs/N-Chars/NULLs Enable Rx
Enable Rx
RxErr OR
gotFCT OR
gotFCT gotN-Char OR
RxErr OR
gotN-Char OR RxErr OR gotTime-Code After
gotTime-Code OR gotFCT OR 12.8 µs
after 12.8 µs gotN-Char OR
gotTime-Code OR Ready
Connecting after 12.8 µs Reset Tx
Send FCTs/NULLs
Enable Rx
Enable Rx
Started [Link Enabled]
gotNULL
Send NULLs
Enable Rx
The error conditions that can occur are shown in red. Any error that is
detected results in a transition to the ErrorReset which starts the error
recovery cycle.
Disconnect Error detection is enabled following link reset, only after the first
bit has been received.
An RxErr can be detected in any state, except ErrorReset, and will result in
a transition to ErrorReset.
69
In the Connecting state it should not be possible to receive N-Chars or time-
codes as the link is only partially connected, an FCT must be received
before any of these other characters, finishing link initialisation and causing
a transition to the Run state. If any N-Chars or time-codes are received in
the Connecting state, it is an error and the state machine moves to the
ErrorReset state.
Thus
is really
3.4.5 Auto-Start
Both ends of a SpaceWire link can be set to Auto-Start, in which case either
end can be started and cause link initialisation.
Without the Auto-Start facility both ends of the link would have to be started
specifically. One end could be started, sending Nulls for a long time before
the other end is also started. This wastes power.
70
[Link Enabled] = ( NOT [Link Disabled] ) AND ([LinkStart] OR (
[AutoStart] AND gotNULL ))
Where:
If the link is disabled, LinkDisabled asserted, the other flags are ignored and
the link will not attempt to start initialisation, until LinkDisabled is de-
asserted.
The "Destination Address" is the first part of the packet to be sent and is a
list of data characters that represents either the identity of the destination
node or the path that the packet has to take through a SpaceWire network
to reach to the destination node. In the case of a point-to-point link directly
between two nodes (no routers in between) the destination address is not
necessary.
71
The "Cargo" is the data to be transferred from source to destination. Any
number of data bytes can be transferred in the cargo of a SpaceWire
packet.
The "End of Packet" (EOP) is used to indicate the end of a packet. The data
character following an End of Packet is the start of the next packet. There is
no limit on the size of a SpaceWire packet.
In addition to the normal end of packet marker (EOP), there is another end
of packet marker, the Error End of Packet marker (EEP), which indicates
that the packet has been terminated prematurely because of an error that
occurred as the packet traversed a SpaceWire link or network. A packet
terminated by an EEP is illustrated in Figure 45.
72
3.6 Link error Recovery
The handling of link errors by the link state machine is described in section
3.4.4. This section considers what happens to a packet that is travelling
across a link when an error occurs.
1. Detect Error
Figure 46 shows the two directions of the link transferring packets, from an
end user buffer passing data to the SpaceWire transmitter, via the
SpaceWire interface transmit FIFO, across the link, into the SpaceWire
interface receive FIFO at the other end of the link, and on into the end user
buffer taking data from the receiver. In the left to right direction of the link the
head of a packet is in the receive user buffer and the tail of the packet with
the EOP is in the transmit end user buffer. An error has just occurred in this
direction of the link.
2. Disconnect Link
The error detected in the receiver on the right hand side of Figure 46,
causes the transmitter at that end to disconnect, resulting in a disconnect
error occurring at the other end of the link. Both ends of the link are now
disconnected as illustrated in Figure 47.
73
BUFFERS FIFOs LINK FIFOs BUFFERS
2. Disconnect
EOP EOP
TX RX
EOP EOP
RX TX
The information in the receive FIFOs has an EEP appended to it, as soon as
there is room in each FIFO to add one. This terminates the received part of
the packet, so that it can continue on its way through the SpaceWire
network. This is illustrated in Figure 48.
The remainder of the packet which has not yet been transferred across the
link has to be discarded, since one or more N-Chars will have been lost or
corrupted due to the error on the link. The tail end of the packet that has not
yet been sent across the link contains valid data, but its destination is not
clear: for example, it could have been an EOP that was lost when the error
occurred on the link. Since the destination is unknown all that can be done
with the tail end of the packet is to discard it. Data is read by the transmitter
and spilt until the EOP (or EEP) is reached. The character following the
EOP will be the destination address at the start of the next packet. The
result after spilling the packet is shown in Figure 49.
74
BUFFERS FIFOs LINK FIFOs BUFFERS
4. Delete Data in TX until next EOP
EOP EOP EEP
TX RX
EEP EOP
RX TX
5. Reconnect
Now the SpaceWire link has tidied up after the error, the link can be re-
initialised and the connection re-established. This is shown in Figure 50.
With the link restarted the next packet, whose head and destination address
is waiting in the transmit FIFO, can be sent. Normal operation has resumed.
This is shown in Figure 51.
75
If the header byte (i.e. first byte after an EOP or EEP) is corrupted, the
entire packet is lost and the data is not propagated across a network. The
routing switch simply disposes of the packet.
If the error occurs in a EOP (or EEP), two packets are affected: the one
before the EOP where all the data is sent but no EOP is received, and the
following one because the link transmitter “spills” the packet until the next
EOP (or EEP).
If the error occurs in a NULL or FCT inserted in the data stream for a
packet, the packet being sent is discarded from that point on. This is
because it is not known what the character was before it was corrupted.
The time taken for complete recovery from the error on the link will depend
upon how long it takes to spill the tail end of the packet being transferred
when the link error occurred, which depends on the size of that packet. The
tail may have to be pulled out from the packet source, through one or more
routers making up a SpaceWire network as it is being split.
The decision about what to do with the packet that terminates with the EEP
is up to the user application.
76
4 SpaceWire Networks
SpaceWire networks are built from SpaceWire links, nodes and routers. The
links and routers are there to connect together nodes so that they can
exchange information and work together to perform some required function.
A node can have more than one SpaceWire interface and may have more
than one logical address.
77
LINK INTERFACE 1
INPUT OUTPUT
PORT 1 PORT 1
INPUT OUTPUT
PORT 2 PORT 2
INPUT OUTPUT
PORT 3 PORT 3
SWITCH
MATRIX
INPUT OUTPUT
PORT 4 PORT 4
INPUT OUTPUT
PORT 5 PORT 5
INPUT OUTPUT
PORT 6 PORT 6
LINK INTERFACE 6
A SpaceWire router transfers packets from the input port of the switch
where the packet arrives, to a particular output port determined by the
packet destination address. A router uses the leading data character of a
packet (one of the destination identifier characters) to determine the output
port of the router to which the packet is to be routed. If there are two input
ports waiting to use a particular output port when the previous packet has
finished being sent then an arbitration mechanism in the output port decides
which input port is to be served.
78
Table 1 Router Addresses
The physical output ports are the actual SpaceWire ports on the router. For
connecting to equipment local to the router it is not always necessary to use
a SpaceWire link. In this case, a FIFO port can be used instead. The FIFO
port is bi-directional and behaves like a SpaceWire port as far as the router,
but there is no SpaceWire link attached to it. Instead, some user application
connects directly to the FIFO port. The physical output ports of a router
include both the SpaceWire ports and the FIFO ports. The physical output
ports are always numbered from one upwards and normally the SpaceWire
ports are first. So a router with four SpaceWire ports and two FIFO ports
would number the SpaceWire ports 1 to 4 and the FIFO ports 5 and 6. FIFO
ports are also called external or parallel ports, meaning external to the
79
SpaceWire network, and parallel as opposed to the serial SpaceWire link. A
router is allowed to have a maximum of 31 physical ports, hence the
physical output ports are numbered 1 to 31 maximum.
Logical addresses are mapped by the routing table to physical output ports.
It is necessary to be able to distinguish between path and logical addresses
so that the router can handle them appropriately. This is done in a simple
way: the value of the leading address character of a packet determines
whether it is a path or logical address, if it is in the range 0 to 31 it is a path
address, if it is in the range 32 to 255 it is a logical address. A path address
is used directly to determine the output port that the packet is to be
forwarded through. A logical address is used as an index into the routing
table from which the physical port number is determined. The packet is then
forwarded through the specific output port.
Logical address 255 is reserved for future applications and should not be
used.
An example routing table for a router with four SpaceWire ports and the
configuration port is illustrated in Figure 53. A ‘1’ in the table maps an
address to an output port number. The configuration port (port 0) is
accessed only via path addressing with the address 0. Ports 1 to 4 are
accessed using path addresses 1 to 4 respectively. Path addresses beyond
address 4 have no meaning in a 4-port router and give rise to a routing
error. Logical addresses can be used to access any of the four output ports
depending on how the routing table is programmed. For example in Figure
53, logical address 33 has been programmed to port 4. Any packets arriving
with address 33 will be routed to output port 4.
80
Address Port 0 Port 1 Port 2 Port 3 Port 4
Configuration 0 1 0 0 0 0
1 0 1 0 0 0
Path
2 0 0 1 0 0
Addressing
…
32 0 0 1 0 0
33 0 0 0 0 1
Logical
34 0 1 0 0 0
Addressing
…
255 0 0 0 0 0
Figure 54. Logical address 33 has two output ports assigned to it. When a
packet with this logical address is received the router has the choice of
routing the packet out of port 3 or port 4.
81
Address Port 0 Port 1 Port 2 Port 3 Port 4
Configuration 0 1 0 0 0 0
1 0 1 0 0 0
Path
2 0 0 1 0 0
Addressing
…
32 0 1 1 0 0
33 0 0 0 1 1
Logical
34 0 1 1 0 0
Addressing
…
255 0 0 0 0 0
No packet buffering
Rapid switching
82
Wormhole routing suffers from one main problem, that of blocking. If the
output port that the packet is to be forwarded through is not ready or is
currently being used, the packet has to wait until it is ready or the packet
currently flowing through it has finished. Since the tail of a packet can be
spread out through the network, not only is the waiting packet halted, but
that packet blocks any other packet in the network that is waiting to use the
links that it is currently occupying. This is illustrated in Figure 55.
1 5 1 5
NODE 1 NODE 6
ROUTER 1 ROUTER 2
2 4 2 4
NODE 2 NODE 5
3 3
NODE 3 NODE 4
83
There are some strategies that help to avoid blocking a network:
Split large packets up into many smaller ones, e.g. an image could
be sent as a series of image lines;
Make sure that the destination is ready before sending the packet,
which can be done using an end-to-end flow control mechanism;
84
network, but if a packet crosses the boundary of one region into another
region, its leading logical address is deleted, exposing another logical
address character that is used in the next region. This is illustrated in Figure
58.
Node Node
Node Node 84 89
Node
64 A 88
Node 211
43
Router
Router Router 3
Node 2
1 Node
71
213 Region
Node 220
Node Node
222
Node Node 33 207
92 201
Region
Node Node 221
64 B 43
Node
Node Router
88
84 4
Node
39
Node Node
97 216
Figure 56 shows a large network. Above the dashed line all the node logical
addresses are unique, but below the dashed line some of the logical
addresses are repeats of those above the line. Node 71 wants to send one
packet to node 64 above the line (node 64 A) and another packet to node 64
below the line (node 64 B). How can this be done? If the routers above the
line are configured to route to node 64 A, a packet sent from node 71 with
destination address 64 will end up at node 64 A.
To reach node 64 B the network is split into two regions: above the line and
below the line. In each region the node logical addresses must all be
unique. Each region is then given a logical address, which should not be
assigned to any of the nodes. The region above the line is given logical
address 220 and that below is given 221. Now to route from node 71 to
85
node 64 B, the packet is given two cascaded logical addresses, the first of
which is the logical address of the destination region (where the target node
lies) and the second of which is the logical address of the destination node
in that region. So for node 64 B this is 221, 64. The packet sent is then:
The routers in region 220 are all configured to route any packet with logical
address 221 first to router 2 and then on to router 4. Router 2 is
programmed so that when the packet is forwarded from router 2 to router 4,
its leading address character is stripped off, revealing the second address
character:
The routers in region 221 are configured to route packets with logical
address 64 to the local node with that logical address, i.e. to node 64 B.
1. Split the network into regions which are each given a logical
address
3. Nodes in two different regions can have the same logical address
4. When routing a packet from one region to another the router must
strip off the leading destination address character. This has to be
specifically configured in the router.
86
4.7 Time-code Broadcast
As well as forwarding SpaceWire packets towards their destinations, a
SpaceWire router also broadcasts time-codes. Time-codes and the way in
which they are broadcast by a SpaceWire router are described in section 5.
87
4.8 Router Configuration
A router is configured via its configuration port. The SpaceWire standard
does not specify how a router should be configured, but it is common
practice for this to be done using the SpaceWire remote memory access
protocol (see section 6.2). This provides a standard means of reading and
writing to registers in the configuration port, but it does not specify the
arrangement and function of registers in the configuration port. Work is
currently being done on a SpaceWire plug and play standard, which aims to
standardise SpaceWire network configuration. In the meantime each router
configuration space has a proprietary arrangement of registers.
88
Control
Logic Routing
Table
SpW Port 1
SpW Port 2
Status/Error
Registers
SpW Port 3
SpW Port 4
Status
SpaceWire Outputs
Interfaces SpW Port 5 Control
Registers
SpW Port 6
Non-blocking
SpW Port 7 Crossbar
Switch Configuration
Port
SpW Port 8
There are eight SpaceWire ports, two external ports and an internal
configuration port in the SpaceWire router. A low latency, worm-hole routing,
non-blocking, crossbar switch enables packets arriving at any SpaceWire
port, external port or generated in the configuration port to be directed out of
any other SpaceWire or external port or to be routed to the configuration
port.
The SpaceWire ports are compliant with the SpaceWire standard [2]
providing high-speed, bi-directional communications. The FIFO ports each
comprise an input FIFO and an output FIFO and can receive and send data
characters and end of packet markers. A time-code port is also provided
along with a time-counter to facilitate the propagation of time-codes, see
section 5. When a valid time-code arrives at a router port it is sent out of all
the other SpaceWire ports and a TICK_OUT signal is generated at the time-
89
code port. The router can operate as a time-code master using the TICK_IN
provided in the time-code port.
90
4.10.2 Watchdog Timers
Similarly a packet could be routed to a port that does not actually exist on
the router, e.g. port 12 on an 8 port router. In this case the router
immediately spills the packet and logs this error in its status registers.
If a packet is routed to a port that is already busy on a router, the router will
wait to send the packet for the watchdog timer period. If it times out it will
spill the packet, but not append an EEP to the end of the packet since no
data from the packet has been sent. This is different from the case in
section 4.10.2, as there the packet is in the process of being sent through a
router output port when it becomes blocked. Here the packet has not yet
been switched to the output port, as that port is busy. If the time out occurs
and the waiting packet is spilt, this error is logged in a status register.
91
4.10.6 Start On Request, Disable On Silence
The SpW-10X has some facilities to support power saving. It can be set to
“disable on silence” and to “start on request”.
Disable on silence will switch off a link when there is no packet to send, after
the watchdog timer interval.
Start on request will start a disabled link when a packet that arrives on
another port wants to be routed across the disabled link. It will attempt to do
this for the watchdog interval and then give up spilling the waiting packet.
Port 5 Port 1
AutoStart AutoStart
Not Running Not Running
Disable on Silence Disable on Silence
1 5 1 5
NODE 1 NODE 6
ROUTER 1 ROUTER 2
2 4 2 4
NODE 2 NODE 5
3 3
92
Node 1
Sends packet
1 5 1 5
NODE 1 NODE 6
ROUTER 1 ROUTER 2
2 4 2 4
NODE 2 NODE 5
3 3
Port 5
automatically
started
1 5 1 5
NODE 1 NODE 6
ROUTER 1 ROUTER 2
2 4 2 4
NODE 2 NODE 5
3 3
93
Connection made
and packet
transferred
1 5 1 5
NODE 1 NODE 6
ROUTER 1 ROUTER 2
2 4 2 4
NODE 2 NODE 5
3 3
After timout
automatically
disabled
1 5 1 5
NODE 1 NODE 6
ROUTER 1 ROUTER 2
2 4 2 4
NODE 2 NODE 5
3 3
There are no more packets to be sent so the link between Router 1 port 5
and Router 2 port 1 falls silent. After the time-out period. Router 1 detects
that there is no more traffic being sent over this link and since it is set to
disable on silence, it disables output port 5.
94
Link
powered down
saving power
1 5 1 5
NODE 1 NODE 6
ROUTER 1 ROUTER 2
2 4 2 4
NODE 2 NODE 5
3 3
4.10.7 Tristate
The output ports can be put in a state similar to tri-state to save some power
when they are not being used. Remember that an LVDS driver is always
sourcing current. Tri-stating it turns off this current and stops power being
wasted. Tri-state can be used with the power saving techniques mentioned
in section 4.10.6.
The SpW-10X router has eight SpaceWire ports. In some systems not all of
these ports are required. It is then possible to turn off the clock tree to a port
to save power. The front end receive part of the port has its clock derived
from the receive signal so there will be no dynamic power consumed in the
receiver if it is not connected. Configuration registers are used to disable the
transmit clocks to unused ports.
95
If there are two input ports in a router waiting to use a particular output port,
an arbitration mechanism is used to select which input port is to be served.
The arbitration mechanism can include a priority scheme. There is no
priority flag available within the header of a SpaceWire packet to specify its
priority level. The SpaceWire header only contains address information, so
packet priority must be associated with a logical address (or with the input
port number). In the routing tables logical addresses may be assigned high
or low priority. High priority logical addresses have preferential access to an
output port when arbitration takes place. A logical address that has been
assigned high priority, acts as a high priority channel across the network
from many possible sources to the one destination. If high and low priority
access to a particular destination is required, two logical addresses are
required for a particular destination, one assigned high priority and the other
low priority. A source can then decide which logical address to use when
sending a packet to a destination, depending on the required priority of the
packet. There is a compromise between the number of destinations that can
be addressed and the number of priority levels. With two priority levels it is
possible to have, say 128 low priority destinations and 96 high priority
destinations within the 224 logical addresses available.
96
5 Time-Codes
SpaceWire time-codes [12] provide a means of synchronising units across a
SpaceWire system with reasonably low jitter. This time information can be
provided as “ticks”, an incrementing value which may be synchronized to
spacecraft time. The time-codes are broadcast rapidly over the SpaceWire
network, alleviating the possible need for a separate time distribution
network.
(P)
P 1 1 1 1 0 T0 T1 T2 T3 T4 T5 T6 T7
ESC DATA CHARACTER
The two control-flags are for general use, they do not have a specific
function for time-code distribution. The only permitted value for the flags is
0b00. The other possible values are reserved.
97
an eight-bit time-code output-port. The eight-bit input and output are split
into two fields: a six-bit time field and two-bit control-flag field.
5.3 Time-counter
Each node and router contains a time-counter which is used to hold the
current 6-bit time value from the time-code time. This counter is also used to
validate an incoming time-code and to decide whether the time-code should
be propagated. In a node, time-code propagation is up to the application
(hardware or software). In a router it is to all the output ports of the router, or
at least all port that have been configured to forward time-codes.
During normal operation the time value in a time-code increments from one
time-code to the next, rolling round from 63 to 0 when it reaches the
maximum possible time value. When a time-code arrives at a SpaceWire
node or router its value should be one more than the current value of the
time-counter at this node or router. The time value of an incoming time-code
is compared to the value of the time-counter. If the time-code is one more
than the current value of the time-counter then the time-code is deemed
valid. The time-counter is then incremented to the value of the time-code
and the arrival of a valid time-code is indicated.
98
If the time value of the time-code is not one more than the time-counter then
the time-code is not valid. In this case the time-counter is set to the value of
the newly arrived time-code but the arrival of the time-code is not flagged.
99
time-codes, checks their validity, updates the time-counter with the new time
value and asserts the TICK_OUT signal. The host system attached to this
end of the SpaceWire link, receives periodic TICK-OUT signals together
with the six-bit time value and the two control-flags. The TICK-OUT signal
can be used to raise an interrupt or event in a software-driven node.
If a SpaceWire interface receives a time-code that is not one more than the
current value of its time-counter then the time-code is not valid and the
interface does not emit a TICK_OUT signal.
If there is a circular connection then the router will receive a time-code with
the same time value as the router time-counter. When this happens the
time-code is ignored. In this way time flows forward through a network
reaching all nodes but is suppressed if it flows back due to a circular
connection.
The initial state of a network is shown in Figure 66. N1 and N2 are nodes.
R1 and R2 are routers. The dotted lines represent the SpaceWire links
between the nodes and routers. N1 is time-master and sends out a time-
100
code whenever its TICK_IN signal is asserted. The numbers in the node and
router boxes represent the current values of their time-counters.
R2
40
N1 R1 R3 N2
40 40 40 40
R2
TICK_IN 40
N1 R1 R3 N2
41 40 40 40
R2
40
N1 R1 R3 N2
41 41 40 40
101
Routers R2 and R3 both receive time-codes sent from R1 and both of these
routers send out a time-code on all other links as shown in Figure 69.
R2
41
N1 R1 R3 N2
41 41 41 40
Figure 69 Routers R2 and R3 Forward Time-codes
Node N2 receives the time-code from R3 and validates that the incoming
time-code has a time value of one more than the node’s time-counter. It
then increments the time-counter and asserts TICK-OUT. See Figure 70.
Time in N2 is updated.
R2
41 TICK_OUT
N1 R1 R3 N2
41 41 41 41
Figure 70 N2 Receives Time-code and Asserts TICK_OUT
R3 will also receive a time-code from R2, which is now not one more than
R3’s time-counter value, so is ignored as shown in Figure 71. This prevents
time-codes from going back through the network. Time-codes move forward
through the network even when there are loops in the network.
R2
41
N1 R1 R3 N2
41 41 41 41
Figure 71 Time-codes Cannot Go Backwards
Node N2 will also receive a time-code from R2 which is not one more than
N2’s time-counter value, so is ignored. This prevents multiple triggering of
the TICK_OUT signal as duplicate copies of the time-code which have taken
102
different paths through the SpaceWire network arrive at the node. See
Figure 72.
R2
41
N1 R1 R3 N2
41 41 41 41
If the time-code is invalid then the time-count is updated to the new value
but the time-code is not propagated in a router and TICK_OUT is not
asserted in a node. This prevents propagation of invalid time-codes across a
network. When the next time-code is received it is expected that the time-
counter matches the time-code and normal operation resumes. Recovery
from missing or invalid time-codes will now be considered.
N1 R1 R2 N2
20 20 19 19
On the next tick N1 sends out the time-code 21. R1 then forwards this time-
code to R2. This is not same as, nor one more than time-counter of R2 so
103
R2 updates its time-counter but does not emit the time-code, as shown in
Figure 74.
N1 R1 R2 N2
21 21 21 19
On the next tick the time-code 22 is sent from N1 to R1, which forwards it on
to R2. At R2 the time-count is now 21, so the incoming time-code is one
more than the time-count hence the time-code is now valid and is
propagated by the router and reaches N2. See Figure 75. When the time-
code reaches N2 it is not one more than N2’s time-count (value 19) so the
time-code is deemed invalid. N2 updates its time count to 22 but does not
give a TICK_OUT.
N1 R1 R2 N2
22 22 22 22
The next tick will result in the time-code 23 propagating across the network
and N2 will produce a TICK_OUT, as shown in Figure 76.
N1 R1 R2 N2 TICK_OUT
23 23 23 23
Figure 76 N2 gets Valid Time-code
It takes several ticks to recover from initial error, depending upon the size of
the network.
104
time-code even though one of the time-codes on its way to R2 gets lost.
This provides a first level of fault tolerance for time-code distribution.
Nodes using the time-code distribution function can either use the
TICK_OUT signal as a periodic timing signal or use the value of the time-
count as an indication of the least-significant 6-bits of system time.
105
time accuracy across a network of significantly better than 10 s may be
difficult to achieve, using the standard time-code mechanism.
5.10.1 Synchronisation
With the provision of this basic time distribution function, application level
protocols can be used to distribute specific time values at full resolution (not
just 6-bits) and to issue time dependent commands etc. The two control
flags that are distributed with the 6-bit time code can be used to broadcast
information to all nodes and routers on the network. However, these two
control flags are reserved in the SpaceWire standard, so they should be
both set to zero, to be compliant with the standard.
Nodes using the system time distribution function can either use the
TICK_OUT signal as a periodic timing signal or use the value of the time-
count as an indication of the least-significant 6-bits of system time.
106
5.10.4 Multiple Time-codes
This approach allowed four different time signals to be provided over the
SpaceWire network.
107
6 SpaceWire Protocols
108
Remote Memory Access Protocol (RMAP) [14] [15], is a particularly
successful example of one of these protocols that operate over SpaceWire.
109
REFERENCES
5. Inmos Transputer
110
10. S.M. Parkes, “High-Speed, Low-Power, Excellent EMC: LVDS for On-
Board Data-handling”, DSP’98, 6th International Workshop on Digital
Signal Processing Techniques for Space Applications, ESTEC,
Noordwijk, The Netherlands, 23-25 September 1998, European Space
Agency (ESA) publication no. WPP-144, paper P16.
12. S.M. Parkes, "The operation and uses of the SpaceWire time-code", in:
ISWS International SpaceWire Seminar 2003 (Noordwijk, The
Netherlands, 4-5 November (CD Rom) 2003) pp.223-230.
111
19. Sentinel 1, http://www.esa.int/esaLP/SEMBRS4KXMF_LPgmes_0.html,
st
viewed on 1 December 2010.
112
31. BepiColombo Mercury Magnetospheric Orbiter,
http://www.stp.isas.jaxa.jp/mercury/p_mmo.html
34. ASNARO,
http://www.usef.or.jp/english/f3_project/asnaro/f3_asnaro.html
42. IEEE Computer Society, “IEEE Standard for a High Performance Serial
Bus”, IEEE Standard 1394-1995, IEEE, August 1996
113
114
About the Author:
www.star-dundee.com
115
116