Abstract Robot and computer-aided surgery platforms

bring a variety of sensors into the operating room. These

sensors generate information to be synchronized and merged
for improving the accuracy and the safety of the surgical
procedure for both patients and operators. In this paper, we
present our work on the development of a sensor management
architecture that is used is to gather and fuse data from
localization systems, such as optical and electromagnetic
trackers and ultrasound imaging devices. The architecture
follows a modular client-server approach and was implemented
within the EU-funded project ROBOCAST (FP7 ICT 215190).
Furthermore it is based on very well-maintained open-source
libraries such as OpenCV and Image-Guided Surgery Toolkit
(IGSTK), which are supported from a worldwide community of
developers and allow a significant reduction of software costs.
We conducted experiments to evaluate the performance of
the sensor manager module. We computed the response time
needed for a client to receive tracking data or video images,
and the time lag between synchronous acquisition with an
optical tracker and ultrasound machine.
Results showed a median delay of 1.9 ms for a client request
of tracking data and about 40 ms for US images; these values
are compatible with the data generation rate (20-30 Hz for
tracking system and 25 fps for PAL video).
Simultaneous acquisitions have been performed with an
optical tracking system and US imaging device: data was
aligned according to the timestamp associated with each sample
and the delay was estimated with a cross-correlation study. A
median value of 230 ms delay was calculated showing that real-
time 3D reconstruction is not feasible (an offline temporal
calibration is needed), although a slow exploration is possible.
In conclusion, as far as asleep patient neurosurgery is
concerned, the proposed setup is indeed useful for registration
error correction because the brain shift occurs with a time
constant of few tens of minutes.

I. INTRODUCTION
Echnologically-advanced surgical platforms exploit
navigation and/or robotic systems to improve the
outcome of procedures in several fields, such as orthopedic
surgery [1], abdominal surgery [2], and neurosurgery [3].

This work was supported by the EU Project Grant ROBOCAST FP7-
ICT-215190 and Scuola Interpolitecnica di Dottorato, Politecnico di
Milano, Bari e Torino.
A. Vaccarella is with Politecnico di Milano, Bioengineering Department,
Neuroengineering and Medical Robotics Laboratory, Piazza Leonardo da
Vinci 32, 20133 Milano, Italy. (e-mail: [email protected])
M.D. Comparetti, G. Ferrigno and E. De Momi are with Politecnico di
Milano, Bioengineering Department, Neuroengineering and Medical
Robotics Laboratory, Milano, Italy.
A. Enquobahrie is with Kitware Inc., Carrboro, NC, USA
Thus, the need for different sensors in the operating room
(OR) has highly increased in the past two decades with the
purpose of improving the accuracy and the safety of surgical
procedures for both patients and operators. The main
typologies of sensors used in the OR include localization
systems, inertial sensors, force sensors, and intra-operative
imaging devices.
Surgical navigation systems usually rely on information
acquired using localization systems (mechanical, optical, or
electromagnetic) in order to localize the pose of surgical
instruments and to overlay them on the CT or MR images to
assist surgeons during procedures [4].
Inertial Measurement Unit (IMU) allows for
determination of an object's position and orientation at a
high rate, even with no line of sight; on the other hand,
IMUs are limited by noise amplification and drift, due to the
double integration of the acceleration signal. They have
recently been proposed in robotic surgery, in addition to an
optical tracking system, to overcome the limitation of
localization systems. In [5] Kalman-based sensor data,
fusion has been proposed for a robot-controlled algorithm
for motion compensation of the patient.
The use of force sensors has been recently investigated in
order to provide haptic feedback to tele-operated surgical
robots; surgical tools have been endowed with strain gauges
[6], [7] or optical, fiber-based force sensors in order to
measure small interaction forces in the range of 2.5 N with a
resolution of 0.04 N [8].
Intra-operative imaging devices, which include Magnetic
Resonance Imaging (MRI), Computed Tomography (CT),
Fluoroscopy, and Ultrasonography (US), are used to update
preoperative images and thus improve the accuracy of
navigation systems; this is of the utmost importance when
dealing with soft tissues. In neurosurgery, even a small
aperture in the skull (mini-invasive procedure also known as
keyhole neurosurgery) results in an intra-operative brain
deformation that is known as brain shift (mainly due to
cerebrospinal-fluid drainage). Different approaches have
been proposed to estimate brain shift based on intra-
operative MRI [9], 2D ultrasound images [10], and 3D
ultrasound reconstruction (Freehand 3D US) [11]. The use of
ultrasound imaging devices is of great interest because of
their relatively low cost, small encumbrance in the OR, low
invasiveness, and high frame rate. In particular, Freehand 3D
US is achieved by tracking the pose in space of an
ultrasound probe, which was previously calibrated with
Sensors Management in Robotic Neurosurgery:
the ROBOCAST project
Alberto Vaccarella, Mirko Daniele Comparetti, Andinet Enquobahrie, Giancarlo Ferrigno and Elena
De Momi
T
978-1-4244-4122-8/11/$26.00 2011 IEEE 2119
33rd Annual International Conference of the IEEE EMBS
Boston, Massachusetts USA, August 30 - September 3, 2011

respect to the localization system. Temporal alignment of
tracking data and US images is a demanding requirement
[12] to ensure accuracy of the reconstruction; this is
achieved by estimating the overall latency between images
and tracking data through a temporal calibration procedure.
Sensor information is crucial, especially when the surgical
intervention is assisted by robots. Robotic systems have been
used in surgery for tool positioning, particularly for
orthopedic or neurosurgical procedures. The robot is
traditionally directly calibrated with respect to the patient
prior to the procedure, so that the relative positions are kept
steady [13]. The use of localization systems allows the
detection of any possible displacement.
The ROBOCAST project (FP7-ICT-2007-215190)
proposes a hybrid, navigated, robotic system for
neurosurgery, where an integrated use of multimodal
tracking systems, force sensors, and an ultrasound imaging
device would improve the safety of the procedure by
providing redundant information for the robot control loop
and for the surgeons [14].
II. SENSOR MANAGER MODULE
ROBOCAST is a robotic system that assists surgeons in
performing brain surgery through a very small hole in the
patients skull (keyhole neurosurgery) for biopsies, Deep
Brain Stimulation (DBS) electrodes implantation, and
cryogenic and electrolytic ablation.



Fig 1. ROBOCAST architecture: TS and US are connected to a laptop
where an IGSTK based application (SM) acquires data and forwards it upon
request to HCI, a workstation in charge of intra-operative navigation, and to
HLC which is responsible for robot control. Communication between SM,
HCI, and HCL is achieved through the ACE-TAO CORBA middleware on
a Gigabit Ethernet LAN.
The Sensor Manager (SM) is a software component
within the distributed architecture of the ROBOCAST
system (Fig 1). It is implemented as a C++ service
application without a user interface running on a dedicated
laptop.
The main purpose of the SM is to gather data from
localization systems (optical and electromagnetic) and
ultrasound imaging devices, and forward such information
upon request to the other components of the system (clients).
As is each component of the ROBOCAST framework, the
SM is part of a client/server architecture based on the
middleware TAO CORBA.
The ACE ORB (TAO - The Ace Orb)
1
is a freely-
available, open-source, and standards-compliant, real-time
C++ implementation of CORBA based upon the Adaptive
Communication Environment (ACE). CORBA enables
separate pieces of software written in different languages
and running on different computers to work with each other
like a single application or set of services. More specifically,
CORBA is a mechanism in software for normalizing the
method-call semantics between application objects residing
either in the same address space (application) or remote
address space (same host, or remote host on a network).
CORBA uses an Interface Definition Language (IDL) to
specify the objects interfaces to the outer world. CORBA
then specifies a mapping from IDL to a specific
implementation language, such as C++ or Java.



Fig 2. The SM is interfaced with tracking systems through the IGSTK
library and with a US machine through the OpenCV library. Client/Server
communication is based on the TAO-CORBA Middleware.

The SM encompasses two main parts: (1) interfacing the
hardware based on the open source libraries "Image-Guided
Surgery ToolKit" (IGSTK)
2
and OpenCV
3
and (2) acting as
a server to provide data and services to other modules in a
CORBA-based network architecture. IGSTK is mainly used
for tracking system management and spatial relationship
hierarchy among reference frames.
Tracking systems are managed in a transparent way for
the user through an xml configuration file, which allows
flexibility in hardware arrangement. IGSTK natively
provides support for NDI Aurora, NDI Polaris (Vicra and
Spectra), Ascension MedSafe, Ascension trakSTAR, and

1
http://www.theaceorb.com/
2
http://www.igstk.org/
3
http://opencv.willowgarage.com/
2120

Atracsys InfiniTrack among the others; for NDI Optotrak
Certus systems, a dedicated class was implemented within
the IGSTK framework [14].
According to the IGSTK architecture, as shown in Fig. 2,
tracking data is continuously acquired in a separate thread
(Tracker Thread, one for each active tracking systems) and
stored in a buffer. The main thread updates spatial object
transformation, reads the stored tracking data from the buffer
at a user-defined frequency (20 Hz), and marks each data
with a timestamp provided by the IGSTK RealTimeClock.
Open-CV allows the acquisition of images from the US
device connected to the SM laptop via an USB frame
grabber (EZ-Grabber, PAL: 720 x 576 @ 25 fps). A
timestamp is also provided for US image reading time from
the unique IGSTK RealTimeClock object instantiated for the
whole application.
IGSTK also provides software components to implement
a spatial object hierarchy, which facilitates spatial
transformation computation between couples of reference
frames. This allows the clients to ask for the transformation
matrix for a reference system associated with an object in
the ROBOCAST scenario, with respect to whatever other
object. This is relevant in a multi-sensor system for surgical
applications, such as ROBOCAST, where multiple reference
frames are involved: optical, electromagnetic, preoperative
images (typically T1 MRI and/or CT scans), US images, and
actuators reference frames, which all need to be calibrated
to a common reference frame in order to have consistent
information. In the ROBOCAST project, the optical
reference frame has been chosen as the global reference;
calibration matrices have been determined to transform data
from any other reference frame to the optical one.

III. SENSOR MANAGER PERFORMANCES
The proposed architecture was tested in order to (1)
measure the response time needed for the client to receive
tracking data or video images and (2) to assess the time lag
between synchronous acquisition with an optical tracker and
ultrasound machine.
For the first experiment, the setup encompasses an optical
localization system (NDI Optotrak Certus), an
electromagnetic tracking system (NDI Aurora), an
Ultrasound imaging device (Aloka Prosound Alpha 7), the
server; a dual-core PC with Windows XP Professional
Service Pack 3 (CPU: Intel T7600, 2.33GHz, RAM: 2 GB)
running the SM; and a quad-core PC with Windows XP
Professional Service Pack 3 (CPU: Intel Q9550, 2.83GHz,
RAM: 2.5 GB) running a testing client application. Client
and server are connected through the ROBOCAST Gigabit
Ethernet LAN based on the TAO-CORBA middleware.
The experimental protocol is the following:
1. On the client, the current time in milliseconds is stored
as t1 through the windows API function, which retrieves
the current value of the high-resolution performance
counter of the personal computer.
2. The client sends a data request to the server with a
blocking remote procedure call (RPC) through the
TAO-CORBA middleware.
3. The server elaborates the request and sends back data.
4. Once the client receives data, the time (t2) is stored on
the client.
5. The time difference (t2 - t1) represents the response
time that is composed by request transmission time,
server elaboration time, and data transmission time.
Each testing session consists of 30 sets of 200 requests.
The experiment was repeated with different tracking systems
(NDI Optotrak Certus, NDI Polaris Vicra, and NDI Auora)
and with the Aloka Prosound Alpha 7 ultrasound machine.
A Kruskal Wallis test showed no statistical difference
between the 30 sets of each system, thus the median value of
6000 samples was computed for each tracking system and
for the US machine; results are reported in Table 1.

Table 1. Response time results. Different tracking systems show similar
values of response delay; image transmission entails one order of
magnitude higher response time.
Tested system Response time
(median value and interquartile range)
NDI Certus 1.92 ms (1.91 1.94 ms)
NDI Aurora 1.93 ms (1.92 1.96 ms)
NDI Vicra 1.94 ms (1.91 1.95 ms)
Aloka US 39.86 ms (35.37 44.54 ms)

In the second experiment, we tested the synchronization of
tracking data and US images acquired through the SM
module with the following protocol. Again, the hardware
adopted consists of the NDI Optotrak Certus optical
localization system, the Aloka Prosound Alpha 7 ultrasound
imaging device, and the Prosurgics PathFinder serial robot.
The probe, localized with the optical tracker, was
immersed in a water tank and moved along the vertical
direction using the Pathfinder robot, which was programmed
to achieve 3 repetitions of vertical motion (lasting 0.83 s at
30 mm/s constant velocity); the experiment was repeated ten
times.
The bottom of the tank is represented on the US image as
a line. A mono-dimensional signal was extracted from the
image sequence, segmenting the line and calculating the
distance in pixels of the central line point from the top of the
image. 3D position data of the probe was acquired at the
same time and normalized with respect to the amplitude of
the performed movement.
The two signals were aligned using the timestamps
provided by the SM module, resampled at 1 kHz. The time
lag was estimated by computing the maximum of the cross
correlation function of the two time series.
2121


Fig. 3. Monodimensional signals derived from tracking data and US
images are reported for a session of three movements (above) and a
zoom of the first rising edge (below).
As shown in Fig. 3, we observed a median delay of 230
ms (first quartile = 223 ms and third quartile = 240 ms).
IV. CONCLUSION
Navigation in surgery relies on different typologies of
sensors, proved to increase the outcome for a huge variety of
surgical interventions.
The sensor management architecture we proposed allows
for the development of a hardware-independent navigation
platform thanks to the transparency of the IGSTK library
and of the CORBA middleware. Additionally, the use of
open-source software allows the development of a cross-
platform, modular application that benefits from a
worldwide community of users and developers.
The client/server architecture we proposed showed a
transmission delay (between data request and data retrieval
on a Gigabit Ethernet network) of approximately 1.9 ms for
the three tested localization systems, which means that a
client can request data with a rate up to 500 Hz, much higher
than localization systems frame rate (typically 20-30Hz).
Also, US images transmission delay (about 40 ms) is in the
range of a PAL frame grabber (25 fps).


Fig. 4. Intra-operative ultrasound navigation during the ROBOCAST final
demo at Ospedale Maggiore, Verona Italy. Optical tracking of the US
probe allows for visualization of the image plane in the 3D scene.
Synchronization between tracking data and US images is
assured by the SM module, which associates position data
and images with timestamps (only 17 ms of inter-quartile
range shows the delay is approximately constant). A median
delay of 230 ms shows that such architecture does not allow
real-time volume reconstruction at this stage, since post
processing of data is required for proper time alignment.
Nevertheless, as shown in Fig. 4, a slow exploration (few
mm per second) is indeed possible and useful whenever soft
tissue interaction is involved; for instance, during
neurosurgical procedures to update the pre-operative plan
once brain-shift has occurred. In particular, during asleep
patient neurosurgery, brain shift is the main source of
inaccuracy, but it occurs with a slow time constant (tens of
minutes); in such a scenario, the proposed setup is indeed
useful for registration error correction.

REFERENCES
[1] D. Kendoff, M. Citak, T. Hfner, S. Chaudhary and C. Krettek,
"Current concepts and applications of computer navigation in
orthopedic trauma surgery," Central European Journal of Medicine,
vol. 2, pp. 392-403, 2007.
[2] E.J. Hanly and M.A. Talamini, "Robotic abdominal surgery," The
American Journal of Surgery, vol. 188, pp. 19-26, 2004.
[3] T. Haidegger, L. Kovacs, G. Fordos, Z. Benyo and P. Kazanzides,
"Future Trends in Robotic Neurosurgery," in 14th Nordic-Baltic
Conference on Biomedical Engineering and Medical Physics, pp. 229-
233, 2008.
[4] P. Grunert, K. Darabi, J. Espinosa and R. Filippi, "Computer-aided
navigation in neurosurgery," Neurosurg. Rev., vol. 26, pp. 73-99,
2003.
[5] K. Cleary and C. Nguyen, "State of the art in surgical robotics: clinical
applications and technology challenges," Computer Aided Surgery,
vol. 6, pp. 312-328, 2001.
[6] A. Tobergte, M. Pomarlan and G. Hirzinger, "Robust multi sensor
pose estimation for medical applications," in IEEE/RSJ International
Conference on Intelligent Robots and Systems, 2009. IROS 2009, pp.
492-497, 2009.
[7] D. De Lorenzo et al. "Miniaturized rigid probe driver with haptic loop
control for neurosurgical interventions," in 2010 3rd IEEE RAS and
EMBS International Conference on Biomedical Robotics and
Biomechatronics (BioRob), pp. 522-527, 2010.
[8] P.J. Berkelman, L.L. Whitcomb, R.H. Taylor and P. Jensen, "A
miniature microsurgical instrument tip force sensor for enhanced force
feedback during robot-assisted manipulation," IEEE Trans. on
Robotics and Automation, vol. 19, pp. 917-921, 2003.
[9] J. Peirs, J. Clijnen, D. Reynaerts, H.V. Brussel, P. Herijgers, B.
Corteville and S. Boone, "A micro optical force sensor for force
feedback during minimally invasive robotic surgery," Sensors and
Actuators A: Physical, vol. 115, pp. 447-455, 2004.
[10] O. Clatz, H. Delingette, I.F. Talos, A.J. Golby, R. Kikinis, F.A. Jolesz,
N. Ayache and S.K. Warfield, "Robust nonrigid registration to capture
brain shift from intraoperative MRI," IEEE Transactions on Medical
Imaging, vol. 24, pp. 1417-1427, 2005.
[11] R. Bucholz et al., "The correction of stereotactic inaccuracy caused by
brain shift using an intraoperative ultrasound device," in CVRMed-
MRCAS'97, pp. 459-466, 1997.
[12] G. Unsgrd, O. Solheim, F. Lindseth and T. Selbekk, "Intra-operative
Imaging with 3D Ultrasound in Neurosurgery," Intraoperative
Imaging, pp. 181-186, 2011.
[13] M.C. Jacobs and M.A. Livingston, "Managing latency in complex
augmented reality systems," in Proceedings of the 1997 symposium on
Interactive 3D graphics, pp. 49-ff., 1997.
[14] A. Vaccarella, P. Cerveri, E. De Momi and G. Ferrigno, "A new
IGSTK-based architecture for the integration of multimodal sensors
and robots in neurosurgical robotics applications," IJCARS, vol. 5,
Supplement 1, pp. 308-309, June 23-26. 2010.
2122