Design and Implementation of a Middleware for

Sentient Spaces
Bijit Hore, Hojjat Jafarpour, Ramesh Jain, Shengyue Ji, Daniel Massaguer
Sharad Mehrotra, Nalini Venkatasubramanian,Utz Westermann

School of Information and Computer Science

University of California
Irvine, CA 92697
Email: {bhore, hjafarpo, jain, shengyuj, dmassagu, sharad, nalini}@ics.uci.edu, [email protected]

Abstract- Surveillance is an important task for guaranteeingfrom the underlying sensors provides numerous advantages.
the security of individuals. Being able to intelligently monitorFirst and foremost, it provides a level of separation that enables
the activity in given spaces is essential to achieve such surveil-
application writers to focus on the application logic instead
lance. Sentient spaces based on a large set of sensors provide of having to directly manipulate sensor streams. Another
the potential for such intelligent monitoring. However, heavily
instrumenting a space with sensors it is not enough to build a advantage is that it provides a framework for the underlying
sentient space. One needs a software architecture that allows middleware and runtime to optimize application execution
programming all these sensors in a transparent and efficient based on variety of application and resource constraints. For
manner. In this paper, we present SATware, a Stream Acquisition instance, an event (such as location of a person of interest in an
and Transformation middleware we are developing to analyze, instrumented space) might be detected through tracking built
query, and transform multimodal sensor data streams to facilitate
flexible development of sentient environments. SATware provides using video cameras. It could also be detected using coarse
a powerful application development environment in which users level triangulation using WiFi access points. Which specific
(i.e., application builders) can focus on the specifics of the sensing technique is utilized can be decided by the run-time
application without having to deal with the technical peculiarities
based on available resources and the applications accuracy
of accessing a large number of diverse sensors via different needs.
protocols.
In this paper, we describe the architecture of SATware, a
I. INTRODUCTION multimodal sensor data stream querying, analysis, and trans-
Advances in computing, communication and sensing tech- formation middleware that aims at realizing a sentient system.
nologies has made it possible to create large scale physical SATware provides applications with a semantically richer level
spaces with diverse embedded sensors, ubiquitous connectiv- of abstraction of the physical world compared to raw sensor
ity, and computation resources that together can enable per- streams, providing a flexible and powerful application devel-
vasive functionality and applications. Today, such applications opment environment. It supports mechanisms for application
are built in a somewhat ad-hoc manner directly on top of multi- builders to specify events of interest to the application, mech-
modal sensor streams and application writers are exposed to anisms to map such events to basic media events detectable
the technical challenges of accessing a large number of dif- directly over sensor streams, a powerful language to compose
ferent sensor types via different protocols over a network. We event streams, and a run-time for detection and transformation
envision a different type of pervasive computing infrastructure of events. SATware is being developed in the context of the
which we refer to as sentient spaces that provides powerful and Responsphere infrastructure at the UC Irvine campus which
flexible abstractions for building pervasive applications. In a is a unique publicly accessible testbed for interdisciplinary re-
sentient space, observers possess capabilities to perceive and search in situation monitoring and awareness in the context of
analyze situation based on available multimodal data from dis- emergency response applications [7]. Currently, Responsphere
parate sources. Sentient spaces provide a much higher level of includes more than 200 sensors of 10 different types deployed
semantic abstraction compared to sensor middleware prevalent over the geographical space that covers about half of the UCI
today. The key concept underlying sentient spaces is events. campus. The sensors range from network pan-tilt-zoom fixed
Events are essentially important occurrences (or state changes video cameras, microphones, wireless motes with accelerom-
in a pervasive space) that are semantically meaningful for the eters, temperature sensors, and motion sensors to RFID tag
application. For instance, entry/exit of people from a building readers and networked people counters. It also includes mo-
might be a meaningful event in the context of a surveil- bile, WiFi-connected sensors such as an autonomous vehicle
lance application. Instead of viewing data as sensor streams,
sentient spaces provide the abstraction of the (semantically
meaningful) event streams which form the building block for
pervasive applications. Decoupling (and abstracting) events

1-4244-1330-3/07/$25.OO 02007 IEEE. 133

and a wearable EvacPack 1 carrying sensing devices like GPS Abstraction. In order to provide an abstraction to ease sensor
devices, gas chromatographs, cameras, and microphones. data stream processing in Responsphere and to allow automatic
Responsphere is currently used to instrument and observe optimization of concurrent processing tasks, declarative lan-
activities such as emergency drills that are conducted in the guages for analyzing, querying, and transforming data streams
UC Irvine Campus. The data collected by sensors (e.g., video, are essential. Not all data stream processing approaches to
audio, etc.) is used by sociologists to observe and analyze sensor stream processing provide such languages (e.g., [10]
human behavior as well as ascertain the effectiveness of new [12]).
technologies, protocols and strategies in crisis situations de- Scalability. In Responsphere, a large number of sensors
ployed within emergency drills. Furthermore, continuous data potentially producing streams with considerably high data
feeds from Responsphere sensors are used to build pervasive rates requiring substantial computational power for process-
applications such as surveillance and infrastructure monitor- ing (such as video streams) pose the difficult challenge of
ing. Although Responsphere provides the main motivation efficiently utilizing both limited available network bandwidth
of our work, it must be noted that SATware is a generic and limited computing resources. Many of the sensor data
framework suiting any application scenario in which highly processing infrastructures proposed in the literature, however,
diverse data streams need to be processed. are based on centralized architectures thwarting scalability to
The paper is organized as follows: using our needs in Responsphere dimensions [5] [15] [20].
Responsphere as the main source of motivation, we highlight Extensibility. With the large number and variety of dif-
elementary notions and issues that multimodal sensor data ferent sensors, developers face numerous challenges such as
stream processing middleware needs to address in Section II. discovering which sensor types are available, what sensor
In Section III, we briefly analyze related work with regard streams they produce, where sensors are located physically
to which extent it addresses (or does not address) the raised and which area they cover, under which addresses they can
issues. In Section IV, we describe the basic architectural be reached on the network, what the network topology is,
building blocks of SATware and the underlying processing what computing nodes are available for stream processing,
model. Section V, concludes this chapter with a summary and etc. A sensor network directory service offering access to this
an outlook to ongoing and future developments of SATware. information is required.
Mobility. The availability of mobile sensors in Respon-
II. MULTIMODAL SENSOR DATA PROCESSING: NOTIONS sphere constitutes a further challenge. While many sensor data
AND ISSUES processing infrastructures are capable of temporally synchro-
The processing of the sensor data produced by the large nizing different streams for multimodal data using different
sensing infrastructure of Responsphere is a challenging task. variants of time window-based joins (e.g., [10] [14] [13]),
The heterogeneity of the infrastructure and its dimensions mobile sensors also require location-based means of stream
provide a very powerful sensing infrastructure that enables a synchronization. For instance, one may want to join the camera
wide variety of applications. However, that same heterogeneity streams of two autonomous vehicles only when they are
and dimensions undermines the ability to use it. Responsphere observing the same area.
produces a set of heterogeneous data streams that range from Power-awareness. Responsphere encompasses sensors that
raw video data to temperature readings. The sensor hardware, are battery-powered and sensors that are constantly connected
software, and their access protocols are also heterogeneous; to the power grid. Battery-powered sensors can also be further
and so are their mobility capabilities, power constraints, and divided into rechargeable sensors and not (easily) rechargeable
processing, storing, and networking capabilities. This hetero- sensors. This heterogeneity implies that different power opti-
geneity together with the wide range of applications that mization approaches have to be considered when accessing
can be implemented in such an infrastructure and the fact each device and balancing the computation.
that Responsphere encompasses a large number of sensors Privacy. As Responsphere covers a public studying and
installed throughout the UC Irvine campus poses the following work environment, privacy of observed individuals is of high
challenges. importance. To avoid misuse and unauthorized access to infor-
Multimodal streams. The sensors in Responsphere produce mation about individuals, it must be possible to enforce privacy
a large diversity of data streams that need to be processed policies already at the level of the sensor data processing
in a unified manner. Existing approaches to data stream infrastructure. However, to our knowledge, privacy issues have
processing focus mostly on a specific kind of stream: e.g., not been playing a significant role so far in sensor data stream
scalar streams as produced by motes (e.g., [1]) or event streams processing research.
as produced by RFID tag readers (e.g., [6]). Only recently, the
processing of raw media streams as produced by cameras and III. RELATED WORK
microphones have been considered as well (e.g., [5] [8] [9]). There has been extensive research in different areas of data
management in sensor networks and pervasive environment
'EvacPack is composed of a backpack with a small computer box with wifi that resulted in implementation of many systems. However,
connectivity, a pair of visualization goggles, a wireless mouse, a wearableeahotessyemhsfcudonaubtofherqi-
keyboard, and a set of sensors including gas sensors, webcam, microphone,eahotessyemhsfcudonaubtofherqi-
compass. ments described in Section II. In this section we provide a

133
brief overview of some of these systems. a user can define different queries on top of video streams
Data Stream Processing. There has been a considerable and MedSMan evaluates them in real-time. In addition to
work on data stream processing systems including Aurora [1], video streams MedSMan supports audio streams; however,
TelegraphCQ [11], and STREAM system [3]. These systems it does not provide an architecture that can be be extended
provide many features such as data model, continuous query to accommodate systems with large number of multimedia
semantic, query language, and scalable and adaptive continu- sensors.
ous query processing. However, most of these systems have IV. SATWARE ARCHITECTURE
concentrated on data streams and do not consider multimedia
streams. This section describes the basic building blocks of SATware.
Sensor Networks. Sensor networks generate high volume These building blocks, shown in Fig. 1, are organized as
of data that need to be analyzed. Much work has been done a stack of layers where each layer provides an extra level
on different aspects of data acquisition and analysis in sensor of abstraction of the sensing infrastructure. User applications
networks which expands a broad spectrum from issues such write queries regarding the pervasive space being sensed
as energy saving to query accuracy [1]. However, most of the (e.g., using a high level language such as CQL). These
work in this field *has* focused
rfo * * type ofr sensors such1
* on single a~~~~ueries are translated into a graph
qs p of operators
p where each
as motes [21]. operator performs a function on a certain stream of data
Multimedia Stream Processing. Recently there have been and produces another stream of data. Translating queries into
systems for processing multimedia streams including IBM operator graphs provides general applicability to the different
Smart Surveillance System (S3) [15], MedSMan [5], IrisNet stream data types we have, expressiveness in that other stream
[8], and Smart Camera Network [9]. These systems provide processing algebras can be mapped to this processing model,
features such as media capture, automatic feature extraction, and natural mapping to a distributed execution. The main
declaration and query language, temporal stream operators, SATware layers are SATLite, SATDeployer, and SATRuntime.
and querying algorithms. In this section we describe IBM S3 @
system and MedSMan in detail. The reader is referred to the Eir
cited references for a detailed description of the other systems.
The IBM Smart Surveillance System is a middleware that
provides video based behavior analysis. It has two main com-
ponents, Smart Surveillance Engine (SSE) and Middleware
for Large Scale Surveillance (MILS). The main functionality
of the SSE is to provide software based surveillance event
detection. It uses video analysis techniques such as object i T)4yer
detection, object tracking and object classification. The MILS
middleware provides data management services for S3. It
converts surveillance video into a relational database table. It
also provides querying capabilities on the surveillance video.
S3 also provides a privacy preserving surveillance feature that Fig. 1. SATware architecture.
can hide personal information to different degrees. Despite all
the rich functionality that it provides, the S3 system is based SATLite provides a language and a language processor for
on a centralized database that may result in scalability issues describing graphs of operators. After the operator topology
in systems with large number of video sources. Also it only has been expressed in SATLite, each operator is assigned a
considers one stream type (video). machine where the operator will be executed. The mapping
MedSMan is a live multimedia management system. It con- of operator graphs to machines and the deployment of such
sists of three main components: Stream Model, Environment operators and establishment of their connections is done by
Model, and Event Model. The stream model in MedSMan the SATDeployer. SATDeployer uses the methods provided
consists of two basic parts: media streams and feature streams. by the SATRuntime layer in order to deploy operators in
A media stream consists of tuples with sequence number, a the network. The SATRuntime layer is distributed along
time interval and media content corresponding to that interval. machines in a network (including sensors) and provides a
A feature stream consists of a sequence number, a timestamp, runtime environment where operators can be injected and
and a feature value that has occurred on that time. MedSMan executed. Through the infrastructure directory, SATRuntime
defines stream operators that can convert media streams to also provides an image of the available sensors, operators,
feature streams and vice versa. In MedSMan an event has and resources. This information is used by SATDeployer to
a start and end time. MedSMan allows a hierarchical event optimize the operator deployment. The following subsections
structure where an event can consist of a set of events, describe in more detail SATware's stream processing model,
Based on its three main components, MedSMan defines a SATRuntime, SATLite, and SATDeployer. The last subsection
query language, which is an extension of CQL [10], to handle introduces the concept of privacy in such a pervasive space
new concepts such as feature streams. Using this language, and its implications for SATware's architecture.

139
A. Sentient building
Before presenting the details of each building block, let us
first look into some details of our instrumentation and some
sample applications. This will establish some context and aid
in illustrating each of SATware's building blocks concepts.
As part of the instrumentation of the Responsphere per-
vasive space [7] we are instrumenting several buildings with
sensors. In one of such buildings, the building where our Fig. 2. A snapshot of coffee cam in smart kitchen application.
offices and labs are located, we already have about 60 cameras
(15 per floor) with microphones, an RFID reader with several
B. Stream Processing model
antennas, several MicaZ wireless sensor nodes (with temper-
ature, acceleration, light sensors, and so on), three mobile This subsection presents the multimodal stream processing
platforms with cameras and other sensors on them, and people model that enables processing of a broad diversity of sensor
counters on the doors of the building. data streams in a scalable and flexible manner. First we present
In such a heavily instrumented environment, we have im- the stream data model adopted in SATware. We then present
plemented a building surveillance application to record the our data stream processing model that provides both flexibility
data being sensed: the SATRecorder. SATRecorder allows a and scalability.
user to visualize where the sensors are located, learn their The large number of sensors producing streams of data and
current readings, and schedule recordings. It provides a GIS- the sharing of the same hardware infrastructure by several
like interface where the user can perform spatial queries with applications makes it indispensable to optimize the use of
a graphical interface. Currently, SATRecorder allows a user the infrastructure resources. The placement of each operator
to record video feeds based on a time-based schedule; in plays an important role in scalability of the system in terms
the future it will allow recording based on automatic event of bandwidth, computing, storage, and energy. For instance,
detection. consider an operator that given an image of a coffee pot
As part of the building instrumentation, we have also instru- outputs the level of coffee in the pot and an operator that
'
mented our kitchen space. Being smaller in size and close to
l~~~~~nforms the user through a GUI about the amount of coffee
our offices, our kitchen space provides us with a controllable in the coffee pot. Placing the image processing operator close
and manageable pervasive space. It is in the context of this to the video camera (i.e., in the same subnet or even in the
camera itself) and the GUI at the user's PC (in a different
coffgdevelroom, her orpo-foctalainae network) is more bandwidth efficient than placing both at the
hei doffeveloped
The Coffee Room Scenario
user's PC. Thus, being able to distribute the operators along
the network enables scalability.
Throughout the paper, we use a sample application to illus- In the coffee room scenario, we compose a query to
trate different concepts and functionalities of SATware. In this determine if a user leaves the coffee pot empty with the burner
application, a kitchen in a shared office space is instrumented on more than three times as the operator graph in Fig 3. In this
with sensors which are used to monitor the resulting "smart example, we have i) an operator (01) that gets video frames
kitchen", including the status of various appliances, in order from the video camera, ii) an operator (02) that gets id tags
to implement different policies of using a shared facility. In from the RFID reader, iii) three operators (03, 04, and 05)
particular, video sensors and RFID readers monitor a coffee that detect the status of the burner, the coffee level, and the
machine to determine policy violators. Examples of such coffee pot position, iv) three filters (06, 07, and 08) that
policies include: "warn a person who leaves the coffee pot detect the events burner switched on, coffee pot level changed
empty with the burner on more than three times", "charge to empty, and coffee pot placed on the burner, v) an operator
people for number of cups of coffee they drink", "determine (09) that detects the burning event if the coffee pot is on the
people who leave the kitchen area dirty", and so on. Fig. 2 burner, empty, and the burner is on, vi) an operator (010) that
depicts a snapshot from one of the cameras. Such events are given a person id and the burning event detects if a person has
used to realize the policies of our shared kitchen facility. let the coffee burn more than three times, and vii) an operator
Note that the coffee room scenario can be extrapolated to (011) that provides a GUI for the user.
monitoring other spaces such as buildings and airports. In 1) Data model: In SATware, data streams go through a
these cases, the events become "leaving an object unattended", variety of transformations. To be able to define SATware's
"exchanging bags", and other suspicious activity. However, processing model, we first need to define the data model for
given its simplicity and controllability, we chose to use the the streams. A stream is an infinite discrete flow of packets and
coffee room scenario for describing the different concepts in we define it as an infinite list of packets: Stream = list(packet).
SATware. In addition, our instrumented coffee room, since it A packet is a tuple of the form packet = (t, c), where t is a
is easy to control and manage, provides us with a space to test timestamp and c is the packet content. The timestamp indicates
SATware. the time that the packet content is related to; the packet content

140D
 synchronizers are nodes with more than one incoming arc.
01 04 Synchronizers synchronize/join streams depending on values
offeecam + X /such as time or location. Destination nodes are not connected
0 X to other SATware nodes but produce output intelligible for a
RFID User human user or other applications (e.g., it outputs into an XML
Tag
file or a Database).
RFIID Reader
Physical deployment. Once an application query has been
Fig. 3. A subgraph for detecting policy violation. designed as a graph of operators, it has to be instantiated in
the SATware system. Instantiating an application query means
to determine which machine executes which operator. Cost
is the data the packet carries. The content can be either data functions are also defined for every operation. These functions
of a simple type (i.e. integer, float, character, byte), a list or express the cost of the operation in such terms as bandwidth,
tuple of a simple type, or an event. memory, CPU, and so on. Based on these assumptions the
Events are more expressive than raw data and are either an mapping function decides the optimal (or good enough) way
interpretation of those or of other events. Given the inherent to deploy operators given the cost of each operator and the
inaccuracy on hardware (sensing circuits, clock synchroniza- current state of SATware in terms of deployed operators and
tion, etc.) and on software (sampling rate, approximations, availability of sensors and resources.
stochastic event detections, etc.) events always carry with
themselves a confidence level. Formally, an event is a tuple C. SATRuntime: The Stream Processing Runtime
with the following form: event = (eventID, confidence, DT).
Apart from eventID that identifies an event and confidence SATware's runtime (SATRuntime) is a reflective distributed
level, events can also carry some extra data (DT) such as runtime which meets the above requirements. It has a central
references to a stream where an event was originated from directory which contains the characteristics and state of each
or quantifiers and qualifiers for the event. of SATware's resources (sensors, machines, and network), and
2) Processing model: Streams are modified by operators. a repository of operators and virtual sensors. Each operator
The set of operators is defined as a subset of functions such is implemented as a mobile agent that can migrate to any
that every function has arguments of type DT, inputs of type of SATware's nodes. Mobile agents are autonomous software
Stream, and an output of type Stream: entities with the capability of dynamically changing their exe-
OP {fDTx ... xDT fDTx ... xDT cution environments in a network-aware fashion [22, 23]. This
Streamx ... xStream -* Stream}. adaptability allows operators to be deployed and dynamically
Virtual sensors. Operator subgraphs can be encapsulated redeployed as needed, to work in isolation when the network
into a single operator: a virtual sensor. This encapsulation is temporarily down, and to migrate operators closer to where
simplifies the design of applications, and produces quicker data is generated to optimize network resources. Moreover,
and bug-free applications. Applications are easier to design the fact that agents can be thought as reactive autonomous
since the complexity of the topologies decreases. Applica- entities that know how to perform certain actions (instead of
tions can be modularized and tested independently. Also re- just objects that encapsulate data) simplifies the application
usability is increased since i) operator topologies (and not design.
only operators) can be now reused, and ii) virtual sensors Fig. 4 depicts the system architecture. The system nodes are
can be replaced without changing the rest of the application. of three types: a) sensor nodes, b) processing nodes, and c) the
An example of a virtual sensor is: WhoLeftCoffeeBurning = directory server. Sensor nodes correspond to the heterogeneous
010(CoffeeBurning, PersonID). set of sensors in the pervasive infrastructure (e.g., Respon-
Using virtual sensor concept, SATware provides a dedicated sphere). These sensors range from RFID readers to video
notion of views for sensor data streams. The concept of cameras and, thus, the programming platform in each of them
virtual sensors is analogous to the data hiding principle in is different. Due to this heterogeneity, SATware supports two
programming languages. ways of obtaining the sensed data. The first way is to create
Operator graph. An application query can be viewed as a an agent gateway that connects to the sensor. For example,
directed graph G such that G is a weakly connected graph consider a network camera that runs a given operating system
formed by a set of vertexes V and a set of arcs A. We usually with a web server. A SATware agent can be programmed such
classify the nodes into source nodes, intermediate nodes, and that it opens an HTTP connection with the camera web-server
destination nodes. Source nodes get data from sensors-they and requests the video being captured. The second way to
transform reality (or even a simulated or a predicted reality) access the cameras is to extend the SATRuntime in the sensor
into a stream of digitalized sensed values. These usually run at node. For example, a compatible runtime (e.g., written in C
the sensor itself or on a PC in the same subnet. Intermediate or Java) is installed in the camera and a SATware agent is
nodes transform data streams. Among the intermediate nodes, injected. The first allows for faster prototyping whether the
we identify two special cases: transformers and synchronizers. second one brings the flexibility, adaptability, and robustness
Transformers are nodes that have only one incoming arc and of mobile agents all the way to the sensor.

142
 erator graphs. Stream querying and transforming are realized
by SATLite language together with SATware agents. There
are two types of SATLite users: a user that types commands
CQL 1 111-
|in SATLite language to deploy, move or terminate agent
.t MAi1111 0 rL operators on the network, and an administrative agent that
generates SATLite statements to control the running of other
agents on different machines. The former usage facilitates
the implementation of a system dashboard; the latter could
support query plan generator for higher level continuous query
languages such as CQL [10] and TelegraphCQ [11].
.........

Consider the smart kitchen scenario. Our purpose is to

detect whether someone left the coffee pot empty and burning
on the coffee machine more than three times. Let us assume
that functions that provide certain transformation and syn-
Fig. 4. SATware distributed architecture. chronization functionality already exist. For example, function
ReadCam reads a video stream from a specified camera;
function DetectBurner detects whether the burner of the coffee
The second type of nodes in SATware are the processing machine is switched on; function And does the logical and
nodes. SATRuntime is installed on each one of the processing operation on multiple boolean inputs.
nodes allowing agents to execute in each of them. SATRuntime The first step is to create an operator instance from existing
provides mobility support and message passing to SATware functions for each corresponding position on the graph. As
agents. When deploying a graph of operators SATRuntime shown in this example there may be multiple instances of a
nodes (not the agents) are explicitly connected according to single function running on different locations and/or running
the topology. This way, agents are programmed in a topology- with different inputs/outputs and parameters. So it is necessary
agnostic manner. Agents receive streams of data and output to assign an identifier for each running instance of the function.
a stream of data, without needing to know the origin or After we have deployed all the agent instances, we connect
destination of such streams. This simplifies the agent design them together using channels. A channel is a data path that
and implementation, increases re-usability, and reduces imple- has one source and one or more destinations. Channels serve
mentation time, errors, and agent size. The third component of as input and output variables for agent functions. Stream
SATware is the directory server. Mainly, the directory server compatibility check in SATLite is similar to type checking
is an interface to the central directory. The directory server in a programming language. Before such checking is issued,
stores a repository of operators and the state of SATware. agent functions (not instances) need to be registered into the
The repository of operators contains the mobile agent code directory service with their input/output stream type. Stream
that, when an agent moves into a SATRuntime node, is declaration will be checked whenever SATLite receives a
downloaded into the runtime if it had not been downloaded data flow setup statement. If compatibility is not satisfied,
there before. The state of SATware contained in the directory the statement will be rejected. Notice that by specifying the
server includes which sensors are available and how to access keyword 'any', the expected stream is compatible with any
them, which processing nodes are available and how much stream types.
resources they are offering, the network topology and its state,
and the current agent deployment. Interfaced via web services, E. SATDeployer: Operator Deployment in SATware
the directory server provides services such as insert, query, SATDeployer is the operator deployment component in
and update types (e.g., stream types, sensor types, machine SATware, responsible for deploying stream processing opera-
types, router types, operator types) and instances (e.g., sensors, tors. SATDeployer translates a query plan (topology) described
machines, routers, operators). It also provides web services in the SATLite language into a deployment plan. Apart from
for deploying operators and graphs of operators. SATRuntime the query plan, SATDeployer needs information about the
and other components such as the query processor interact current operator deployment, the available sensors, and the
with the directory server so both application and processing state of the network and machines available to SATware.
nodes constraints (e.g., bandwidth, storage) can be met and Different objectives can be defined for operator placement in
optimized. In addition to assisting on deploying applications, a distributed stream processing system. Minimizing communi-
the directory server provides a permanent storage area where cation cost, latency, or operator computation cost are some of
SATware agents can save persistent data (e.g., the encrypted the objectives that can be considered in operator deployment
automatons described in Section IV-F). for executing queries.
The main step of query deployment in SATDeployer is to
D. SATLite Stream Query and Transformwation Language map the query plan graph to the nodes in the network. The
SATLite stream query and transformation language in SAT- result of this mapping is an operator deployment graph. Each
ware provides an interface for describing and instantiating op- node of the deployment graph corresponds to a machine in the

142
SATware network and depicts the operations that should be need to pay for each cup. The goal of a surveillance application
performed in that machine. The deployment graph is formed could be to detect events that violate such policies. In general,
in such a way that the objective of the operator placement is policy-violating events are composite events, composed of
satisfied. After forming the deployment graph, SATDeployer, one or more sequences of basic events. We can denote such
using the API provided by SATRuntime, sends the operators to a composite event by a finite-state automaton where the
the corresponding nodes in the SATware network and connects transitions between its states happen on some basic event. An
them to each other according to the deployment graph. Then application will have to maintain partial information about the
the stream processing to answer the query starts. "state" (automatons) of the pervasive environment in order
Consider a query to detect people who do not pay attention to detect the occurrence of policy-violating composite events.
if the pot is burning. The query should be as follows: Therefore, the question that arises is the following: "Is it
SELECT PersonID from KitchenEvents where PotBurning = possible to design an event-detection system in a manner that
true; reveals an individual's identity (if at all) only when he violates
A possible instantiation of this query would use streams a policy applicable to him and not at any time before that".
from two sensors, an RFID reader and a camera. From the Fig. 5 shows an automaton for detecting the event when
RFID reader we obtain the identity of who is in the kitchen. an individual has had more than three cups of coffee. So
With the camera, we detect whether the coffee pot is on the and SF denote the start and the end states of the automaton
burner, the burner is on, and the coffee pot is empty. When respectively. The basic event el = < u, coffee-room, coffee-
there is someone in the kitchen and the pot is left empty on cup, exit > denotes the act of leaving the coffee-room with
the burner and the burner is on, we want to get the identity of a cup of coffee. The self-loops labeled -,el denote all other
the person who was in the kitchen. The query plan generated basic events and act as filtering edges.
by SATLite for this query is depicted in Fig 3.
After receiving the query plan from SATLite, it is the re- ,
sponsibility of SATDeployer to decide which operator should
be placed on which machine. Consider a network with three
processing nodes. One possible deployment plan is to perform W
03, .., 09 on the Processing node 1, 02 on Processing node 2,
and 010 on the Processing node 3. However, depending on the Fig. 5. Automaton for detecting that an individual has had more than three
network situation and other existing operations SATDeployer cups of coffee.
may decide to place 03,.., 09 on the Processing node 3.
Much research has been conducted on the operator place- We assume the adversary to be an inquisitive but non-
ment problem in recent years [16] [17] [18] [19]. Most of interfering (passive) entity that can observe all stored data and
the existing work in operator placement concentrates on mini- intercept all communication between two components of the
mizing communication load including bandwidth consumption system. Such an adversary is also assumed to be knowledge-
and latency and it ignores the processing load caused by able of the policies and the corresponding automatons that
performing operators in the nodes. However, in SATware implement these policies. In real life, such an adversary could
applications there are operations, such as event extraction from be an insider, like a system administrator.
video stream, that have a high processing load which cannot To keep things simple, we will restrict the notion of privacy
be ignored. simply to that of anonymity and the application semantics
The other important feature that SATware considers is the to that of implementing deterministic finite-state automatons
reuse of the existing operators in the network. Many incoming (representing composite events). We say the system ensures
queries might share some operators. In all of the existing k-anonymity of individuals if no observable pattern (of au-
operator placement systems the placement for each query is tomaton access/manipulation plus stored data/meta-data) leads
done independently, which may result in replicated execution to identifying an individual to within a set of at most k
of shared operators among queries. This results in higher individuals. The problem now is that of designing a system that
processing load on the nodes. Since SATDeployer has the can store, retrieve, and advance automatons in a k-anonymous
knowledge of the deployment graph of the existing queries, manner, i.e., where the adversary can at best attribute any event
by reusing existing operators it can considerably reduce the e in the stream to a set of k or more individuals even after
processing load in SATware. observing a very large number of events after e.
A generalization/clustering based scheme to anonymize
F: Privacy event streams for a surveillance application are discussed in
In this section we illustrate how the SATware system can detail in [24] which is easily applicable in the SATware system
be configured to implement human-centric applications in as well. But, unlike in that case, the trusted nodes in the
a privacy-preserving manner. To illustrate, let us consider SATware system will also require implementing operators that
individuals in the pervasive space have roles assigned to them can scrub data streams of identifying information in such a
such that a set of policies apply to each role, e.g., students can manner that other (untrusted) nodes are able to perform their
have at most three free cups of coffee in a day, professors will tasks on the resulting stream. For example, a video stream

143
containing an individual as well as the coffee pot might be different network protocols between SATRuntimes to cope
processed at a trusted node first which scrubs (hides) the with different types of data. Last, privacy is an important issue
human figure from the video frames (Fig 6) before passing that has to be taken into account when building pervasive
on the stream to another node that is responsible for detecting spaces. Here we provide a technical solution for preserving
whether the coffee pot is empty or full. privacy when detecting human-centric events. However, our
solution only deal with a subset of all possible problems.
Investigating further privacy preserving related issues is also
part of our future research.
REFERENCES
[1] Samuel Madden, Michael J. Franklin, Joseph M. Hellerstein, Wei Hong.
TinyDB: an acquisitional query processing system for sensor networks.,
ACM Trans. Database Syst. 30(1): 122-173 (2005).
[2] U. Srivastava, K. Munagala and J. Widom. Operator Placement for In-
Network Stream Query Processing., ACM PODS 2005.
[3] D. Abadi, D. Carney, U. Cetintemel, and et al. Aurora: A new
model and architecture for data stream management., VLDB Journal,
12(2:12039,Agust203
[4] Paolo Costa and Gian Pletro Picco and Silvana Rossetto Publish-
Subscribe on Sensor Networks: A Semi-probabilistic Approach., Pro-
ceedings of the 2nd IEEE International Conference on Mobile Adhoc
Fig. 6. Human identity anonymized on a video frame. and Sensor Systems (MASS05), Washington DC, USA, 2005.
[5] B. Liu, A. Gupta, R. Jain. MedSMan: a Streaming Data Management
System over Live Multimedia., In Proc. of the 13th ACM Intl. Conference
V. CONCLUSIONS AND FUTURE WORK on Multimedia (MULTIMEDIA 05), Singapore, Nov. 2005.
[6] Joseph M. Hellerstein, Wei Hong and Samuel R. Madden. The sensor
Sentient spaces have the potential to allow intelligent spectrum: technology, trends, and requirements., ACM SIGMOD Record
privacy-preserving surveillance. Such spaces are permeated [7] ACM SIGMOD Record, Volume 32 , Issue 4 (December 2003)
http://www.responsphere.org/
with multimodal sensors that capture the environment and the [8] http://www.intel-iris.net/research.html
activities within. In contrast to most sensor middleware preva- [9] http://wsnl.stanford.edu/index.php
[10] Arvind Arasu, Shivnath Babu, Jennifer Widom. The CQL continuous
lent today, sentient spaces' users are provided with streams query language: semantic foundations and query execution., VLDB J.
of events rather than streams of data. Events provide a higher 15(2): 121-142 (2006)
abstraction than raw data and are semantically meaningful. [11] Sirish Chandrasekaran, Owen Cooper, Amol Deshpande, Michael J.
Franklin, Joseph M. Hellerstein, Wei Hong, Sailesh Krishnamurthy,
The higher level of abstraction provided by sentient spaces Samuel Madden, Frederick Reiss and Mehul A. Shah. TelegraphCQ:
empowers users with the capabilities to monitor entities (such Continuous Datafiow Processing., SIGMOD 2003.
as spaces and individuals) without needing to know (and [12] A. Demers, J. Gehrke, M. Hong, et al. Towards Expressive Pub-
lish/Subscribe Systems., In Proc. of the 10th Intl. Conference on Ex-
deal with) the details of an underlying heterogeneous sensor tending Database Technology (EDBT 2006), Munich, Germany, March
network. 2006.
This paper presents and discusses each of the challenges [13] A. Arasu, S. Babu, J. Widom. An Abstract Semantics and Concrete Lan-
guage for Continuous Queries over Streams and Relations., Technical
involved in building a sentient space. Given that no current Report 2002-57, Stanford University, Stanford, California, 2002.
system addresses all of them, we then propose SATware: [14] B. Babcock, S. Babu, M. Datar, R. Motwani, and J. Widom. Models and
a middleware for sentient spaces. SATware's basic blocks issues in data stream systems., In Symposium on Principles of Database
Systems, pages 116, Madison,Wisconsin, May 2002.
include a distributed runtime system (SATRuntime) that per- [15] http://www.research.ibm.com/peoplevision/index.html
mits injection, execution, and interconnection of data stream [16] Z. Abrams and J. Liu. Greedy is Good: On Service Tree Placement for
processing operators, a declarative language (SATLite) for In-Network Stream Processing., IEEE ICDCS 2006.
composition of such operators, an operator deployment module [17] X. Gu, P. Yu and K. Nahrstedt.Optimal Component Composition for
Scalable Stream Processing., IEEE ICDCS 2005.
(SATDeployer), and an infrastructure directory for storing the [18] P. Pietzuch, J. Ledlie, J. Shneidman, M. Roussopoulos, M. Welsh, and
availability of resources. M. Seltzer. Network-Aware Operator Placement for Stream-Processing
Systems., IEEE ICDE 2006.
Currently, SATware is being implemented i a pervasive [19] L. Amini, N. Jain, A. Sehgal,J. Silber, 0. Verscheure.Adaptive Control
space being build in our campus. There, we have instrumented of xtremescale Stream Processing Systems., IEEE ICDCS 2006.
a shared kitchen space with a set of sensors including two D- [20] D. Carney, U. Centintemel, M. Cherniack, et al. Monitoring Streams
A New Class of Data Management Applications., In Proc. of the 28th
link wireless Internet cameras, two Linksys Internet IP web- Intl. Conference on Very Large Data Bases (VLDB 2002), Hong Kong,
cams, and an RFID reader. Communication between different China, Aug. 2002.
parts of system is done through IEEE 802.11 and 100Mbps [21] Crossbow Technology, Inc. MPR/MIB User's Manual, 2004.
[22] Karnik, N. M. and Tripathi, A. R. (1998), Design Issues in Mobile-
Ethernet LANs. Our first applications and proof-of-concept Agent Programming Systems., IEEE Concurrency, Vol. 6, No. 3, pp.
demos have been developed in that kitchen. 52-61, July-September.
As part of our on-going work,we are building larger appli- [23] Fuggetta, A., Picco, G. P. and Vigna, G. Understanding Code Mobility,
' .
cations that involve the rest of the instrumented space. Future
~~~~~~~~~IEEE
Transactions on Software Engineering, Vol. 24, No. 5, pp. 352-
361, May 1998.
work involves the research and implementation of optimization [24] "Privacy-Preserving Event Detection for Media Spaces" ICS-UCI
algorithms for operator deployment, as well as supporting technical-report, 2006.

1 44