CS-541

Wireless Sensor Networks

Lecture 8: Data models and data acquisition

Spring Semester 2016-2017

Prof Panagiotis Tsakalides, Dr Athanasia Panousopoulou, Dr Gregory Tsagkatakis

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 1
University of Crete, Computer Science Department
Multivariate based WSN models

Sensor ID
Time 10

15

20

25

30

35

40

45

50
10 20 30 40 50 60 70 80 90 100

Sampling Instance

Spatial Field
CS-541 Wireless Sensor Networks
Spring Semester 2016-2017 2
University of Crete, Computer Science Department
Sampling a WSN

Sensor
?
? ?

Time
? ? ?
? ? ?
?
? ?
? ? ?

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 3
University of Crete, Computer Science Department
Incomplete Matrices

Problem “first” appeared in Netflix challenge

• Given user-movie rating
• Guess missing entries

John Anne Scot Mark Alice

Chicago 2 5 ? ? ?
Matrix 5 ? 5 ? ?
Star wars ? ? 5 ? 1
Inception ? 3 ? 2 ?
Alien 4 1 ? ? ?
Pulp Fiction ? ? 4 ? 2

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 4
University of Crete, Computer Science Department
Matrix Rank
The rank of a matrix M is the size of the largest collection of linearly
independent columns of M (the column rank) or the size of the largest
collection of linearly independent rows of M (the row rank)
• Row Echelon Form
A matrix is in row echelon form if
(i) all nonzero rows are above any rows of all
zeroes
(ii) The leading coefficient of a nonzero row is
always strictly to the right of the leading
coefficient of the row above it

Rank=2

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 5
University of Crete, Computer Science Department
Matrix Rank
• The rank of an m × n matrix is a nonnegative integer and cannot be
greater than either m or n. That is, rank(M) ≤ min(m, n).
• A matrix that has a rank as large as possible is said to have full rank;
otherwise, the matrix is rank deficient.

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 6
University of Crete, Computer Science Department
Matrix Rank

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 7
University of Crete, Computer Science Department
Singular Value Decomposition (SVD)

Given any mn matrix M, algorithm to find matrices U, Σ, and V such

that M = U Σ VT
• U: left singular vectors (orthonormal) M  UV T

• Σ: diagonal containing singular values

m m m n V is nn
• V: right singular vectors (orthonormal)

   
    T
    s1 0 0  
   0  0  
  
M U V
  
    0 0 sn  

   
   

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 8
University of Crete, Computer Science Department
Singular Value Decomposition (SVD)

Properties
• The si are called the singular values of M
• If M is singular, some of the si will be 0
• In general rank(M) = number of nonzero si
• SVD is mostly unique (up to permutation of SV)

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 9
University of Crete, Computer Science Department
Low rank approximations

• Denoising

• Dimensionality reduction

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 10
University of Crete, Computer Science Department
Low rank approximation

Matrix norms
  mij
2
• Frobenius norm can be computed from SVD M F
i j

  si
2
• Changes to a matrix ↔ changes to singular values M F
i

Low rank approximation

Approximation problem: Find Mk of rank k such that
Mk  min M X F
X :rank ( X )  k

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 11
University of Crete, Computer Science Department
Singular Value Decomposition (SVD)

• Solution via SVD k

M  U diag ( 1 ,...,  k ,0,...,0)V T

set smallest r-k

singular values to zero

M k  i 1 i u v
k T
column notation: sum
i i
of rank 1 matrices

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 12
University of Crete, Computer Science Department
Approximation error

• How good (bad) is this approximation?

• It’s the best possible, measured by the Frobenius norm of
the error:

min M X F
 M  Mk F
  k 1
X :rank ( X )  k

where the i are ordered such that i  i+1.

Suggests why Frobenius error drops as k increased.

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 13
University of Crete, Computer Science Department
Data model
 WSN data modeling
 Spatio-temporal correlations <-> Low rank measurement matrix

Low rank

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 14
University of Crete, Computer Science Department
The case of missing values
1

Percentage of zeros
0,8
Power consumption
0,6
Packet losses
0,4
Temporal sampling of WSN
0,2
• Sampling rate
0
• De-synchronization 10 30 60 120 180

• Temporal resolution Sampling Rate (min)

1 2 3 1 2 3 1 2 3

4 5 6 4 5 6 4 5 6

7 8 9 7 8 9 7 8 9

14:15
13:00

13:30

14:00

14:30

15:00

13:45
13:00

14:00

15:00

13:00

13:15

13:30

14:00
CS-541 Wireless Sensor Networks
Spring Semester 2016-2017 15
University of Crete, Computer Science Department
Matrix completion

low rank matrix with low rank

missing entries matrix

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 16
University of Crete, Computer Science Department
 CS-541 Wireless Sensor Networks
Spring Semester 2016-2017 17
University of Crete, Computer Science Department
Matrix Completion (MC)

Let be a measurement
matrix consisting of s measurements from n different
sources.
Recovery of M is possible from k<<ns random entries
if matrix M is low rank and
To recover the unknown matrix, solve:

Rank constraint makes problem NP-hard….

CS-541 Wireless Sensor Networks
Spring Semester 2016-2017 18
University of Crete, Computer Science Department
Sampling operator

Sampling operator

• Not all low-rank matrices can be recovered from

partial measurements!
• … a matrix containing zeroes everywhere except the top-
right corner.
• This matrix is low rank, but it cannot be recovered from
knowledge of only a fraction of its entries!

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 19
University of Crete, Computer Science Department
Matrix Coherence

The coherence of subspace U of Rn and having

dimension r with respect to the canonical basis {ei} is
n
defined as:  (U )  max 1i n Uei
2

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 20
University of Crete, Computer Science Department
Formal definition of key assumptions

• Consider an underlying matrix M of size n1 by n2. Let

the SVD of M be given as follows:
r
M   k uk vkT
k 1
• We make the following assumptions about M:
(A0) 1 r /(n1n2 ) , 1  0 r

(A1) The maximum entry in the n1 by n2 matrix kk

u v
k 1
T

is upper bounded by
0 such that max(  (U ),  (V ))  0

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 21
University of Crete, Computer Science Department
What do these assumptions mean

(A0) means that the singular vectors of the matrix

are sufficiently incoherent with the canonical basis.
(A1) means that the singular vectors of the matrix
are not spiky
• canonical basis vectors are spiky signals – the spike
has magnitude 1 and the rest of the signal is 0;
• a vector of n elements with all values equal to
1/square-root(n) is not spiky.

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 22
University of Crete, Computer Science Department
What is the trace-norm of a matrix?

• The nuclear / trace norm of a matrix is the sum of

its singular values.

• It is a softened version of the rank of a matrix

• Similar to the L0 → L1-norm of a vector
• Minimization of the trace-norm is a convex
optimization problem and can be solved efficiently.
• This is similar to the L1-norm optimization (in
compressive sensing) being efficiently solvable.
CS-541 Wireless Sensor Networks
Spring Semester 2016-2017 23
University of Crete, Computer Science Department
Matrix Completion (MC)

Relaxation

* 2 (2  p ) min(n1 , n2 )
Performance M M 4   2 ,
F p
m 
where p  fraction of known entries  
n1n2 n1n2

Noisy case

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 24
University of Crete, Computer Science Department
Recovery guarantees

the trace-norm minimizer

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 25
University of Crete, Computer Science Department
 Matrix Completion solvers
• Matrix Completion via ALM
• Objective

• Reformulation

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 26
University of Crete, Computer Science Department
Matrix Completion solvers

• Let y=A(M)

• Iterative Hard Thresholding

SVD

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 27
University of Crete, Computer Science Department
 CS and MC
Sparse recovery Rank minimization
Unknown Vector x Matrix A

Observations y = Ax y = L[A] (linear map)

Combinatorial
objective

Convex
relaxation

Algorithmic Linear programming Semidefinite programming

tools

Yi Ma et al, “Matrix Extensions to

Sparse Recovery”, CVPR2009
CS-541 Wireless Sensor Networks
Spring Semester 2016-2017 28
University of Crete, Computer Science Department
Applications of MC

 Recommendation systems
• Matrix (user, preference/quality/intention)
 Sensor localization
• Matrix (location, physical quantity)
 Data recovery in Wireless Sensor Networks
• Matrix (sensor, time)

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 29
University of Crete, Computer Science Department
Data Gathering

• STCDG: An Efficient Data Gathering Algorithm Based on Matrix

Completion for Wireless Sensor Networks

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 30
University of Crete, Computer Science Department
 CS-541 Wireless Sensor Networks
Spring Semester 2016-2017 31
University of Crete, Computer Science Department
Body sensor network

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 32
University of Crete, Computer Science Department
 RTT estimation

Decentralized Matrix Factorization by Stochastic Gradient Descent (DMFSGD),

Estimation of end-to-end network distances
• Network nodes exchange messages with each other
• Each node collects and processes local measurements

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 33
University of Crete, Computer Science Department
 CS-541 Wireless Sensor Networks
Spring Semester 2016-2017
University of Crete, Computer Science Department 34
 CS-541 Wireless Sensor Networks
Spring Semester 2016-2017
University of Crete, Computer Science Department 35
 LQM Estimation

Estimation Error
Sensor Node

Sensor Node

Effect of shadowing (in dB)

Grid location Grid location

Estimation Error
• Dataset
• Testbed @ FORTH (144m2, 1x1m grid)
• RSSI values (channel quality)
• 13 IEEE802.11b/g channels
Number of nodes
CS-541 Wireless Sensor Networks
Spring Semester 2016-2017 36
University of Crete, Computer Science Department
 Robust PCA

missing + low rank sparse

corrupted matrix corruptions
entries

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 37
University of Crete, Computer Science Department
1-Bit MC

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 38
University of Crete, Computer Science Department
Distributed vs. Centralized Storage
A A

•Centralized

Time
Time
 Access to resources
B
 Controlled environment B

Time
• Decentralized

Time
 Increased network lifetime
 Autonomy A B C D A B C D
Time

Time
Performance comparison
• Per sensor vs. collective
• Temporal resolution
CS-541 Wireless Sensor Networks
Spring Semester 2016-2017 39
University of Crete, Computer Science Department
 9 10
10 10
matrix [50] x [72] - data per hour Sensor 1 - Single Sensor Recovery
matrix [50] x [36] - data per 2 hours Sensor 1 - Collective Sensor Recovery
matrix [50] x [18] - data per 4 hours Sensor 2 - Single Sensor Recovery
9
matrix [50] x [9] - data per 8 hours 10 Sensor 2 - Collective Sensor Recovery
8
10
MSE

MSE
8
10

7
10
7
10

6 6
10 10
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Fill Ratio Fill Ratio
10 9
10 10
Sensor 1: [50] x [72] Sensor 2: [50] x [72]
Sensor 1: [50] x [36] Sensor 2: [50] x [36]
Sensor 1: [50] x [18] Sensor 2: [50] x [18]
9
10 Sensor 1: [50] x [9] Sensor 2: [50] x [9]
8
10
MSE

MSE
8
10

7
10
7
10

6 6
10 10
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Fill Ratio Fill Ratio
CS-541 Wireless Sensor Networks 40
Spring Semester 2016-2017
University of Crete, Computer Science Department
Tensors analysis

Includes materials from: Introduction to tensor, tensor factorization and its

applications, by Mu Li, iPAL Group Meeting, Sept. 17, 2010
CS-541 Wireless Sensor Networks
Spring Semester 2016-2017 41
University of Crete, Computer Science Department
Fiber and slice

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 42
University of Crete, Computer Science Department
Tensor unfoldings: Matricization and
vectorization
• Matricization: convert a tensor to a matrix

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 43
University of Crete, Computer Science Department
 Time factor
Tensor factorization

V
Day factor
Hours
[1-24] X ≈ U
Day [1-365]
Location factor
Location [1-100]

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 44
University of Crete, Computer Science Department
Tensor factorization

• CANDECOMP/PARAFAC factorization (CP)

• Rank 1 Tensor models

• Tensor factorization: an extension of SVD and PCA of matrix.

• CP factorization:

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 45
University of Crete, Computer Science Department
Tensor completion
• Similar to the matrix case

• Tensor nuclear norm

• Nuclear norm minimization

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 46
University of Crete, Computer Science Department
Applications in Sensor Network Data

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 47
University of Crete, Computer Science Department
 Problem statement
 Network: undirected, connected graph

? ?
? ?
? ? n
?
? ? ?

Goal: Given per node and single-hop exchanges, find

(P1)

 Challenges
 Nuclear norm is not separable
Mardani M. et al. , “Distributed
 Global optimization variable Nuclear Norm Minimization
for Matrix Completion” SPAWC 2012

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 48
University of Crete, Computer Science Department
 Separable regularization
 Key result [Recht et al’11]

Lxρ

≥rank[X]
 New formulation equivalent to (P1)

(P2)

 Nonconvex; reduces complexity:

Proposition 1. If stationary pt. of (P2) and ,

then is a global optimum of (P1).

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 49
University of Crete, Computer Science Department
 Distributed estimator

(P3)
Consensus with
neighboring nodes

 Network connectivity (P2) (P3)

 Alternating-directions method of multipliers (ADMM) solver

 Method [Glowinski-Marrocco’75], [Gabay-Mercier’76]
 Learning over networks [Schizas et al’07]

 Primal variables per agent :

n
 Message passing:

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 50
University of Crete, Computer Science Department
 Real data
 Network distance prediction [Liau et al’12]

 Abilene network data (Aug 18-22,2011)

 End-to-end latency matrix
 N=9, L=T=N
 80% missing data

Relative error: 10%

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 51
University of Crete, Computer Science Department
Reading material

Keshavan, Raghunandan H., Andrea Montanari, and Sewoong Oh.

"Low-rank matrix completion with noisy observations: a quantitative
comparison.“ Communication, Control, and Computing, 2009. Allerton
2009. 47th Annual Allerton Conference on. IEEE, 2009.
Cheng, Jie, et al. "STCDG: An efficient data gathering algorithm based
on matrix completion for wireless sensor networks." Wireless
Communications, IEEE Transactions on 12.2 (2013): 850-861.
Gürsun, Gonca, and Mark Crovella. "On traffic matrix completion in the
internet." In Proceedings of the 2012 ACM conference on Internet
measurement conference, pp. 399-412. ACM, 2012.
G. Tzagkarakis, G. Tsagkatakis, D. Alonso, E. Celada, C. Asensio, A.
Panousopoulou, P. Tsakalides, B. Beferull-Lozano, “Signal and Data
Processing Techniques for Industrial Cyber-Physical Systems”, 2015

CS-541 Wireless Sensor Networks

Spring Semester 2016-2017 52
University of Crete, Computer Science Department