PROXSCAL

PROXSCAL performs multidimensional scaling of proximity data to find a least-

squares representation of the objects in a low-dimensional space. Individual
differences models can be specified for multiple sources. A majorization algorithm
guarantees monotone convergence for optionally transformed, metric and
nonmetric data under a variety of models and constraints.
Detailled mathematical derivations concerning the algorithm can be found in
Commandeur and Heiser (1993).

Notation
The following notation is used throughout this chapter, unless stated otherwise.
For the dimensions of the vectors and matrices are used:

n Number of objects
m Number of sources
p Number of dimensions
s Number of independent variables
h maximum(s, p)
l Length of transformation vector
r Degree of spline
t Number of interior knots for spline

The input and input-related variables are:

∆k n × n matrix with raw proximities for source k

Wk n × n matrix with weights for source k

E n × s matrix with raw independent variables

F n × p matrix with fixed coordinates

Output and output-related variables are:

1
2

ˆ
D n × n matrix with transformed proximities for source k
k

Z n × p matrix with common space coordinates

Ak p × p matrix with space weights for source k

Xk n × p matrix with individual space coordinates for source k

Q n × h matrix with transformed independent variables

B h × p matrix with regression weights for independent variables

S l × (r + t ) matrix of coefficients for the spline basis

Special matrices and functions are:

J I − 11T / 1T1 , centering matrix of appropriate size

D ( Xk ) n × n matrix with distances, with elements {dijk } ,

where d ijk = (x ik − x jk )( xik − x jk )

 − wijk for i ≠ j

n × n matrix with elements {vijk } , where vijk = n

∑
Vk
 wilk for i = j
 l ≠i

B ( Xk ) n × n × m matrix with elements {bijk } , where

 − wijk f (δ ijk )
 if dij ( X k ) > 0 and i ≠ j
 d ij ( X k )

bijk = 0 if d ij ( X k ) = 0 and i ≠ j
 n
− b
 ∑ ilk if i = j
l ≠i

Introduction
The following loss function is minimized by PROXSCAL,
1 m n
σ 2 ≡ ∑∑ wijk  dˆijk − dij ( X k )  ,
2
(1.1)
m k =1 i < j

which is the weighted mean squared error between the transformed proximities
and the distances of n object within m sources. The transformation function for the
 3

proximities provides nonnegative, monotonically nondecreasing values for the

transformed proximities dˆijk . The distances dij ( X k ) are simply the Euclidean
distances between the object points, with the coordinates in the rows of Xk .
The main algorithm consists of the following major steps:
1. find initial configurations X k , and evaluate the loss function;
2. find an update for the configurations Xk ;

3. find an update for the transformed proximities dˆijk ;

4. evaluate the loss function; if some predefined stop criterion is satisfied, stop;
otherwise, go to step 2.

Preliminaries
At the start of the procedure, several preliminary computations are performed to
handle missing weights or proximities, and initialize the raw proximities.

Missings
On input, missing values may occur for both weights and proximities. If a weight
is missing, it is set equal to zero. If a proximity is missing, the corresponding
weight is set equal to zero.

Proximities
Only the upper or lower triangular part (without the diagonal) of the proximity
matrix is needed. In case both triangles are given, the weighted mean of both
triangles is used. Next, the raw proximities are transformed such that similarities
become dissimilarities by multiplying with −1 , taking into account the
conditionality, and setting the smallest dissimilarity equal to zero.

Transformations
For ordinal transformations, the nonmissing proximities are replaced by their
ascending rank numbers, also taking into account the conditionality. For spline
transformations, the spline basis S is computed.
4

Normalization
The proximities are normalized such that the weighted squared proximities equal
the sum of the weights, again, taking into account the conditionality.

Step 1: Initial Configuration

PROXSCAL allows for several initial configurations. Before determining the
initial configuration, missings are handled, and the raw proximities are initialized.
Finally, after one of the starts described below, the common space Z is centered
on the origin and optimally dilated in accordance with the normalized proximities.

Simplex Start

The simplex start consists of a rank-p approximation of the matrix V − B ( J ) . Set

H , an n × p columnwise orthogonal matrix, satisfying H T H = I p equal to I p ,
where I p denotes the matrix with the first p columns of the identity matrix. The
nonzero rows are selected in such a way that the first Z = B(J )H contains the p
columns of B(J ) with the largest diagonal elements. The following steps are
computed in turn, until convergence is reached:
1. For a fixed Z , H = PQ T , where PQ T is taken from the singular value
decomposition B(J )Z = PLQ T ;

2. For a fixed H , Z = 2−1/ 2 V − B (J )H , where V − is the pseudo-inverse of V .

For a restricted common space Z , the second step is adjusted in order to fullfill
the restictions. This procedure was introduced in Heiser (1985).

Torgerson Start
The proximities are aggregated over sources, squared, double centered and
multiplied with −0.5 , after which an eigenvalue decomposition is used to
determine the coordinate values, thus
−0.5JD* J = Q 4 T ,

where elements of D* are defined as

−1
 m  m 
dij* =  ∑ wijk dˆijk2   ∑ wijk 
 k =1   k =1 
 5

followed by Z = Q 1/ 2 , where only the first p positive ordered eigenvalues

( λ1 ≥ λ2 ≥ K ≥ λn ) and eigenvectors are used. This technique, classical scaling, is
due to Torgerson (1952, 1958) and Gower (1966) and also known under the names
Torgerson scaling or Torgerson-Gower scaling.

(Multiple) Random Start

The coordinate values are randomly generated from a uniform distribution using
the default random number generator from the SPSS system.

User-Provided Start
The coordinate values provided by the user are used.

Step 2: Configuration Update

Update for the Common Space

The common space Z is related to the individual spaces X k ( k = 1,..., m ) through

the model X k = ZA k , where A k are matrices containing space weights.
Assume that weight matrix A k is of full rank. Only considering Z defines (1.1) as
σ 2 ( z ) = c + z T Hz − 2z T t, (3.1)

where
z ≡ vec ( Z ) ,

∑ ( A k A Tk ⊗ Vk ),
m
1
H≡ (3.2)
m k =1
1 m 
t ≡ vec  ∑ B ( X k ) X k A kT  ,
 k =1
m 
for which a solution is found as
z = H−t (3.3)
Several special cases exist for which (3.3) can be simplified. First, the weights
matrices Wk may all be equal, or even all equal to one. In these cases H will
simplify, as will the pseudo-inverse of H . Another simplification is concerned
with the different models, reflected in restrictions for the space weights. Equation
6

(3.3) provides a solution for Z , where A k is of full rank. This model is the
generalized Euclidean model, also known as IDIOSCAL (Carroll and Chang,
1972). The weighted Euclidean model, or INDSCAL, restricts A k to be diagonal,
which does simplify H , but not the pseudo-inverse. The identity model requires
A k = I for all k, and does simplify H and its pseudo-inverse, for the kronecker
product ⊗ vanishes.
To avoid computing the pseudo-inverse of a large matrix, PROXSCAL uses
three technical simplifications when appropriate. First, the pseudo-inverse can be
replaced by a proper inverse by adding the nullspace, taking the proper inverse and
then subtracting the nullspace again as
H− = ( H + N) − N
−1

where N = (11T ) / (1T1) .

Furthermore, a dimensionwise approach (Heiser and Stoop, 1986) is used
which results in a solution for dimension a of Z as
z a = Va− z a ,

where
1 m
Va = ∑ Vk eTa A k A Tk ea ,
m k =1
where e a is the a-th column of an identity matrix, and
1 m
za = ∑ B(Xk )Xk A Tk − Vk Pa A k ATk  ea ,
m k =1 

with Pa an n × p matrix equal to Z , but with the a-th column containing zeros.
Still, the proper inverse of a n × n matrix is required. The final simplification is
concerned with a majorization function in which the largest eigenvalue of V
allows for an easy update (Heiser, 1987; Groenen, Heiser, and Meulman, 1999).
Instead of the largest eigenvalue itself, an upper bound is used for this scalar
(Wolkowicz and Styan, 1980).

Update for the Space Weights

An update for the space weights A k ( k = 1,..., m ) for the generalized Euclidean
model is given by
A k = ( ZT Vk Z ) (Z B (X ) X ) .
-1 T
k k (3.4)
 7

Suppose, Pk L k QTk is the singular value decomposition of A k , for which the

diagonal matrix with singular values L k is in nonincreasing order. Then, for the
reduced rank model, the best r ( r < p ) rank approximation of A k is given by
R k TkT , where R k contains the first r columns of Pk L k , and Tk contains the first
r columns of Q k .

For the weighted Euclidean model, (3.4) reduces to a diagonal matrix

A k = diag ( Z T Vk Z ) diag ( Z T B ( X k ) X k ) .
-1

The space weights for the identity model need no update, since A k = I for all k.
Simplifications can be obtained if all weights W are equal to one and for the
reduced rank model, which can be done in r dimensions, as explained in Heiser
and Stoop (1986).

Restrictions

Fixed coordinates

If some of the coordinates of Z are fixed by the user, then only the free
coordinates of Z need to be updated. The dimensionwise approach is taken one
step further, which results in an update for object i on dimension a as
1 1 m 1 p  m   1
zia+ = T eiT  ∑ B( X k ) X k A Tk e a − ∑  ∑ eTj A k A Tk e a Vk z j  − T eiT Va z% ia
ei Va ei  m k =1 m j ≠ a  k =1   ei Va ei

where the ath column of Z is divided into z a = z% ia + zia ei , with ei the ith column of
1 m T
the identity matrix, and Va = ∑
m k =1
e j A k A Tk e a Vk .

This update procedure will only locally minimize (3.1) and repeatedly cycling
through all free coordinates until convergence is reached, will provide global
optimization. After all free coordinates have been updated, Z is centered on the
origin. On output, the configuration is adapted as to coincide with the initial fixed
coordinates.

Independent variables

Independent variables Q are used to express the coordinates of the common space
Z as a weighted sum of these independent variables as
8

h
Z = QB = ∑ q j b Tj .
j =1

An update for Z is found by performing the following calculations for j = 1,..., h :

h
1. U j = ∑ q k b Tk
k≠ j

1 m 1 m
2. Tj = C − ∑
m k =1
Vk U j A k A T
k , where C = ∑
m k =1
B ( Xk ) Xk A Tk

−1
1 m T
3. update b j as b j =  ∑ q j Vk q j A k A k  Tj q j
T

 m k =1 
4. optionally, compute optimally transformed variables by regressing
1  1  1 m
q% j = Tj b j +  I − V j  q j , where V j = ∑ b Tj A k A kTb j Vk and
k1  k1  m k =1
k1 is greater than or equal to the largest eigenvalue of V j , on the original
variable q j . Missing elements in the original variable are replaced with the
corresponding values from q% j .
h
Finally, set Z = QB = ∑ q j b Tj .
j =1

Independent variables restrictions were introduced for the MDS model in Bentler
and Weeks (1978), Bloxom (1978), de Leeuw and Heiser (1980) and Meulman
and Heiser (1984). If there are more dimensions (p) than independent variables (s),
p-s dummy variables are created and treated completely free in the analysis. The
transformations for the independent variables from Step 4 are identical to the
transformations of the proximities, except that the nonnegativety constraint does
not apply. After transformation, the variables q are centered on the origin,
normalized on n, and the reverse normalization is applied to the regression weights
b.
 9

Step 3: Transformation Update

Conditionality
Two types of conditionalities exist in PROXSCAL. Conditionality refers to the
possible comparison of proximities in the transformation step. For unconditional
transformations, all proximities are allowed to be compared with each other,
irrespective of the source. Matrix-conditional transformations only allow for
comparison of proximities within one matrix k, in PROXSCAL refered to as one
source k. Here, the transformation is computed for each source seperately (thus m
times).

Transformation Functions
All transformation functions in PROXSCAL result in nonnegative values for the
transformed proximities. After the transformation, the transformed proximities are
normalized and the common space is optimally dilated accordingly. The following
transformations are available.

Ratio

ˆ =
D . No transformation is necessary, since the scale of D̂ is adjusted in the
normalization step.

Interval

ˆ = α + β . Both α and β are computed using linear regression, in such a

D
way that both parameters are nonnegative.

Ordinal

ˆ = WMON ( , : ) . Weighted monotone regression (WMON) is computed

D
using the up-and-down-blocks minimum violators algorithm (Kruskal, 1964;
Barlow et al., 1972). For the secondary approach to ties, ties are kept tied, the
proximities within tieblocks are first contracted and expanded afterwards.
10

Spline

( )
ˆ = Sb . PROXSCAL uses monotone spline transformations (Ramsay,
vec D
1988). In this case, the spline transformation gives a smooth nondecreasing
piecewise polynomial transformation. It is computed as a weighted regression of
D on the spline basis S . Regression weights b are restricted to be nonnegative
and computed using nonnegative alternating least squares (Groenen, van Os and
Meulman, 2000).

Normalization
After transformation, the transformed proximities are normalized such that the
sum-of-squares of the weighted transformed proximities are equal to mn(n-1)/2 in
the unconditional case and equal to n(n-1)/2 in the matrix-conditional case.

Step 4: Termination
After evaluation of the loss function, the old function value and new function
values are used to decide whether iterations should continue. If the new function
value is smaller than or equal to the minimum Stress value MINSTRESS, provided
by the user, iterations are terminated. Also, if the difference in consecutive Stress
values is smaller than or equal to the convergence criterion DIFFSTRESS,
provided by the user, iterations are terminated. Finally, iterations are terminated if
the current number of iterations, exceeds the maximum number of iterations
MAXITER, also provided by the user. In all other cases, iterations continue.

Remaining Issues
Acceleration
For the identity model without further restictions, the common space can be
updated with acceleration as Z new = 2Z update − Z old , also refered to as the
relaxed update.

Lowering dimensionality
For a restart in p-1 dimensions, the p-1 most important dimensions need to be
identified. For the identity model, the first p-1 principal axes are used. For the
 11

weighted Euclidean model, the p-1 most important space weights are used, and for
the generalized Euclidean and reduced rank models, the p-1 largest singular values
of the space weights determine the remaining dimensions.

Stress measures
The following statistics are used for the computation of the Stress measures:

( )
m n
ˆ = ∑∑ w dˆ 2
η2 D ijk ijk
k =1 i < j

( )
m n
ˆ = ∑∑ w dˆ 4
η4 D ijk ijk
k =1 i < j
m n
η ( X ) = ∑∑ wijk d ij2 ( Xk )
2

k =1 i < j
m n
η 4 ( X ) = ∑∑ wijk d ij4 ( Xk )
k =1 i < j
m n
ρ ( X ) = ∑∑ wijk dˆijk dij ( X k )
k =1 i < j
m n
ρ 2 ( X ) = ∑∑ wijk dˆijk2 d ij2 ( Xk )
k =1 i < j
2

κ 2 ( X ) = ∑∑ wijk ( d ij ( Xk ) − d ( X ) )
m n

k =1 i < j

where d ( X ) is the average distance.

The loss function minimized by PROXSCAL, normalized raw Stress, is given by:

σ = 2
( )
ˆ + η 2 (α X ) − 2 ρ (α X )
η2 D
α=
ρ ( X)
( )
, with .
η D
ˆ 2
η 2 ( X)

Note that at a local minimum of X , α is equal to one. The other Fit and Stress
measures provided by PROXSCAL are given by:

Stress-I:
( )
ˆ + η 2 (α X ) − 2 ρ (α X )
η2 D
, with α=
( ).
η2 D
ˆ
η (α X )
2
ρ ( X)
12

Stress-II:
( )
ˆ + η 2 (α X ) − 2 ρ (α X )
η2 D
, with α=
( ).
η2 D
ˆ
κ (α X )
2
ρ ( X)

ˆ + η 4 (α X ) − 2 ρ 2 (α X ) , with α 2 = ρ ( X ) .
( )
2

S-Stress: η4 D
η 4 ( X)
Dispersion Accounted For (DAF): 1 − σ .
2

Tucker’s coefficient of congruence: 1-σ 2 .

Decomposition of normalized raw Stress

Each part of normalized raw Stress, as described before, is assigned to objects and
sources. Either sum over objects or sum over sources are equal to total normalized
raw Stress.

Transformations on output
On output, whenever fixed coordinates or independent variables do not apply, the
models are not unique. In these cases transformations of the common space and
the space weights are in order.
For the identity model, the common space Z is rotated to principal axes. For the
Z = nZ ( diag Z T Z )
−1/ 2
weighted Euclidean model, so that

diag ( Z T Z ) = nI , and reverse tranformations are applied to the space weights

A k . Further, the sum over sources of the squared space weights are put in
descending order as to specify the importance of the dimensions. For the
generalized Euclidean model, the Cholesky decomposition Z T Z = LLT specifies
Z = n Z ( LT ) , so that Z T Z = nI .
−1
the common space on output as
 13

References
Barlow, R. E., Bortholomew, D. J., Bremner, J. M. and Brunk, H. D. (1972).
Statistical inference under order restrictions. New York: Wiley.

Bentler, P. M. and Weeks, D. G. (1978). Restricted multidimensional scaling

models. Journal of Mathematical Psychology, 17, 138-151.

Bloxom, B. (1978). Constrained multidimensional scaling in n spaces.

Psychometrika, 43, 397-408.

Carroll, J. D. and Chang, J. J. (1972, March). IDIOSCAL (Individual Differences In

Orientation SCALing): A generalization of INDSCAL allowing idiosyncratic
reference systems as well as analytic approximation to INDSCAL. Paper presented
at the spring meeting of the Psychometric Society. Princeton, NJ.

Commandeur, J. J. F. and Heiser, W. J. (1993). Mathematical derivations in the

proximity scaling (PROXSCAL) of symmetric data matrices (Tech. Rep. No. RR-
93-03). Leiden, The Netherlands: Department of Data Theory, Leiden University.

De Leeuw, J. (1977). Applications of convex analysis to multidimensional scaling.

In J. R. Barra, F. Brodeau, G. Romier, and B. van Cutsem (Eds.), Recent
developments in statistics (pp. 133-145). Amsterdam, The Netherlands: North-
Holland.

De Leeuw, J. and Heiser, W. J. (1980). Multidimensional scaling with restrictions

on the configuration. In P. R. Krishnaiah (Ed.), Multivariate analysis (Vol. V, pp.
501-522). Amsterdam, The Netherlands: North-Holland.

Gower, J. C. (1966). Some distance properties of latent root and vector methods
used in multivariate analysis. Biometrika, 53, 325-338.

Groenen, P. J. F., Heiser, W. J. and Meulman, J. J. (1999). Global optimization in

least squares multidimensional scaling by distance smoothing. Journal of
Classification, 16, 225-254.

Groenen, P. J. F., van Os, B. and Meulman, J. J. (2000). Optimal scaling by

alternating length-constained nonnegative least squares, with application to
distance-based analysis. Psychometrika, 65, 511-524.
14

Heiser, W. J. (1985). A general MDS initialization procedure using the SMACOF

algorithm-model with constraints (Tech. Rep. No. RR-85-23). Leiden, The
Netherlands: Department of Data Theory, Leiden University.

Heiser, W. J. (1987). Joint ordination of species and sites: The unfolding

technique. In P. Legendre and L. Legendre (Eds.), Developments in numerical
ecology (pp. 189-221). Berlin, Heidelberg: Springer-Verlag.

Heiser, W. J. and De Leeuw, J. (1986). SMACOF-I (Tech. Rep. No. UG-86-02).

Leiden, The Netherlands: Department of Data Theory, Leiden University.

Heiser, W. J. and Stoop, I. (1986). Explicit SMACOF algorithms for individual

differences scaling (Tech. Rep. No. RR-86-14). Leiden, The Netherlands:
Department of Data Theory, Leiden University.

Kruskal, J. B. (1964). Nonmetric multidimensional scaling: A numerical method.

Psychometrika, 29, 28-42.

Meulman, J. J. and Heiser, W. J. (1984). Constrained Multidimensional Scaling:

more Directions than Dimensions. In T. Havranek et al (Eds.), COMPSTAT 1984
(pp. 137-142). Wien: Physica Verlag.

Ramsay, J. O. (1988). Monotone regression splines in action. Statistical Science,

3(4), 425-461.

Stoop, I., Heiser, W. J. and De Leeuw, J. (1981). How to use SMACOF-IA. Leiden,
The Netherlands: Department of Data Theory, Leiden University.

Stoop, I. and De Leeuw, J. (1982). How to use SMACOF-IB. Leiden, The

Netherlands: Department of Data Theory, Leiden University.

Torgerson, W. S. (1952). Multidimensional scaling: I. Theory and method.

Psychometrika, 17, 401-419.

Torgerson, W. S. (1958). Theory and methods of scaling. New York: Wiley.

Wolkowicz, H. and Styan, G. P. H. (1980). Bounds for eigenvalues using traces.

Linear algebra and its applications, 29, 471-506.