714 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO.

3, MAY 1997

Supervised Neural Networks for the

Classification of Structures
Alessandro Sperduti and Antonina Starita, Member, IEEE

Abstract—Until now neural networks have been used for clas-

sifying unstructured patterns and sequences. However, standard
neural networks and statistical methods are usually believed
to be inadequate when dealing with complex structures be-
cause of their feature-based approach. In fact, feature-based
approaches usually fail to give satisfactory solutions because of
the sensitivity of the approach to the a priori selection of the
features, and the incapacity to represent any specific information
on the relationships among the components of the structures.
However, we show that neural networks can, in fact, represent
and classify structured patterns. The key idea underpinning
our approach is the use of the so called “generalized recursive
neuron,” which is essentially a generalization to structures of
a recurrent neuron. By using generalized recursive neurons,
all the supervised networks developed for the classification of
sequences, such as backpropagation through time networks, real-
time recurrent networks, simple recurrent networks, recurrent
cascade correlation networks, and neural trees can, on the whole,
be generalized to structures. The results obtained by some of
the above networks (with generalized recursive neurons) on the
classification of logic terms are presented.
Index Terms— Classification of graphs, gradient methods,
graph theory, learning systems, recurrent neural networks,
structured domains, supervised classification.

I. INTRODUCTION

S TRUCTURED domains are characterized by complex

patterns which are usually represented as lists, trees,
and graphs of variable sizes and complexity. The ability
to recognize and classify these patterns is fundamental for
several applications, such as medical and technical diag-
noses (discovery and manipulation of structured dependencies, Fig. 1. Example of a conceptual graph encoding a medical concept.
constraints, explanations), molecular biology and chemistry
[classification of chemical structures, DNA analysis, quan-
While neural networks are able to classify static information
titative structure-property relationship (QSPR), quantitative
or temporal sequences, the current state of the art does not
structure-activity relationship (QSAR)], automated reasoning
allow the efficient classification of structures of different sizes.
(robust matching, manipulation of logical terms, proof plans,
Some advances in this area have recently been presented
search space reduction), software engineering (quality testing,
in [33], where preliminary work on the classification of
modularization of software), geometrical and spatial reasoning
logical terms, represented as directed labeled graphs, was
(robotics, structured representation of objects in space, figure
reported. The aim, in that case, was the realization of a
animation, layouting of objects), speech and text processing
hybrid (symbolic/connectionist) theorem prover, where the
(robust parsing, semantic disambiguation, organizing and find-
connectionist part had to learn a heuristic for a specific domain
ing structure in texts and speech), and other applications that
in order to speed-up the search for a proof. Other related
use, generate or manipulate structures.
techniques for dealing with structured patterns can be found,
for example, in [21] and [27]–[29].
Manuscript received February 5, 1996; revised August 12, 1996 and The aim of this paper is to show, at least in principle, that
December 22, 1996. neural networks can deal with structured domains. Some basic
The authors are with the University of Pisa, Dipartimento di Informatica,
56125 Pisa, Italy. concepts, which we believe are very useful for expanding the
Publisher Item Identifier S 1045-9227(97)02749-5. computational capabilities of neural networks to structured do-
1045–9227/97$10.00  1997 IEEE

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
SPERDUTI AND STARITA: SUPERVISED NEURAL NETWORKS 715

domains. However, we will show that neural networks can, in

fact, represent and classify structured patterns.

A. Preliminaries
Graphs: We consider finite directed vertex labeled graphs
without multiple edges. For a set of labels , a graph (over
) is specified by a finite set of vertices, a set of
ordered couples of (called the set of edges), and
a function from to (called the labeling function).
Note that graphs may have loops, and labels are not restricted
to being binary. Specifically, labels may also be real-valued
vectors. A graph (over ) is a subgraph of if
, and . For a finite set # denotes
its cardinality. Given a graph and any vertex , the
function returns the number of edges leaving
from , i.e.,
, while the function returns the number of
edges entering , i.e.,
. Given a total order on the edges leaving from
, the vertex in is the vertex pointed by
Fig. 2. Chemical structures represented as trees. the th pointer leaving from . The valence of a graph
is defined as . A labeled directed
mains, are given. Specifically, we propose a generalization of acyclic graph (labeled DAG) is a graph, as defined above,
a recurrent neuron—the generalized recursive neuron—which without loops. A vertex is called a supersource for
is able to build a map from a domain of structures to the set if every vertex in can be reached by a path starting from .
of reals. This newly defined neuron allows the formalization The root of a tree (which is a special case of a directed graph)
of several supervised models for structures which stem very is always the (unique) supersource of the tree.
naturally from well known models, such as backpropagation Structured Domain, Target Function, and Training Set: We
through time networks, real-time recurrent networks, simple define a structured domain (over ) as any (possibly
recurrent networks, recurrent cascade correlation networks, infinite) set of graphs (over ). The valence of a domain
and neural trees. is defined as the maximum among the out-degrees of the
This paper is organized as follows. In Section II we intro- graphs belonging to . Since we are dealing with learning,
duce structured domains and some preliminary concepts on we need to define the target function we want to learn. In
graphs and neural networks. The general problem of how approximation tasks, a target function over is defined
to encode labeled graphs for classification is discussed in as any function , where is the output dimension,
Section III. The generalized recursive neurons are defined in while in (binary) classification tasks we have
Section IV, where some related concepts are discussed. Sev- (or .) A training set on a domain is
eral supervised learning algorithms, derived by standard and defined as a set of couples , where and
well-known learning techniques, are presented in Section V. is a target function defined on .
Simulation results for a subset of the proposed algorithms are Standard and Recurrent Neurons: The output of a
reported in Section VI, and conclusions drawn in Section VII. standard neuron is given by

II. STRUCTURED DOMAINS (1)

This paper deals with structured patterns which can be
represented as directed labeled graphs. Some examples of these
where is some nonlinear squashing function applied to
kinds of patterns are shown in Figs. 1–4, which report some
the weighted sum of inputs .1 A recurrent neuron with a
typical examples from structured domains such as conceptual
single self-recurrent connection, on the other hand, computes
graphs representing medical concepts (Fig. 1), chemical struc-
its output as follows:
tures (Fig. 2), bubble chamber events (Fig. 3), and logical
terms (Fig. 4). All these structures can also be represented
by using a feature-based approach. Feature-based approaches, (2)
however, are very sensitive to the features selected for the
representation and are inable to represent any specific infor-
where is applied to the weighted sum of inputs plus the
mation about the relationships among components. Standard
self-weight, , times the previous output. The above formula
neural networks, as well as statistical methods, are usually
bound to this kind of representation and are thus generally 1 The threshold of the neuron is included in the weight vector by expanding
not considered as being adequate for dealing with structured the input vector with a component always equal to one.

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
716 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 3, MAY 1997

(a) (b)
Fig. 3. Bubble chamber events: (a) coded events and (b) corresponding tree representation.

This can be done by implementing the encoding process

through an additional neural network which is trained, along-
side the neural network performing the classification, to learn
the best way to encode the graphs for the given classification
task. Unfortunately, standard neurons as well as recurrent
neurons are not powerful enough to be used in such a network.
This is because the former were devised for processing un-
structured patterns, while the latter can only naturally process
sequences. In the next section we thus need to define a
generalization of recurrent neurons which allow us to formally
Fig. 4. Example of logic terms represented as labeled directed acyclic
define how labeled directed graphs can be encoded by a neural
graphs. network.

IV. THE FIRST-ORDER GENERALIZED RECURSIVE NEURON

can be extended both considering several interconnected re-
Standard and recurrent neurons are not suited to dealing
current neurons and delayed versions of the outputs. For the
with labeled structures. In fact, neural networks using these
sake of presentation, we will skip these extensions.
kind of neurons can deal with approximation and classification
problems in structured domains only by using a complex and
III. THE ENCODING PROBLEM
very unnatural encoding scheme which maps structures onto
In this paper we face the problem of devising neural network fixed-size unstructured patterns or sequences. We propose to
architectures and learning algorithms for the classification solve this inadequacy of the present state of the art in neural
of structured patterns, i.e., labeled graphs. We are given a networks by introducing the generalized recursive neuron.
structured domain over , and a training set The generalized recursive neuron is an extension of the
representing a classification task. recurrent neuron where instead of just considering the output
Fig. 5 reports the standard way to approach this problem by of the unit in the previous time step, we consider the outputs
using a standard neural network. Each graph is encoded as a of the unit for all the vertices which are pointed by the current
fixed-size vector which is then given as input to a feedforward input vertex (see Fig. 6). The output of the generalized
neural network for classification. This approach is motivated recursive neuron for a vertex of a graph is defined as
by the fact that neural networks only have a fixed number of
input units while graphs are variable in size. The encoding
process is usually defined a priori and does not depend on
the classification task. For example, in molecular biology
and chemistry, where chemical compounds are represented as (3)
labeled graphs, the encoding process is performed through the where is a sigmoidal function, is the number of units
definition of topological indexes [14], [22], which are designed encoding the label attached to the current input ,
by a very expensive trial and error approach. and are the weights on the recursive connections. Thus, the
The a priori definition of the encoding process has two output of the neuron for a vertex is computed recursively
main drawbacks. on the output computed for all the vertices pointed by it.
1) The relevance of different features of the graphs may Notice that, if the valence of the considered domain is ,
change dramatically for different learning tasks. then the generalized recursive neuron will have recursive
2) Since the encoding process has to work for any classifica- connections, even if not all of them will be used for computing
tion task, each graph must get a different representation; the output of a vertex with .
this may result in vectorial representations which are very To clarify how a generalized recursive neuron works, it may
difficult to classify. be useful to show how it relates to a recurrent neuron.
To overcome the above difficulties, we propose to adapt the 1) A recurrent neuron can be considered as a generalized
encoding process to the specific classification task at hand. recursive neuron applied to lists, i.e., labeled graphs with

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
SPERDUTI AND STARITA: SUPERVISED NEURAL NETWORKS 717

Fig. 5. Classification of graphs by neural networks. The input graph is encoded as a fixed-size vector which is given as input to a feedforward neural
network for classification.

valence one; in that case, the position of a vertex within where , and
the list corresponds to the time of processing; e.g., given , yields the following set of equations:
the list

where , the set of equations

derived by the recursive application of (3) is

cycle

where the output for the vertices involved in the cycle,

where is the weight on the recurrent connection; i.e., and , are interdependent; this means
by making the time of processing explicit, the above that, while can be readily computed, can only
equations can be rewritten in the following forms: be computed after the subset of equations referring to the
cycle has been solved.
In the next section we introduce the concept of an encoding
network which will allow us to give a general solution to the
problem of the adaptive encoding of directed graphs. Before
proceeding, however, let us see how (3) is defined when
considering interconnected generalized recursive neurons.
Equation (3) becomes

which correspond to the equations obtained by using a (4)

recurrent neuron; where
2) From the above equations it is clear that the output
. In the following,
of the recurrent neuron at time depends only on the
when referring to the output of a generalized neuron we will
output at time ; this is due to the fact that lists do
drop the upper index.
not contain cycles; in general, however, graphs contain
cycles and a generalized recursive neuron must deal with
A. Generation of Neural Representations for Graphs
them. Specifically, the outputs of the generalized recursive
neuron for the vertices belonging to a cycle are obtained To understand how generalized recursive neurons can gener-
as the solutions of a system of interdependent equations; ate representations for directed graphs, let us consider a single
e.g., the graph generalized recursive neuron and a single graph . The
following two conditions must hold.
Number of Connections: the generalized recursive neuron
must have as many recursive connections as the valence of
graph .

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
718 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 3, MAY 1997

Fig. 6. Neuron models for different input domains. The standard neuron is suited for the processing of unstructured patterns, the recurrent neuron for the
processing of sequences of patterns, and finally the proposed generalized recursive neuron can deal very naturally with structured patterns.

Fig. 7. The encoding network for an acyclic graph. The graph is represented by the output of the encoding network.

Supersource: graph must have a reference supersource. network is recurrent (see Fig. 8), and the neural representation
Notice that if graph does not have a supersource, then it is considered to be well formed only if converges to a
is still possible to define a convention for adding to graph stationary value.
an extra vertex (with a minimal number of outgoing edges), The encoding network fully describes how the represen-
tation for the structure is computed and it will be used in
such that is a supersource for the new graph (see Appendix
the following to derive the learning rules for the generalized
A for an algorithm).
recursive connections.
If the above conditions are satisfied, we can adopt the
When considering a structured domain, the number of
convention that graph is represented by , i.e., the recursive connections of must be equal to the valence of
output of computed for the supersource . Consequently, the domain. The extension to a set of generalized neurons
due to the recursive nature of (3), it follows that the neural is trivial: if the valence of the domain is , each generalized
representation for an acyclic graph is computed by a feed- neuron will have groups of recursive connections each.
forward network (encoding network) obtained by replicating
the same generalized recursive neuron and connecting these B. Optimized Training Set for Directed Acyclic Graph
copies according to the topology of the structure (see Fig. 7). When considering DAG’s, the training set can be organized
If the structure contains cycles, then the resulting encoding so to improve the computational efficiency of both the reduced

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
SPERDUTI AND STARITA: SUPERVISED NEURAL NETWORKS 719

Fig. 8. The encoding network for a cyclicgraph. In this case, the encoding network is recurrent and the graph is represented by the output of the
encoding network at a fixed point of the network dynamics.

Fig. 9. Optimization of the training set: the set ofstructures (in this case, trees) is transformed into the corresponding minimal DAG, which is then used
to generate thesorted training set. The sorted training set is then transformed into a set of sorted vectors using the numeric codes for the labels and
targets, and used as a training set for the network.

representations and the learning rules. In fact, given a training reduction in space complexity. In some cases, such reduction
set of DAG’s, if there are graphs which share can even be exponential.
a common subgraph , then we need to explicitly represent
only once. The optimization of the training set can be C. Well-Formedness of Neural Representations
performed in two stages (see Fig. 9 for an example).
When considering cyclic graphs, to guarantee that each
1) All the DAG’s in the training set are merged into a single encoded graph gets a proper representation through the en-
minimal DAG, i.e., a DAG with a minimal number of coding network, we have to guarantee that for every initial
vertices. state the trajectory of the encoding network converges to
2) A topological sort on the vertices of the minimal DAG an equilibrium, otherwise it would be impossible to process
is performed to determine the updating order on the a nonstationary representation. This is particularly important
vertices for the network. when considering cyclic graphs. In fact, acyclic graphs are
Both stages can be done in linear time with respect to the size guaranteed to get a convergent representation because the
of all DAG’s and the size of the minimal DAG, respectively.2 resulting encoding network is feedforward, while cyclic graphs
Specifically, stage 1) can be done by removing all the duplicate are encoded by using a recurrent network. Consequently,
subgraphs through a special subgraph-indexing mechanism the well-formedness of representations can be obtained by
(which can be implemented in linear time). defining conditions that guarantee the convergence of the
The advantage of having a sorted training set is that all the encoding network. Regarding this issue, it is well known that
reduced representations (and also their derivatives with respect if the weight matrix is symmetric, an additive network with
to the weights, as we will see when considering learning) first-order connections possesses a Lyapunov function and is
can be computed by a single ordered scan of the training set. convergent ([5], [15]). Moreover, Almeida [1] proved a more
Moreover, the use of the minimal DAG leads to a considerable general symmetry condition than the symmetry of the weight
matrix, i.e., a system satisfying detailed balance
2 This analysis was done by C. Goller. (5)

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
720 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 3, MAY 1997

Fig. 10. A possible architecture for the classification of structures: the

generalized recursive neurons generate the neural representation for the
structures, which are then classified by a standard feedforward network.
Fig. 11. The functional scheme of the proposed network. The encoder
and the cClassifier are considered as two distinct entities which exchange
is guaranteed to possess a Lyapunov function as well. On the information: the encoder forward the neural representations of the structures
to the classifier; in turn the classifier returns to the encoder the deltas, which
other hand, if the norm of the weight matrix (not necessarily are then used by the encoder to adapt its weights.
symmetric) is sufficiently small, e.g., satisfying

(6) contributions of corresponding copies of the same weight and

then changing all copies by the total amount.
Backpropagation through time can easily be extended to
the network’s dynamics can be shown to go to a unique structures. The basic idea is to use generalized recursive
equilibrium for a given input ([2]). neurons to encode the structures; the representations obtained
The above results are sufficient when the encoding network are then classified or used to approximate an unknown function
is static, however we will see that in several cases, for example, by a standard feedforward network. An example of this kind
in classification tasks, the encoding network changes with of network for classification is given in Fig. 10.
learning. In these cases, these results can be exploited to define Given an input graph , the network output can be
the initial weight matrix, but there is no guarantee that learning expressed as the composition of the encoding function
will preserve the stability of the encoding network. and the classification (or approximation) function (see
Fig. 11):
V. SUPERVISED MODELS (7)
In this section, we discuss how several standard supervised
algorithms for neural networks can be extended to structures. Learning for the set of weights can be implemented by
plain backpropagation on the feedforward network realizing
A. Backpropagation Through Structure
The task addressed by backpropagation through time net- Error
(8)
works [23] is to produce particular output sequences in re-
sponse to specific input sequences. These networks are, gen-
erally, fully recurrent, in which any unit may be connected where , i.e., the input to the feedforward network,
to any other. Consequently, they cannot be trained by using while learning for the set of weights, , realizing can
plain backpropagation. A trick, however, can be used to turn be implemented by
an arbitrary recurrent network into an equivalent feedforward Error
network when the input sequences have a maximum length (9)
. In this case, all the units of the network can be duplicated
times (unfolding of time), so that the state of a unit in the where the first term represents the error coming from the
recurrent network at time is held by the th copy of the feedforward network and the second one represents the error
same unit in the feedforward network. By preserving the same due to the encoding network.
weight values through the layers of the feedforward network, How the second term is actually computed depends on the
it is not difficult to see that the two networks will behave family of structures in the training set. If the training set
identically for time steps. The feedforward network can consists of DAG’s, then plain backpropagation can be used
be trained by backpropagation, taking care to preserve the on the encoding network. Otherwise, i.e., if there are graphs
identity constraint between weights of different layers. This with cycles, then recurrent backpropagation [20] must be used.
can be guaranteed by adding together the individual gradient Consequently, we treat these two cases separately.

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
SPERDUTI AND STARITA: SUPERVISED NEURAL NETWORKS 721

Case I—DAG’s: This case has been treated by Goller and with
Küchler in [12]. Since the training set contains only DAG’s,
can be computed by backpropagating the error from the ..
. (16)
feedforward network through the encoding network of each
structure. As in backpropagation through time, the gradient
contributions of corresponding copies of the same weight are repetitions of
collected for each structure. The total amount is then used
to change all the copies of the same weight. If learning is and is defined according to the topology
performed by structure, then the weights are updated after of the graph.
the presentation of each individual structure, otherwise, the In this context, the input of the adjoint network is
Error , and the learning rules become
gradient contributions are collected through the whole training
set and the weights changed after all the structures in the
training set have been presented to the network. (17)
Case II—Cyclic Graphs: When considering cyclic graphs,
can only be computed by resorting to recurrent back- (18)
propagation. In fact, if the input structure contains cycles, then
the resulting encoding network is cyclic. To hold the constraint on the weights, all the changes
In the standard formulation, a recurrent backpropagation referring to the same weight are added and then all copies
network with units is defined as of the same weight are changed by the total amount. Note
(10) that each structure gives rise to a new adjoint network and
independently contributes to the variations of the weights.
where is the input vector for the network, and Moreover, the above formulation is more general than the
the weight matrix. The learning rule for one previously discussed and it can also be used for DAG’s,
a weight of the network is given by in which case the adjoint network becomes a feedforward
network representing the backpropagation of errors.
(11)
An important variation to the learning rules presented above
is constituted by teacher forcing, a technique proposed by
where all the quantities are taken at a fixed point of the Pineda [20] to force the creation of new fixed points in
recurrent network, is the error between the current output standard recurrent networks. While the effects of this technique
of unit and the desired one, ( is a is well understood in standard recurrent networks, it is not
Kronecker delta), and the quantity clear to us how it can influence the dynamics of the encoding
(12) networks. In fact, while a new encoding network for each
graph in the training set is generated, all the encoding networks
are interdependent since the weights on the connections are
can be computed by relaxing the adjoint network , i.e., a shared. A more accurate study on how this sharing of resources
network obtained from by reversing the direction of the affects the dynamics of the system when using teacher forcing
connections. The weight from neuron to neuron in is needed.
is replaced by from neuron to neuron in . The
activation functions of are linear and the output units in B. Extension of Real-Time Recurrent Learning
become input units in with as input.
Given a cyclic graph , let be the number The extension of real-time recurrent learning [34] to gener-
of generalized neurons and the output at time of these alized recursive neurons does not present particular problems
neurons for . Then we can define when considering graphs without cycles. Cyclic graphs, in-
stead, give rise to a training algorithm that can be considered
only loosely in real time.
(13) In order to be concise, we will only show how derivatives
..
. can be computed in real time, leaving to the reader the
development of the learning rules, according to the chosen
network architecture and error function.
as the global state vector for our recurrent network, where for
Case I—DAG’s: Let be the number of generalized
convention is the output of the neurons representing the
neurons, the supersource of , and
supersource of .
To account for the labels, we have to slightly modify (10)
(14)
where (19)
be the representation of according to the encoding network,
.. where
. (15)
(20)

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
722 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 3, MAY 1997

and is the valence of the domain. The derivatives of with Note that both the strongly connected components of a
respect to and can be computed graph and its component graph can be computed by
from (19) using two depth-first searches in in time [6].
Moreover, a strongly connected component of corresponds
to a maximal subset of interconnected units in the encoding
network for , while the component graph describes the
functional dependences between these subsets of units. Thus,
the computation of the derivatives in real time can be done by
observing the following.
(21)
1) Equations (21) and (22) are still valid.
where is the Kronecker delta, 2) Each strongly connected component of the graph corre-
and , is the th row of sponds to a set of interdependent equations.
3) The component graph defines the dependences between
sets of equations.
For example, let consider a single generalized recursive neuron
and the graph
(22)

where is the Kronecker delta,

, and .
These equations are recursive on the structure , and can be whose component graph is
computed by noting that if is such that ,
then
where
and (23)

This allows the computation of the derivatives in real time

alongside the computation of the neural representations for
the graphs.
Let us consider the derivatives . They are the
Case II—Cyclic Graphs: When considering a cyclic graph
solutions of the set of equations [derived by (22)]
, the situation is very similar to the one described
in Section V-A.2. However, as soon as the output of the (corresponding to
encoding network for a vertex of has been computed,
the corresponding derivatives must be computed as well. corresponding to
Of course, if there is a cycle in which
involves all the vertices in , then given any can
corresponding to
be computed only when the encoding network is relaxed, i.e.,
is computed for every . In this case, the learning which can be decomposed according to the component graph.
rules are the same as in Section V-A.2. Specifically, by substituting the solution for in the equation
A different situation is encountered when does not contain for , the subset of equations corresponding to the strongly
global cycles, i.e., cycles involving all the vertices in . In connected component can be solved through a relaxation
this case, it is useful to resort to two strictly related concepts, process. Finally, once the solution value for is known,
the strongly connected components of a graph and the related the last equation corresponding to can be solved by
definition of a component graph. direct substitution. In general, by taking into account the
Definition V.1—(Strongly Connected Component): A dependences established by the component graph, both the
strongly connected component of a directed graph output of the neuron and its derivatives can be computed in
is a maximal set of vertices such that for every pair real time with respect to the component graph.
of vertices and in , there exists a path from to and
viceversa, i.e., and are reachable from each other. C. LRAAM-Based Networks and Simple Recurrent Networks
Definition V.2—(Component Graph): The (acyclic) com- In this section, we present a class of networks which are
ponent graph of is defined as the graph based on the LRAAM model [33] (see Appendix B). Learning
, where contains one vertex for in these networks is implemented via the combination of a
each strongly connected component of , and contains supervised procedure with an unsupervised one. Since it is
the edge if there is a directed edge from a vertex in based on truncated gradients, it is suited for both acyclic and
the strongly connected component of corresponding to cyclic graphs. We will see that this class of networks contains,
to a vertex in the strongly connected component of as a special case, a network which turns out to be the extension
corresponding to . of a simple recurrent network [7] to structures.

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
SPERDUTI AND STARITA: SUPERVISED NEURAL NETWORKS 723

(a)

(a)

(b)
Fig. 13. LRAAM-based networks for the classification of structures.

(b)
The reason for allowing different degrees of interaction
G
Fig. 12. (a) A labeled directed graph . The strongly connected components
of G are shown as shaded regions. (b) The acyclic component graph SCC G between the classification and the representation tasks is due
obtained by shrinking each strongly connected component of G
to a single to the necessity of having different degrees of adaptation of
vertex. the compressed representations to the requirements of the
classification task. If no interaction at all is allowed, i.e., the
In this type of network (see Fig. 13), the first part of the LRAAM is trained first and then its weights frozen
network is constituted by an LRAAM (note the double arrows , the compressed representations will be such that similar
on the connections) whose task is to devise a compressed representations will correspond to similar structures, while if
representation for each structure. This compressed represen- full interaction is allowed, i.e., the LRAAM and the classifier
tation is obtained by using the standard learning algorithm for are trained simultaneously, the compressed representations will
LRAAM [32]. The classification task is then performed in the be such that structures in the same class will get very similar
second part of the network through a multilayer feedforward representations.5 On the other hand, when considering DAG’s,
network with one or more hidden layers (network ) or a by setting , where is the number
simple sigmoidal neuron (network ). of vertices traversed by the longest path in the structures,
Several options for training networks and are available. the classifier error will be backpropagated across the whole
These options can be characterized by the proportion of the structure, thus implementing the backpropagation through the
two different learning rates (for the classifier and the LRAAM) structure defined in Section V-A.
and by the different degrees , and in the presence of the It is interesting to note that the SRN by Elman can be
following two basic features. obtained as a special case of network . In fact, when
considering network (with and ) for the
• The training of the classifier is not started until per-
classification of lists (sequences) the same architecture is ob-
cent of the training set has been correctly encoded and
tained, with the difference that there are connections from the
subsequently decoded by the LRAAM.
hidden layer of the LRAAM back to the input layer6, i.e., the
• The error from the classifier is backpropagated across
decoding part of the LRAAM. Thus, when considering lists,
levels of the structures encoded by the LRAAM.3
the only difference between a SRN and network is in the
Note that, even if the training of the classifier is started unsupervised learning performed by the LRAAM. However,
only when all the structures in the training set are properly when forcing the learning parameters for the LRAAM to
encoded and decoded, the classifier’s error can still change the be null, we obtain the same learning algorithm as in SRN.
compressed representations which, however, are maintained Consequently, we can claim that SRN is a special case of
consistent4 by learning in the LRAAM. network . This can be better understood by looking at the
right-hand side of Fig. 14, which shows network in terms of
3 The backpropagation of the error across several levels of the structures can
be implemented by unfolding the encoder of the LRAAM (the set of weights 5 Moreover, in this case, there is no guarantee that the LRAAM will be able
from the input to the hidden layer) according to the topology of the structures. to encode and to decode consistently all the structures in the training set, since
4 A consistent compressed representation is a representation of a structure training is stopped when the classification task has been performed correctly.
which contains all the information sufficient for the reconstruction of the 6 The output layer of the LRAAM can be considered as being the same as
whole structure. the input layer.

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
724 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 3, MAY 1997

(a) (b)
Fig. 14. (a) The network B, with x = 0 and y = 1 can be considered as (b) a generalization of the simple recurrent network by Elman.

elements of an SRN. Of course, the copy function for network th hidden unit. The output of the output neuron is
is not as simple as the one used in an SRN, since the computed as
correct relationships among components of the structures to
be classified must be preserved.7 (25)

D. Cascade-Correlation for Structures

where is the weight on the connection from the th (frozen)
The cascade-correlation algorithm [9] creates a standard hidden unit to the output unit.
neural network using an incremental approach for the clas- Learning is performed as in standard cascade-correlation,
sification of unstructured patterns. The starting network is with the difference that, according to (24), the derivatives of
a network without hidden nodes trained with a least mean with respect to the weights are now
square algorithm. If network is not able to solve the
problem, a hidden unit is added such that the correlation
between the output of the unit and the residual error of network
is maximized.8 The weights of are frozen and the
remaining weights are retrained. If the obtained network (26)
cannot solve the problem new hidden units are added which
are connected (with frozen weights) with all the inputs and
previously installed hidden units. The resulting network is
a cascade of nodes. Fahlman extended the algorithm to the
classification of sequences, obtaining good results [18]. In the
following, we show that the cascade-correlation can be further
extended to structures by using generalized recursive neurons.
For the sake of simplicity, we will discuss the case of acyclic
graphs, leaving to the reader the extension to cyclic graphs (27)
(see Section V-A, cyclic graphs).
The output of the th hidden unit, in our framework, can
be computed as

(28)

(24)
where
is the derivative of . The above
where is the weight of the th hidden unit associated equations are recurrent on the structures and can be computed
with the output of the th hidden unit computed on the by observing that for a leaf vertex (26) reduces to
th component pointed by , and is the weight of the , and all the remaining derivatives are null. Consequently,
connection from the th (frozen) hidden unit, , and the we only need to store the output values of the unit and its
7 The copy function needs a stack for the memorization of compressed derivatives for each component of a structure. Fig. 15 shows
representations. The control signals for the stack are defined by the encoding- the evolution of a network with two pointer fields.
decoding task.
8 Since the maximization of the correlation is obtained using a gradient
Note that, if the hidden units have self-recurrent connections
ascent technique on a surface with several maxima, a pool of hidden units is only, the matrix defined by the weights is not triangular,
trained and the best one selected. but diagonal.

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
SPERDUTI AND STARITA: SUPERVISED NEURAL NETWORKS 725

Fig. 15. The evolution of a network with two pointer fields. The units in the label are fully connected with the hidden units and the output unit.

The learning and classification algorithms for binary neural

trees are described in Fig. (17). Algorithms for general trees
can be found in [24]. The extension of these algorithms to
structures is straightforward: the standard discriminator (or
network) associated to each node of the tree is replaced by a
generalized recursive discriminator (or network) which can be
trained with any of the learning algorithms we have presented
so far.

VI. EXPERIMENTAL RESULTS

We have tested some of the proposed architectures on
several classification tasks involving logic terms. Note that
logic terms may be represented very naturally as DAG’s (see
Fig. 4) and consequently, classification problems involving
logic terms belong to the class of structured problems de-
scribed at the beginning of the paper. In the next subsection,
we present the classification problems. Then, we discuss the
results obtained with LRAAM-based networks and cascade-
correlation for structures.
Fig. 16. An example of neural tree.

A. Description of the Classification Problems

Table I summarizes the characteristics of each problem. The
E. Extension of Neural Trees
first column reports the name of the problem, the second the set
Neural trees have recently been proposed as a fast learning of symbols (with associated arity) compounding the terms, the
method in classification tasks. They are decision trees [4] third shows the rule(s) used to generate the positive examples
where the splitting of the data for each vertex, i.e., the of the problem,9 the fourth reports the number of terms in
classification of a pattern according to some features, is the training and test set respectively, the fifth the number of
performed by a perceptron [26] or a more complex neural subterms in the training and test sets, and the last the maximum
network [24]. After learning, each vertex at every level of the depth10 of terms in the training and test sets. For example,
tree corresponds to an exclusive subset of the training data and let us consider the problem inst1 long. All the terms in
the leaf nodes of the tree completely partition the training set. the training and test sets are componded by three different
In the operative mode, the internal nodes route the input pattern constants, i.e., , and , and a functional with
to the appropriate leaf node which represents the class of it. two arguments. Terms which have to be classified positively
An example of how a binary neural tree splits the training set are required to have the first argument equal to the second
is shown in Fig. 16. one , e.g., both and
One advantage of the neural tree approach is that the tree have to be classified
structure is constructed dynamically during learning and not positively, while has to be classified negatively.
linked to a static structure like a standard feedforward network. For this problem, the training set contains 202 terms, while the
Moreover, it allows incremental learning, since subtrees can be test set contains 98 terms; the total number of distinct subterms
added as well as deleted to recognize new classes of patterns for the training and test sets is 403 and 204, respectively. Note
or to improve generalization. Both supervised [3], [24]–[26] 9 Note that the terms are all ground.
and unsupervised [16], [18], [19] splitting of the data have 10 We define the depth of a term as the maximum number of edges between
been proposed. We are interested in supervised methods. the root and leaf nodes in the term’s LDAG-representation.

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
726 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 3, MAY 1997

Fig. 17. The learning and classification algorithms for binary neural trees.

TABLE I
DESCRIPTION OF A SET OF CLASSIFICATION PROBLEMS INVOLVING LOGIC TERMS

that the total number of subterms for the training set will be of a specific unification pattern. Specifically, in the unifi-
equal to the number of vertices of the minimal DAG obtained cation patterns for the problems inst1 and inst1 long,
after the optimization of the training set. Finally, the maximal the variable occurs twice making these problems much
depth of the terms, both in the training and test sets, is six. more difficult than inst4 long, because any classifier for
For each problem about the same number of positive and these problems would have to compare arbitrary subterms
negative examples is given. Both positive and negative ex- corresponding to .
amples were generated randomly. Training and test sets are As a final remark, it must be pointed out that the number of
disjoint and were generated by the same algorithm. Some training examples is a very small fraction of the total number
examples of logic terms represented as trees are shown in of terms characterizing each problem. In fact, the total number
Figs. 18–20. of distinct terms belonging to a problem is defined both by
Note that the set of proposed problems ranges from the de- the number and arity of the symbols, and by the maximal
tection of a particular atom (label) in a term to the satisfaction depth allowed for the terms. Making some calculations, it

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
SPERDUTI AND STARITA: SUPERVISED NEURAL NETWORKS 727

Fig. 18. Samples of trees representing terms for problem inst4 long. The positive terms are shown on the left, while negative terms are shown on
the right. The vertices and edges characterizing the positive terms are drawn in bold.

Fig. 19. Samples of trees representing terms for problem inst1 long. The positive term is shown on the left, while the negative term is shown on the right.

is not difficult to compute that the total number of terms and the learning parameters, thus it should be possible to
allowed for problem inst1 is equal to 21 465. This number, improve on the reported results. The first column of the table
however, grows exponentially with the maximal depth, e.g., if shows the name of the problem, the second the number of units
the maximal depth is set to four, the total number of distinct used to represent the labels, the third the number of hidden
terms is larger than . This gives an idea of how small is units, the fourth the learning parameters ( is the learning
the training set for problem inst1 long, where the maximal parameter for the LRAAM, the learning parameter for the
depth is six. classifier, the momentum for the LRAAM), the fifth the
percentage of terms in the training set which the LRAAM was
B. LRAAM-Based Networks able to properly encode and decode, the sixth the percentage
Table III reports the best results we obtained for the majority of terms in the training set correctly classified, the seventh the
of the problems described in Table I, over four different net- percentage of terms in the test set correctly classified, and the
work settings (both in number of hidden units for the LRAAM eighth the number of epochs the network employed to reach
and learning parameters) for the LRAAM-network with a the reported performances.
single unit as classifier. The simulations were stopped after The results highlight that some problems get a very satisfac-
30 000 epochs, apart from problem inst1 long for which tory solution even if the LRAAM performs poorly. Moreover,
we used a bound of 80 000 epochs, or when the classification this behavior does not seem to be related to the complexity
problem over the training set has been completely solved. We of the classification problem, since both problems involv-
made no extended effort to optimize the size of the network ing the simple detection of an atom (label) in the terms

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
728 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 3, MAY 1997

Fig. 20. Samples of trees representing terms for problem termocc1 very-long. The positive term is shown on the left, while the negative term
is shown on the right.

TABLE II we used. The basic LRAAM organizes the representational

RESULTS OBTAINED ON THE TEST SETS FOR SOME OF THE CLASSIFICATION space in such a way that similar structures get similar reduced
PROBLEMS USING LRAAM-BASED NETWORKS WITH DIFFERENT NUMBERS OF
HIDDEN UNITS AND DIFFERENT SETTINGS FOR THE LEARNING PARAMETERS representations (see [28] for more details). This happens
because, although the LRAAM is trained in supervised mode
both on the output of the network and on the relationships
among components (i.e., the information about the pointers),
the network is auto-associative and thus it decides by itself
the representation for the pointers. Consequently, the learning
mode for the LRAAM is on the whole unsupervised. When
a classifier is introduced (as in our system), the training of
the LRAAM is no longer unsupervised, since the error of the
classifier constrains the learning. The resulting learning regime
is somewhere between an unsupervised and a supervised mode.
In order to understand the differences between representa-
tions devised by a basic LRAAM and the ones devised in the
present paper, we trained a basic LRAAM (of the same size
as the LRAAM used by our network) on the training set of
inst1, then we computed the first, second, and third principal
components of the reduced representations obtained both for
(lblocc1 long) and problems involving the satisfaction the training and test sets.12 These principal components are
of a specific unification rule (inst4 long, inst7) can plotted in Fig. 24. Note that the representations obtained
be solved without a fully developed LRAAM. Thus, it is mainly cluster themselves in specific points of the space.
clear that the classification of the terms is exclusively based Terms of the same depth constitute a single cluster, and terms
on the encoding power of the LRAAM’s encoder, which is of different depths are in different clusters. The same plot
influenced both by the LRAAM error and the classification for the reduced representations devised by our network (as
error. However, even if the LRAAM’s decoder is not directly from Table III, row 3) is presented in Fig. 25. The overall
involved in the classification task, it helps the classification differences with respect to the basic LRAAM plot consist in a
process since it forces the network to generate different concentration of more than half (57%) of the positive examples
representations for terms in different classes.11 of the training and test sets in a well defined cluster, while the
In order to give a feeling of how learning proceeds, the remaining representations are spread over two main subspaces.
performance of the networks during training is shown for The well-defined cluster can be understood as the set of
the problems termocc1, inst4 and inst4 long in representations for which there was no large-scale interference
Figs. 21–23, where the encoding–decoding performance curve between the decoder of the LRAAM and the classifier (this
of the LRAAM on the training set is reported, together with allowed the formation of the cluster), while the remaining
the classification curves on the training and test set. representations do not preserve the cluster structure since
Reduced Representations for Classification: In this section they have to satisfy competitive constraints deriving from the
we briefly discuss the representational differences between classifier and the decoder. Specifically, the classifier tends to
a basic LRAAM (without a classifier) and the architecture cluster the representations into two well-defined clusters (one
11 The decoder error forces the LRAAM network to develop a different for each class), while the LRAAM decoder tends to develop
representation for each term, however, when the error coming from the
classifier is very strong, terms in the same class may get almost identical 12 We considered only the reduced representations for terms. No reduced
representations. representation for subterms was included in the analysis.

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
SPERDUTI AND STARITA: SUPERVISED NEURAL NETWORKS 729

TABLE III
THE BEST RESULTS OBTAINED FOR SOME OF THE CLASSIFICATION PROBLEMS BY AN LRAAM-BASED NETWORK

Fig. 21. Performance curves for termocc1.

distinct reduced representations since they must be decoded accordance with the better performance in encoding-decoding
into different terms. of the latter than the former. Of course, this does not constitute
The above considerations on the final representations for enough evidence to conclude that the relationship between the
the terms are valid only if the LRAAM reaches a good variance of the clusters and the performance of the LRAAM
encoding-decoding performance on the training set. However, is demonstrated. However, it does seem to be enough to call
as reported in Table III, some classification problems can for a more accurate study on this issue.
be solved even if the LRAAM performs poorly. In this
case, the reduced representations contain almost exclusively C. Cascade-Correlation for Structures
information about the classification task. Figs. 26 and 27 The results obtained by cascade-correlation for structures,
report the results of a principal components analysis on the shown in Table IV, are obtained for a subset of the problems
representations developed for the problems inst4 long using a pool of eight units. The networks used have both
and inst7, respectively. In the former, the first and sec- triangular and diagonal recursive connections matrices and no
ond principal components suffice for a correct solution of connections between hidden units. We decided to remove the
the classification problem. In the latter, the second principal connections between hidden units to reduce the probability of
component alone gives enough information for the solution overfitting.
of the problem. Moreover, notice how the representations We made no extended effort to optimize the learning pa-
developed for inst7 clustered with smaller variance than the rameters and the number of units in the pool, thus it should
representations developed for inst4 long, and how this is in be possible to significantly improve on the reported results.

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
730 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 3, MAY 1997

Fig. 22. Performance curves for inst4.

Fig. 23. Performance curves for inst4 long.

VII. CONCLUSION defined for standard neurons, can be adapted to deal with
structures. We believe that other learning procedures, besides
We have proposed a generalization of the standard neuron, those covered in this paper, can be adapted as well.
namely the generalized recursive neuron, for extending the The proposed approach to learning in structured domains
computational capability of neural networks to the processing can be adopted for automatic inference in syntactic and struc-
of structures. On the basis of the generalized recursive neu- tural pattern recognition [10], [13]. Specifically, in this paper,
ron, we have shown how several of the learning algorithms we have demonstrated the possibility to perform classification

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
SPERDUTI AND STARITA: SUPERVISED NEURAL NETWORKS 731

Fig. 24. The first, second, and third principal components of the reduced representations, devised by a basic LRAAM on the training and test sets of the
inst1 problem, yield a nice three-dimensional view of the term’s representations.

Fig. 25. Results of the principal components analysis (first, second, and third principal components) of the reduced representations developed by the
proposed network (LRAAM + Classifier) for the inst1 problem.

tasks involving logic terms. Note that automatic inference can (QSPR and QSAR), where it can be used for the automatic
also be obtained by using inductive logic programming [17]. determination of topological indexes [14], [22], which are
The proposed approach, however, has its own specific pecu- usually designed through a very expensive trial and error
liarity, since it can approximate functions from a structured approach.
domain (which may have real-valued vectors as labels) to In conclusion, the proposed architectures extend the process-
the reals. Specifically, we believe that the proposed approach ing capabilities of neural networks, allowing the processing
can fruitfully be applied to molecular biology and chemistry of structured patterns which can be of variable size and

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
732 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 3, MAY 1997

Fig. 26. Results of the principal components analysis (first and second principal components) of the reduced representations developed by the proposed
network (LRAAM + Classifier) for the inst4-long problem. The resulting representations are clearly linearly separable.

Fig. 27. Results of the principal components analysis (first, and second principal component) of the reduced representations developed by the proposed
network (LRAAM + Classifier) for the inst7 problem. The resulting representations can be separated using only the second principal component.

complexity. However, some of the proposed architectures the network. These limitations, in the context of standard
do nevertheless have computational limitations. For example, recurrent cascade-correlation (RCC), have been discussed in
cascade-correlation for structures has computational limita- [11], where it is demonstrated that certain finite state automata
tions due to the fact that frozen hidden units cannot receive cannot be implemented by networks built by RCC using
input from hidden units introduced after their insertion into monotone activation functions. Since our algorithm reduces

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
SPERDUTI AND STARITA: SUPERVISED NEURAL NETWORKS 733

TABLE IV
RESULTS OBTAINED ON THE TEST SETS FOR SOME OF THE CLASSIFICATION PROBLEMS USING BOTH NETWORKS WITH TRIANGULAR AND
DIAGONAL RECURSIVE CONNECTION MATRICES. THE SAME NUMBER OF UNITS (8) IN THE POOL WAS USED FOR ALL NETWORKS

to standard RCC when considering sequences, it follows that in different ways according to the complexity of the structured
it has limitations as well. domain. For example, if the labels which appear in a graph
are all distinct, then we can define a total order on (the
set of labels) and note that the set
APPENDIX
ALGORITHM FOR THE ADDITION OF A SUPERSOURCE for each
In the following we define the algorithm for the addition contains a single element which can be used as representative
of a supersource to a graph. The algorithm uses the concepts (i.e., ). If the same label can have multiple occurrences, we
of strongly connected components of a graph and the related can use information on the out-degree of the vertices
definition of a component graph (see Section V-B.2 for formal
definitions).
for each
A. Algorithm Supersource
and further on the in-degree of the vertices
Input: A graph
Output: A graph with supersource
Begin for each
• Compute the component graph of ; If the set still contains more than one vertex, then it can be
• Define the set refined by resorting to more sophisticated criteria (see [31]).
; Theorem 1: The algorithm Supersource applied to a graph
• Define for each vertex the set containing adds to the minimum number of edges to
all the vertices (in ) within the strongly connected create a supersource.
component corresponding to ; Proof: If no edge is added by the algorithm
• For each define , Supersource. Otherwise both a new vertex , the supersource,
i.e., is a vertex representing the strongly and a new edge for each strongly connected component
connected component corresponding to ; of with are created. The vertex is
• if ( (i.e., )) a supersource, since, by definition, all the verices in can
then is the supersource, , and ; be reached by a path starting from . Moreover, if one edge
else leaving from is removed, say , the vertices belonging
• Create a new vertex (the supersource) and set to the strongly connected component cannot be reached by
; any other vertex since . This demonstrates
• Set , where that all the added edges are needed.
.
End. LABELING RAAM
In the algorithm above we left the procedure The labeling RAAM (LRAAM) [27], [28], [30], is an exten-
undefined since it can be defined sion of the RAAM model [21] which allows one to encode

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
734 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 3, MAY 1997

Fig. 29. The network for a general LRAAM. The first layer in the network
implements an encoder; the second layer, the corresponding decoder.

Fig. 28. The result of algorithm Supersource applied to the graph shown in
Fig. 12(a).

labeled structures. The general structure of the network for an

LRAAM is shown in Fig. 29. Fig. 30. An example of a labeled directed graph.
The network is trained by backpropagation to learn the iden-
tity function. The idea is to obtain a compressed representation
(hidden layer activation) of a node of a labeled directed graph representations for the nonvoid pointers in the training set are
by allocating a part of the input (output) of the network to updated depending on the hidden activation obtained in the
represent the label ( units), and the rest to represent one or previous epoch for each pattern: . The
more pointers. This representation is then used as a pointer void representations are, on the other hand, copied from the
to the node. To allow the recursive use of these compressed output: .
representations, the part of the input (output) layer which If the backpropagation algorithm converges to perfect learn-
represents a pointer must be the same size as the hidden layer ing, i.e., the total error goes to zero, it can be stated that
( units). Thus, a general LRAAM is implemented by an
feedforward network, where ,
and is the number of pointer fields. nil nil nil nil nil nil
Labeled directed graphs can easily be encoded using an
LRAAM. Each node in the graph only needs to be represented Once training is complete, the patterns of activation rep-
as a record, with one field for the label and one field for each resenting pointers can be used to retrieve information. Thus,
pointer to a connected node. The pointers only need to be for example, if the activity of the hidden units of the net-
logical pointers, since their actual values will be the patterns of work is clamped to , the output of the network becomes
hidden activation in the network. At the beginning of learning, , thus enabling further retrieval of information
their values are set at random. A graph is represented by a list by decoding or , and so on. Note that multiple labeled
of these records, and this list is the initial training set for the directed graphs can be encoded in the same LRAAM.
LRAAM. During training the representations of the pointers When the LRAAM is used inside an LRAAM-based net-
are continuously updated depending on the hidden activations. work for the classification of terms, it is more convenient
Consequently, the training set is dynamic. to force a single representation for the void pointer, i.e., a
For example, the network for the graph shown in Fig. 30 null vector. This allows the network to encode graphs which
can be trained as follows: do not belong to the training set without looking for a valid
representation for the void pointer.
input hidden output
ACKNOWLEDGMENT
The authors would like to thank C. Goller for the generation
nil nil nil nil of the training and test sets used in this paper and D. Majidi
nil nil for the implementation of the cascade-correlation algorithm.
where and are the label of and the pointer to the th REFERENCES
node, respectively; and represents the time, or epoch, of train-
ing. At the beginning of training the representations [1] L. B. Almeida, “A learning rule for asynchronous perceptrions with
feedback in a combinatorial environment,” in Proc. IEEE 1st Annu. Int.
for the nonvoid pointers and void pointers Conf. Neural Networks, M. Caudil and C. Butler, Eds. New York:
in the training set are set at random. After each epoch, the IEEE Press, 1987, pp. 609–618.

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.
SPERDUTI AND STARITA: SUPERVISED NEURAL NETWORKS 735

[2] A. Atiya, “Learning on a general network,” in Neural Information [27] A. Sperduti, “Encoding of labeled graphs by labeling RAAM,” in
Processing Systems, D. Z. Anderson, Ed. New York: AIP, 1988, pp. Advances in Neural Information Processing Systems, J. D. Cowan, G.
22–30. Tesauro, and J. Alspector, Eds. San Mateo, CA: Morgan Kaufmann,
[3] L. Atlas et al., “A performance comparison of trained multilayer vol. 6, pp. 1125–1132, 1994.
perceptrons and trained classification trees,” Proc. IEEE, vol. 78, pp. [28] , “Labeling RAAM,” Connection Sci., vol. 6, no. 4, pp. 429–459,
1614–1619, 1992. 1994.
[4] L. Breiman, J. Friedman, R. Olshen, and C. Stone, Classification and [29] , “Stability properties of labeling recursive autoassociative mem-
Regression Trees. Belmont, CA: Wadsworth, 1984. ory,” IEEE Trans. Neural Networks, vol. 6, pp. 1452–1460, 1995.
[5] M. A. Cohen and S. Grossberg, “Absolute stability of global pattern [30] A. Sperduti and A. Starita, “An example of neural code: Neural
formation and parallel memory storage by competitive neural networks,” trees implemented by LRAAM’s,” in Proc. Int. Conf. Neural Networks
IEEE Trans. Syst., Man, Cybern., vol. SMC-13, pp. 815–826, 1983. Genetic Algorithms, Innsbruck, Austria, 1993, pp. 33–39.
[6] T. H. Cormen, C. E. Leiserson, and R. L. Rivest, Introduction to [31] , “Encoding of graphs for neural network processing,” Diparti-
Algorithms. Cambridge, MA: MIT Press, 1990. mento di Informatica, Università di Pisa, Italy, Tech. Rep., 1996.
[7] J. L. Elman, “Finding structure in time,” Cognitive Sci., vol. 14, pp. [32] A. Sperduti, A. Starita, and C. Goller, “Fixed length representation of
179–211, 1990. terms in hybrid reasoning systems, report i: Classification of ground
[8] S. E. Fahlman, “The recurrent cascade-correlation architecture,” terms,” Dipartimento di Informatica, Università di Pisa, Italy, Tech.
Carnegie Mellon Univ., Pittsburgh, PA, Tech. Rep. CMU-CS-91-100, Rep. TR-19/94, 1994.
1991. [33] A. Sperduti, A. Starita, and C. Goller, “Learning distributed representa-
[9] S. E. Fahlman and C. Lebiere, “The cascade-correlation learning archi- tions for the classification of terms,” in Proc. Int. Joint Conf. Artificial
tecture,” in Advances in Neural Information Processing Systems, D. S. Intell., 1995, pp. 509–515.
Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, vol. 2, 1990, pp. [34] R. J. Williams and D. Zipser, “A learning algorithm for continually
524–532. running fully recurrent neural networks,” Neural Computa., vol. 1, pp.
[10] K. S. Fu, Syntactical Pattern Recognition and Applications. Engle- 270–280, 1989.
wood Cliffs, NJ: Prentice-Hall, 1982.
[11] C. L. Giles, D. Chen, G. Z. Sun, H. H. Chen, Y. C. Lee, and M.
W. Goudreau, “Constructive learning of recurrent neural networks:
Limitations of recurrent casade correlation and a simple solution,” IEEE
Trans. Neural Networks, vol. 6, pp. 829–836, 1995. Alessandro Sperduti received the “laurea” and
[12] C. Goller and A. Küchler, Learning Task-Dependent Distributed Doctoral degrees in 1988 and 1993, respectively,
Structure-Representations by Backpropagation Through Structure, all in computer science from the University of Pisa,
Institut für Informatik, Technische Universität München, Germany, Italy.
AR Rep. AR-95-02, 1995. In 1993 he spent a period at the International
[13] R. C. Gonzalez and M. G. Thomason, Syntactical Pattern Recognition. Computer Science Institute, Berkeley, supported by
Reading, MA: Addison-Wesley, 1978. a postdoctoral fellowship. In 1994 he returned to
[14] L. H. Hall and L. B. Kier, “Reviews in computational chemistry,” the Computer Science Department, University of
The Molecular Connectivity Chi Indexes and Kappa Shape Indexes Pisa, where he is presently Assistant Professor. His
in Structure-Property Modeling. New York: VCH, 1991, ch. 9, pp. research interests include data sensory fusion, image
367–422. processing, neural networks, hybrid systems. In the
[15] J. J. Hopfield, “Neurons with graded response have collective computa- field of hybrid systems his work has focused on the integration of symbolic
tional properties like those of two-state neurons,” in Proc. Nat. Academy and connectionist systems.
Sci., 1984, pp. 3088–3092. Dr. Sperduti has contributed to the organization of several workshops on
[16] T. Li, L. Fang, and A. Jennings, “Structurally adaptive self-organizing
this subject and also served on the program committee of conferences on
neural trees,” in Proc. Int. Joint Conf. Neural Networks, 1992, pp.
Neural Networks.
329–334.
[17] S. Muggleton and L. De Raedt, “Inductive logic programming: Theory
and methods,” J. Logic Programming, vol. 19, no. 20, pp. 629–679,
1994.
[18] M. P. Perrone, “A soft-competitive splitting rule for adaptive tree-
Antonina Starita (A’91–M’96) received the
structured neural networks,” in Proc. Int. Joint Conf. Neural Networks,
“laurea” degree in physics from the University of
1992, pp. 689–693.
Naples, Italy, and the Doctoral degree in computer
[19] M. P. Perrone and N. Intrator, “Unsupervised splitting rules for neural
science from the University of Pisa, Italy.
tree classifiers,” in Proc. Int. Joint Conf. Neural Networks, 1992, pp.
She was Research Fellow of the Italian National
820–825.
[20] F. J. Pineda, “Dynamics and architecture for neural computation,” J. Council of Research at the Information Processing
Complexity, vol. 4, pp. 216–245, 1988. Institute of Pisa and then she became a Researcher
[21] J. B. Pollack, “Recursive distributed representations,” Artificial Intell., for the same institution. Since 1980, she has become
vol. 46, nos. 1–2, pp. 77–106, 1990. Associate Professor at the Computer Science
[22] D. H. Rouvray, Computational Chemical Graph Theory. New York: Department of the University of Pisa, where she is
Nova Sci., 1990, p. 9. in charge of the two regular courses, “Knowledge
[23] D. E. Rumelhart and J. L. McClelland, Parallel Distributed Processing: Acquisition and Expert Systems” and “Neural Networks,” at the University
Explorations in the Microstructure of Cognition. Cambridge, MA: MIT of Pisa. She has been active in the areas of biomedical signal processing,
Press, 1986. rehabilitation engineering, and biomedical applications of signal processing,
[24] A. Sankar and R. Mammone, “Neural tree networks,” Neural Networks: responsible for many research projects at the national and international level.
Theory and Applications. New York: Academic, 1991, pp. 281–302. Her current scientific interests are now shifted to AI methodologies, robotics,
[25] I. K. Sethi, “Entropy nets: From decision trees to neural networks,” neural networks, hybrid systems, and sensory integration in artificial systems.
Proc. IEEE, vol. 78, pp. 1605–1613, 1990. Dr. Starita is member of the Italian Medical and Biological Engineering
[26] J. A. Sirat and J.-P. Nadal, “Neural trees: A new tool for classification,” Society, the International Medical and Biological Engineering Society, and
Network, vol. 1, pp. 423–438, 1990. of the INNS (International Neural Network Society).

Authorized licensed use limited to: CHONNAM NATIONAL UNIVERSITY. Downloaded on September 28,2023 at 06:02:45 UTC from IEEE Xplore. Restrictions apply.