Fugue:

A Functional Language
for Sound Synthesis
Roger B. Dannenberg, Carnegie Mellon University
Christopher Lee Fraley, Microsoft Corporation
Peter Velikonja, West Virginia University zyxw
zyxwvu
- zyxwvutsrqpon
T
raditional software synthesis systems -which include Music V,' cmusic,2
and Csound?- are based on the same principles. Composers use a score
language to describe a list of notes or sounds to be synthesized. Each note
specifies a starting time, a duration, an instrument name, and a list of parameters

.".' ''
zyxwvut
Fugue provides
zyx
specific to the instrument (such as pitch, loudness, and articulation). A separate
orchestra language defines a set of instruments, each of which specifies a particular
signal processing algorithm. An instrument is defined as a set of interconnected
signal processing steps known as unit generators. Typical unit generators are
osci I1 a t o rs, f i I t e rs. adders, and m u It i pl ie rs.
Each note in the wore language invokes an instance (esentially a copy) ol' a n
instrument in the orchestra. This instrument instance computes sound for the
duration of the note according to the parameters of the note statement. The
synthesis system adds the resulting sound to that of the other notes in the score and
functions to create and writes them all to a disk file. After the computation completes, the system can read
the synthesized music from the disk in real time and convert it into analog form for
manipulate sounds as listening.
abstract, immutable This approach has been used without much change for over two decades. It
offers excellent and robust ideas, but it also has some weaknesses. An obvious
objects. The interactive problem is the separation of the score and orchestra languages.
language supports In this article, we present Fugue. Unlike the traditional approach, which re-
quires separate languages for different tasks, Fugue lets composers express signal
behavioral abstraction, processing algorithms for sound synthesis, musical scores, and higher level musical
procedures all in one language.
so composers can Although it extends the traditional sound synthesis approach2 with concepts
manage complex borrowed from functional programming: Fugue retains the advantages of the
Music V class of systems. One advantage of Music V is that composers can specify

36
musical structures.
zyxwvutsrqponml
the starting time and duration of each note in the score. These temporal attributes

00 18-9 162/91/0700-0036$01.OO 0 1991 IEEE COMPUTER

are implicit parameters to each instru- can use one language for both instru- bowing, the violinist adds more bow
ment instance that tell the system when ments and scores. strokes to extend the duration. Clearly,
to start and how many samples t o com- With this unification, composers can the notion of stretching is abstract, and
pute. Fugue supports an extension of express scores and instruments more the software instrument designer must

zyxwvutsrqpon
this technique. flexibly. In traditional synthesis systems, control its implementation. The user of
Another important contribution of it is difficult t o alter a phrase of many the instrument, however, need not be
Music V is the notion of combining unit notes by a single volume envelope. be- aware of the implementation details.
generators into a signal flow graph. cause this requires a hierarchical nest- T o support nested expressions in
Fugue implements this feature effi- ing of notes within the volume envelope. Fugue, we viewed a starting time o r
ciently. In Fugue, however, scores and signals duration as a transformation rather than
are all nested expressions, making hi- as an absolute parameter. We extended

zyx
erarchical structures very natural. the concept to allow transformations of
Programs as scores Composers can also use Fugue for articulation, loudness, and pitch. Com-
signal analysis to determine aspects of a posers can make additional qualities
Historically, musical scores have been score. Signal analysis is a common transformable and extend the system
static data structures created by com- practice in computer music, but signal with new transformation operators.
posers. Traditional scores do not ex- analysis programs are rarely integrated Programmers o r composers define
press computation beyond a few simple into traditional sound synthesis lan- behuviors that describe how to generate
abbreviations t o indicate repeats or al- guages. a sound within a transformation context.
ternate endings. There is a good reason Perhaps Fugue’s most valuable feature A context in Fugue reflects the cumula-
for this. Traditional scores are data in- is that it encourages composers t o de- tive effect of nested transformations on
tensive and contain much detailed in- velop personal musical vocabularies, environmental parameters such as cur-
formation because composers want to unencumbered by a language-specified rent time. stretch. transposition, and
express the results of their creativity, model of how music or music computa- overall loudness. Behaviors can be hi-
rather than show the process of cre- tion should be structured. Of course. erarchical compositions of other behav-
ation. any language, including Fugue, is bound iors. Once defined, a behavior can have
Consequently, lists of notes and their to influence how composers approach many instances, each of which can be
attributes have been the standard form programming or composition. Howev- evaluated in a different context and/or
of machine-readable scores for many er, Fugue supports the Music V model with different parameters. In this way,
years. As data structures, note lists can of computation as a subset, so we can at composers can define concepts such as
be transformed in time, in pitch, or least claim an improvement in flexibil- “drum roll” o r “glissando” once and
along any other dimension defined by ity and generality. apply them in many different contexts.
note attributes. It is generally easier to Fugue allows the composer to treat A behavior realized according to a
generate and manipulate data struc- simple instruments a n d sounds as context is called a behavioral ub-
tures than programs. For example, building blocks for more complex sound straction.’ We present a few examples
making all the notes in a section louder events. This development process is t o show how Fugue uses behavioral
is easy if the notes are represented as supported by its abstraction capabilities abstractions. In a sequence of three
data. But note lists can suffer from the and an interactive language interpreter. sounds
fact that they are not programs. In par-
ticular. there is a schism between the (seq (tremoloA3) (cue wind) (oscBf3))
“score” and the “orchestra”: The score Behavioral abstraction
controls while the orchestra executes; cue is a behavior that simply plays a
the functions are not integrated. An important feature of Music V is sound at a given time, and tremolo and
A common way to address this di- that note starting times and durations osc play pitches ( A and B-flat below
lemma is to use programs to compute determine when and how long the sys- middle C). Osc and cue are built-in be-
note lists. This presents at least two tem instantiates an instrument. In ad- haviors, while tremolo is defined by the
problems. Composers must deal with dition to unifying the score and orchestra, composer in terms of built-in behaviors.
yet another language (as if two were Fugue also supports this approach. Wind is a sound, perhaps loaded from a
not enough). and the ultimate result Otherwise, temporal aspects of com- sound file.
does not close the gap between the positions might be very difficult to ex- If we want to hear the same sequence
score and orchestra. For example, note- press. half as loud and with the wind sound
generating procedures cannot use the Starting times and durations in Music delayed by 2 seconds, we write
results of signal processing functions in V provide a special kind of abstraction:
the orchestra, and instruments in the Instruments define a class of behaviors (loud 0.5
orchestra cannot call on the note-gen- that can have any starting time or du- (seq (tremoloA3) (at2.0(cuewind))
erating capabilities of the score. ration. Abstraction is important because (osc Bf3)))
Fugue offers a new approach. Fugue it is not always obvious what to do to
scores are actually program expressions make a note longer. For example, a Suppose we wish t o change the pitched
that. when evaluated. return audio or violinist typically lengthens a note by sounds of the sequence. We write
control signals. Fugue also defines in- drawing the bow across the string for a
struments in terms of expressions that longer period of time, but if the note is (transpose 3 (seq (tremolo A3)
denote audio signals. Thus. composers a tremolo, with rapid back-and-forth (cue wind) (osc Bf3)))

July 199 I 37
 sustained tone (as in the technique used
by sampling synthesizers). Quickly
swapping periods during this cycling
produces a tone with a time-varying
spectrum, a technique used extensively
in the latter half of the piece. In the first
half, the composer used tones generat-
ed from extracted periods to create
chords, glissandi, and chorus effects.
Spomin illustrates the advantage of
using an integrated language for ex-
pressing score information as well as
signal processing. Composers can dele-
gate signal processing routines to low-
level functions and then work more ab-
stractly using high-level functions.
Moreover, Fugue modules, modified to
output printing information rather than
digital samples, can produce the graph-
ical portion of a score. Figure 1 gives an
example.
The code in Figure 2 shows the layer-

zyxw
ing of several levels of abstraction. In
this example a short slice is cut from the
source sound and cycled to form a grain.
Grains of sound form pebbles, which
are strung on a necklace. The second
band from the top in Figure 1 shows
necklaces modified at the grain level to
rise in pitch over time. Each grain is
represented by a short line, angled to
indicate where (in the source) a slice

zyxwvutsrq
was extracted. The third, fourth, and
fifth bands show timing and amplitude
information. The top band shows glis-

zyxwvuts
sandi of tones built from extracted peri-
ods.
Figure 1. Page 25 from the score of Spomin. To generate the score, we replaced Without Fugue's abstraction capabil-
Fugue's sound generation modules with modules that output graphics com-
ities and computational support, the top
mands. The modules ensure that the score accurately reflects the sound. The fi-
level of the score would consist of many
nal score shown here includes manually added music symbols.
thousands of notes, each corresponding
to a tiny grain of sound. Such a score is
technically feasible but impractical to

zyxwvutsr
construct or edit by hand.
Because Fugue allows composers to
This says to transpose the sequence up An example combine components to form complex
by three semitones. However, the cue structures, they can control large sound
abstraction overrides and prevents the events with a few commands. The inter-
transposition of wind because cue is The composition Spomin (an unpub- active environment provides rapid
intended for use with unpitched sounds. lished score by P. Velikonja) uses thou-
Hence. only the tremolo and osc behav- sands of transformations of a single
iors are affected. We could replace cue human vocal utterance (see the box at
with a behavior that would allow the right). The composer used Fugue sound-
wind sound to transpose. processing primitives to manipulate the
In general, we defined transforma- source sound to varying degrees, so some
tions and built-in behaviors to work sounds are clearly vocal while other
like traditional unit generators. As a more highly processed sounds bear lit-
result, most instrument definitions in tle relation to their vocal source. Using
Fugue support the standard transfor- Fugue, the composer extracted slices of
mations implicitly. But composers can the source, in some cases down to the
always customize the default behavior level of an individual period. Once ex-
as needed. tracted, a period can be cycled to form a

38 COMPUTER
feedback, so composers have greater
control over sound materials. Fugue’s
supportive framework helps composers
explore new musical forms and compo-
sitional methods. Composers can define

zyxwvutsrqpon
a musical syntax so that they can com-
pose music as a process rather than as a

zyxwvutsrqp
simple series of notes. Some might ob-
ject that Lisp is too great an obstacle for
noncomputer scientists, so it is worth
mentioning that Spomin is the first full
piece generated using Fugue and was
the composer’s first exposure to Lisp.

Implementation
We implemented Fugue in a combi-
nation of C and XLisp to run on Unix
workstations. (Written by David Betz,
XLisp is an interpreter which is in turn
implemented in C.) We used XLisp
because it is fairly easy to extend with
new data types, and it is also easy to
interface with C programs for signal
processing. Lisp provides convenient
and powerful interaction, and C effi-
ciently implements low-level function-
ality. It might be possible to use a more
efficient compiled Lisp. However, most
of the computation time is taken by
signal processing primitives, so the Lisp
interpretation represents only a small
overhead.
We implemented Fugue’s transfor-
mation context in Lisp. Operators such
as Transpose are macros that bind an
element of the context (*Transpose* in
this case) and then evaluate the em- Figure 2. An example Fugue program taken from Spomin, showing multiple
bedded expression. The binding is re- levels of abstraction. Any level can be invoked interactively for testing, or from
stored upon exit from the transforma- within a higher level expression serving as a score.
tion. Composers can introduce new
synthesis techniques into Fugue by
combining existing behaviors or by
writing new sound synthesis algorithms tra or to smooth envelopes, and use Fugue doesn’t actually compute any
in C and calling them from Lisp. multiplication uniformly for gain con- samples until absolutely neosssary, and
Our implementation includes multi- trol, amplitude envelopes, or audio-rate when it does, it can combine opera-
ple sample rates and lazy evaluation to amplitude modulation. tions. For example, Fugue commonly
increase time and memory efficiency. We implemented sounds in Fugue as allocates only one array to hold the
Multiple sample rates allow for “con- an extension to Lisp. Sounds are im- result of many additions. This tech-
trol” signals at a low sample rate, as in mutable values, meaning that once a nique avoids many needless ‘copyoper-
the Music-11 system and Csound,3 re- composer creates a sound, it cannot be ations and is completely hidden from
ducing time and memory requirements. altered. Therefore, the implementation the composer.
When the system must manipulate two cannot add several sounds directly into
sounds with different sample rates, it a single buffer. Instead, each addition
uses linear interpolation by default to of two sounds produces a new sound. Example of lazy
resample the lower sample rate signal Using lazy evaluation,6 we avoid the evaluation
to the higher sample rate. Composers inefficiency of immutable values. When
can explicitly specify other types of in- composers perform additions (and
terpolation. Because there is no distinc- many other operations), our implemen- To demonstrate how Fugue’s imple-
tion between control and audio signals, tation merely builds a small data struc- mentation of lazy evaluation works, we
composers can use filters to modify spec- ture describing the desired operation. show what memory structures look like

July 1991 39
I zyxwvutsrqponmlkj
zyxwvutsrqponmlk I one in which the actual samples have

zyxwvu
been computed and storage for samples
has been allocated, as shown in Figure
6. This last step is the only time the
system allocates new storage for sample
data and actually computes a new sound
Figure 3. Sequence of operations for the lazy evaluation example. sample.

Mysound L zyx
zyxwvutsrqp
From 0.0 sec.
Imagine a typical computation of the
form

for i := 1 to 100 d o
mypiece := mypiece t
MakeNote(i);

In Fugue, a roughly equivalent program

would be

Figure 4. Memory structure of My- (seqrep (i 100) (make-note i))

sound after the first operation in Fig-
ure 3, assuming the duration of My- where seqrep is a control construct that
sound is 1 second. The system loaded concatenates some number of instances
the samples from a file. of a behavior - in this case 100 copies
of make-note.
Typically, MakeNote(i) generates a
relatively short signal to be added to a
much longer mypiece. Without lazy

I
” :7Scale
End
2.0
0.0 sec.
2.0 sec.
2.0 sec.
I
evaluation, each addition requires the
system to

(1) allocate memory at least the size

of mypiece to hold the sum of the
two signals,
(2) copy mypiece into the new mem-
ory area, and
(3) add the result of MakeNote to

zyxwvutsrqpon form the new signal.

zyxwvutsrqzyx
Mysound Allocating memory and copying signals
make this computation costly.
Scale 1.0 I Scale 1.0 Scale 1.0 With lazy evaluation, each “lazy” ad-

II I
From 0.0 sec. From 0.0 sec. From 0 0 sec.
1.O sec. To 1.0 sec. To 1.O sec dition simply adds another level to a
End
To
Shift
1.0 sec.
0.0 sec. shift
End 1.0 sec.
0.0 sec.
End
Shift
1.0 sec
1.0 sec.
tree of sum nodes like the one in Figure
Stretch
SRate
1.0
16 kHz I Stretch
~SRate
1.0
16 kHz
Stretch
SRate
10
16 kHz
5. T o produce the final result, the sys-
tem traverses the tree to determine the
size of the result, allocates memory, and
/ I adds the leaves of the summation tree.
Samples for “mysound”
This technique eliminates memory allo-
cation and signal copying to form inter-
Figure 5. Memory structure of Demo after the second operation in Figure 3. mediate results.
The transformations and summation are reflected in the data structure, so the We considered but did not implement
system doesn’t need to compute new sound samples. another approach for efficient execution.
In the alternate approach, the composer
explicitly allocates a buffer to hold the
final sum of the MakeNote signals, and
at each step of a sequence of operations. tion are involved, essentially no compu- the system allows the buffer to be modi-
Figure 3 shows the operations. The first tation takes place. Figure 5 shows the fied by an add-signal operation. This
line sets the variable Mysound toa sound resulting configuration. The system has approach violates the principle that sig-
stored in the file “mysound.” Figure 4 performed no addition; instead, Fugue nals are immutable, and it places more
shows the resulting configuration. The represents the sum as a data structure. storagemanagement burden on thecom-
second line evaluates a score consisting Finally, the third line forces the system poser. Lazy evaluation, on the other hand,
of two copies of Mysound in sequence. to produce samples for Demo. It replac- allows Fugue to exhibit clean and simple
Because only time-shifting and addi- es the representation for Demo with semantics without loss of efficiency.

40 COMPUTER
Future directions zyxwvu
zyxwvutsr
We need to extend Fugue with more
sound functions as in Moore’s cmusic

zyxw
(distributed by the University of Califor-
nia at San Diego), Vercoe’s Csound (dis-
emo

End
Shift
Stretch
2.0 sec.
0.0 sec.
1.0
References
1. M.V. Mathews, The Technology of
Computer Music, MIT Press, Boston,
1969.

2. F.R. Moore, Elements of Computer Mu-

zyxwvuts
tributed by the MIT Media Lab), Next’s sic, Prentice Hall, Englewood Cliffs, N.J.,
SRate 16 kHz
Sound Kit,’ and Lansky’s Cmix (distrib- 1990.
uted by the Princeton University Music
Department). These systems are popu- 3. B. Vercoe, “Csound: A Manual for the
lar partly because of the library of syn- Audio Processing System and Support-
thesis techniques they provide. ing Programs,” MIT Media Lab, MIT,
We may investigate the use of Fugue und Cambridge, Mass., 1986.
in parallel computation. Because of its
functional style, Fugue programs con- 4. A.J. Field andP.G. Harrison,Functional
Programming, Addison-Wesley, Read-
tain explicit parallelism in the sim (“si- ing, Mass., 1988.
multaneous”) construct. Even when
there are data dependencies such as in 5. R.B. Dannenberg, “Expressing Tempo-
the seq construct, lazy evaluation often ral Behavior Declaratively,” Carnegie
defers signal computations so that the Mellon Computer Science 25th Anniver-
data dependencies can be resolved im- t sary Proc., Addison-Wesley, Reading,
Samples for “mysound” Mass., 1991.
mediately. Then the signal processing
can proceed in parallel. If we implement
6. R.B. Dannenberg and C.L. Fraley,
sounds in Fugue as streams, we can Figure 6. Memory structure of Demo “Fugue: Composition and Sound Syn-
achieve even more parallelism by lazily after the third operation in Figure 3. thesis with Lazy Evaluation and Behav-
evaluating streams. This could dramat- The system computes samples accord- ioral Abstraction,” Proc. Int’l Computer
ically reduce the memory requirements ing to the transformations in Figure 5 Music C o n f , Computer Music Assoc.,
for intermediate results in Fugue ex- San Francisco, 1989, pp. 76-79.
and caches the resulting samples, re-
pressions. Even with large virtual mem- placing the previous data structure
ories and automatic garbage collection, 7. D. Jaffe and L. Boynton, “An Overview
with a new one. of the Sound and Music Kit for the Next
storage is a serious problem in the cur- Computer,” Computer MusicJ., Vol. 13,
rent implementation. No. 2, Summer 1989, pp. 48-55.
An exciting potential of the lazy eval-
uation of streams is real-time execu- this simpler. Fugue should also sup- 8. R.B. Dannenberg, “A Runtime System
tion.8 This would require real-time port the use of MIDI (musical instru- for Arctic,” Proc. Int’l Computer Music
garbage collection as well. We have not ment digital interface) files as scores Conf., Computer Music Assoc., San
so that composers can use existing Francisco, 1990, pp. 185.187.
yet explored many opportunities for the
compilation and optimization of Fugue music editors as data sources. Another
behaviors. improvement would b e t o allow 9. R.B. Dannenberg, “The Canon Score
time-varying transformations,y using Language,” Computer MusicJ., Vol. 13,
No. 1, Spring 1989, pp. 47-56.
signals in place of real numbers to
achieve musical effects such as accele-
rando (gradual increase in overall tem-

M
any of the ideas we use in
Fugue seem appropriate for po) and crescendo (gradual increase in
computer graphics and com- overall loudness). Also, Fugue must
puter animation. The idea of behavioral support multidimensional signals. We
abstraction seems to fit nicely with graph- plan to make these changes in a future
ical transformations (for example, version. W
“make this truck longer”) and also with
action in animations (“run faster”). In
computer animation, Fugue’s notions
of explicit timing and constructs for
Acknowledgments

zyxwvut
parallel and sequential behavior might
be useful. For images, new constructs
might be added to represent spatial as We thank Carnegie Mellon University and
well as temporal relationships. Yamaha Music Technologies for their sup- Roger B. Dannenberg is a senior research
We could extend Fugue’s semantics
in several ways. Currently, only some-
one with a fair understanding of how
contexts are implemented can extend
the context in Fugue. We should make

July 1991
zyxwvutsr
port of this work. Dean Rubine provided
Lisp consultation for the production of
Spomin, and we used his Strips library to
produce the graphical score. George Polly
contributed some of Fugue’s signal process-
ing functions.
computer scientist at Carnegie Mellon Uni-
versity. His research interests include pro-
gramming-language design and implemen-
tation, and the application of computer
science techniques to the generation, con-
trol, and composition of computer music. He

41
 -
-
=
-
-
- zy
zyxw
Carnegie Mellon University
Software Engineering Institute

SEI Conference on
sofim
is codirector of the Piano Tutor project, whose
goal is applying music understanding and ex-
pert system technology to music education.
He frequently performsjazz and experimental
music on trumpet or electronically.
Dannenberg received a BSEE from Rice
University in 1977, an MS in computer engi-
neering from Case Western Reserve Universi-
ty in 1979, and a PhD in computer science from
Carnegie Mellon in 1982. He is a member of Phi
Beta Kappa, Sigma Xi, Tau Beta Pi, Phi Mu
Alpha, ACM, and SIGCHI, and research coor-
dinator for the Computer Music Association. zyx
P
T zyxwvutsr are Maintenance Project
aduate Courses in Software
Christopher Lee Fraley is a computer de4ign
engineer at Microsoft Corporation, where he
is working on graphical user interface build-
ers. His interests include computer applica-
tions in music and poetry. He received his BA
in computer engineering from Carnegie Mel-
Ion University in 1989.

g for Software Engineers

Computing Curricula 1991:

Its Implicationsfor Software Engineering
Education
Computer Based Systems Engineering
Teaching Styles for Software Engineering
Teaching Project Management Bottom Up
Seven Lessons to Teach Design
Peter Velikonja is an assistant professor of
Desi n Evolution: Implications for oboe and theory at West Virginia University
Aca$emia and Industry and is active as a performer and composer. He
has performed with several major orchestras,
Teaching Software Design in the Freshman including the Chicago Symphony and the Met-
Year ropolitan Opera Orchestra. At the School of
Computer Science at Carnegie Mellon Uni-
Teaching Software Engineering for Real-Time versity. he writes music using digital synthesis
Design and develops other music software.
Velikonja received his training at North-
Industrial-StrengthCASE Tools for western University; the Folkwang Musikhoch-
Software Engineering Classes schuleinEssen,Germany,onaFulbrightgrant;
and the Mannes College in New York City.
What We Have Learned About
Software Engineering Expertise

-m
For
registration
Instructionfor Software Engineering Expertise
Knowledge Elicitation for
Software Engineering Expertise
1
Dannenberg can be reached at the School of
Computer Science, Carnegie Mellon Univer-
sity, Pittsburgh, PA 15213-3890. His e-mail
address is [email protected].
information Helen E. Joyce
contact-
(412) 268-6504
I COMPUTER