Practical Applications Of Sparse Modeling

Practical Applications Of Sparse Modeling

download

https://ebookbell.com/product/practical-applications-of-sparse-
modeling-practical-applications-of-sparse-modeling-56401682

Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Practical Applications Of Medical Geology Malcolm Siegel Olle Selinus

https://ebookbell.com/product/practical-applications-of-medical-
geology-malcolm-siegel-olle-selinus-46404368

Practical Applications Of Coaching And Mentoring In Dentistry Janine

Brooks

https://ebookbell.com/product/practical-applications-of-coaching-and-
mentoring-in-dentistry-janine-brooks-46654252

Practical Applications Of Computational Biology And Bioinformatics

16th International Conference Pacbb 2022 1st Ed 2023 Florentino
Fdezriverola

https://ebookbell.com/product/practical-applications-of-computational-
biology-and-bioinformatics-16th-international-conference-
pacbb-2022-1st-ed-2023-florentino-fdezriverola-46787820

Practical Applications Of Physical Chemistry In Food Science And

Technology A K Haghi

https://ebookbell.com/product/practical-applications-of-physical-
chemistry-in-food-science-and-technology-a-k-haghi-48681378
Practical Applications Of Computational Biology And Bioinformatics
17th International Conference Pacbb 2023 Miguel Rocha

https://ebookbell.com/product/practical-applications-of-computational-
biology-and-bioinformatics-17th-international-conference-
pacbb-2023-miguel-rocha-50849258

Practical Applications Of Intravenous Fluids In Surgical Patients 2nd

Edition Shaila Shodhan Kamat

https://ebookbell.com/product/practical-applications-of-intravenous-
fluids-in-surgical-patients-2nd-edition-shaila-shodhan-kamat-51883550

Practical Applications Of Radioactivity And Nuclear Radiations

Lowenthal Gc

https://ebookbell.com/product/practical-applications-of-radioactivity-
and-nuclear-radiations-lowenthal-gc-2045580

Practical Applications Of Asymptotic Techniques In Electromagnetics

Hardvdr Francisco Saez De Adana

https://ebookbell.com/product/practical-applications-of-asymptotic-
techniques-in-electromagnetics-hardvdr-francisco-saez-de-adana-2134262

Practical Applications Of Microresonators In Optics And Photonics

Optical Science And Engineering 1st Edition Andrey Matsko

https://ebookbell.com/product/practical-applications-of-
microresonators-in-optics-and-photonics-optical-science-and-
engineering-1st-edition-andrey-matsko-2212780
2014
c Massachusetts Institute of Technology

All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means
(including photocopying, recording, or information storage and retrieval) without permission in writing from
the publisher.

MIT Press books may be purchased at special quantity discounts for business or sales promotional use. For
information, please email special_sales@ mitpress.mit.edu.

This book was set in the LATEX programming language by the author. Printed and bound in the United States of
America.

Library of Congress Cataloging-in-Publication Data

Practical applications of sparse modeling / edited by Irina Rish,
Guillermo A. Cecchi, Aurelie Lozano, and Alexandru Niculescu-Mizil.
pages cm. — (Neural information processing series)
Includes bibliographical references and index.
ISBN 978-0-262-02772-4 (hardcover : alk. paper) 1. Mathematical
models. 2. Sampling (Statistics). 3. Data reduction. 4.
Sparse matrices. I. Rish, Irina. 1969– editor of compilation.
TA342.P73 2014
003’.74—dc23
2014003812

10 9 8 7 6 5 4 3 2 1
Series Foreword

The yearly Neural Information Processing System (NIPS) workshops bring together sci-
entists with broadly varying backgrounds in statistics, mathematics, computer science,
physics, electrical engineering, neuroscience, and cognitive science, unified by a com-
mon desire to develop novel computational and statistical strategies for information
processing and to understand the mechanisms for information processing in the brain.
In contrast to conferences, these workshops maintain a flexible format that both allows
and encourages the presentation and discussion of work in progress. They thus serve as
an incubator for the development of important new ideas in this rapidly evolving field.
The series editors, in consultation with workshop organizers and members of the NIPS
Foundation Board, select specific workshop topics on the basis of scientific excellence,
intellectual breadth, and technical impact. Collections of papers chosen and edited by
the organizers of specific workshops are built around pedagogical introductory chapters,
and research monographs provide comprehensive descriptions of workshop-related top-
ics, to create a series of books that provides a timely, authoritative account of the latest
developments in the exciting field of neural computation.

Michael I. Jordan and Thomas G. Dietterich

 C H A P T E R 1
Introduction
Irina Rish, Guillermo A. Cecchi, Aurelie Lozano, and
Alexandru Niculescu-Mizil

Sparse modeling is a rapidly developing area at the intersection of statistical learning

and signal processing that has recently produced an impressively large body of novel
theoretical results, efficient algorithms, and successful practical applications. From a
statistical point of view, sparse modeling is motivated by the age-old variable selection
problem concerned with finding a relatively small number of most predictive variables
in high-dimensional data sets. This objective is particularly important for improving
the interpretability of predictive models in scientific applications such as computa-
tional biology; for example, identifying a subset of genes relevant to a particular dis-
ease can potentially improve our understanding of underlying biological processes and
contribute to better diagnostic methods. Moreover, variable selection provides an effec-
tive way of avoiding the “curse of dimensionality” because it prevents overfitting and
reduces computational complexity in high-dimensional but small-sample data sets.
A closely related motivation for sparse modeling arises in signal processing
applications, such as image processing, concerned with the efficient recovery of high-
dimensional unobserved signals from a limited number of measurements. As in the
statistical setting, it is assumed that most of the signal’s coordinates are zero (or close
to zero), and thus the effective dimensionality of a problem is much smaller than its
ambient dimension. Thus, a seemingly intractable problem of reconstructing a high-
dimensional signal from a small number of measurements can be solved by restricting
attention to only sparse solutions. Moreover, while the ultimate sparse recovery—the
smallest subset selection, also known as l0 -norm optimization—is an NP-hard combina-
torial problem, sparse solutions can often be found in a computationally efficient way
by using convex relaxation such as l1 -norm minimization and other sparsity-inducing
priors. The applications of sparse modeling are wide-ranging, including compressed
sensing, computational biology, neuroscience, image processing, and social network
analysis.
2 Chapter 1 Irina Rish and colleagues

However, is the promise of sparse modeling fully realized in practice? Despite the
significant advances in the field, a number of open issues remain when sparse mod-
eling meets real-life applications. For example, achieving stability and reproducibil-
ity of sparse models is essential for their interpretability, particularly in computational
biology and other scientific applications. Scalability of sparse learning and sparse sig-
nal recovery algorithms is essential when the number of variables goes much beyond
thousands, as, for example, in neuroimaging applications such as functional magnetic
resonance imaging (fMRI) analysis. Novel, more complex types of structure, dictated by
the nature of applications, require the choice of novel regularizers (so-called structured
sparsity). Moreover, feature construction, or finding a proper dictionary allowing for
sparse representations, remains a critical issue in many practical domains.
The aim of this book is to discuss a range of practical applications of sparse model-
ing, from biology and neuroscience to topic modeling in video analysis, and to provide
an overview of state-of-the-art approaches developed for tackling the challenges pre-
sented by these applications. This book is based on the contributions presented at the
NIPS-2010 Workshop on Practical Applications of Sparse Modeling and several invited
chapters.
The book is structured as follows. Chapter 2 provides a brief overview of some
challenging issues arising in computational biology, one of the traditional applications
of sparse modeling, where the primary goal is to identify biological variables such as
genes and proteins that are most relevant (ultimately, causally related) to a biologi-
cal phenomenon of interest. The chapter introduces several biological fields, such as
genomics, proteomics, metabolomics, and transcriptomics, and discusses some high-
dimensional problems arising in these areas, including genome-wide association stud-
ies (GWAS), gene expression (DNA microarray) data analysis, reverse engineering of
cellular networks, and metabolic network reconstruction. Also, neuroimaging applica-
tions, that is, statistical analysis of fMRI, EEG, PET, and other brain imaging data that
involve prediction of mental states and localizing brain areas most relevant to a par-
ticular mental activity are introduced here as another rich source of high-dimensional,
small-sample problems that can benefit from sparse techniques. Overall, the goal of
chapter 2 is to provide biological background for the subsequent five chapters, which
focus on particular aspects of sparse modeling in applications to biology and neuro-
science.
Chapter 3 discusses several key properties of applications that influence the
choice of the sparse methods: (1) the amount of correlation among the predictive vari-
ables, (2) the expected level of sparsity (the fraction of important variables versus the
total number of predictors), and (3) the primary objective of predictive modeling, such
as accurate recovery of the true underlying sparsity pattern versus an accurate predic-
tion of the target variable. Chapter 3 focuses on two popular biological problems—the
genome-wide association studies (GWAS) and gene expression (DNA microarray) data
analysis—as examples of practical applications with different properties. A simplify-
ing assumption that is traditionally adopted in GWAS and often realized in practice
is that only a very small number of almost uncorrelated input variables (predictors),
 Introduction 3

corresponding to single-nucleotide polymorphisms (SNPs), are truly relevant to a given

phenotype (output variable). It is argued that under this assumption, simple univari-
ate (or filter-based) variable selection approaches tend to work well and are competitive
with l1 -regularized methods such as Lasso. (Note, however, that in more complex GWAS
problem scenarios that do not fit into these simplistic assumptions, more sophisticated
sparse methods can be justified, as discussed in chapter 4). On the other hand, gene
expression data tend to exhibit a complex correlation structure across the variables, and
the sparsity level may not be as extreme as in case of traditional GWAS. Chapter 3 argues
that in such scenarios a simple univariate approach is insufficient, and better results are
achieved by more sophisticated embedded variable selection methods such as Lasso and
its various augmented versions, including the Random Lasso method.
Chapter 4 continues exploring sparse predictive modeling in GWAS applications.
Its focus is on more complex scenarios not covered by traditional assumptions. Namely,
in-depth consideration is given to various dependencies among both inputs and out-
puts of the regression models; these dependencies can be captured by group-sparse
approaches such as structured input, structured output, and structured input-output
regression. Structured approaches incorporate prior knowledge about the relations over
the groups of regression coefficients, from simple nonintersecting groups to overlap-
ping groups and hierarchical structures; block-wise regularizers such as l1 /l2 -norm and
l1 /l` -norm are typically used to enforce group-level sparsity. The proposed methods
are shown to be better than traditional univariate techniques and basic Lasso at captur-
ing nontrivial GWAS properties that include linkage disequilibrium (correlated inputs),
epistasis (nonlinear interactions among SNPs in their influence on phenotypic traits),
and population stratification (the presence of multiple populations associated with dif-
ferent statistical properties), all combined in a single data set. Chapter 4 discusses a
variety of structured regression problem formulations and optimization techniques for
solving them, and presents numerous empirical results on both simulated examples and
on practical GWAS applications.
Chapter 5 discusses application of sparse recovery methods to the analysis of
protein mass spectrometry data. The objective is to extract biologically relevant com-
ponents (peptides) from the raw protein mass spectrum (MS). This problem can be for-
mulated as a sparse recovery and addressed by standard sparse recovery methods, such
as, for example, l1 -regularized linear regression (Lasso) or greedy algorithms such as
orthogonal matching pursuit (OMP). However, certain domain-specific properties must
be taken into account in order to achieve good performance: two key properties of the
mass spectrometry data are non-negativity and heteroscedastic noise (combination of
several noise types with different statistical properties). In order to accommodate those
properties, chapter 5 proposes augmented versions of the standard sparse techniques
and demonstrates their advantages in experiments. It is interesting to note that in some
situations non-negativity alone can serve as a powerful constraint for enforcing sparsity,
as suggested by theoretical results presented in that chapter. Moreover, empirical results
support the theory, demonstrating that a simple thresholding approach combined with
non-negative least squares minimization can outperform standard approaches that use
4 Chapter 1 Irina Rish and colleagues

an explicit sparsity-enforcing regularization. Finally, chapter 5 discusses practical sit-

uations in MS applications when some standard assumptions made in sparse recovery
framework (e.g., absence of model misspecifications and an upper bound on the coher-
ence of the dictionary) are not satisfied and proposes a postprocessing procedure for
handling such situations.
The following two chapters, chapter 6 and chapter 7, focus on stability of sparse
models in brain imaging applications, such as fMRI. In fMRI, an MR scanner nonin-
vasively records a subject’s blood-oxygenation-level-dependent (BOLD) signal, known
to be correlated with neural activity, as a subject performs certain tasks (e.g., viewing
a picture or reading a sentence) or is exposed to some other kind of stimulus. Such
scans produce a sequence of 3-D images, where each image typically has on the order
of 10,000–100,000 subvolumes, or voxels, and the sequence typically contains a few
hundred time points, or TRs (time repetitions). One of the key questions in fMRI anal-
ysis is to discover which brain areas (spatial clusters of voxels) are most relevant to the
applied stimulus or observed behavior. The traditional approach in the field addresses
this question by univariate voxel selection, based on individual voxel correlations with
stimulus or mental state; the voxels correlated above a certain threshold are said to
be activated by the task and are typically presented as brain activation maps. A more
recent trend, as discussed in both chapters, is to employ multivariate sparse regression
techniques, such as Lasso and Elastic Net, for discovering relevant subsets of voxels
simultaneously with learning a predictive regression model. However, no matter how
such sparse voxel patterns (or brain maps) are produced, their stability (reproducibility)
across multiple experiments of similar kind is essential for neuroscientific interpretabil-
ity. Chapter 6 focuses on the trade-off between the predictive accuracy of various linear
models—including sparse linear regression, Fisher discriminant analysis (FDA), and
support vector machine (SVM) models—and their stability, which is measured here as
the correlation between the (vectorized) maps, or sparse patterns, learned over sub-
sampled data sets. That chapter argues that the accuracy versus interpretability plots
that can by obtained by varying parameters, such as, for example, sparsity level, are a
reasonable substitute for ROC curves, since the latter are impossible to obtain without
knowledge of the true underlying sparse pattern. Overall conclusions are that both sta-
bility and predictive accuracy must necessarily be taken into account in fMRI analysis
and that optimization of regularization parameter(s) with those two criteria in mind may
be much more important than the choice of a particular discriminant model.
Chapter 7 builds on the stability studies presented in the previous chapter and in
other related work. While the stability (also called reliability in that chapter) property
appears to be as important as predictive accuracy in brain imaging studies, the question
remains how to best measure the stability of sparse brain models or, more generally, any
brain maps representing task-relevant voxels. In chapter 7 it is argued that simply using
vector similarity metrics such as overlap or correlation (see chapter 6) does not account
for the statistical significance of the similarity observed across different brain maps. To
resolve this issue, it is proposed to use as a null hypothesis a random map that pre-
serves the overall level of functional activity and spatial structure, and to estimate the
 Introduction 5

significance of stability based on such null hypothesis. This method appears to signif-
icantly impact the stability results and provides a better, significance-based approach to
stability evaluation. Also, chapter 7 proposes that spatial smoothing be used as a simple
way of improving stability without sacrificing much of prediction accuracy. Studies of
predictive accuracy versus model stability, as defined in that chapter, also demonstrate
that the two metrics can be positively correlated, though highly nonlinearly; thus, as
observed in prior work, including chapter 6, equally predictive models may have quite
different stability, and clearly more stable ones are preferred for the purpose of neuro-
scientific interpretation.
Since highly efficient sparse recovery techniques are essential in large-scale appli-
cations, chapter 8 focuses on improving the efficiency of sparse recovery methods by
using sequential testing approaches. Unlike traditional (nonsequential) sparse recov-
ery, sequential (adaptive) approaches make use of information about previously taken
measurements of an unknown sparse signal when deciding on the next measurement.
While the standard procedures require the number of measurements logarithmic in the
dimension of the signal in order to recover the signal accurately, sequential procedures
require the number of measurements logarithmic in the sparsity level, that is, the num-
ber of nonzeros. This can lead to a dramatic reduction in the number of measurements
when the signals are sufficiently sparse. The chapter considers two motivating appli-
cations: a biological one, concerned with identifying a small subset of a large number
of genes (e.g., more than 13,000 genes in a fruit fly) that are involved in virus replica-
tion using single-deletion strains, and an engineering application known as cognitive
radio, where the task is to quickly perform spectrum sensing (identification of currently
unused bands of the radio spectrum). Chapter 8 discusses the advantages of a novel
sequential testing procedure, sequential thresholding, which does not require knowl-
edge of underlying data distributions and the sparsity level (unlike the standard sequen-
tial probability ratio test (SPRT)), is very simple to implement, and yet is nearly optimal.
The chapter also provides a historic overview of the sequential testing field and sum-
marizes key theoretical results in this domain.
Algorithmic aspects of sparse recovery are further explored in chapter 9. Two
novel sparse recovery methods are proposed that, unlike most of their predecessors,
combine two types of sparsity-enforcing regularizers, or priors: convex l1 -norm and
nonconvex l0 -norm (the number of nonzeros, or sparsity level). Interestingly, this com-
bination results in better empirical performance as compared to state-of-the-art Lasso
solvers and also allows better theoretical sparse recovery guarantees based on weaker
assumptions than traditionally used in sparse recovery. One of the algorithms, called
the game-theoretic approximate matching estimator (GAME), reformulates the sparse
approximation problem that combines both l1 - and l0 -norm regularizers as a zero-sum
game and solves it efficiently. The second algorithm, combinatorial selection and least
absolute shrinkage (CLASH), leads to even better empirical performance than GAME but
requires stronger assumptions on the measurement matrix for estimation guarantees.
Chapter 10 considers the problem of learning sparse latent models, that is, mod-
els including unobserved, or latent, variables. This problem is often encountered in
6 Chapter 1 Irina Rish and colleagues

applications such as text or image analysis, where one might be interested in finding
a relatively small subset of (hidden) topics or dictionary elements that accurately approx-
imate given data samples. That chapter advocates using Bayesian sparsity-enforcing
methods with various sparsity-enforcing priors that go beyond the standard Laplace
prior corresponding to popular l1 -norm minimization. (Note that maximizing Laplace
log-likelihood is equivalent to minimizing the l1 -norm, and thus maximum a posteri-
ori (MAP) inference with Laplace prior is equivalent to standard l1 -norm minimiza-
tion.) Specifically, Chapter 10 focuses on the spike-and-slab prior and demonstrates
on multiple real-life data sets, including analysis of natural scenes, human judgments,
newsgroup text, and SNPs data, that this approach consistently outperforms the
l1 -norm-based methods in terms of predictive accuracy. However, this is a classic exam-
ple of accuracy versus (computational) efficiency trade-off, since Bayesian approaches
based on Markov Chain Monte Carlo (MCMC) inference can be considerably slower
than the l1 optimization. Overall, the message of the chapter is that the Laplace prior
that gives rise to l1 -norm formulations is just one out of many possible ways of enforc-
ing sparsity, and depending on a particular application and modeling goals, other pri-
ors may be preferred. While the current literature on sparse modeling is heavily biased
towards l1 -norm-based approaches, chapter 10 provides a convincing argument for more
widespread use of alternative sparsity-enforcing techniques.
Learning latent variable models, or topic models, is also the focus of chapter 11.
This chapter is motivated by computer vision applications, such as scene analysis and
event detection from video. An example considered here involves the scene analysis of
traffic videos taken at a busy intersection, where many vehicle- and pedestrian-related
activities occur simultaneously, and one would like to identify key activity components,
or sequences (motifs), corresponding to the car and pedestrian movements and perform
event detection. The problem is similar to the detection of changing topics using topic
models in text analysis but is also much more complex and challenging, since there
are multiple simultaneous activities, and no prior knowledge is given about the number
of such activities in the scene. The chapter reviews some sparsity-enforcing methods
for topic modeling and focuses specifically on a topic-based method for temporal activ-
ity mining that extracts temporal patterns from documents where multiple activities
occur simultaneously. Sparsity is enforced on the motif start time distributions of the
probabilistic latent sequential motif (PLSM) model, using information-theoretic formu-
lation. Empirical results on simulated data and real-life video suggest that sparsity con-
straint improves the performance of the method and makes the model more robust in the
presence of noise.
 C H A P T E R 2
The Challenges of Systems Biology
Pablo Meyer and Guillermo A. Cecchi

Biology oozes with complexity, from viruses to multicellular organisms. While the
complete physiology of a vertebrate animal, with its brain included, may apparently
dwarf that of a single cell, the intricacy of the interlocking mechanisms that account for
generic and type-specific cellular mechanisms is bewildering in itself. Eukaryotic cells,
for instance, need to coordinate a vast number of processes such as DNA transcription
into RNA, translation of RNA into the amino acid chains that make up proteins, trans-
port of proteins in and out of the nucleus, energy storage, regulation of protein synthesis
in response to sensed external signaling and genetically determined programs. The ini-
tial response to this complexity in the early years of modern molecular biology was
to develop a theoretical perspective that associated specific cellular functions and dis-
eases, such as circadian rhythms or cancer, with one or a handful of genes. It is still
quite common to find journalistic accounts and even scholarly articles on the “gene
for X.” For similar reasons, neuroscience has been also dominated by the grandmother
cell doctrine, the idea that each sufficiently elaborate mental function is reflected in the
activity of a specific neuron.
However, over the past two decades molecular biology has experienced a qual-
itative increase in the amount of data produced to answer its key scientific ques-
tions, forcing the transformation from molecular to systems biology. Molecular biology
tries to discover the missing molecular links between phenotype and genotype, that
is, to find the genes responsible for a particular phenotype/disease. The revolution of
genome sequencing led to new computational methodologies allowing the comparison
and study of species at the whole genome level (Loots 2008). Hence genes responsi-
ble for innate immunity in the fruit fly could be inferred in humans via gene sequence
comparison. Gene comparisons, however, are not enough. The function of genes does
not rely only on their sequence but also on their spatiotemporal expression resulting
from complex regulatory processes. With the advent of high-throughput technologies,
omics1 data types have provided quantitative data for thousands of cellular components
across a variety of scales, or systems. For instance, genomics provides data on a cell’s
8 Chapter 2 Pablo Meyer and Guillermo A. Cecchi

DNA sequence, transcriptomics on the mRNA expression of cells, proteomics on a cell’s

protein composition, and metabolomics on a cell’s metabolite abundance. Recently, to
the ability for massively sequencing genomes have been added the abilities to multiplex
single cell event observations with technologies such as flow cytometry, measurements
of multiple protein phosphorylation states responsible for different regulations, global
measures of cell messenger RNA and metabolites, protein-DNA interactions responsible
for transcription regulation, protein-protein interactions. A similar explosion in data
is experienced at the physiological level with advances in imaging (MRI, fMRI, PET,
EEG, MEG) and at the patient level with the advent of electronic health records that
allow doctors to have the whole medical history of their patients along with genomic
information.
One of the ultimate goals of systems biology is to provide a mechanistic under-
standing of biological systems from these high-throughput data. A central challenge for
its development is the integration of the data to generate predictive computational mod-
els. Computational methods are needed to reduce this dimensionality across the wide
spectrum of omics data to achieve an accurate understanding of the underlying biolog-
ical processes (Çakir et al. 2006; Pfau, Christian, and Ebenhoh 2011). Systems biology
has hence emerged as a new paradigm to address these problems. The novelty it has
brought consists in embracing the complexity of a system as the key for understand-
ing it. Rather than taking a reductionist approach, in which the system is subdivided
into its irreducible components, which are studied in isolation from the rest of the sys-
tem, systems biology aims at finding the key to the working of a system in the system
itself, postulating that in subdividing the system for study we miss essential features.
The availability of experimental data at several layers of cellular organization (gene
regulation, signaling, metabolism, mechanical properties) raises the question of integra-
tion of these measurements into a model that not only represents what we measured of
the system but that in principle embodies the possibility of making predictions of yet
unmeasured states of the cellular system. We now discuss in more detail subdisciplines
of biology that jump-started the acquisition of data of various kinds: genomics providing
data on DNA sequences, transcriptomics on mRNA expression, proteomics on proteins
composing a cell, and metabolomics on chemicals/metabolites abundance.

1 GENOMICS, GENE EXPRESSION, AND

NEXT-GENERATION SEQUENCING
The goal of the human genome project was to generate a sequence of bases A,T,C,G
for every expressed gene of the roughly 22,000 representing 1.5 percent of the human
genome (Lander et al. 2001). DNA sequencing has been extended to different species,
allowing the quest through genome comparison of regions purified through natural
selection (Lindblad-Toh et al. 2011) and to many human individuals through projects
 The Challenges of Systems Biology 9

such as the 1000 genomes project (Clarke et al. 2012). It has also allowed genome-wide
association studies (GWAS), where researchers take an unbiased survey of common
single-nucleotide polymorphisms (SNPs) across the genome and look for alleles whose
presence correlates with phenotypic traits such as disease.
SNPs are defined by a single-nucleotide variant in a DNA fragment of the genome
across individuals of the same species or in paired chromosomes of the same individual.
While SNPs tend to be found more in noncoding regions, increasing evidence indicates
that these regions are functionally relevant. It is expected that differences between indi-
viduals in susceptibility to disease and response to treatment are associated with these
genetic variations. GWAS are designed to scan the entire genome for these associations
between SNPs and disease, emerging potentially from millions of single-nucleotide vari-
ants. The utter dimensionality of the genome as target for variants poses a significant
challenge from a computational point of view, compounded with the current lack of
generative models that can connect SNPs and function in a mechanistic way.
As an example, the first GWAS study reported that patients with macular degen-
eration carry two SNPs with altered allele frequency compared to the healthy control
group (Klein et al. 2005). As the example highlights, this approach faces the challenge
of detecting a handful of variables out of several thousands or tens of thousands. More-
over, the molecular mechanisms that link these SNPs with the disease are completely
unclear, as is the extent to which other SNPs, perhaps with individually weaker sta-
tistical associations, may also contribute collectively to patients’ susceptibility to the
disease. However, hundreds of disease-related gene candidates have been found since
then, although most have only a modest effect (McCarthy et al. 2008). A more recent
example, using sparse (l1 -regularized) regression techniques, identified the risk loci
common to five major psychiatric disorders (schizophrenia, major depression, autism
spectrum, bipolar, and attention deficit hyperactivity disorders) and a subset of affected
genes involved in calcium channel signaling, which at least points in the direction of
biological interpretability (Smoller et al. 2013).
Genome sequencing has also facilitated the production of DNA microarrays to
generate genome-wide gene expression profiles based on the Watson-Crick base pair
complementarity of DNA. mRNA extracted from tissues or cells is commonly reverse-
transcribed into cDNA and hybridized into small glass or silicon arrays where a
short section of each of the expressed genes has been attached. The amount of DNA
hybridized is measured with fluorescent markers attached to the short DNA sections
printed in the arrays and reflect the amount of mRNA present in the biological sample.
This field of functional genomics has extended the classical gene-by-gene approach to
find sets of genes that are differentially expressed in cases of disease, such as in breast
cancer where 70 genes are used as a signature for diagnosis and prevention (van ’t Veer
et al. 2002). The extent of functional genomics growth is exemplified in the database
ArrayExpress containing publicly accessible microarray data from 2,284 different exper-
iments, 97,006 assays in 20,458 conditions. DNA sequencing and gene expression have
been recently engulfed in the revolution of new sequencing techniques (Gunderson et al.
2004; Rothberg et al. 2011) by which sequence and expression levels can be extracted;
10 Chapter 2 Pablo Meyer and Guillermo A. Cecchi

they rely on a higher number of sequencing repeats per nucleotide, also called depth.
Deep-sequencing of mRNA transcripts, also called RNA-seq, can detect 25 percent more
genes than microarrays as well as previously unidentified splicing events (Sultan et al.
2008).

2 METABOLIC NETWORK RECONSTRUCTION

In contrast to the widespread availability of genetic data, including DNA and RNA
expression profiles, it is only recently that researchers focused their attention in the
cell metabolome, that is, the full set of small-molecule metabolites, including metabolic
intermediates, hormones, and other signaling molecules, as well as secondary metabo-
lites, contained in a single organism. Beyond its basic scientific value, the state of the
metabolome is highly relevant to the understanding of the regulation of nutrients and
energy use in various diseases. Currently, the abundance of up to 2,200 different cell
metabolites can be determined using mass spectrometry platforms, mainly liquid chro-
matography (LC)-MS/MS and gas chromatography (GC)-MS. Since the most common
compounds, reactions, and enzymes have been biochemically determined, it is easy to
generate a metabolic network for a specific organism or cell from a set of biochemical
reactions. Once a network is defined, mathematical tools such as constraint-based flux
balance analysis modeling (FBA) (Ramakrishna et al. 2001), elementary mode analysis
(EMA) (Schuster, Fell, and Dandekar 2000) and graph theory (Planes and Beasley 2008)
have been used to analyze the flow of metabolites through the metabolic network by
integrating transcriptomic and metabolomic data into genome-scale metabolic network
reconstructions (Blazier and Papin 2012).
The overarching assumption behind these models is that mRNA levels can be used
as indicators of enzyme activity in the context of a specific cell or organism metabolic
state. Although metabolome-wide studies in response to perturbations are common in
bacterial cells, standard fractionation approaches are not possible for metabolomics
in eukaryotic cells because they result in mixing of metabolites between subcellular
compartments. Quantitative predictions of metabolome-wide responses to perturbations
in eukaryotic cells is complicated by compartmentalization of the cytoplasm, which
impedes a detailed quantification of metabolites necessary for a correct FBA (Reaves
and Rabinowitz 2011). In order to find algorithms that explain, for example, cancer
metabolism control, it is necessary to consider the contribution of multiple nutrients
and include quantitative information about the variability in protein levels and enzyme
activity as well as the localization of the active enzymes.

3 COMPUTATIONAL MODELS OF GENE TRANSLATION

Gene translation, the process of producing proteins from mRNA inside ribosomes, is
used by all known life forms, such as eukaryotes (including multicellular organisms)
and prokaryotes (bacteria and archaea) and is also frequently induced by viruses to
 The Challenges of Systems Biology 11

generate proteins. Consisting of the three main steps of initiation, elongation, and ter-
mination, translation is a central cellular process with ramifications related to all biolog-
ical and clinical research, including human health (Kimchi-Sarfaty et al. 2007; Coleman
et al. 2008; Lee et al. 2006; Bahir et al. 2009; van Weringh et al. 2011; Vogel et al. 2010;
Pearson 2011; Lavner and Kotlar 2011; Comeron 2006), biotechnology (Gustafsson,
Govindarajan, and Minshull 2004; Kudla et al. 2009; Plotkin and Kudla 2010; Supek
and Smuc 2010), evolution (Bahir et al. 2009; van Weringh et al. 2011; Drummond and
Wilke 2008, 2009; Shah and Gilchrist 2010a, 2010b; Plata, Gottesman, and Vitkup 2010;
Bulmer 1991; Sharp and Li 1987), functional genomics (Danpure 1995; Lindblad-Toh
et al. 2011; Schmeing et al. 2011; Warnecke and Hurst 2010; Zhou, Weems, and Wilke
2009; F. Zhang et al. 2010; Fredrick and Ibba 2010), and systems biology (Bahir et al.
2009; Shah and Gilchrist 2010a; Fredrick and Ibba 2010; Z. Zhang et al. 2010; Man and
Pilpel 2007; Cannarozzi et al. 2010; Schmidt et al. 2007; Elf et al. 2003). There has been
a long-standing debate regarding the rate-limiting stage of translation and whether ini-
tiation or elongation is the bottleneck (Gustafsson, Govindarajan, and Minshull 2004;
Kudla et al. 2009; Burgess-Brown et al. 2008; Supek and Smuc 2010). If the initiation
step is relatively slow compared to elongation, codon bias (i.e., which bases in the third
position are preferred by ribosomes) should not affect the translation rate. However, if
initiation is fast relative to elongation, codon bias should have substantial influence on
protein levels. Additionally, determining which variables of mRNA transcripts are rel-
evant to initiation efficiency is yet not fully resolved, with recently reassessed features
such as mRNA folding strength (Tuller, Waldman et al. 2010) and the nucleotide con-
text of the first start codon ATG at the beginning of the open reading frame (ORF) (Kozak
2005) providing only very weak correlations with protein levels. Finally, it is not clear
if ORF features affect the elongation rate or which features are relevant to elongation
or how they affect translation efficiency (Kudla et al. 2009; Tuller, Waldman et al. 2010;
Welch et al. 2009; Ingolia, Lareau and Weissman 2011; Frenkel-Morgenstern et al. 2012).
Various features related to the translation process (e.g., protein levels, ribosomal
densities, initiation rates) have been taken into account in various model organisms (see
(Tuller, Waldman et al. 2010; Tuller, Kupiec and Ruppin 2007; Zur and Tuller 2012a;
Tuller 2011; Tuller, Veksler et al. 2011; Zur and Tuller 2012b; Reuveni et al. 2011; Tuller,
Carmi et al. 2010)) and to engineer gene translation (Dana and Tuller 2011). A gen-
eral predictor can be based on the different features of the untranslated region (UTR)
(e.g., small ORFs in the UTR named uORFs, GC content, mRNA folding in different parts
of the UTR), the ORF (e.g., codon frequencies and order, amino acid bias, ORF length),
mRNA levels, number of available ribosomes, and degradation rates when available.
Predictors may also be based on machine learning approaches or biophysical models
(Kudla et al. 2009; Welch et al. 2009; Reuveni et al. 2011). The challenge in inferring
causal relations between features of the transcripts and their expression levels is related
to the fact that highly expressed genes are often under evolutionary selection for various
features that do not improve translation. Thus, these features have significant correla-
tion with protein levels of a gene that is not causal or one that does not affect translation
efficiency. For example, highly expressed genes are under selection for features such
as increased mRNA self-folding to prevent aggregation of mRNA molecules (because of
12 Chapter 2 Pablo Meyer and Guillermo A. Cecchi

potential interaction with other genes), even though for a certain gene not interacting
with other mRNA molecules, increased mRNA folding may actually decrease translation
efficiency (Tuller, Veksler et al. 2011; Zur and Tuller 2012b).

4 REVERSE ENGINEERING OF CELLULAR NETWORKS

Different methods to reverse engineer cellular networks represent molecular data in
statistical, graphical, or mechanistic models in an attempt to integrate the informa-
tion into a cogent structure allowing for a synthetic description of the measurements
(e.g., a network, a heat map). The models are used to predict the system’s behavior
under previously unseen conditions or to classify the data into groups, such as healthy
subjects versus the subjects affected by a disease. More important, however, the fun-
damental goal of this approach is to infer models of cellular interactions that can be
mechanistically interpreted, so as to guide experimental interventions for analysis as
well as, eventually, clinical treatment.
A wide range of different reverse engineering methods has been developed over
the past 20 years, although it is not always clear which methods are most valuable from a
practical perspective. A step toward raising the confidence on high-throughput data sets
is to have better experimental and analytical techniques that yield accurate and repro-
ducible data with known error rates. For example, verification of mass spectrometry
proteomic measurements has proved difficult because the results of the measurements
may depend strongly on sample preparation, on detection method, on the biological
context in which the measurements were made, and so on. But the complex networks
that translate genotype into phenotype are also highly sensitive to biological context
and to environmental influences. Typically, this is controlled by signaling networks, for
example, through the action of proteins or kinases phosphorylating other proteins, and
other similar interaction mechanisms. In particular, one such mechanism essential for
cell function is transcriptional regulation, that is, the requirement that gene expression
levels be regulated by the need of the cell to respond to environmental changes and its
own program and cell fate. This process has been shown to involve a very large number
of genes called transcription factors, whose function is simply to regulate the expres-
sion of other genes. These other regulated genes might be, in many cases, transcription
factors themselves, which led researchers in the field to conceptualize transcriptional
regulation as the emergent function of a network, and as such determined both by con-
nectivity and logic.
Transcription factors act by binding to specific sequences of DNA, the binding
domains where transcription is initiated, in such a way that they can control the tran-
scription of DNA into mRNA. By altering the interaction between DNA and the RNA
polymerase, their effect can be to promote or activate the expression of the target gene,
or repress or deactivate it. That is, rather than controlling the activity of proteins in their
metabolic space, transcription regulation controls them at their source.
 The Challenges of Systems Biology 13

The number of transcription factors is relatively large, and the density per gene
depends on the specific species. The human genome contains more than 2,500 bind-
ing sites, in high likelihood corresponding to a similar number of transcription factors.
Transcriptional regulation is involved in most cellular functions: in mature, fully differ-
entiated cells they control housekeeping, mostly through the precise timing of expres-
sion; they regulate the processes associated with the development of an organism and
the differentiation of cells; they are a necessary mechanism for cells to response and
adapt to environmental challenges or normal signals. However, from the point of view
of computational complexity, a remarkable feature of transcription factors is that they
also act on themselves. That is, they form a network of interactions of a highly dynamic
nature (time is of the essence for regulatory purposes), which only in some cases can
be reduced to Boolean functions of a handful of inputs. As such, the small motifs they
form lend themselves to engineering-type analysis as signal processing and detection
devices (Alon 2006). Larger-scale network motifs, however, have been more difficult
to interpret, and their study has relied on statistical characterization and comparison
with generic network models such as small-world and scale-free topologies (Jeong et al.
2000). Moreover, these larger motifs pose a significant computational problem because
search algorithms scale supralinearly with the number of nodes in the network (Ma’ayan
et al. 2008).
The challenges associated with the analysis of reconstructed networks are
compounded with the basic problem of validating the reconstruction itself. Given the
intricate nature of interactions giving rise to function, traditional approaches to net-
work validation based on targeted biochemical interventions, for instance, knock-ins
and knock-outs, are of limited applicability. The notion of model validation through
prediction has taken root recently in the systems community. In particular, the Dia-
logue on Reverse Engineering Assessments and Methods (DREAM) is a project designed
to evaluate model predictions and pathway inference algorithms in systems biology
(Stolovitzky, Prill, and Califano 2009). DREAM is structured in the form of challenges
that comprise open problems presented to the community, whose solutions are known
to the organizers but not to the participants. Participants submit their predictions of
the solutions to the challenges, which are evaluated by the organizers so that rigor-
ous scrutiny of scientific research based on community involvement is possible. In
its most recent edition, the DREAM consortium evaluated more that 30 network infer-
ence methods on microarray data from eukaryotic and prokaryotic cells. Sparse regres-
sion methods performed particularly well for linear network motifs (cascades), whereas
more complex motifs such as loops proved quite difficult across all inference meth-
ods. Interestingly, eukaryotic networks also proved more difficult than prokaryotic ones,
possibly related to that higher degree of post-transcriptional regulation in the former,
which makes the correlation between the levels of mRNA of transcription factors and
their corresponding targets weaker than in the latter. However, the method aggregation
approach resulted in a significantly improved reconstruction accuracy: by integrating
predictions from multiple methods, networks including close to 1,700 transcriptional
interactions were identified with high precision for each of E. coli (prokaryotic) and
14 Chapter 2 Pablo Meyer and Guillermo A. Cecchi

S. aureus (eukaryotic) cells. Moreover, the study identified more than 50 novel interac-
tions, of which close to half were experimentally confirmed (Marbach et al. 2012).

5 OUTLOOK
The high-dimensional nature of cellular processes and the inevitable sources of noise
in data make learning statistical models in this field particularly prone to generalization
errors. Thus, regularization approaches, such as sparse regression, become an essential
tool for improving prediction accuracy as well as for the validation and parameter esti-
mation of mechanistic, interpretable models in biomedical and clinical applications.
Specific applications of sparse modeling in the context of systems biology are discussed
in chapters 3, 4, and 5.
So far, we have focused on systems biology, but similar challenges are confronted
by researchers trying to make sense of neuroscientific data, in particular, those produced
by multielectrode arrays and brain imaging. The technology of arrays is in accelerated
development, and while at present arrays consist of fewer than 1,000 electrodes, typi-
cally sampled at the high-end spiking frequency of 1 kHz, potentially a few orders of
magnitude more electrodes may be recorded and sampled at higher frequencies if mem-
brane potentials are considered (Nicolelis and Lebedev 2009). However, it is in the con-
text of brain imaging that sparse modeling has shown the most promising results (Carroll
et al. 2009). In particular, fMRI can at present record the activity of about 30,000 brain
voxels, sampled at 0.5 to 1 Hz. Given that for humans scanning time is typically lim-
ited to a few minutes, samples are limited to less than 1,000 independent volumes, and
therefore multivariate models are severely underdetermined. Chapters 6 and 7 address
issues arising in sparse modeling of fMRI data, particularly the stability of sparse models
across multiple subjects and experiments.
Finally, the near future is very likely to witness the increasing convergence of sys-
tems biology data with other organism-level measurements, such as heart and brain
imaging, as well as the myriad behavioral markers routinely utilized by clinicians
(e.g., temperature, blood pressure, skin conductance, tremors, speech). We envision an
integrated approach to the simultaneous characterization of genotypic and phenotypic
features related to diseases ranging from Alzheimer’s and Parkinson’s to autism and
schizophrenia, for the purpose of better prognosis and drug development. In this hypo-
thetical (but realistic) landscape of flooding data, sparse modeling will be an essential
tool for the challenges of an augmented systems biology.

NOTE
1. Omics is a general term referring to biological subfields such as genomics, proteomics, metabolomics,
and transcriptomics. Genomics is a subfield of genetics focused on sequencing, assembling, and analyz-
ing the function and structure of genomes, that is, the complete set of DNA within a single cell of an
 The Challenges of Systems Biology 15

organism. Proteomics is studying the structure and function of proteins, and metabolomics is concerned
with chemical processes involving metabolites, (the intermediates and products of metabolism). The tran-
scriptome is the set of all RNA molecules (mRNA, rRNA, tRNA, and other noncoding RNA); the fields of
transcriptomics, or expression profiling, analyzes the expression level of mRNAs in a given population of
cells, often using methods such as DNA microarray technology.

REFERENCES
Alon, U. An Introduction to Systems Biology. Chapman and Hall, 2006.

Bahir, I., et al. Viral adaptation to host: A proteome-based analysis of codon usage and
amino acid preferences. Molecular System Biology 5(311):1–14, 2009.

Blazier, A. S., and J. A. Papin. Integration of expression data in genome-scale metabolic

network reconstructions. Frontiers in Physiology 3:299, 2012.

Bulmer, M. The selection-mutation-drift theory of synonymous codon usage. Genetics

129(3):897–907, 1991.

Burgess-Brown, N. A., et al. Codon optimization can improve expression of human

genes in Escherichia coli: A multi-gene study. Protein Expression and Purification
59(1):94–102, 2008.

Çakir, T., et al. Integration of metabolome data with metabolic networks reveals reporter
reactions. Molecular Systems Biology 2(Oct.), 2006.

Cannarozzi, G., et al. A role for codon order in translation dynamics. Cell 141(2):355–
367, 2010.

Carroll, M. K., et al. Neuroimage 44(1):112–122, 2009.

Clarke, L., et al. The 1000 genomes project: Data management and community access.
Nature Methods 9(5):459–462, 2012.

Coleman, J. R., et al. Virus attenuation by genome-scale changes in codon pair bias.
Science 320(5884):1784–1787, 2008.

Comeron, J. M. Weak selection and recent mutational changes influence polymorphic

synonymous mutations in humans. Proceedings of the National Academy of Sciences
103(18):6940–6945, 2006.

Dana, A., and T. Tuller. Efficient manipulations of synonymous mutations for controlling
translation rate. Journal of Computational Biology 19(2):200–231, 2011.

Danpure, C. J. How can the products of a single gene be localized to more than one
intracellular compartment? Trends in Cell Biology 5(6):230–238, 1995.

———. Mistranslation-induced protein misfolding as a dominant constraint on coding-

sequence evolution. Cell 134(2):341–352, 2008.
16 Chapter 2 Pablo Meyer and Guillermo A. Cecchi

Drummond, D. A., and C. O. Wilke. The evolutionary consequences of erroneous protein

synthesis. Nature Reviews Genetics 10(10):715–724, 2009.

Elf, J., et al. Selective charging of tRNA isoacceptors explains patterns of codon usage.
Science 300(5626):1718–1722, 2003.

Fredrick, K., and M. Ibba. How the sequence of a gene can tune its translation. Cell
141(2):227–229, 2010.

Frenkel-Morgenstern, M., et al. Genes adopt nonoptimal codon usage to generate cell
cycle-dependent oscillations in protein levels. Molecular Systems Biology 8(572):572,
2012.

Gunderson, K. L., et al. Decoding randomly ordered DNA arrays. Genome Research
14(5):870–877, 2004.

Gustafsson, C., S. Govindarajan, and J. Minshull. Codon bias and heterologous protein
expression. Trends in Biotechnology 22(7):346–353, 2004.

Ingolia, N. T., L. F. Lareau, and J. S. Weissman. Ribosome profiling of mouse embry-

onic stem cells reveals the complexity and dynamics of mammalian proteomes. Cell
147(4):789–802, 2011.

Jeong, H., et al. The large-scale organization of metabolic networks. Nature 407:651–654,
2000.

Kimchi-Sarfaty, C., et al. A silent polymorphism in the MDR1 gene changes substrate
specificity. Science 315(5811):525–528, 2007.

Klein, R. J., et al. Complement factor H polymorphism in age-related macular degenera-

tion. Science 308(5720):385–389, 2005.

Kochetov, A. V. Alternative translation start sites and their significance for eukaryotic
proteomes. Molecular Biology 40(5):705–712, 2006.

Kozak, M. Regulation of translation via mRNA structure in prokaryotes and eukaryotes.

Gene 361:13–37, 2005.

Kudla, G., et al. Coding-sequence determinants of gene expression in Escherichia coli.

Science 324(5924):255–258, 2009.

Lander, E. S., et al. Initial sequencing and analysis of the human genome. Nature
409(6822):860–921, 2001.

Lavner, Y., and D. Kotlar. Codon bias as a factor in regulating expression via translation
rate in the human genome. Gene 345(1):127–138, 2005.

Lee, J. W., et al. Editing-defective tRNA synthetase causes protein misfolding and neu-
rodegeneration. Nature 443(7107):50–55, 2006.

Lindblad-Toh, K., et al. A high-resolution map of human evolutionary constraint using

29 mammals. Nature 478(7370):476–482, 2011.
 The Challenges of Systems Biology 17

Loots, G. G. Genomic identification of regulatory elements by evolutionary sequence

comparison and functional analysis. In Advances in Genetics, vol. 61, ed. V. van
Heyningen and R. Hill, 269–293. Academic Press, 2008.

Ma’ayan, A., et al. Ordered cyclic motifs contribute to dynamic stability in bio-
logical and engineered networks. Proceedings of the National Academy of Sciences
105(49):19235–19240, 2008.

Man, O., and Y. Pilpel. Differential translation efficiency of orthologous genes is

involved in phenotypic divergence of yeast species. Nature Genetics 39(3):415–421,
2007.

Marbach, D., et al. Wisdom of crowds for robust gene network inference. Nature Methods
9(8):796–804, 2012.

McCarthy, M. I., et al. Genome-wide association studies for complex traits: Consensus,
uncertainty and challenges. Nature Reviews Genetics 9(5):356–369, 2008.

Nicolelis, M. A. L., and M. A. Lebedev. Principles of neural ensemble physiology under-

lying the operation of brain-machine interfaces. Nature Reviews Neuroscience 10:530–
540, 2009.

Pearson, C. E. Repeat associated non-ATG translation initiation: One DNA, two

transcripts, seven reading frames, potentially nine toxic entities. PLoS Genetics
7(3):e1002018, 2011.

Pfau, T., N. Christian, and O. Ebenhoh. Systems approaches to modelling pathways and
networks. Briefings in Functional Genomics 10(5):266–279, 2011.

Planes, F. J., and J. E. Beasley. A critical examination of stoichiometric and path-finding

approaches to metabolic pathways. Briefings in Bioinformatics 9(5):422–436, 2008.

Plata, G., M. E. Gottesman, and D. Vitkup. The rate of the molecular clock and the cost
of gratuitous protein synthesis. Genome Biology 11(9):R98, 2010.

Plotkin, J. B., and G. Kudla. Synonymous but not the same: The causes and conse-
quences of codon bias. Nature Reviews Genetics 12(1):32–42, 2010.

Ramakrishna, R., et al. Flux-balance analysis of mitochondrial energy metabolism: Con-

sequences of systemic stoichiometric constraints. American Journal of Physiology. Reg-
ulatory, Integrative and Comparative Physiology 280(3):R695–R704, 2001.

Reaves, M. L., and J. D. Rabinowitz. Metabolomics in systems microbiology. Current

Opinion in Biotechnology 22(1):17–25, 2011.

Reuveni, S., et al. Genome-scale analysis of translation elongation with a ribosome flow
model. PLoS Computational Biology 7(9):e1002127, 2011.

Rothberg, J. M., et al. An integrated semiconductor device enabling nonoptical genome

sequencing. Nature 475(7356):348–352, 2011.
18 Chapter 2 Pablo Meyer and Guillermo A. Cecchi

Schmeing, T. M., et al. How mutations in tRNA distant from the anticodon affect the
fidelity of decoding. Nature Structural and Molecular Biology 18(4):432–436, 2011.

Schmidt, M. W., et al. Comparative proteomic and transcriptomic profiling of the fission
yeast Schizosaccharomyces pombe. Molecular Systems Biology 3:79, 2007.

Schuster, S., D. A. Fell, and T. Dandekar. A general definition of metabolic pathways

useful for systematic organization and analysis of complex metabolic networks. Nature
Biotechnology 18(3):326–332, 2000.

Shah, P., and M. A. Gilchrist. Effect of correlated tRNA abundances on translation errors
and evolution of codon usage bias. PLoS Genetics 6(9):e1001128, 2010a.

———. Explaining complex codon usage patterns with selection for translational effi-
ciency, mutation bias, and genetic drift. Proceedings of the National Academy of Sci-
ences 108(25):10231–10236, 2010b.

Sharp, P. M. and W. H. Li. The rate of synonymous substitution in enterobacterial genes

is inversely related to codon usage bias. Molecular Biology and Evolution 4(3):222–230,
1987.

Smoller, J. W., et al. Identification of risk loci with shared effects on five major psychi-
atric disorders: A genome-wide analysis. Lancet 381(9875):1371–1379, 2013.

Stolovitzky, G., R. J. Prill, and A. Califano. Lessons from the DREAM2 Challenges.
Annals of the New York Academy of Sciences 1158:159–195, 2009.

Sultan, M., et al. A global view of gene activity and alternative splicing by deep sequenc-
ing of the human transcriptome. Science 321(5891):956–960, 2008.

Supek, F., and T. Smuc. On relevance of codon usage to expression of synthetic and
natural genes in Escherichia coli. Genetics 185(3):1129–1134, 2010.

Tuller, T. A. Comprehensive computational model for analyzing gene translation. In

Proceedings of ISMB/ECCB: Late Breaking Research Presentation Schedule, 2011.

Tuller, T., A. Carmi et al. An evolutionarily conserved mechanism for controlling the
efficiency of protein translation. Cell 141(2):344–354, 2010.

Tuller, T., M. Kupiec, and E. Ruppin. Determinants of protein abundance and translation
efficiency in S. cerevisiae. PLoS Computational Biology 3(12):2510–2519, 2007.

Tuller, T., I. Veksler, et al. Composite effects of gene determinants on the translation
speed and density of ribosomes. Genome Biology 12(11):R110, 2011.

Tuller, T., Y. Waldman, et al. Translation efficiency is determined by both codon bias
and folding energy. Proceedings of the National Academy of Sciences 107(8):3645–3650,
2010.

van Weringh, A., et al. HIV-1 modulates the tRNA pool to improve translation efficiency.
Molecular Biology and Evolution 28(6):1827–1834, 2011.
 The Challenges of Systems Biology 19

van ’t Veer, L. J., et al. Gene expression profiling predicts clinical outcome of breast
cancer. Nature 415(6871):530–536, 2002.

Vogel, C., et al. Sequence signatures and mRNA concentration can explain two-thirds of
protein abundance variation in a human cell line. Molecular System Biology 6(400):1–9,
2010.

Warnecke, T., and L. D. Hurst. GroEL dependency affects codon usage: –Support for a
critical role of misfolding in gene evolution. Molecular Systems Biology 6(340):1–11,
2010.

Welch, M., et al. Design parameters to control synthetic gene expression in Escherichia
coli. PLoS One 4(9):1–10, 2009.

Zhang, F., et al. Differential arginylation of actin isoforms is regulated by coding

sequence-dependent degradation. Science 329(5998):1534–1537, 2010.

Zhang, Z., et al. Nonsense-mediated decay targets have multiple sequence-related fea-
tures that can inhibit translation. Molecular Systems Biology 6(442):1–9, 2010.

Zhou, T., M. Weems, and C. O. Wilke. Translationally optimal codons associate with
structurally sensitive sites in proteins. Molecular Biology and Evolution 26(7):1571–
1580, 2009.

Zur, H., and T. Tuller. RFMapp: Ribosome flow model application. Bioinformatics
28(12):1663–1664, 2012a.

———. Strong association between mRNA folding strength and protein abundance in S.
cerevisiae. EMBO Reports 13:272–277, 2012b.
 C H A P T E R 3
Practical Sparse Modeling
An Overview and Two Examples
from Genetics
Saharon Rosset

1 SPARSE MODELING ROAD MAP

The sparse modeling assumption states that the true relation of the response Y to the
covariates x1 , . . . , xp (also called predictive variables, or predictors) is a function of a
small number of the covariates,

E(Y|x) 5 f (xj1 , . . . , xjq ),

with q , ,p. It is typically assumed further that the relation is linear,


q

E(Y|x) 5 b l x jl ,
l51

although this assumption can be extended, if necessary, to nonlinear dependencies,

such as, for example, in logistic regression and other generalized linear models.
This notion of sparsity is relevant and appropriate in many real-life domains,
including signal processing (Donoho 2006; Candès 2006; Elad 2010). Here, we con-
centrate on applications of sparsity in genetics and particularly on two major classes
of problems where sparsity is regularly assumed: genome-wide association studies
(GWAS) and gene microarray data analysis. In GWAS the phenotype is measured for a
large panel of individuals (typically several thousands), and a large number (hundreds
of thousands to a few millions) of single-nucleotide polymorphisms (SNPs) throughout
the genome are genotyped in all these participants. The goal is to identify SNPs that are
statistically associated with the phenotype and ultimately to build statistical models
22 Chapter 3 Saharon Rosset

to capture the effect of genotype on the phenotype. It is usually assumed (and invari-
ably confirmed by GWAS results) that only a small number of SNPs are associated with
any specific phenotype. Thus, the GWAS-based model describing the dependence of the
phenotype on SNP genotypes is expected to be sparse, usually extremely sparse1 . This
example is discussed further in the next section.
A second class of relevant problems is gene microarray modeling. Before the
advent of GWAS, the major technology geared toward finding connections between
genetic and phenotypic information was to measure gene expression levels in differ-
ent individuals or different tissues. In this mode, the quantities being measured are the
expressions or activity levels of actual proteins. Proteins are encoded by genes, which
are fragments of the genome. Hence, gene expression experiments can be thought of as
measuring the association between genomic regions and phenotypes except that this is
done through the actual biological mechanisms as expressed in proteins rather than by
direct inspection of genetic sequences, as in GWAS. Not surprisingly, gene expression
analysis also typically assumes that only a few genes are actually directly related to
the phenotype of interest. Thus, this is also a sparse modeling situation, although the
statistical setup has some major differences from the GWAS.
Fundamentally, sparse recovery approaches pursue the following two major
goals:

• Correct recovery of the identities of the covariates that actually participate in

the function f
• Accurate estimation of the model f , both in terms of parameter estimation and
prediction accuracy

Earlier methods for sparse recovery used to fall into one of the two main cate-
gories:

• Methods based on exact or approximate combinatorial enumeration over the

space of possible sparse models and selection from this set based on model
performance
• Methods based on univariate modeling of the relation between single covariates
and the response y, then selection of a small set of covariates showing strong
association with y for inclusion in the sparse model

In feature selection nomenclature, the first approach is referred to as the wrapper

approach, and the second is called the filter approach (Guyon and Elisseeff 2003).
Beyond wrappers and filters, a new type of sparse recovery technique has become
extremely popular in the past few years. This approach uses convex optimization to
solve 1 -norm-regularized regression problems, as it was shown that such regularization
encourages solution sparsity. Sparse recovery methods include Lasso, Dantzig selector,
and multiple other techniques (Tibshirani 1996; Donoho 2006; Candès and Tao 2007).
These methods are often referred to as embedded variable selection approaches, since
 Practical Sparse Modeling 23

they combine variable selection with estimation of model parameters by setting some
of the parameters to zero. Such sparse techniques can provably succeed in situations
where both wrapper and filter methods are unlikely to result in successful recovery.
A detailed technical review of this class of methods is omitted here; rather, a qualitative
description of these approaches and their properties is given. 1 -type methods all share
some version of the same basic (and quite intuitive) conditions for success in sparse
recovery:

• Sufficient sparsity—typically the number of covariates that participate in the

solution is required to be O(n), where n is the sample size, since the well-
known results from the compressed sensing literature state that an accurate
recovery of a sparse signal is possible when the number of samples is O(q log p),
where q is the number of nonzeros and p is the number of covariates.
• Low correlation among the covariates with nonzero regression coefficients and
between them and the other variables. Different versions of this condition are
termed incoherence (Candès and Plan 2009), irrepresentability (Meinshausen
and Yu 2009), and so on.

It is clear that these two dimensions—level of sparsity and degree of correlation

between the covariates—are critical in determining whether it is possible to recover
sparse models from data, and which specific approaches are likely to be successful. The
problem space is here qualitatively divided into the following regions, using sparsity
and correlation levels as the main two axes:

• Situations that can be addressed by simple feature selection wrapper/filter

approaches
• Situations that are appropriate for 1 -based sparse recovery approaches
• Situations where sparse recovery is unlikely to be possible

Three qualitative levels of sparsity are considered: very sparse, where the number
of important variables is O(1); sparse, where the number is O(n), n being the number
of samples, and not sparse otherwise. Three qualitative levels of correlation between
nonzero covariates and other covariates are considered: uncorrelated/orthogonal, low
correlation, as defined in the 1 sparse recovery literature, and high correlation. The
genetic motivating applications can be characterized in terms of these dimensions: in
the GWAS example, it is typically assumed that the model is very sparse and the nonzero
covariates (SNPs) almost uncorrelated between them and with almost all zero covariates;
in the gene expression modeling example, it is typically assumed that large groups of
covariates (genes) may have high correlation within them but low correlations between
groups, and so the sparse situation pertains (Leung and Cavalieri 2003).
Considering which sparse recovery approaches fit which situation, one can make
some observations: (1) in the very sparse situation, combinatorial wrapper approaches

are often likely to do well, in particular, if one assumes q is very small and pq is
24 Chapter 3 Saharon Rosset

F i g u r e 3.1
A schematic view of sparse modeling scenarios.

a manageable enumeration; and (2) in the uncorrelated/orthogonal situation, marginal

univariate (filter) approaches are expected to do well in identifying important covari-
ates. In fact, it can easily be shown that Lasso is equivalent to univariate regression
when the covariates are orthogonal from a variable selection perspective (Tibshirani
1996). In that respect, the 1 -type methods can be thought of as reducing to univariate
modeling when there is no correlation.
The conclusions from this qualitative discussion are summarized in figure 3.1.
Easy situations, where filter or wrapper methods are likely to do well, are shown in
green. Medium situations, where filter or wrapper methods will be challenged or are
unlikely to do well, are shown in yellow. This includes high correlation, very sparse
scenarios (where wrapper methods are appropriate, but successful sparse recovery may
be difficult), and sparse, low-correlation scenarios, which are the space where 1 -type
methods are most useful. The red regions are hard ones, where there is no realistic
chance of successful sparse recovery.

2 EXAMPLE 1: GENOME-WIDE ASSOCIATION

STUDIES (GWAS)
As previously discussed, in GWAS one can assume that the number of features
(SNPs) p range from the hundreds of thousands to a few millions, the number of
 Practical Sparse Modeling 25

observations (individuals) n in the thousands, and there are also the following statis-
tical characteristics:

• Frequently, only a very small number of SNPs are associated with the pheno-
type y, typically ten or fever. Thus, it is clearly the very sparse scenario.
• The vast majority of SNP pairs are uncorrelated. This is owing to the recombina-
tion process driving the SNP-SNP correlation in the genome. SNPs that are far
from each other on the genome, and certainly SNPs on different chromosomes,
are in linkage equilibrium, meaning they are completely uncorrelated, because
of being separated by many recombination events in the genetic history of the
sample being considered. Hence one can assume that each SNP is correlated
only with a tiny fraction of all other SNPs, and typically all truly associated
SNPs are uncorrelated between them. However, keep in mind that every SNP
typically has some neighboring SNPs that are in high correlation with it.

The standard methodology in analyzing GWAS data is to perform p univariate

tests of association between each SNP and the response (Wellcome Trust 2007; Manolio
2010). Appropriate univariate models are chosen to accommodate the specific prob-
lem setting, like linear regression, logistic regression, or chi-square tests of association.
Each model is evaluated using the p-value for the effect being tested (SNP coefficient
in the linear regression, chi-square statistic, and so on). The p-values from the univari-
ate models or tests are ranked, and after appropriate multiple comparison corrections,
as warranted, the top results are declared significant, therefore indicating likely true
association. It should be noted that because of the correlation structure, it is typical
that each significant finding is actually expressed as multiple neighboring SNPs that
are significantly associated, and the typical policy is to select the most associated SNP
in the region, consistent with the view that each region is likely to have only one true
association.
The significant results from GWAS are usually used as motivation and guid-
ance for follow-up studies, aimed at revalidating the findings and further examining
the potential biological/genetic mechanisms underlying the discovered associations
(Manolio 2010).
When examining this process as a sparse modeling exercise, several questions
arise:

• Is the approach of performing univariate tests instead of joint modeling justi-

fied? What can one gain from performing multivariate analysis?
• Is the ranking and selection of SNPs based on p-values rather than on other
commonly used model evaluation criteria (like likelihood) justified? Can a dif-
ferent approach give better results?
• How should the methodology of selection be related to the nature of the follow-
up studies to be performed?
26 Chapter 3 Saharon Rosset

The first two questions are addressed here, starting from the second: Is selec-
tion by p-value justified? To frame the discussion theoretically, let’s assume a standard
univariate linear regression formulation, where

y 5 bT x 1 e , e ∼ N(0, s 2 ),

and assume for simplicity s 2 is known and there is only one truly associated SNP. In
other words, we assume all bj are zero except for one. The coordinates xj can be highly
correlated and the dimensionality of the problem is not too high (i.e., assume concentra-
tion in the genomic region around the true association). The primary goal is to identify
the SNP j0 with the true association.
A statistical approach in this situation is to use the maximum likelihood (ML) esti-
mation. Assuming the noise distribution is Gaussian and only one coefficient is nonzero,
it is easy to see that ML estimation in this case amounts to finding the univariate model
with the minimal residual sum of square (RSS):


n
jˆ0 5 arg min (yi 2 bj xij )2 .
j,bj
i51

How does this compare to selecting jˆ0 as the SNP attaining the minimal p-value in per-
forming a z-test on the coefficient of the SNP (or equivalently, a test for the univariate
model against the null model)? As it turns out, the two are completely equivalent in this
case, in the sense that ranking of the SNPs according to RSS is identical to their ranking
according to z-test p-values. To see this, denote for SNP j the sum of squares of x·j by
  
Sxxj 5 i xij2 2 nx̄·j , and denote Sxyj 5 i xij yi 2 nx̄·j ȳ and Syy 5 i yi2 2 nȳ.
Then the coefficient of the regression of y on xj is bj 5 Sxyj /Sxxj and the p-value
of the z-test is
 
2|bj |
pj 5 2 ∗ F  ,
s 2 /Sxxj

where F(·) is the cumulative standard normal distribution function. Note that this
expression is a monotone function of

|bj | Sxyj
 ~ .
s 2 /Sxx j Sxxj

From the standard theory of linear regression it follows that the best RSS for the uni-
variate model with SNP j is

 2
nSxyj2 Sxyj
RSS(b̂j ) 5 Syy 2 5 Syy 2 n  ,
Sxxj Sxxj
 Practical Sparse Modeling 27


which is also clearly a monotone function of Sxyj / Sxxj . Thus, selecting the lowest
p value or using ML is mathematically equivalent.
This perfect equivalence breaks down once one moves away from the simplest lin-
ear regression setting. For example, consider a logistic regression setup, where GWAS
typically uses the Wald statistic for p value calculation (McCullagh and Nelder 1989).
This is based on a quadratic approximation of the likelihood around the estimate.
Selecting the SNP that gives the lowest p value is no longer equivalent to selecting
the one that gives the best likelihood in a univariate model. One would intuitively
expect that the maximum likelihood approach would be slightly better than the p-value-
based approach. To demonstrate that this is indeed the case, consider a simplistic sim-
ulation. Assume there are two SNPs, with xi1 ∼ N(0, 1) and xi2 5 xi1 1 r · N(0, 1), and
P(yi 5 1|xi ) 5 exp(xi1 )/(1 1 exp(xi1 )). Thus, SNP 1 is the true association, but the two
SNPs are correlated with

√
cor(x·1 , x·2 ) 5 1/ 1 1 r 2 .

Examine the rate of success of both approaches in identifying SNP 1 as the more highly
associated, as a function of r. Results are given in figure 3.2. As expected, the success
rate of both approaches is similar, but the approach based on likelihood is slightly better
for all values of r.
0.90

likelihood
p−value
0.85
0.80
% correct
0.75
0.70
0.65
0.60

0.2 0.4 0.6 0.8 1.0

Noise level

F i g u r e 3.2
Percentage of cases the correct true association is identified by maximum likelihood and Wald test p value in a
logistic regression setup. The maximum likelihood criterion is slightly superior for all levels of correlation.
28 Chapter 3 Saharon Rosset

To summarize the discussion of the use of p-values for model selection: this crite-
rion is generally similar to using maximum likelihood but could be inferior, depending
on the approximations used for calculating p-value, which may break down the equiv-
alence.
The other question to be addressed pertains to the use of univariate models, as
opposed to multivariate sparse modeling approaches like Lasso (Tibshirani 1996). Con-
sider again a genomic region with correlated SNPs, where at most one SNP is associated,
and one would like to compare the use of univariate models to find the associated SNP
to the use of Lasso or similar methods. The Lasso formulation,


b̂(l) 5 arg min (yi 2 bT xi )2 1 lb1 , (3.1)
b
i

includes a regularization parameter l. At l 5 ` the solution is all zeros; as l → 0 the

solution converges to the least squares solution. Specifically, at large enough l the solu-
tion would contain only a single nonzero coefficient. It is easy to verify that this first
variable is the maximizer of the empirical covariance, that is, Sxyj (Efron et al. 2004). In
other words, if all xj are prestandardized to have the same Sxxj , then the univariate and
Lasso approaches amount to selecting the same first covariate.
For lower values of l, a reasonable approach using Lasso and assuming a single
association is to select the largest absolute coefficient in b̂(l) as jˆ0 . It is now relevant to
inquire if this approach could prove superior to the univariate approach in identifying
the correct association. To test this question, a simulation study was performed, this
time with three covariates. We have xi1 ∼ N(0, 1), yi 5 2 1 5xi1 1 ei , ei ∼ N(0, 1) is the
true association signal, and two correlated variables and defined as xi2 5 xi1 1 di2 , di2 ∼
N(0, 0.01) and similarly xi3 5 xi2 1 di3 , di3 ∼ N(0, 0.01). The success of four approaches
in detecting the first variable as the true association is examined:

• The univariate regression approach in GWAS

• Regular least squares, where the maximal coefficient is chosen
• Lasso with standardized covariates for various regularization levels, where the
maximal Lasso coefficient is chosen
• Lasso with nonstandardized covariates

Figure 3.3 presents the results. The x-axis is the Lasso constraint (in its Lagrange-
equivalent constrained form), and the y-axis is the percentage of correct identification
of the first explanatory variable as the best association. The univariate approach and
the standardized Lasso with small constraint (high penalty) are much better than the
other two approaches. On the simulation data, there were a few examples where the
standardized Lasso added the wrong variable first but then for higher constraint values
the order of absolute coefficients reversed and the first variable was correctly chosen.
Hence, there is a range of constraint around 0.4 where the Lasso does very slightly bet-
ter than univariate. The generality of this phenomenon requires further research. Not
 Practical Sparse Modeling 29

0.90
0.85
0.80
% correct selection
0.75
0.70
0.65

Univariate
Least squares
LASSO−standardized
0.60

LASSO−nonstand

0.0 0.2 0.4 0.6 0.8 1.0

Lasso constraint

F i g u r e 3.3
Success of different variable selection schemes on a simulated GWAS example.

surprisingly, the least squares approach and the nonstandardized Lasso are far inferior
in their model selection performance.
To summarize the analysis of univariate GWAS tests, it has been shown that the
common practice of using p-values for selection is generally similar to using maximum
likelihood, although the latter may be slightly superior in some cases. Also, under the
assumption of a very sparse problem with almost uncorrelated variables, the univariate
approach works quite well and is comparable to multivariate approaches such as Lasso
for the purpose of identifying the associated SNPs.
The third question, how the selection should be affected by follow-up study
design, has not been discussed. As a simple example, if planned follow-up work is
a search for the biological mechanisms underlying statistical associations, then it may
make sense to bias modeling toward identification of associations in biologically plausi-
ble genomic regions (such as inside genes). This can be accomplished by using Bayesian
priors or other intuitive weighting schemes (Cantor, Lange, and Sinsheimer 2010). Fur-
ther discussion of this aspect is outside the scope of this chapter.

3 EXAMPLE 2: GENE MICROARRAY DATA ANALYSIS

Microarray technology actually precedes the emergence of the GWAS approach (Schena
et al. 1995). The analyzed data comprise expression levels of genes—how much of
each protein (equivalently, gene) is expressed in each sample. Different samples can
30 Chapter 3 Saharon Rosset

be different individuals, different tissues, or even the same tissue under different envi-
ronmental conditions. The most prevalent goal in analyzing gene expression data is to
identify which genes are associated with the response of interest, which can be disease
status, as in GWAS (in which case, the same case control design as in GWAS can be
used), a measure of the environmental conditions being applied (such as concentration
of sugar or temperature), and so on. The number of samples (n) is usually in the tens
or low hundreds, and the number of genes (p) is usually in the thousands or tens of
thousands; hence one is in the p . .n situation of wide data.
As in GWAS, it is usually assumed that the true association relation between
gene expression and the response is sparse or very sparse, in the sense that the true
dependence (e.g., conditional expectation) of the response on the gene expression can
be almost fully modeled using a few true genes. However, the correlation structure
among expressions of genes is much more complex than the correlation among SNPs,
since genes are organized in pathways and networks (Davidson and Levin 2005), which
interact and co-regulate in complex ways. It is usually not assumed that these interac-
tions and the resulting correlation structure are known; hence, one can consider this an
example of a sparse modeling scenario with arbitrary complex correlations between
the explanatory variables. In particular, one cannot assume that the few true genes
are uncorrelated as in the GWAS case. Hence, univariate approaches are unlikely to
properly address this situation, and although they had originally been used for gene
expression analysis, in particular, for identification of differentially expressed genes
(Leung and Cavalieri 2003), they have been surpassed in this task, too, by multivariate
approaches, which have been demonstrated to be much more effective (Meinshausen
2007; Wang et al. 2011). It should be noted that combinatorial variable selection wrap-
per approaches are unlikely to be relevant, since enumerating all sparse models with
several dozens of nonzero coefficients out of thousands is clearly intractable.
Another important difference between GWAS and gene expression analysis is that
in the latter case we are often interested in building an actual prediction model to
describe the relation between gene expression and the response rather than just identi-
fying the associated genes for further study (Leung and Cavalieri 2003). This also affects
the choice of models.
Since we are seeking a sparse prediction model in high dimension with limited
samples, Lasso-type methods are a natural approach to consider. The standard Lasso has
some major shortcomings in this situation:

• With p . .n, Lasso regularized models are limited to choosing at most n genes
in the model (Efron et al. 2004). This can become a problem in gene expres-
sion modeling with very few samples. Furthermore, Lasso typically selects
one representative from each group of highly correlated explanatory variables
(in gene expression, this could represent genes in a specific pathway). This is
not necessarily desirable, as there could be multiple independent associations
in the same path, or separating the true association from other genes that are
 Practical Sparse Modeling 31

highly correlated with it can be very difficult. Hence a selection of a single gene
can be arbitrary or nonrepresentative.
• If one is interested in prediction, then the shrinkage Lasso performs on
its selected variables is likely to a lead to suboptimal predictive model
(Meinshausen 2007).

Several Lasso extensions have used gene expression as a motivating application

and these specific problems as motivation for their proposed algorithmic extensions:

• Elastic Net (Zou and Hastie 2005), which adds a second quadratic penalty to
the Lasso formulation in (3.1), thus allowing solution with more than n distinct
features and similar coefficients for highly correlated features.
• Adaptive Lasso (Zou 2006), which adds weighting to the Lasso penalty of each
feature, using the least square coefficients as weights. This leads to favorable
theoretical properties and has also shown improved empirical performance.
• Relaxed Lasso (Meinshausen 2007), which uses Lasso for variable selection but
then fits a less regularized model in these variables only, thus partly avoiding
the excessive shrinkage behavior.
• VISA (Radchenko and James 2008), which implements a more involved version
of the same idea, of performing less shrinkage on the “good” variables Lasso
identifies than warranted by the Lasso solution.
• Random Lasso (Wang et al. 2011).

Random Lasso is described here in more detail, and the relative performance of these
algorithms is demonstrated on simulated and real gene expression data, following Wang
et al. (2011).

4 RANDOM LASSO
When many highly correlated features are present, one wants to consider the portion of
them that is useful for predictive modeling purposes. Lasso-type regularization tends to
pick one of them semiarbitrarily, which can be considered a model instability issue.
The statistics literature offers some recipes for dealing with instability, most pop-
ular among them Breiman’s proposals of Bagging and Random Forest (Breiman 2001).
The basic idea is to generate a variety of slightly modified versions of the data or mod-
ified versions of the model-fitting algorithm, generating a variety of different prediction
models that approximately fit the data. Then averaging these models has a stabilizing
effect, as one hopes that models not chosen for the original data would occasionally
get chosen when the data are changed. Empirically, this usually leads to much more
accurate prediction models (Breiman 2001).
32 Chapter 3 Saharon Rosset

As Breiman noted, linear modeling approaches are not subject to improvement

from Bagging, but since Lasso is not a linear approach in this sense (because of the
regularization), it can be subjected to Bagging-type modifications.
The first part of the Random Lasso (RLasso) is basically applying two-way boot-
strap aggregating, which can be considered a hybrid of Bagging and Random Forest.
The second part repeats the same exercise, but with variables weighted according to
their importance in the first part, to accomplish stronger variable selection. Here is a
brief schematic description of the algorithm:

1. Iterate B1 times:

a. Bootstrap sample the data and sub sample the features (two-dimensional sam-
pling).

b. Fit a Lasso model to the current sample.

2. Average the coefficients of all resulting models.

3. Generate an importance measure for each variable, typically proportional to its aver-
age coefficient.

4. Perform a second iteration, this time B2 times:

a. Bootstrap sample the data and subsample the features according to their impor-
tance measure.

b. Fit a Lasso model to the current sample.

5. The final model is the average of the B2 models from the second stage.

Detailed discussion of the motivation behind the exact formulation of the algo-
rithm is beyond the scope of this chapter, but a comparison of the various Lasso exten-
sions is shown here on simulation and real gene expression data.
In the simulation scenario there are p 5 40 variables. The first ten coefficients are
nonzero. The correlation between each pair of the first ten variables is set to be 0.9. The
remaining 30 variables are independent with each other and also independent with the
first ten variables. Let

b 5 (3, 3, 3, 3, 3, 22, 22, 22, 22, 22, 0, ..., 0)

and

y 5 bT x 1 e, , e ∼ N(0, 9).
 Practical Sparse Modeling 33

T a b l e 3.1
Variable selection frequencies (%) of different methods for the simulation example

Lasso ALasso Enet Relaxo VISA RLasso

n 5 50
IV 35 38 60 29 28 98
UV 20 11 13 9 7 17
RME 666 613 562 608 610 299

n 5 100
IV 69 82 76 62 62 99
UV 52 21 35 36 37 30
RME 505 313 471 487 487 132

IV, important variables; UV, unimportant variables; RME, relative model error (lower is better).

T a b l e 3.2
Analysis of the glioblastoma data set

Method No. of Genes Selected Mean Prediction Error

Lasso 29 1.118 (0.205)
Adaptive Lasso 33 1.143 (0.211)
Relaxed Lasso 23 1.054 (0.194)
Elastic net 28 1.113 (0.204)
VISA 15 0.997 (0.188)
Random Lasso 58 0.950 (0.210)

The signal-to-noise ratio is about 3.2.

Table 3.1 shows the performance of the various algorithms in selecting the impor-
tant variables 1–10 (IV) and the unimportant variables 11–30 (UV) and also the relative
model error (RME), as defined in (Wang et al. 2011). The performance is averaged over
100 simulations. As can be seen, RLasso is far superior to all competing methods on
IV and RME and competitive in UV. The Wang et al. paper contains many other simu-
lation setups, including some where RLasso is inferior to some of the alternatives, and
discussion of the underlying reasons. Note, however, that for most realistic simulation
scenarios that are gene expression motivated, RLasso performs best.
Finally, all methods were also applied to a famous real gene expression data set,
where the examined response is the log-survival time of glioblastoma patients (Freije
et al. 2004). One data set with n 5 50 patients was used for training the models, and
the other with n 5 61 patients for comparing predictive performance. The number of
genes is p 5 3600, reduced to p 5 1000 by initial filtering. Table 3.2 shows the results:
number of genes selected and mean squared prediction error. As can be seen, RLasso
chooses more genes than other methods (though still less than 6 percent of genes) and
achieves the best predictive performance.
34 Chapter 3 Saharon Rosset

5 SUMMARY
Practical applications of sparse modeling can possess quite different properties, and
thus selection of appropriate sparse methods should strongly depend on the problem
at hand. Particularly, as shown in this chapter, the specific type of sparsity and the
correlation structure across the covariates, or predictive variables, are two important
considerations, as well as the desired performance metrics for the model: successful
variable selection, favorable predictive performance, or both.
Two common problems from computational biology were considered as examples:
GWAS and gene expression analysis. In the case of the GWAS problem, where the main
goal is to identify associated SNPs for follow-up studies, the commonly used univariate
filter approach often appears to be sufficient, under the common assumptions of extreme
sparsity and uncorrelated covariates. However, in the case of gene expression analysis,
where the correlation structure among the variables is more complex, and both vari-
able selection and good predictive performance are equally important, a more complex
methodology is required. Accordingly, variants of Lasso were surveyed that aim to take
the specifics of the problem into account and accomplish both goals.
Chapter 4 focuses on GWAS problems, extending the traditional sparse
approaches discussed here to cases of more complex relation structure among the
covariates and also among the multiple output variables in GWAS.

NOTE
1. In recent years, random- and mixed-effect models have been used to demonstrate that there are likely
many more associations between genotype and phenotype that we are currently unable to discover (Yang
et al. 2010; Lee et al. 2011). Since current studies lack power to identify the specific SNPs underlying
these associations, this intriguing direction is outside of the scope of our discussion, which focuses on
the traditional fixed effects regression framework.

REFERENCES
Breiman, L. Random forests. Machine Learning 45:5–32, 2001.

Candès, E. Compressive sampling. In Proceedings of the International Congress of Math-

ematicians, vol 3, 1433–1452, 2006.

Candès, E., and Y. Plan. Near-ideal model selection by 11 minimization. Annals of

Statistics 37(5A):2145–2177, 2009.

Candès, E., and T. Tao. The Dantzig selector: Statistical estimation when p is much larger
than n. Annals of Statistics 35(6):2313–2351, 2007.
 Practical Sparse Modeling 35

Cantor, R. M., K. Lange, and J. S. Sinsheimer. Prioritizing GWAS results: A review of

statistical methods and recommendations for their application. American Journal of
Human Genetics, 86(1):6–22, 2010.

Davidson, E., and M. Levin. Gene regulatory networks. Proceedings of the National
Academy of Sciences 102(14):4935, 2005.

Donoho, D. Compressed sensing. IEEE Transactions on Information Theory 52(4):1289–

1306, 2006.

Efron, B., T. Hastie, I. Johnstone, and R. Tibshirani. Least angle regression. Annals of
Statistics 32(2):407–499, 2004.

Elad. M, Sparse and Redundant Representations: From Theory to Applications in Signal

and Image Processing. Springer, 2010.

Freije, W. A., F. E. Castro-Vargas, Z. Fang, S. Horvath, T. Cloughesy, L. M. Liau et al.

Gene expression profiling of gliomas strongly predicts survival. Cancer Research 64:
6503–6510, 2004.

Guyon, I., and A. Elisseeff. An introduction to variable and feature selection. Journal of
Machine Learning Research 3:1157–1182, 2003.

Lee, S. H., N. R. Wray, M. E. Goddard, and P. M. Visscher. Estimating missing heritability

for disease from genome-wide association studies. American Journal of Human Genetics
88(3):294–305, 2011.

Leung, Y. F., and D. Cavalieri. Fundamentals of cDNA microarray data analysis. Trends
in Genetics 19(11):649–659, 2003.

Manolio, T. A. Genomewide association studies and assessment of the risk of disease.

New England Journal of Medicine 363(2):166–176, 2010.

McCullagh, P., and J. Nelder. Generalized Linear Models. Chapman and Hall, 1989.

Meinshausen, N. Relaxed Lasso. Computational Statistics and Data Analysis 52(1):374–

393, 2007.

Meinshausen, N., and B. Yu. Lasso-type recovery of sparse representations for high-
dimensional data. Annals of Statistics 37(1):246–270, 2009.

Radchenko, P., and G. M. James. Variable inclusion and shrinkage algorithms. Journal of
the American Statistical Association, 103(483):1304–1315, 2008.

Schena, M., D. Shalon, R. W. Davis, and P. O. Brown. Quantitative monitoring of gene

expression patterns with a complementary DNA microarray. Science 270(5235):467–
470, 1995.

Tibshirani, R. Regression shrinkage and selection via the Lasso. Journal of the Royal
Statistical Society 58(1):267–288, 1996.
36 Chapter 3 Saharon Rosset

Wang, S., B. Nan, S. Rosset, and J. Zhu. Random lasso. Annals of Applied Statistics
5(1):468–485, 2011.

Wellcome Trust Case Control Consortium. Genome-wide association study of 14,000

cases of seven common diseases and 3,000 shared controls. Nature 447(7145):661–678,
2007.

Yang, J., B. Benyamin, B. P. McEvoy, S. Gordon, A. K. Henders, D. R. Nyholt, P. A.

Madden, A. C. Heath, N. G. Martin, G. W. Montgomery, M. E. Goddard, and P. M.
Visscher. Common SNPs explain a large proportion of the heritability for human height.
Nature Genetics 42:565–569, 2010.

Zou, H. The adaptive Lasso and its oracle properties. Journal of the American Statistical
Association 101(476):1418–1429, 2006.

Zou, H., and T. Hastie. Regularization and variable selection via the Elastic Net. Journal
of the Royal Statistical Society Series B, 67(2):301–320, 2005.
High-Dimensional Sparse Structured
C H A P T E R 4
Input-Output Models, with
Applications to GWAS
Eric P. Xing, Mladen Kolar, Seyoung Kim, and Xi Chen

Genome-wide association studies (GWAS) are a popular approach to discovering the

genetic causes of many complex diseases, such as cancer, asthma, and diabetes. In a
typical study one tries to discover an association function from a small set of causal
variables, known as single-nucleotide polymorphisms (SNPs), out of a few million
candidates, to a set of genes whose expression levels are interdependent in a complex
manner. The problem is statistically challenging because there are a large number of
potential causal markers compared to the number of individuals from a population.
One of the main challenges is maximizing the power of procedures for identifying causal
SNPs while suppressing false positives. In this chapter, we present a number of methods
that leverage prior knowledge and underlying structure of the problem to improve the
statistical power of association analysis.
Traditionally, a simple single-marker test has been widely used for detecting an
association, (see chapter 3). This test examines the correlation between the given out-
puts and each feature, one feature at a time, to compute p-values of SNPs, finding SNPs
with low p-values significant. An issue with this approach is that it considers only one
SNP at a time, whereas many complex diseases are now believed to be controlled by
multiple genetic loci. An alternative approach is based on multivariate linear regres-
sion, where all SNPs are considered jointly in a single statistical model. In this model,
phenotype is regressed onto SNPs, and the regression coefficients are used to deter-
mine significance of the association for each SNP. Unfortunately, the multivariate linear
regression method does not provide parsimonious models, which are sought by practi-
tioners for their ease of interpretation.
Parsimonious models can be obtained by performing variable selection in linear
models. Efficient variable selection can be performed using penalized linear regression
where the 1 -norm of the regression coefficients is used to set a number of regression
coefficients to zero. The resulting estimator is known as the Lasso in the statistical lit-
erature (Tibshirani 1996) and has recently been actively studied. There is a large liter-
ature on efficient, specialized convex program solvers for the Lasso (see, e.g., Fu 1998;
38 Chapter 4 Eric P. Xing and Colleagues

Efron et al. 2004; Beck and Teboulle 2009, and references therein) as well as theory on
generalization properties and variable selection consistency (see, e.g., Wainwright 2009;
Zhao and Yu 2006; Bickel, Ritov, and Tsybakov 2009; Zhang 2009).
Although a widely studied and popular procedure, Lasso was shown to be limited
in its power for selecting SNPs that are truly influencing complex traits. The main reason
is that regularization with the 1 -norm is equivalent to the assumption that the regres-
sion coefficients are independent variables (following Laplace priors) and hence cannot
model more complex relations among the predictors, such as, for example, group selec-
tion. Similarly, Lasso does not model potentially nontrivial relations among multiple
outputs. In practice, however, relations and structures among input or output variables
exist, which should be leveraged to improve the estimation procedure. For example,
module structures in gene co-expression patterns are often captured by gene networks
or hierarchical clustering trees. Thus, in an investigation for genetic effects on gene
expression traits, the module structures could be leveraged to improve the statistical
power by considering multiple related gene expression traits jointly to identify SNPs
influencing gene modules. Regarding input structures, it is well known in genetics that
in genomes there exist local correlation structures known as linkage disequilibrium,
nonlinear interaction among SNPs in their influence on traits, and population structure
often captured by different genotype frequencies in different populations.
These problems can be approached using structurally penalized linear regression,
where the penalty reflects some prior knowledge or structure of the problem, such as
relations among input or output variables. Early work considered variables to be par-
titioned in nonoverlapping groups, which reflects prior knowledge that blocks of vari-
ables should be selected or ignored jointly. The resulting estimator, in the context of
multivariate regression, is called group Lasso (M. Yuan and Lin 2006). The grouped
penalty was shown to improve both predictive performance and interpretability of the
models (Lounici et al. 2010; Huang and Zhang 2010). More complex prior knowledge
can be encoded by allowing groups to overlap (see, e.g., Zhao, Rocha, and Yu 2009;
Jacob, Obozinski, and Vert 2009; Jenatton, Audibert, and Bach 2009/2011; Bach et al.
2011). Another structural penalty arising in applications to GWAS is the total variation
penalty, which in the context of multivariate linear regression results in the fused Lasso
(Tibshirani et al. 2005). It assumed that there is a natural ordering of the input vari-
ables, and the total variation penalty is used to encode the prior information that nearby
regression coefficients have similar values.
These structural penalties also arise in the context of multitask learning. In GWAS
it is common to observe multiple traits that are all related to the same set of input vari-
ables. In this context it is useful to use multioutput multivariate regression models to
further reduce the number of falsely selected input variables. The simplest multitask
model assumes that the output variables are only related by sharing the same feature
set. In this context one can use the nonoverlapping group penalty to select the rele-
vant variables for all tasks (see, e.g., Turlach, Venables, and Wright 2005; Liu, Palatucci,
and Zhang 2009; Obozinski, Taskar, and Jordan 2010; Lounici et al. 2009, Kolar, Laf-
ferty, and Wasserman 2011 and references therein). With additional prior knowledge one
 High-Dimensional Sparse Structured Input-Output Models 39

can use overlapping group penalties (Kim and Xing 2010) or fusion penalties (Kim and
Xing 2009).
Therefore, given structures on either or both the input and output sides of a regres-
sion problem, what we need to consider in GWAS is a sparse structured input-output
regression model of high dimensionality. General interior point convex program solvers
can be used to find parameters of the structurally penalized regression models. However,
interior point methods are not suitable for solving relevant real-world problems arising
in GWAS. Although they provide high accuracy solutions, they are not scalable to high-
dimensional problems because they do not exploit the special structure of the penalties
commonly used in practice. For large-scale problems, it is found that first-order meth-
ods, especially proximal gradient algorithms can effectively exploit the special structure
of the typical convex programs and can be efficiently applied to the problems arising in
GWAS.
In the remainder of this chapter, we review various designs of penalties used to
incorporate prior knowledge in the inputs and outputs of the aforementioned structured
input-output regression models used in GWAS, followed by a survey of convex opti-
mization algorithms applicable to estimating such models in general. Then we provide
details on the proximal methods that are particularly effective in solving the convex
problems in high-dimensional settings in GWAS, followed by an empirical comparison
of different optimization approaches on simulation data. We conclude with a number
of illustrative examples of applying the structured input-output models to GWAS under
various contexts.

1 PROBLEM SETUP AND NOTATIONS

We introduce some formulations and notations that will be used throughout the chapter.
Assume a sample of N instances, each represented by a J-dimensional input vector and
a K-dimensional output vector. Let X denote the N 3 J input matrix, whose column
corresponds to observations for the jth input xj 5 (xj1 , . . . , xjN )T . In GWAS, each element
xji of the input matrix takes values from {0, 1, 2} according to the number of minor alleles
at the jth locus of the ith individual. Let Y denote the N 3 K output matrix, whose
column is a vector of observations for the kth output yk 5 (yk1 , . . . , ykN )T . For each of the
K output variables, we assume a linear model:

yk 5 X␤ k 1 ⑀ k , ∀k 5 1, . . . , K, (4.1)

where ␤ k is a vector of J regression coefficients (b1k , . . . , bJk )T for the kth output, and ⑀ k is
a vector of N independent error terms having mean 0 and a constant variance. We center
 
the yk ’s and xj ’s such that i yki 5 0 and i xji 5 0, and consider the model without an
intercept. Let B 5 (␤ 1 , . . . , ␤ K ) denote the J 3 K matrix of regression coefficients for all
K outputs.
40 Chapter 4 Eric P. Xing and Colleagues

As discussed, when J is large and the number of inputs relevant to the output is
small, ordinary multivariate regression does not perform well and the penalized linear
regression should be used. Throughout the chapter we consider problems of form

B̂ 5 argmin (f (B) ≡ (B) 1 V(B)) , (4.2)

where

1 1
(B) 5 Y 2 XB2F 5 (yk 2 X␤ k )T · (yk 2 X␤ k ) (4.3)
2 2
k

is the quadratic loss function and V : RJ 3K → R is a penalty that encodes prior knowl-
edge about the problem into the optimization procedure.
Lasso offers an effective feature selection method for the model in eq. (4.1). Lasso
Lasso
estimator B̂ can be obtained by solving the optimization problem in eq. (4.2) with
the following penalty:

 j
VLasso (B) 5 l |bk |. (4.4)
j k

Lasso
The estimator B̂ will be sparse in the sense that a number of its elements will exactly
Lasso
equal zero. The sparsity of B̂ is controlled by a tuning parameter l. Setting l to
larger values leads to a smaller number of nonzero regression coefficients. The resulting
estimator is good in situations where one has only information that the true parameter
B has few nonzero elements. However, the penalty VLasso does not offer a mechanism
to explicitly couple the estimates of the regression coefficients for correlated output
variables nor to incorporate information about correlation between input variables.

2 STRUCTURED PENALTY ON THE INPUTS

While the standard Lasso penalty does not assume any structure among the input vari-
ables, in this section we discuss penalties that can be used to leverage prior information
about relations between different inputs or outputs in the model in eq. (4.1). In partic-
ular, the problem of learning with multiple related outputs is known as the multitask
learning. For the sake of notational simplicity, let us first consider a single-task setting
where the output is a single vector y ∈ RN 31 . The linear model in eq. (4.1) becomes

y 5 X␤ 1 ⑀ , (4.5)

J
and VLasso (␤ ) 5 l␤ 1 5 l j51 |bj |.
 High-Dimensional Sparse Structured Input-Output Models 41

To facilitate the estimation of structured sparsity pattern, we further introduce

a structured sparsity–inducing penalty Vstruct (␤ ) on top of VLasso (␤ ) and formulate the
structured sparse learning problem as

min f (␤ ) 5 (␤ ) 1 Vstruct (␤ ) 1 VLasso (␤ ), (4.6)

␤

where (␤ ) 5 12 y 2 X␤ 22 is the squared convex loss.

As examples of such structured penalties, we consider two broad categories of
penalties Vstruct (␤ ) based on two different types of functional forms, namely, an over-
lapping group Lasso penalty based on the 1 /2 mixed-norm and a graph-guided fusion
penalty. These two types of penalties cover a broad set of structured sparsity–inducing
penalties that have been introduced in the literature (M. Yuan and Lin 2006; Jenatton
et al. 2009/2011; Kim and Xing 2010; Zhao, Rocha, and Yu 2009; Tibshirani et al. 2005;
Kim, Sohn, and Xing 2009). In our discussion, we impose 1 regularization VLasso (␤ )
along with the structured sparsity–inducing penalty in order to explicitly enforce indi-
vidual feature level sparsity, although our optimization algorithm can be applied in a
similar way regardless of whether there is VLasso (␤ ) or not. While the structure among
inputs or outputs can be leveraged, we first discuss the problem of estimating structured
input regression and then extend it to structured output and structured input-output
regression.

2.1 Overlapping Group Lasso Penalty

Assume that the set of groups of inputs G 5 {g1 , . . . , g|G| } is defined as a subset of the
power set of {1, . . . , J} and is available as prior knowledge. Note that members of G
(groups) are allowed to overlap. The overlapping group Lasso penalty based on the 1 /2
mixed-norm (Jenatton et al. 2009/2011) is defined as


Vstruct (␤ ) ≡ g wg ␤ g 2 , (4.7)
g∈G

where ␤ g ∈ R|g| is the subvector of ␤ for the inputs in group g; wg is the predefined
weight for group g; and  · 2 is the vector 2 -norm. The 1 /2 mixed-norm penalty V(␤ )
plays the role of setting all the coefficients within each group to zero or nonzero values.
The widely used hierarchical tree-structured penalty (Zhao, Rocha, and Yu 2009) is a
special case of eq. (4.7). It is worthwhile to note that the 1 /` mixed-norm penalty can
also achieve the similar grouping effect. Although our approach can also be used for the
1 /` penalty as well, we focus on the 1 /2 penalty.

We also note that the penalty Vstruct (␤ ) ≡ g g∈G wg ␤ g 2 enforces group-level
sparsity but not sparsity within each group. More precisely, if the estimated ␤ˆ g 2 5 0,
each b̂j for j ∈ g will be nonzero. With the 1 regularization VLasso (␤ ) on top of Vstruct (␤ )
as in eq. (4.6), we not only select groups but also variables within each group. Simon
et al. (2012) give more details.
42 Chapter 4 Eric P. Xing and Colleagues

2.2 Graph-Guided Fusion Penalty

Let us assume the structure of J input variables is available as a graph G with a set
of nodes V 5 {1, . . . , J} and a set of edges E. Let rml ∈ R denote the weight of the
edge e 5 (m, l) ∈ E, corresponding to the correlation between the two inputs for nodes
m and l. The graph-guided fusion penalty is defined as

Vstruct (␤ ) 5 g t(rml )|bm 2 sign(rml )bl |, (4.8)
e5(m,l)∈E,m,l

where t(r) weights the fusion penalty for each edge e 5 (m, l), such that bm and bl for
highly correlated inputs with larger |rml | receive a greater fusion effect. We consider
t(r) 5 |r|, but any monotonically increasing function of the absolute values of correla-
tions can be used. The sign(rml ) indicates that for two positively correlated nodes, the
corresponding coefficients tend to influence the output in the same direction, and for
two negatively correlated nodes, the effects (bm and bl ) take the opposite direction. Since
this fusion effect is calibrated by the edge weight, the graph-guided fusion penalty in
eq. (4.8) encourages highly correlated inputs corresponding to a densely connected sub-
network in G to be jointly selected as relevant. We notice that if rml 5 1 for all e 5 (m, l),
the penalty function in eq. (4.8) reduces to

Vstruct (␤ ) 5 g |bm 2 bl |. (4.9)
e5(m,l)∈E,m,l

 21
The standard fused Lasso penalty (Tibshirani et al. 2005) defined as g Jj51 |bj11 2 bj | is
a special case of eq. (4.9), where the graph structure is confined to be a chain and the
widely used fused signal approximator refers to the simple case where the design matrix
X is orthogonal.

3 OPTIMIZATION ALGORITHMS
In this section, we discuss numerical procedures for solving the optimization problem
in eq. (4.6) with penalties introduced in the previous sections. The problem in eq. (4.6)
is convex, and there are a number of methods that can be used to find a minimizer. Gen-
eral techniques like subgradient methods and interior point methods (IPMs) for second-
order cone programs (SOCPs) can be used. However, these methods are not suitable for
high-dimensional problems arising in practical applications because of their slow con-
vergence rate or poor scalability. On the other hand, block gradient methods and proxi-
mal gradient methods, although not as general, do exploit the structure of the penalties
and can scale well to large problems. In the following section, we first discuss some
general methods for solving convex programs and then focus on proximal methods.
Each optimization algorithm is measured by its convergence rate, that is, the number of
iterations t to achieve an e-accurate solution: f (␤ t ) 2 f (␤ ∗ ) # e, where ␤ ∗ is one of the
minimizers of f (␤ ).
 High-Dimensional Sparse Structured Input-Output Models 43

3.1 Subgradient Descent

Subgradient descent is a general method that can be applied to any unconstrained
convex optimization problem. The method requires that a subgradient of the penalty
V(␤ ) 5 Vstruct (␤ ) 1 VLasso (␤ ) can be computed efficiently. A subgradient of a convex
function V : RJ 31 → R at ␤ is defined as an element of the following set:

äV(␤ ) :5 {z ∈ RJ 31 |V(␤ ) 1 zT (␤  2 ␤ ) # V(␤  ) for all ␤  ∈ RJ 31 }.

The method involves updating the estimate ␤ t11 with the following iterations:

c1
␤ t11 5 ␤ t 2 äf (␤ t ), (4.10)
t c2

where

äf (␤ t ) 5 ∇(␤ t ) 1 z 5 XT (X␤ t 2 y) 1 z with z ∈ äV(␤ t ),

where c1 in eq. (4.10) is a constant parameter and c2 5 1 for strongly convex loss (␤ )
and c2 5 1/2 for nonstrongly convex loss (␤ ). The updates are equivalent to the usual
gradient descent with the gradient substituted with a subgradient. The algorithm con-
verges under suitable conditions, but this convergence is slow. In particular, the con-
   
vergence rate for subgradient descent is O 1e for strongly convex loss and O e12 for
nonstrongly convex loss (␤ ). In high-dimensional settings with J @ N, XT X is rank-
deficient and hence (␤ ) is nonstrongly convex. Therefore, the vanilla subgradient
 
descent has a slow convergence rate of O e12 in our problems.

3.2 Block Coordinate Descent

When there is no overlap between different groups in group Lasso penalty, block coor-
dinate descent can be applied to solve eq. (4.6). According to the subgradient conditions
for eq. (4.6), the optimal ␤ g should satisfy

2(Xg )T (y 2 Xg␤ g ) 1 äVstruct (␤ g ) 1 äVLasso (␤ g ) 5 0. (4.11)
g

This optimality condition can be obtained for each block coefficient ␤ g , and using this
condition, we can derive an optimization procedure that iteratively computes an opti-
mal ␤ g fixing other coefficients. The general optimization procedure is as follows: for
each group g, we check the group sparsity condition that ␤ g 5 0. If it is true, no update
is needed for ␤ g . Otherwise, we solve eq. (4.6) over ␤ g with all other coefficients fixed.
This step can be efficiently solved by using a standard optimization technique such as
accelerated generalized gradient descent (Simon et al. 2012; Beck and Teboulle 2009).
This procedure is continued until a convergence condition is met. Block coordinate
descent is efficient to solve eq. (4.6) only with nonoverlapping group Lasso penalty.
However, this method cannot be used for overlapping group Lasso penalty owing to the
lack of convergence guarantee (Tseng and Yun 2009).
44 Chapter 4 Eric P. Xing and Colleagues

3.3 Second-Order Cone Program or Quadratic Program

Reformulation
On the other hand, the structured sparse learning problems can also be easily formu-
lated into second-order cone programs (SOCPs) or quadratic programs (QPs). Take the
overlapping group Lasso penalty as an example; the corresponding structured sparse
learning problem can be formulated into an SOCP as follows:

1  
J
min s1g w g tg 1 l qj
2 g∈G j51

s.t. X␤ 2 y22 # s; ␤ g 2 # tg ∀ g ∈ G; |bj | # qj j 5 1, . . . , J.

We can also formulate the optimization problem with graph-guided fusion penalty
into a QP by letting bj 5 q1 2 1 2 1 2
j 2 qj with qj , qj $ 0 and bm 2 sign(rml )bl 5 sml 2 sml with
1 2
sml , sml $ 0.
The benefit of these approaches is that the standard IPMs along with many read-
ily available toolboxes (e.g., SDPT3 (Tütüncü, Toh, and Todd 2003)) can be directly
used to solve the convex problems. Even though IPMs achieve a faster convergence rate
  
of O log 1e and can lead to solutions with very high precision, solving the Newton
linear system at each iteration of an IPM is computationally too expensive. Therefore,
IPMs can only be used to solve small or medium-scale problems.

3.4 Proximal Gradient Method

Proximal gradient methods represent a generic, arguably more simplistic and efficient
family of first-order methods for solving a composite minimization problem of the form

min (␤ ) 1 V(␤ ),

␤

where the function (␤ ) is a differentiable convex function and V(␤ ) is a nonsmooth
penalty. Proximal gradient methods, which are descendants of the classical projected
gradient algorithms, have become popular because they only utilize the gradient infor-
mation and hence can scale up to very large problems. A typical iteration of the algo-
rithm looks like

L
␤ t11 5 argmin␤ (␤ t ) 1 ∇(␤ t ), ␤ 2 ␤ t 1 ␤ 2 ␤ t 22 1 V(␤ ), (4.12)
2

where L . 0 is a parameter that should upper bound the Lipschitz constant of ∇(␤ ).
This step is often called the proximal operator, proximal mapping, or simply projection
step.
Efficiency of this iterative algorithm relies on the ability to efficiently solve the
proximal operator exactly without any error. When there is an exact solution of the
 High-Dimensional Sparse Structured Input-Output Models 45

proximal operator, it can be shown that the proximal gradient method with an accel-
eration scheme (Nesterov 2007; Beck and Teboulle 2009) leads to a convergence rate
 
of O e12 , and this rate is optimal under the first-order black-box model (Nesterov 2003).
The proximal operator in eq. (4.12) can be rewritten as

1 1 1
min ␤ 2 (␤ t 2 ∇(␤ t ))22 1 V(␤ );
␤ 2 L L

therefore it is important to find solutions to the following optimization problem:

1 1
␤ˆ 5 arg min ␤ 2 v22 1 V(␤ ).
␤ 2 L

This can be done in a closed form for the Lasso type penalty, i.e., when V(␤ ) 5
VLasso (␤ ) 5 l␤ 1 the solution ␤ˆ can be obtained by the soft-thresholding operator
(Friedman, Hastie, and Tibshirani 2010):
 
l
b̂j 5 sign(vj ) max 0, |vj | 2 . (4.13)
L

A closed-form solution can also be obtained for the 1 /2 mixed-norm penalties

for nonoverlapping groups. In particular, when V(␤ ) 5 Vstruct (␤ ) 5 g g∈G ␤ g 2 with
nonoverlapping groups, the closed-form solution of the proximal operator takes the
form (Duchi and Singer 2009)
 
g
␤ˆ g 5 max 0, 1 2 vg .
Lvg

3.5 Smoothing Proximal Gradient Method

For many complex structured sparsity–inducing penalties commonly used in GWAS,
such as penalties in eq. (4.7) and eq. (4.8), there is no exact solution for the proximal
operator discussed in the previous section, and hence the proximal gradient method
cannot be directly applied. To address this challenge, the smoothing proximal gradi-
ent (SPG) method, which is based on Nesterov’s smoothing technique, was proposed
in Chen et al. (2012). Such a method can efficiently solve structured sparse learning
problems in eq. (4.6) with a wide spectrum of Vstruct (␤ ).
Another motivation behind the SPG method is the nonseparability of ␤ in the non-
smooth penalty function Vstruct (␤ ) in eq. (4.6). For example, for both the overlapping
group Lasso penalty in and the graph-guided fusion penalty, Vstruct (␤ ) encodes non-
separable constraints over ␤ , which prohibit direct application of efficient first-order
methods such as a block gradient algorithm. However, a closer examination of formu-
lations of Vstruct (␤ ) for both structures reveals that although they are seemingly very
different, they can in fact be reformulated into the same form. A key idea behind SPG
is to decouple the nonseparable constraints in the structured sparsity–inducing penal-
ties via a simple linear transformation of ␤ via the dual norm. Based on that, a smooth
46 Chapter 4 Eric P. Xing and Colleagues

approximation to Vstruct (␤ ) can be introduced using the technique from Nesterov (2005)
such that its gradient with respect to ␤ can be easily calculated.

SPG in Single-Response Structured Sparse Regression

We first introduce the key ideas behind the linear transformation and smooth approxi-
mation procedures underlying SPG in the simple context of single-response regression
with either overlapping group Lasso penalty or graph-guided fusion penalty.

Deconvolving Overlapping Group Lasso Penalty

Since the dual norm of 2 -norm is also an 2 -norm, ␤ g 2 5 max␣ g 2 #1 ␣ Tg ␤ g , where
 T
␣ g ∈ R|g| is the vector of auxiliary variables associated with ␤ g . Let ␣ 5 ␣ Tg1 , . . . , ␣ Tg|G| .

Then ␣ is a vector of length g∈G |g| with domain Q ≡ {␣ | ␣ g 2 # 1, ∀g ∈ G}, where Q
is the Cartesian product of unit balls in Euclidean space and thus a closed and convex
set. The overlapping group Lasso penalty in eq. (4.7) can be rewritten as

 
Vstruct (␤ ) 5 g wg max ␣ Tg ␤ g 5 max gwg␣ Tg ␤ g 5 max ␣ T C␤ , (4.14)
␣ g 2 #1 ␣ ∈Q ␣ ∈Q
g∈G g∈G


where C ∈ R g∈G |g|3J is a matrix defined as follows. The rows of C are indexed by all
pairs of (i, g) ∈ {(i, g)|i ∈ g, i ∈ {1, . . . , J}}, the columns are indexed by j ∈ {1, . . . , J}, and
each element of C is given as

gwg if i 5 j,
C(i,g),j 5 (4.15)
0 otherwise.

Note that C is a highly sparse matrix with only a single nonzero element in each row

and g∈G |g| nonzero elements in the entire matrix, and hence it can be stored with only
a small amount of memory during the optimization procedure.

Deconvolving Graph-Guided Fusion Penalty

The graph-guided fusion penalty in eq. (4.8) can be written as follows:


g t(rml )|bm 2 sign(rml )bl | ≡ C␤ 1 ,
e5(m,l)∈E,m,l

where C ∈ R|E|3J is the edge-vertex incident matrix:

⎧
⎪
⎨ g · t(rml )
⎪ if j 5 m
Ce5(m,l),j 5 2g · sign(rml )t(rml ) if j 5 l (4.16)
⎪
⎪
⎩0 otherwise.
 High-Dimensional Sparse Structured Input-Output Models 47

Again, note that C is a highly sparse matrix with 2 · |E| nonzero elements. Since the
dual norm of the ` -norm is the 1 -norm, the graph-guided fusion penalty can be further
rewritten as
C␤ 1 ≡ max ␣ T C␤ , (4.17)
␣ ` #1

where ␣ ∈ Q 5 {␣ |␣ ` # 1, ␣ ∈ R|E| } is a vector of auxiliary variables associated with

C␤ 1 , and  · ` is the ` -norm defined as the maximum absolute value of all entries in
the vector.

Smooth Approximation
With the reformulation using the dual norm, all different forms of structured sparsity–
inducing penalties can be formulated into a maximization problem of the form

Vstruct (␤ ) 5 max ␣ T C␤ . (4.18)

␣ ∈Q

However, it is still a nonsmooth function of ␤ , and this makes the optimization chal-
lenging. To tackle this problem, a smooth approximation of Vstruct (␤ ) can be constructed
using Nesterov’s smoothing technique (Nesterov 2005):

 
Vmstruct (␤ ) 5 max ␣ T C␤ 2 md(␣ ) , (4.19)
␣ ∈Q

where d(␣ ) is defined as 12 ␣ 22 , and m is the positive smoothness parameter that controls
the quality of the approximation:

Vstruct (␤ ) 2 mD # Vmstruct (␤ ) # Vstruct (␤ ),

where D 5 max␣ ∈Q d(␣ ). Given the desired accuracy e, the convergence result suggests
e
m 5 2D to achieve the best convergence rate.
The function Vmstruct (␤ ) is smooth in b with a simple form of the gradient:

∇Vmstruct (␤ ) 5 C T ␣ ∗ , (4.20)

where ␣ ∗ is the optimal solution to eq. (4.19). The optimal ␣ ∗ can be easily obtained in a
closed form for a number of penalties of interest. In particular, for the overlapping group
gwg ␤
Lasso penalty, ␣ ∗ is composed of {␣ ∗g }g∈G for each group g ∈ G and ␣ ∗g 5 S( m g ). Here
S is the projection operator that projects any vector u to the 2 ball:

⎧
⎨ u
u2 . 1,
u2
S(u) 5
⎩u u2 # 1.
Random documents with unrelated
content Scribd suggests to you:
compromising its fidelity or efficiency, she made very attractive by its
literary qualities and its entertaining and instructive miscellany.
Mrs. Maria W. Chapman, who wielded gracefully a trenchant pen,
plied it busily in our cause with great effect. Her successive numbers
of “Right and Wrong in Boston” were too incisive not to touch the
feelings of the good people of that metropolis, which claimed to be
the birthplace of American independence, but had ceased to be
jealous for “the inalienable rights of man.” Year after year her
“Liberty Bell” rung out the clearest notes of personal, civil, and
spiritual liberty, and she compiled our Antislavery Hymn Book,—“The
Songs of the Free,”—effusions of her own and her sisters’ warm
hearts, and of their kindred spirits in this country and England.
But though the excellent women whom I have named, and many
more like them, constantly attended our meetings, and often
suggested the best things that were said and done at them, they
could not be persuaded to utter their thoughts aloud. They were
bound to silence by the almost universal sentiment and custom
which forbade “women to speak in meeting.”
In 1836 two ladies of a distinguished family in South Carolina—
Sarah and Angelina E. Grimké—came to New York, under a deep
sense of obligation to do what they could in the service of that class
of persons with whose utter enslavement they had been familiar
from childhood. They were members of the “Society of Friends,” and
were moved by the Holy Spirit, as the event proved, to come on this
mission of love. They made themselves acquainted with the
Abolitionists, our principles, measures, and spirit. These commended
themselves so entirely to their consciences and benevolent feelings
that they advocated them with great earnestness, and enforced their
truth by numerous facts drawn from their own past experience and
observation.
In the fall of 1836 Miss A. E. Grimké published an “Appeal to the
Women of the South,” on the subject of slavery. This evinced such a
thorough acquaintance with the American system of oppression, and
so deep a conviction of its fearful sinfulness, that Professor Elizur
Wright, then Corresponding Secretary of the American Antislavery
Society, urged her and her sister Sarah to come to the city of New
York and address ladies in their sewing-circles, and in parlors, to
which they might be invited to meet antislavery ladies and their
friends. No man was better able than Professor Wright to appreciate
the value of the contributions which these South Carolina ladies
were prepared to make to the cause of impartial liberty and
outraged humanity. As early as 1833, while Professor of Mathematics
and Natural Philosophy in Western Reserve College, he published an
elaborate and powerful pamphlet on “The Sin of Slave-holding,”
which we accounted one of our most important tracts. Commended
by him and by others who had read her “Appeal,” Miss Grimké and
her sister attracted the antislavery women of New York in such
numbers that soon no parlor or drawing-room was large enough to
accommodate those who were eager to hear them. The Rev. Dr.
Dunbar, therefore, offered them the use of the vestry or lecture-
room of his church for their meetings, and they were held there
several times. Such, however, was the interest created by their
addresses, that the vestry was too small for their audiences.
Accordingly, the Rev. Henry G. Ludlow opened his church to them
and their hearers, of whom a continually increasing number were
gentlemen.
Early in 1837 the Massachusetts Antislavery Society invited these
ladies to come to Boston to address meetings of those of their own
sex. But it was impossible to keep them thus exclusive, and soon,
wherever they were advertised to speak, there a large concourse of
men as well as women was sure to be assembled. This was an
added offence, which our opposers were not slow to mark, nor to
condemn in any small measure. It showed plainly enough that “the
Abolitionists were ready to set at naught the order and decorum of
the Christian church.”
My readers may smile when I confess to them that at first I was
myself not a little disturbed in my sense of propriety. But I took the
matter into serious consideration. I looked the facts fully in the face.
Here were millions of our countrymen held in the most abject, cruel
bondage. More than half of them were females, whose condition in
some respects was more horrible than that of the males. The people
of the North had consented to this gigantic wrong with those of the
South, and those who had risen up to oppose it were denounced as
enemies of their country, were persecuted, their property and their
persons violated. The pulpit for the most part was dumb, the press
was everywhere, with small exceptions, wielded in the service of the
oppressors, the political parties were vying with each other in
obsequiousness to the slaveholding oligarchy, and the petitions of
the slaves and their advocates were contemptuously and angrily
spurned from the legislature of the Republic. Surely, the condition of
our country was wretched and most perilous. I remembered that in
the greatest emergencies of nations women had again and again
come forth from the retirement to which they were consigned, or in
which they preferred to dwell, and had spoken the word or done the
deed which the crises demanded. Surely, the friends of humanity, of
the right and the true, never needed help more than we needed it.
And here had come two well-informed persons of exalted character
from the midst of slavedom to testify to the correctness of our
allegations against slavery, and tell of more of its horrors than we
knew. And shall they not be heard because they are women? I saw,
I felt it was a miserable prejudice that would forbid woman to speak
or to act in behalf of the suffering, the outraged, just as her heart
may prompt and as God has given her power. So I sat me down and
penned as earnest a letter as I could write to the Misses Grimké,
inviting them to come to my house, then in South Scituate, to stay
with us as long as their engagements would permit, to speak to the
people from my pulpit, from the pulpit of my excellent cousin, Rev.
E. Q. Sewall, Scituate, and from as many other pulpits in the county
of Plymouth as might be opened to them.
They came to us the last week of October, 1837, and tarried
eight days. It was a week of highest, purest enjoyment to me and
my precious wife, and most profitable to the community.
On Sunday evening Angelina addressed a full house from my
pulpit for two hours in strains of wise remark and eloquent appeal,
which settled the question of the propriety of her “speaking in
meeting.”
The next afternoon she spoke to a large audience in Mr. Sewall’s
meeting-house in Scituate, for an hour and a half, evidently to their
great acceptance. The following Wednesday I took the sisters to
Duxbury, where, in the Methodist Church that evening, Angelina held
six hundred hearers in fixed attention for two hours, and received
from them frequent audible (as well as visible) expressions of assent
and sympathy.
On Friday afternoon I went with them to the Baptist meeting-
house in Hanover, where a crowd was already assembled to hear
them. Sarah Grimké, the state of whose voice had prevented her
speaking on either of the former occasions, gave a most impressive
discourse of more than an hour’s length on the dangers of slavery,
revealing to us some things which only those who had lived in the
prison-house could have learnt. Angelina followed in a speech of
nearly an hour, in which she made the duty and safety of immediate
emancipation appear so plainly that the wayfaring man though a fool
must have seen the truth. If there was a person there who went
away unaffected, he would not have been moved though an angel
instead of Angelina had spoken to him. I said then, I have often said
since, that I never have heard from any other lips, male or female,
such eloquence as that of her closing appeal. Several gentlemen
who had come from Hingham, not disposed nor expecting to be
pleased, rushed up to me when the audience began to depart, and
after berating me roundly for “going about the neighborhood with
these women setting public sentiment at naught and violating the
decorum of the church,” said “there can be no doubt that they have
a right to speak in public, and they ought to be heard; do bring
them to Hingham as soon as may be. Our meeting-house shall be at
their service.” Accordingly, the next day I took them thither, and they
spoke there with great effect on Sunday evening, November 5th,
from the pulpit of the Unitarian Church, then occupied by Rev.
Charles Brooks.
 The experience of that week dispelled my Pauline prejudice. I
needed no other warrant for the course the Misses Grimké were
pursuing than the evidence they gave of their power to speak so as
to instruct and deeply impress those who listened to them. I could
not believe that God gave them such talents as they evinced to be
buried in a napkin. I could not think they would be justified in
withholding what was so obviously given them to say on the great
iniquity of our country, because they were women. And ever since
that day I have been steadfast in the opinion that the daughters of
men ought to be just as thoroughly and highly educated as the sons,
that their physical, mental, and moral powers should be as fully
developed, and that they should be allowed and encouraged to
engage in any employment, enter into any profession, for which they
have properly qualified themselves, and that women ought to be
paid the same compensation as men for services of any kind equally
well performed. This radical opinion is spreading rapidly in this
country and in England, and it will ultimately prevail, just as surely
as that God is impartial and that “in Christ Jesus there is neither
bond nor free, neither male nor female.” And yet it has been, and is,
as strenuously opposed and as harshly denounced as was our
demand of the immediate emancipation of the enslaved. Men and
women, press and pulpit, statesmen and clergymen, legislative and
ecclesiastical bodies have raised the cry of alarm, and pronounced
the advocates of the equal rights of women dangerous persons,
disorganizers, infidels.
The first combined assault was made upon “The Rights of
Women” by the Pastoral Association of Massachusetts in the fall of
1837 or the spring of 1838, in their spiritual bull against the
antislavery labors of the Misses Grimké, which it utterly condemned
as unchristian and demoralizing. This, of course, made it the duty, as
it was pleasure, of the New England Abolitionists to stand by those
excellent women, who had rendered such inestimable services to the
cause of the enslaved, the down-trodden, the despised millions of
our countrymen. Therefore, at the next New England Antislavery
Convention, held in Boston, May, 1838, attended by delegates from
eleven States, it was “Voted, That all persons present, or who may
be present, at subsequent meetings, whether men or women, who
agree with us in sentiment on the subject of slavery, be invited to
become members and participate in the proceedings of the
Convention.”
This gave rise to a long and very animated discussion, but was
passed by a very large majority. Immediately eight Orthodox
clergymen requested to have their names erased from the roll of
that Convention, and seven others, including some of our faithful
fellow-laborers, presented a protest against the vote, which, by their
request, was entered upon the records, and published with the
doings of the Convention.
At that same great gathering a committee of three persons was
appointed to prepare and transmit a memorial to each and all of the
ecclesiastical associations in New England, of every sect, beseeching
them to testify against the further continuance in our country of
slavery, and take such measures as they might deem best to induce
the members of their several denominations who were guilty of the
dreadful iniquity to consider and turn away from it. One of that
committee was a much respected woman, as well qualified as either
of her associates to discharge the duties assigned them. An excellent
memorial was prepared and presented in accordance with the vote.
But it was very coldly received by some, and rudely treated by
others of the ecclesiastical bodies to which it was sent. On the
presentation of it to the Rhode Island Congregational Consociation, a
scene of great excitement ensued. The memorial was treated with
all possible indignity. Most of the brethren who had been earnest for
the reception of it, and for such action as it requested, when they
were informed that one of the committee by whom the memorial
was prepared was a woman, united in a vote “to turn the illegitimate
product from the house, and obliterate from the records all traces of
its entrance.” No deliberative assembly ever behaved in a more
indecorous manner. And those who were most active in trampling
upon that respectful petition in behalf of bleeding humanity were the
professed ministers of Him who came to preach deliverance to the
captive. “O tempora! O mores!!”

“THE PASTORAL LETTER” AND “THE CLERICAL

APPEAL.”
Abolitionists from the first were persons of both sexes and all
complexions, of every class in society, of every religious
denomination, of each of the three learned professions, of both
political parties, and of all the various trades and occupations in
which men and women engage. Although it is too true that most
ministers, especially in the cities, were slow to espouse the cause of
the oppressed, yet it is due to them to say that, taking the country
through, there were, in proportion to their numbers, more of that
profession than of either of the others who embraced the doctrine of
“immediate emancipation,” advocated it publicly, wrote columns,
pamphlets, and volumes in its defence, and suffered no little obloquy
and persecution for so doing. And they were, as I have said, of
every Protestant sect. Whenever a complete history of our
antislavery conflict shall be written, grateful and admiring mention
will be made of the valuable services and generous sacrifices of
many ministers whose names may not appear in my slight sketches.
These various individuals were evidently moved by one spirit,
drawn together by the conviction that there was a great, a fearful
iniquity involved in the enslavement of millions of the inhabitants of
our land, that if the God-given rights of humanity were (as the
founders of our Republic declared them to be) inalienable, then
those men, who were holding human beings as their chattels, were
setting the will and authority of the Almighty at defiance, and would
bring themselves to ruin. Moreover, there was a deep conviction
awakened in the hearts of those who openly espoused the cause of
the bondmen, that the people of the North were verily guilty in
consenting to their enslavement; and, as the States and the
churches refused to interfere for their deliverance, it was left for
individuals and voluntary associations to do what might be done, so
to correct public opinion and awaken the public conscience that
slavery could not be tolerated in the land.
Further than this there was little agreement among the early
Abolitionists. But this proved to be a mighty solvent. And for years
the wonderful, the beautiful, the Christian sight was seen,—
Trinitarians and Unitarians, Methodists and Universalists, Baptists
and Quakers, laboring together in the cause of suffering fellow-
beings, with so much earnestness that they had set aside, for the
while, their theological and ritualistic peculiarities, and seemed to
rejoice in their release from those narrow enclosures. Coming out of
our hall on the second evening of our Convention in Philadelphia, in
December, 1833, a young Orthodox minister took my arm with an
affectionate pressure, and said, “Brother May, I never thought that I
could feel towards a Unitarian as I feel towards you.” My reply was:
“Dear M., if professing Christians were only real Christians, engaged
in the work of the Lord, they could not find the time nor the heart to
quarrel about creeds and rites.” Wherever I went, preaching the
gospel of impartial liberty, I was as cordially received by Orthodox as
by Unitarian Abolitionists, until I came to have a much more
brotherly feeling towards an antislavery Presbyterian or Baptist or
Methodist than I did towards a Unitarian who was proslavery, or
indifferent to the wrongs of the bondmen. And this feeling was
obviously reciprocated. I was repeatedly invited to preach in the
pulpits of Orthodox ministers, and to commune with Orthodox
churches. Once I attended a church in company with Miss Ann G.
Chapman, one of the most single-minded and true-hearted of
women. The invitation to the Lord’s table was given in such words as
virtually excluded us. Of course we arose and departed. But so soon
as the service was over both the minister and deacon (beloved
antislavery brethren) came to my lodgings to assure me that the
exclusion was not intended, and that whenever Miss Chapman and
myself might again be at their church on a similar occasion, they
hoped that we would commune there.
 I give these facts, and could give many more like them, to show
the anti-sectarian tendency of the antislavery reform. This was
perceived by many of “the wise and prudent” leaders of the sects,
and was evidently watched by them with a jealous eye. As the
number of Abolitionists increased, and our influence in the churches
came to be felt more and more, many of those leaders joined
antislavery societies, partly, no doubt, because they had been
brought to see the truth of our doctrines and the importance of the
work we were laboring to accomplish, but also in part, if not chiefly
(as I was afterwards forced to suspect), because they wished to
maintain the ascendency over their sects, and to prevent the
obliteration of the lines which separated them from such as they
were pleased to consider unsound in faith.
We were greatly encouraged and gladdened by the accessions
we received in 1835 and 1836. Many ministers of the evangelical
sects joined us, not a few of them Doctors of Divinity. And the
obligations of Christians to the bondmen in our land, and the
discipline that should be brought to bear on those professing
Christians who were holding them in slavery, became the subjects of
earnest debate in several of the large ecclesiastical bodies. But we
found these new-comers were much disposed to object to the liberty
that was allowed on our platform. Generally the president or
chairman of our meetings would call upon some one to invoke the
divine blessing upon our undertaking. Sometimes, in deference to
our Quaker brethren, we would sit in silence until the Spirit moved
some one to offer prayer. Then again, persons who were not
members of any religious denomination, nay, even some who were
suspected of being, if not known to be, unbelievers, infidels, were
permitted to co-operate with us, to contribute to our funds, to take
part in our deliberations, and to be put upon our committees. This
was a scandal in the estimation of those of the “straitest sect.” Our
only reply was, that as so many, who made the highest professions
of Christian faith, turned a deaf ear to the cries of the millions who
were suffering the greatest wrongs, we were grateful for the
assistance of such as made no professions. Not those who cried
Lord, Lord, but those who were eager to do the will of the impartial
Father, were the persons we valued most.
But nothing gave so much offence as the admission of women to
speak in our meetings, to act on our committees, and to co-operate
with us in any way they saw fit. In my last I gave some account of
the rupture it caused in our New England Antislavery Convention in
1838. This was foreshadowed the year previous. Some time in the
summer of 1837 the General Association of Massachusetts issued a
“Pastoral Letter to the churches under their care,” intended to avert
the alarming evils which were coming upon them from the over-
heated zeal of the Abolitionists. First, the extraordinary document
mourns over the loss of deference to the pastoral office, which is
enjoined in Scripture, and which is essential to the best influence of
the ministry. At this day, when all but Roman Catholics and High
Church Episcopalians are wondering at, if not amused by, the
dealing of Bishop Potter with Mr. Tyng, it may surprise my readers to
be told that thirty years ago the Orthodox Congregational ministers
of Massachusetts set up the same claim of authority in their several
parishes, that the diocesan of New York and New Jersey demands
for his clergymen. “One way,” they said in their Pastoral Letter, “one
way in which the respect due to the pastoral office has been in some
cases violated, is in encouraging lecturers or preachers on certain
topics of reform to present their subjects within the parochial limits
of settled pastors, without their consent.” “Your minister is ordained
of God to be your teacher, and is commanded to feed that flock over
which the Holy Ghost hath made him overseer. If there are certain
topics upon which he does not preach with the frequency, or in the
manner that would please you, it is a violation of sacred and
important RIGHTS to encourage a stranger to present them.”
“Deference and subordination are essential to the happiness of
society, and peculiarly so in the relation of a people to their pastor.”
Happily for those who may come after us, we Abolitionists have
done much to emancipate the people from such spiritual bondage,
and secure to them the privilege of seeking after knowledge
wherever it may be found, and yielding themselves to good
influences, let them come through whatever channel they may.
But the “Pastoral Letter” dwelt at greater length upon the
dangers which threatened the female character with wide-spread
and permanent injury. Forgetting that women were the bravest, as
well as the most devoted and affectionate of the first disciples of
Jesus, that in all ages since they have been prominent among the
confessors of Christianity, and that in our day they do more than
men to uphold the churches,—forgetting these facts, the frightened
authors and signers of that letter uttered themselves thus: “The
power of woman is in her dependence, flowing from the
consciousness of that weakness which God has given her for her
protection, and which keeps her in those departments of life that
form the characters of individuals and of the nation.... But, when she
assumes the place and tone of man as a public reformer, our care
and protection of her seem unnecessary; we put ourselves in self-
defence against her; she yields the power which God has given her
for protection, and her character becomes unnatural. If the vine,
whose strength and beauty is to lean upon the trellis-work and half
conceal its clusters, thinks to assume the independence and the
overshading nature of the elm, it will not only cease to bear fruit,
but will fall in shame and dishonor into the dust.” Did not those
ministers know—were there not in their day wives who sustained
their husbands instead of leaning upon them? women who were the
stay and staff of the men of their families—their mental and moral
stamina? There have been such women in all other times; we have
known and do know such women now. If our antislavery conflict has
done nothing else, it has shown that there is neither orthodox nor
heterodox, neither white nor black, neither male nor female, but all
are one in the work of the Lord.
Undismayed by the censure and warning of so exalted a body as
the General Association, we Abolitionists continued to labor as we
had done, pursuing the same measures, using the same
instrumentalities, employing as our agents and lecturers women no
less than men, whom we found able as well as willing to do good
service. And to several, besides those I have already named, the
bondmen and their advocates were immeasurably indebted. Abby
Kelly (now Mrs. Foster) performed for years an incredible amount of
labor. Her manner of speaking in her best days was singularly
effective. Her knowledge of the subject was complete, her facts
were pertinent, her arguments forcible, her criticisms were keen, her
condemnation was terrible. Few of our agents of either sex did more
work while her strength lasted, or did it better.
Susan B. Anthony was one of the living spirits of our financial
department, indomitable in her purposes, ingenious in her plans,
untiring in her exertions, she not only kept herself continually at
work, but spurred all about her to new effort. She has often herself
spoken to excellent effect, and more frequently stimulated others to
their best efforts.
Miss Sallie Holley has seldom consented to speak in our largest
assemblies, or in our cities. But we have very frequently heard of her
diligent labors in the rural districts, and of the good fruits she has
gathered there. Her eloquence is particularly dignified and
impressive.
I should love to tell of Lucy Stone, and Antoinette L. Brown, and
Mrs. E. C. Stanton, and Ernestine L. Rose, all wise women and
attractive speakers, but their word and work has been given more to
the advocacy of “Woman’s Rights.” The reformation for which they
have toiled so long and so well, though the offspring of Abolitionism,
is still more radical; and to the history of it volumes will hereafter be
devoted.
I can here only name Miss Anna E. Dickinson, now one of the
most attractive of the popular lecturers. Although another of the
women who have been brought out of their retirement by the
exigency of the times, yet she came upon the platform about the
period at which I intend these recollections shall cease.
As surely as the conflict with slavery has been found to be
irrepressible, so surely will it be found to be impossible to suppress
the conflict for the rights of women until they shall be securely
placed where the Creator intended them to stand, on an entire
equality with men in their domestic, social, legal, and political
relations.
Not long after the “Pastoral Letter,” there came forth from some
of the members of the Massachusetts General Association a still
more pointed attack upon The Liberator, Mr. Garrison and his
associates, one which would have been very damaging if it had not
been so easily repelled. It was entitled the “Appeal of Clerical
Abolitionists on Antislavery Measures,” signed by two Orthodox
ministers of Boston, and three in the vicinity of that city. As these
gentlemen had belonged to the Antislavery Society, and two of them
had been vehement if not fierce in their advocacy of our doctrines, it
would seem that they must have known whereof they affirmed.
They prefaced their Appeal with a declaration of their lively interest
in the cause of the oppressed, their clear perception of the
sinfulness and their detestation of slavery. Then they went on to
accuse the leading Abolitionists, 1st, of hasty, unsparing, and almost
ferocious denunciation “of a certain reverend gentleman because he
had resided in the South,” without having taken pains to ascertain
whether he had been a slaveholder or not; 2d, They accused us of
“hasty insinuations” against an Orthodox minister of high standing in
Boston, that he was a slaveholder, without having had any proof of
the truth of the reports we may have heard so damaging to the
reverend gentleman’s reputation. Their third, fourth, and fifth
accusations were, that we had demanded of ministers what we had
no right to require of them; had abused them for not doing as we
called upon them to do, and, through our zeal in the cause of the
enslaved, we had become indifferent to other Christian enterprises,
and would withdraw from them the regards of those who co-
operated with us, and that we had censured and denounced
excellent Christian ministers and church-members because they
were not prepared to enter fully into the work of antislavery
societies.
 This document, coming from such persons, of course was the
occasion of no little excitement. Our enemies exulted over it as
testimony against us, given by those who had been in our councils
and well knew what spirit animated us. Others who had been timid
friends, or half inclined to join our ranks, were at first repulsed from
us by the apprehension that there was too much truth in these
charges.
But as soon as possible elaborate and thorough replies were
published to this Appeal, denying the truth of each of the above-
named accusations, and showing them to be false. One of the
replies was written by Mr. Garrison, in his clear and trenchant style,
and showed up the inconsistency as well as the falseness of the
accusations by ample quotations from the writings and speeches of
Mr. Fitch, the author of the Appeal. The other reply was from the
pen of Rev. A. A. Phelps.
This good orthodox brother was then the General Agent of the
Antislavery Society, and therefore felt it to be incumbent upon him to
repel charges so unjust and so injurious. No one but Mr. Garrison
was so competent as he to do this. From an early period Mr. Phelps
had been engaged in this great reform. In 1833 or 1834 he
published a volume on the subject, which showed how thoroughly
he understood the principles, how deeply he was imbued with the
spirit, of the undertaking. He gave years of undivided attention to
the cause, and by the labors of his pen and his voice rendered
essential services. His reply to the Appeal was complete, exhaustive,
unanswerable. And thus what was intended to do us harm was
overruled for our good. It gave a fair and proper occasion for the
fullest exposition to the public of our doctrines, our measures, and of
the spirit in which we intended to prosecute them.
I am most happy to conclude this narrative by stating, because it
is so highly honorable to Rev. Charles Fitch, the author of the
Appeal, that some time afterwards he saw and frankly confessed his
fault. On the 9th of January, 1840, in a letter addressed to Mr.
Garrison, after a very proper introduction to such a confession, Mr.
Fitch said:—
“I feel bound in duty to say to you, sir, that to gain the good will
of man was the only object I had in view in everything which I did
relative to the ‘Clerical Appeal.’ As I now look back upon it, in the
light in which it has of late been spread before my own mind (as I
doubt not by the Spirit of God), I can clearly see that in all that
matter I had no regard for the glory of God or the good of man. If
you can make any use of this communication that you think will be
an honor to Him, or a service to the cause of truth, dispose of it at
your pleasure.”
It surely will do good to republish this magnanimous, noble,
Christian confession of the wrong that was attempted to be done by
that “Clerical Appeal.”

DR. CHARLES FOLLEN.

The name of Dr. Follen will send a grateful thrill through the
memory of every one who really knew him. He was a dear son of
God, and attracted all but such as were repulsed by the spirit of
righteousness and freedom. He was a native of that country which
gave birth to Luther. The light of civil and religious liberty kindled in
Wittenberg shone upon his cradle. He was the son of Protestant
parents, and received a religious education with little reference to
the dogmas of any sect. He was born in the early years of the
French Revolution,—that event which at first revived the hopes of
the oppressed subjects of European despots. The Germans,
especially those of the smaller members of the Confederacy, hailed
the prospect of more liberal institutions in France as the harbinger of
a better day for themselves. Charles Follen was just then at the age
to receive into the depths of his soul the generous sentiments that
were uttered by the purest, best men of Germany. His father, an
enlightened civilian and liberal Christian, encouraged the growing
ardor of his son in the cause of freedom and humanity.
When, therefore, the German States, finding themselves
deceived by Bonaparte, united with one accord to oppose him,
Charles Follen, then a student at the University of Giesen, and only
nineteen years of age, came forward to act his first public part in the
great struggle for civil liberty. He entered the allied army in a
volunteer corps of young men, and endured the fatigues and
incurred the dangers of those battle-fields, on which were witnessed
the death-throes of the first Napoleon’s ambition. I have heard him
describe his feelings, and what he believed to be the feelings of his
youthful comrades, in that so-called “holy war of the people.” They
refused to wear the trappings of soldiers. They needed not “the
pomp and circumstance of war” to rouse or sustain the purpose of
their souls. They came into the field of mortal strife as men, not
soldiers, to contend for liberty, not laurels. Whenever he spoke of
that momentous period of his life, a solemnity came over the calm,
sweet face of Dr. Follen, his utterance was subdued, his whole frame
pervaded by a deep emotion, so that, much as I differed from him in
my opinion of that resort to carnal weapons, I could not doubt that
he had thrown himself into the dread conflict with a self-sacrificing, I
had almost said, a holy spirit. Körner, “the patriot poet of Germany,”
was his personal friend, and it is a touching incident that some of his
last mental efforts were most successful translations into our
language of the breathing thoughts and burning words of that
enthusiast of liberty.
Although the issue of the French Revolution cast down the hope
of the friends of freedom, that hope was not destroyed. True they
had been deceived. But they could not doubt that freedom was a
reality, the birthright of man. When, therefore, the real design of the
self-styled “Holy Alliance” between Russia, Austria, and Prussia
became manifest, many of the choicest spirits who had united under
their banner to overthrow the tyrant of France uprose to withstand
them. None were more resolute, few became more conspicuous,
than the still youthful Follen, who had scarcely entered upon his
professional career. He boldly claimed for his fellow-subjects of
Hesse Darmstadt a mitigation of the feudal tenures under which
they were oppressed. Thus he incurred the displeasure of the Grand
Duke. But the farmers of that country gratefully acknowledged the
importance of his service in letters that are still extant.
In 1817, when twenty-two years of age, he took his degree of
Doctor of Laws, and became a teacher in the University of Jena.
Here he found an atmosphere congenial to his free spirit. The most
distinguished professors there were friends of liberal institutions.
And the Duke of Saxe-Weimar was for a while indulgent towards
them. At Jena appeared the first periodical publications that
disturbed the diplomatists of Frankfort and Vienna. To these
publications Dr. Follen contributed, and, even among such men as
Dr. Oken and Professors Fries and Luden, he distinguished himself as
an advocate of the rights of man.
The sovereigns of Austria and Prussia were alarmed. The
professors of the University at Jena were proscribed, and the young
men of Austria and Prussia who were students there were required
to leave the infected spot. The persecution of Dr. Follen was carried
further. An attempt was made to involve him in the guilt of the
deluded murderer of Kotzebue, “that unblushing hireling of the
Russian Autocrat,” and he was arrested on the charge. He was fully
exonerated, but the spirit which dictated his arrest made it
uncomfortable for him to remain in Germany.
He went to Switzerland, the resort of the free spirits of that day,
and was appointed Professor of Civil Law at the University of Basle.
Here he continued, both in his lectures and through the press, to
give utterance to his liberal opinions. Consequently, in August, 1824,
the governments of Prussia, Austria, and Russia demanded of the
government of Basle to deliver him up, with the other Professors of
Law in their university. At first this demand was refused. But, being
afterwards enforced by a threat of the serious displeasure of the
allied powers, it was yielded to, and Dr. Follen was compelled to
depart, with no reproach upon his character but that which was cast
upon it by the enemies of freedom. Exiled from Germany as the
dreaded foe of the oppressors of his country, hunted by the allied
sovereigns out of Europe, as if their thrones were insecure while he
dwelt on the same continent with themselves—surely the man who
made himself such a terror to despots was entitled to a carte-
blanche on the confidence of freemen!
Thus recommended, he came to our country in December, 1824,
a few months after the arrival of Lafayette. The illustrious
Frenchman came to feast his eyes and rejoice his heart with the
sight of the astonishing growth and unexampled prosperity of the
nation for whose deliverance from a foreign yoke he had in his early
manhood lavished his fortune and exposed his life. The illustrious
German came, as it proved, to assist in a great moral enterprise, the
success of which was indispensably necessary to complete the
American Revolution, and verify the truths which it declared to the
world.
Nearly a year after his arrival he spent in Philadelphia perfecting
himself in the language of our country. But by the advice of
Lafayette, who highly esteemed him, he came to Boston, and in
December, 1825, was appointed teacher of the German language in
Harvard College, where, in 1830, he was raised to a professorship of
German literature.
He had not been long in the United States before he was struck
by the contrast between our institutions and our habits of thought
and conversation. He was surprised that he so seldom met with a
free mind, or saw an individual who acted independently. Most
persons seemed to be in bonds to a political party or a religious sect,
or both. “I perceive,” said he to an intimate friend, “that liberty in
this country is a fact rather than a principle.”
Such a soul as Dr. Follen could not be indifferent to any
movement tending to liberate more than three millions of people in
the country, of which he had become a citizen, from the most abject
cruel slavery, and his fellow-citizens from the awful iniquity of
keeping them in such bondage. The bugle-blast of The Liberator in
1831 summoned him to the conflict. Worldly wisdom, prudential
considerations, would have withheld him if he had been like too
many other men. He had then been in a professor’s chair at
Cambridge about a year. He had married a lady worthy of his love.
He had become a father. He had made many friends. He was
admired for his rich and varied endowments, his extensive and
accurate knowledge, and sound understanding. He was honored for
his exertions and sacrifices in the cause of liberty in Europe. He was
cherished as an invaluable acquisition to the literature of our
country, and as a most successful teacher of youth. How obvious,
then, that he had as many reasons as any, and more reasons than
most, for remaining quiet, contenting himself with an occasional sigh
over the wrongs of the slaves, or an eloquent condemnation of
slavery in the abstract, or the utterance of the form of prayer,—that
the Sovereign Disposer of all events would, in his own good time,
cause every yoke to be broken and oppression to cease. He was
occupying a sphere of great responsibility, where, as was intimated
to him, he might find enough to fill even the large measure of his
ability for labor. Then he was wholly dependent upon his own
exertions for the support of his family. Moreover, being a foreigner
by birth, he was reminded that it was less decorous in him, than it
might be in others, to meddle with the “delicate question” which
touched so vitally the institutions of a very sensitive portion of the
country.
But Charles Follen was a genuine man. In godly sincerity he felt
as well as said, “that whatever affected the welfare of mankind was
a matter of concern to himself.” He was astonished at the apathy of
so large a portion of the respectable and professedly religious of our
country to the wretched condition of more than a sixth part of the
population, to the disastrous influence of their enslavement upon the
characters of their immediate oppressors, upon the well-being of the
whole Republic, and the cause of liberty throughout the world.
When, therefore, the words of Garrison came to his ears, “he
rejoiced in spirit and said, I thank thee, O Father, that thou hast hid
these things from the wise and prudent, and hast revealed them
unto the babes; even so, Father, for so it seemed good in thy sight.”
He sought out the editor of The Liberator. He clambered up into his
little chamber in Merchants’ Hall, where were his writing-desk, his
types, his printing-press; and where, with the faithful partner of his
early toils, Isaac Knapp, he was living like the four children of Israel
in the midst of the corruptions of Babylon, living on pulse and water.
This was a sight to fill with hope Follen’s sagacious soul. While,
therefore, many who counted themselves servants of God and
friends of humanity thought, or affected to think, that no good could
come out of such a Nazareth, he often went to The Liberator office
to converse with and encourage the young man who had dared to
brave the contumely and detestation of the world in “preaching
deliverance to the captives and liberty to them that are bruised.”
He stopped not to inquire how it might affect his temporal
interests, or even his good name, to espouse so unpopular a cause.
“Some men,” said he, “are so afraid of doing wrong that they never
do right.” The shameful fact, that the cause of millions of enslaved
human beings in a country that made such high pretensions to
liberty as ours was unpopular, so astonished and alarmed him that
he felt all the more called to rise above personal considerations.
Therefore, soon after the New England Antislavery Society was
instituted, he made known his intention to join it. Some friends
remonstrated. They admonished him that so doing would be very
detrimental to his professional success. He hesitated a little while on
account of his wife. But that gifted, high-minded, whole-hearted lady
reproved the hesitation, and bade him act in accordance with his
sense of duty, and in keeping with his long devotion to the cause of
liberty and humanity. He joined the society, became one of its vice-
presidents, was an efficient officer, and rendered us invaluable
services. At that time I became intimately acquainted with him, and
soon learned to love him tenderly and respect him profoundly.
The apprehensions of his friends proved to be too well founded.
The funds for the support of his professorship at Cambridge were
withheld; and he was obliged to retire from a position which had
been most agreeable to himself, for which he was admirably
qualified, and in which he had been exceedingly useful. It was a
severe trial to his feelings, and the loss of his salary subjected him
to no little inconvenience. But liberty, the rights of man, and his
sense of duty were more precious to him than physical comforts or
even life.
In May, 1834, was held in Boston the first New England
Antislavery Convention. It was a large gathering. Dr. Follen was one
of the committee of arrangements, and evinced great interest in
L
making the meeting effective. He was also appointed Chairman of
the “address” that was ordered “to the people of the United States,”
and was the writer of it. His spirit breathes throughout it. It showed
how wholly committed he was to the enterprise of the Abolitionists,
how thoroughly he understood the principles on which we had from
the first relied, and how unfeignedly he desired to make them
acceptable to his fellow-citizens by the most lucid exposition of
them, and the most earnest presentation of their importance.
In 1835 and 1836 I was the General Agent of the Society. This
brought me into a much closer connection with him. It was during
the most stormy period,—the time that tried men’s souls. I have
given some account of it in previous articles, and have made some
allusions to Dr. Follen’s fidelity and fearlessness. He never quailed.
His countenance always wore its accustomed expression of calm
determination. He aided us by his counsels, animated us by his
resolute spirit, and strengthened us by the heart-refreshing tones of
his voice. In this crisis it was, at our annual meeting in January,
1836, that he made his bravest speech. There was not a word, not a
tone, not a look of compromise in it. He met our opponents at the
very points where some of our friends thought us deserving of
blame, and he manfully maintained every inch of our ground. That
speech may be found in the Appendix to the Memoir of his life. It is
not easy even for us to recall, and it is impossible to give to those
who were not Abolitionists then, a clear idea of the state of the
community at the time the above-named speech was made. The
culmination of our trials was the sanction which the Governor of
Massachusetts gave to the opinion of one of the judges, that we had
committed acts that were punishable at common law. I have given
some description of the scenes that were witnessed in the Hall of
Representatives. Dr. Follen distinguished himself there. We can never
cease to be grateful to him for his pertinacity in withstanding the
aggressive overbearance of the Chairman of the joint-committee of
the Senate and House appointed to consider our remonstrance
against Governor Everett’s condemnation of us. I have sometimes
thought it was the turning-point of our affairs in the old
Commonwealth.
Soon afterwards Dr. Follen removed to New York and became
pastor of the first Unitarian church. It was a situation so eligible, and
in every respect so desirable to him, that many supposed he would
suffer his Abolitionism to become latent, or at least would refrain
from giving full and free expression to it in the pulpit. They knew not
the man. He did there as he had done elsewhere. Modestly, mildly,
yet distinctly, he avowed his antislavery sentiments, and endeavored
to make his hearers perceive how imperative was the obligation
pressing upon them as patriots, scarcely less than as Christians, to
do all in their power to exterminate slavery from our country. He was
chosen a member of the Executive Committee of the American
Antislavery Society, and promptly accepted the appointment. The
members of that Board testified that “his sound judgment, his
discriminating intellect, his amenity of manners, and his
uncommonly single-hearted integrity greatly endeared him to his
associates.” Yet was the offence he gave by his antislavery preaching
such that, after about two years, his services were dispensed with
by the Unitarian church.
He returned to Massachusetts, and soon interested so highly the
liberal Christians at East Lexington that he was invited to become
their pastor. They set about in 1839 the building of a meeting-house,
in accordance with his taste, and after a plan which I believe he
furnished. The 15th day of January, 1840, was fixed upon as the day
for the dedication, and Dr. Channing was engaged to preach on the
occasion.
 In December Dr. Follen went to New York and delivered a course
of lectures. On the evening of the 13th of January he embarked on
board the ill-omened steamer Lexington to return. She took fire in
the night, and all the passengers and crew excepting three perished
in the flames, or in their attempts to escape from them. Dr. Follen,
alas! was not one of the three.
The grief and consternation caused by that awful catastrophe
need not be described. Few if any persons in the community had so
great cause for sorrow as the Abolitionists. One of the towers of our
strength had fallen. The greatness of our loss was dwelt upon at the
annual meeting of the Massachusetts Society a few days afterward,
and it was unanimously voted: “That an address on the life and
character of Charles Follen, and in particular upon his early and
eminent services to the cause of abolition, be delivered by such
person and at such time and place as the Board of Managers shall
appoint.” Their appointment fell upon me, and I was requested to
give notice so soon as my eulogy should be written. I gave such a
notice early in February, when I was informed by the managers that
they had not yet been able to procure a suitable place, for such a
service as they wished to have in connection with my discourse.
They had applied for the use of every one of the Unitarian and for
several of the Orthodox churches in Boston, and all had been
refused them. It was said that Dr. Channing did obtain from the
trustees of Federal Street Church consent that the eulogy on Dr.
Follen, whom he esteemed so highly, might be pronounced from his
pulpit. But another meeting of the trustees, or of the proprietors,
was called, and that permission was revoked. More sad still the
meeting-house at East Lexington, which had been built under his
direction, which he was coming from New York to dedicate, and in
which he was to have preached as the pastor of the church if his life
had been spared,—even that meeting-house was refused for a
eulogy and other appropriate exercises in commemoration of the
early and eminent services of Dr. Follen to the cause of freedom and
humanity in Europe, and more especially in our country. Such was
the temper of that time, such the opposition of the people in and
about the metropolis of New England to Mr. Garrison and his
associates.
In consequence of this treatment by the churches, and as a
protest against it, the Board of Managers determined to defer the
delivery of the eulogy, until the meeting-house of some religious
body in Boston should be granted for that purpose. No door was
unbarred to us for more than two months. In April one of our fellow-
laborers, Hon. Amasa Walker, having become one of the proprietors
of Marlborough Chapel, succeeded in getting permission for the
Massachusetts Antislavery Society, and other friends of Dr. Follen, to
meet in that central and very ample room on the evening of the 17th
of April, there to express in prayer, in eulogy, and hymns our
gratitude to the Father of spirits for the gift of such a brother, so
able, so devoted, so self-sacrificing; to attempt some delineation of
his admirable character, some acknowledgment of his inestimable
services, and thus make manifest our deep sense of bereavement
and loss occasioned by his sudden and as we supposed dreadful
death.
It so happened that the 17th of April, 1840, was Good Friday,—a
most appropriate day on which to mourn the death and
commemorate the glorious life of one who had been so true a
disciple of Him, who was crucified on Calvary for his fidelity to God
and to the redemption of man.
The assemblage was large, estimated by some at two thousand.
A prayer was offered by Rev. Henry Ware, Jr.,—such a prayer as we
expected would rise from the large, liberal, loving, devout heart of
that excellent man. A most appropriate hymn, written by himself,
was then read by Rev. John Pierpont. After my discourse was
delivered another touching hymn from the pen, or rather the heart,
of Mrs. Maria W. Chapman was read by Rev. Dr. Channing, and sung
very impressively by the congregation, after which the services were
closed by a benediction from Rev. J. V. Himes, a zealous antislavery
brother of the Christian denomination.
 JOHN G. WHITTIER AND THE ANTISLAVERY
POETS.
All great reformations have had their bards. The Hebrew
prophets were poets. They clothed their terrible denunciations of
national iniquities and their confident predictions of the ultimate
triumph of truth and righteousness in imagery so vivid that it will
never fade. Mr. Garrison was bathed in their spirit when a child by
his pious mother. He is a poet and an ardent lover of poetry. The
columns of The Liberator, from the beginning, were every week
enriched by gems in verse, not unfrequently the product of his own
rapt soul. No sentiment inspires men to such exalted strains as the
love of liberty. Many of the early Abolitionists uttered themselves in
fervid lines of poetry,—Mrs. M. W. Chapman, Mrs. E. L. Follen, Miss
E. M. Chandler, Miss A. G. Chapman, Misses C. and A. E. Weston,
Mrs. L. M. Child, Mrs. Maria Lowell, Miss Mary Ann Collier, and
others, male and female. In 1836—the time that tried men’s souls—
Mrs. Chapman gathered into a volume the effusions of the above-
named, together with those of kindred spirits in other lands and
other times. The volume was entitled, “Songs of the Free and Hymns
of Christian Freedom.” Many of these songs and hymns will live so
long as oppression of every kind is abhorred, and men aspire after
true liberty. This book was a powerful weapon in our moral welfare.
My memory glows with the recollections of the fervor, and often
obvious effect, with which we used to sing in true accord the 13th
hymn, by Miss E. M. Chandler:—
“Think of our country’s glory
All dimmed with Afric’s tears!
Her broad flag stained and gory
With the hoarded guilt of years!”

Or the 15th, by Mr. Garrison:—

 “The hour of freedom! come it must.
O, hasten it in mercy, Heaven!
When all who grovel in the dust
Shall stand erect, their fetters riven.”

Or the 7th, by Mrs. Follen:—

“‘What mean ye, that ye bruise and bind
My people,’ saith the Lord;
‘And starve your craving brother’s mind,
That asks to hear my word?’”

Or the 102d, by Mrs. Chapman:—

“Hark! hark! to the trumpet call,—
‘Arise in the name of God most high!’
On ready hearts the deep notes fall,
And firm and full is the strong reply:
‘The hour is at hand to do and dare!
Bound with the bondmen now are we!
We may not utter the patriot’s prayer,
Or bend in the house of God the knee!’”

Or that stirring song, by Mr. Garrison:—

“I am an Abolitionist;
I glory in the name.”

The singing of such hymns and songs as these was like the
bugle’s blast to an army ready for battle. No one seemed unmoved.
If there were any faint hearts amongst us, they were hidden by the
flush of excitement and sympathy.
In 1838 or 1839 Mrs. Chapman, assisted by her sisters, the
Misses Weston, and Mrs. Child, commenced the publication of The
Liberty Bell. A volume with this title was issued annually by them for
ten or twelve years, especially for sale at the yearly antislavery fair.
These volumes were full of poetry in prose and verse. The editors
levied contributions upon the true-hearted of other countries besides
our own, and enriched their pages with articles from the pens of all
the above-named, and from Whittier, Pierpont, Lowell, Longfellow,
Phillips, Quincy, Clarke, Sewall, Adams, Channing, Bradburn,
Pillsbury, Rogers, Wright, Parker, Stowe, Emerson, Furness,
Higginson, Sargent, Jackson, Stone, Whipple, our own countrymen
and women; and Bowring, Martineau, Thompson, Browning, Combe,
Sturge, Webb, Lady Byron, and others, of England; and Arago,
Michelet, Monod, Beaumont, Souvestre, Paschoud, and others, of
France. It would not be easy to find elsewhere so full a treasury of
mental and moral jewels.
The names of most of our illustrious American poets appear in
The Liberty Bell more or less frequently. To all of them we were and
are much indebted. James Russell Lowell was never, I believe, a
member of the Antislavery Society. He was seldom seen at our
meetings. But his muse rendered us essential services. His poems
—“The Present Crisis,” “On the Capture of Fugitive Slaves near
Washington,” “On the Death of Charles T. Torrey,” “To John G.
Palfrey,” and especially his “Lines to William L. Garrison,” and his
“Stanzas sung at the Antislavery Picnic in Dedham, August 1,
1843”—committed him fully to the cause of freedom,—the cause of
our enslaved countrymen.
Rev. John Pierpont gave us his hand at an earlier day. He took
upon himself “our reproach” in 1836, when we most needed help. I
have already made grateful mention of his “Word from a Petitioner,”
sent to me by the hand of the heroic Francis Jackson in the midst of
the convention of the constituents of Hon. J. Q. Adams, called at
Quincy to assure their brave, invincible representative of their deep,
admiring sense of obligation to him for his persistent and almost
single-handed defence of the sacred right of petition on the floor of
Congress.
Mr. Pierpont’s next was a tocsin in deed as well as in name. He
was impelled to strike his lyre by the alarm he justly felt at the
tidings from Alton of the destruction of Mr. Lovejoy’s antislavery
printing-office, and the murder of the devoted proprietor. His
indignation was roused yet more by the burning of “Pennsylvania
Hall” in Philadelphia, and the shameful fact that at the same time,
1838, no church or decent hall could be obtained in Boston for “love
or money,” in which to hold an antislavery meeting; but we were
compelled to resort to an inconvenient and insufficient room over
the stable of Marlborough Hotel.
His next powerful effusion was The Gag, a caustic and scathing
satire upon the Hon. C. G. Atherton, of New Hampshire, for his base
attempt in the House of Representatives at Washington to put an
entire stop to any discussion of the subject of slavery.
His next piece was The Chain, a most touching comparison of
the wrongs and sufferings of the slaves with other evils that injured
men have been made to endure.
Then followed The Fugitive Slave’s Apostrophe to the North Star,
which showed how deeply he sympathized with the many hundreds
of our countrymen who, to escape from slavery, had toiled through
dismal swamps, thick-set canebrakes, deep rivers, tangled forests,
alone, by night, hungry, almost naked and penniless, guided only by
the steady light of the polar star, which some kind friend had taught
them to distinguish, and had assured them would be an unerring
leader to a land of liberty. They who have heard the narratives of
such as have so escaped need not be told that Mr. Pierpont must
M
have had the tale poured through his ear into his generous heart.
But of all our American poets, John G. Whittier has from first to
last done most for the abolition of slavery. All my antislavery
brethren, I doubt not, will unite with me to crown him our laureate.
From 1832 to the close of our dreadful war in 1865 his harp of
liberty was never hung up. Not an important occasion escaped him.
Every significant incident drew from his heart some pertinent and
often very impressive or rousing verses. His name appears in the
first volume of The Liberator, with high commendations of his poetry
and his character. As early as 1831 he was attracted to Mr. Garrison
by sympathy with his avowed purpose to abolish slavery. Their
acquaintance soon ripened into a heartfelt friendship, as he declared
in the following lines, written in 1833:—
“Champion of those who groan beneath
Oppression’s iron hand:
In view of penury, hate, and death,
I see thee fearless stand.
Still bearing up thy lofty brow,
In the steadfast strength of truth,
In manhood sealing well the vow
And promise of thy youth.
* * * * *
“I love thee with a brother’s love;
I feel my pulses thrill,
To mark thy spirit soar above
The cloud of human ill.
My heart hath leaped to answer thine,
And echo back thy words,
As leaps the warrior’s at the shine
And flash of kindred swords!
* * * * *
“Go on—the dagger’s point may glare
Amid thy pathway’s gloom,—
The fate which sternly threatens there
Is glorious martyrdom!
Then onward with a martyr’s zeal;
And wait thy sure reward,
When man to man no more shall kneel,
And God alone be Lord!”

Mr. Whittier proved the sincerity of these professions. He joined

the first antislavery society and became an active official.
Notwithstanding his dislike of public speaking, he sometimes
lectured at that early day, when so few were found willing to avow
and advocate the right of the enslaved to immediate liberation from
bondage without the condition of removal to Liberia. Mr. Whittier
attended the convention at Philadelphia in December, 1833, that
formed the American Antislavery Society. He was one of the
secretaries of that body, and a member, with Mr. Garrison, of the
committee appointed to prepare the “Declaration of our Sentiments
and Purposes.” Although, as I have elsewhere stated, Mr. Garrison
wrote almost every sentence of that admirable document just as it
now stands, yet I well remember the intense interest with which Mr.
Whittier scrutinized it, and how heartily he indorsed it.
In 1834, by his invitation I visited Haverhill, where he then
resided. I was his guest, and lectured under his auspices in
explanation and defence of our abolition doctrines and plans. Again
the next year, after the mob spirit had broken out, I went to
Haverhill by his invitation, and he shared with me in the perils which
I have described on a former page.
In January, 1836, Mr. Whittier attended the annual meeting of
the Massachusetts Antislavery Society, and boarded the while in the
house where I was living. He heard Dr. Follen’s great speech on that
occasion, and came home so much affected by it that, either that
night or the next morning, he wrote those “Stanzas for the Times,”
which are among the best of his productions:—
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

ebookbell.com