nature methods

Article https://doi.org/10.1038/s41592-024-02574-2

Resolving tissue complexity by multimodal

spatial omics modeling with MISO

Received: 31 January 2024 Kyle Coleman 1,2 , Amelia Schroeder2, Melanie Loth2, Daiwei Zhang2,
Jeong Hwan Park 3, Ji-Youn Sung4, Niklas Blank5,6, Alexis J. Cowan5,
Accepted: 21 November 2024
Xuyu Qian7, Jianfeng Chen8, Jiahui Jiang9,10, Hanying Yan 2, Laith Z. Samarah11,
Published online: xx xx xxxx Jean R. Clemenceau12, Inyeop Jang12, Minji Kim12, Isabel Barnfather 12,
Joshua D. Rabinowitz 11, Yanxiang Deng 13,14, Edward B. Lee 13,
Check for updates
Alexander Lazar 15, Jianjun Gao8, Emma E. Furth13, Tae Hyun Hwang12,
Linghua Wang 9,16, Christoph A. Thaiss 5, Jian Hu 17 & Mingyao Li 2,13

Spatial molecular profiling has provided biomedical researchers valuable

opportunities to better understand the relationship between cellular
localization and tissue function. Effectively modeling multimodal spatial
omics data is crucial for understanding tissue complexity and underlying
biology. Furthermore, improvements in spatial resolution have led to
the advent of technologies that can generate spatial molecular data with
subcellular resolution, requiring the development of computationally
efficient methods that can handle the resulting large-scale datasets. MISO
(MultI-modal Spatial Omics) is a versatile algorithm for feature extraction
and clustering, capable of integrating multiple modalities from diverse
spatial omics experiments with high spatial resolution. Its effectiveness
is demonstrated across various datasets, encompassing gene expression,
protein expression, epigenetics, metabolomics and tissue histology
modalities. MISO outperforms existing methods in identifying biologically
relevant spatial domains, representing a substantial advancement in
multimodal spatial omics analysis. Moreover, MISO’s computational
efficiency ensures its scalability to handle large-scale datasets generated by
subcellular resolution spatial omics technologies.

Multimodal spatial omics represents a paradigm shift in understand- These technologies provide at least one spatially resolved omics modal-
ing complex biological systems by integrating diverse omics modali- ity and are often accompanied by a high-resolution hematoxylin and
ties within their original tissue contexts1. This approach provides a eosin (H&E)-stained histology image, yielding two or more modalities
comprehensive view of cellular and tissue functions, which is cru- from the same tissue slice. The integration of multiple modalities is
cial for unraveling the molecular mechanisms of diseases. Recently pivotal in obtaining a more holistic and accurate understanding of
developed multimodal spatial omics technologies include spatial tissue architecture and function at cellular and molecular levels.
transcriptomics2–4 (transcriptomics and histology), spatial epigenome– A critical step in multimodal spatial omics data analysis is to
transcriptome co-profiling5 (chromatin accessibility, histone modifi- amalgamate features from different modalities to create cohesive
cation and transcriptomics), spatial gene and protein expression6–8 low-dimensional data representations that can be harnessed for down-
(transcriptomics, proteomics and histology) and spatial transcriptom- stream analyses such as spatial domain detection by clustering. Current
ics and metabolomics9 (transcriptomics, metabolomics and histology). methods, however, either ignore spatial information10,11 or are limited

A full list of affiliations appears at the end of the paper. e-mail: [email protected]; [email protected]; [email protected]

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

Input
Adjacency matrix Encoder Decoder Construction of input features K-means clustering
modalities

Omic modality 2 Omic modality 1 T T 2

||x − xˆ ||2
Overview of MISO workflow

C C 2
||x − xˆ ||2
×
⊗ ⊗

⊗ ⊗
H&E image

H H 2
||x − xˆ ||2

⊗ ⊗

Fig. 1 | MISO workflow for analysis of spatial multi-omics dataset with paired are extracted from the multilayer perceptrons, and interactions between each
histology image. MISO starts by constructing an adjacency matrix for each pair of modalities are computed. The features of low-quality modalities are
modality. This adjacency matrix and the spot-level features for that modality removed, and the final group of embeddings is used as input for the k-means
are used as input for a multilayer perceptron that is trained to minimize spectral clustering algorithm.
clustering and reconstruction loss functions. The modality-specific embeddings

to integrating a subset of modalities in spatial omics experiments, A crucial step in MISO is the filtration of embeddings from low-quality
typically handling just two modalities, requiring extensive hyperpa- modalities, which are essential in enhancing the robustness of the analy-
rameter tuning, and being sensitive to modalities of lower quality. sis. The refined subset of embeddings is then used as input for a k-means
For instance, MUSE12, a spatial clustering algorithm, is designed for clustering algorithm to separate the spots into distinct clusters, each
combining transcriptomics and imaging data in spatial transcriptom- reflective of the unique characteristics defined by the input modali-
ics, while SpatialGlue13, a spatial multi-omics clustering algorithm, ties. MISO is a computationally fast algorithm, processing medium-
caters to three distinct experiment types: SPOTS8, Stereo-CITE-seq7 scale (1,000–10,000 spots/cells) datasets in under 1 min on a single
and spatial assay for transposase-accessible chromatin and RNA using graphics processing unit. MISO also scales effectively to large datasets
sequencing (spatial ATAC–RNA-seq)5. SpatialGlue, although adept at (>10,000 spots/cells) and allows for evaluation of datasets containing
integrating two omics modalities, does not support tissue histology hundreds of thousands of capture locations (Supplementary Fig. 1).
imaging data as an input. As multimodal spatial omics technologies
advance and continue to produce data with more than two omics Application to bladder cancer 10x Visium V2 data
modalities along with tissue histology imaging data, there is a grow- To demonstrate MISO’s robust integration capabilities, we applied it to
ing need for methods that can seamlessly integrate information from a wide range of multimodal spatial omics datasets. Our initial evaluation
diverse spatial omics experiments. focused on a bladder cancer sample generated using 10x Visium (V2)14,
Here we present MISO (MultI-modal Spatial Omics), a feature which includes both spatial transcriptomics and tissue histology image
extraction and spatial clustering algorithm designed for comprehen- data (Fig. 2a). Our primary objective in analyzing this dataset was to
sive modality integration, including both omics and tissue histology assess MISO’s ability to identify fine-grained, disease-relevant tissue
imaging data, in any multimodal spatial omics experiment. MISO’s structures. Specifically, we focused on the segmentation of two tertiary
efficacy is demonstrated across various datasets, such as spot-level lymphoid structures (TLSs), denoted as TLS1 and TLS2 (Fig. 2a). TLSs
and single-cell-level spatial transcriptomics (10x Visium, Xenium and are aggregates of immune cells that develop in nonlymphoid tissues
Visium HD), spatial ATAC–RNA-seq, spatial CUT&Tag–RNA-seq, spatial in response to chronic inflammation15. The presence of TLSs in tumors
gene and protein expression, and spatial transcriptomics and metabo- has been associated with improved prognosis for multiple types of
lomics. MISO excels in accurately integrating different modalities and cancer, underscoring their potential importance in developing novel
segregating spots into distinct, biologically relevant spatial domains. therapeutic interventions16. TLSs contain high endothelial venules
A notable advantage of MISO is its minimal requirement for hyperpa- (HEVs), specialized blood vessels that recruit lymphocytes from the
rameter tuning, coupled with its ability to mitigate artifacts associated bloodstream to form the TLS17. An annotation of the HEVs within TLS1
with low-quality modalities, a common challenge in real-world studies. and TLS2, determined using nuclei segmentation, is shown in Fig. 2a.
This makes MISO not only a powerful tool for multimodal data analysis Although HEVs are crucial in the immune response to tumor, they
but also a robust solution for researchers exploring the complexities are also very small, typically measuring only 40 µm in diameter18. Spots
of spatial omics. sequenced using 10x Visium have a diameter of 55 µm, with 100 µm
between the centers of adjacent spots, resulting in gaps between spots
Results (Fig. 2b). Consequently, even if a Visium spot encompasses an HEV,
Overview of MISO it likely contains a mixture of transcripts from immune cells within
The MISO workflow (Fig. 1) begins with the extraction of low-dimensional the TLS, potentially obscuring the gene expression signatures of the
embeddings for each modality. This is achieved using modality-specific endothelial cells lining the HEV boundary. To enhance the detection
multilayer perceptrons, which are trained to minimize spectral clus- of HEVs, we increased the spatial resolution of the Visium gene expres-
tering and reconstruction loss functions. MISO then calculates the sion data using iStar19, producing gene expression data at 4 × 4 µm2
outer product between pairs of modality-specific embeddings to con- super-pixels with no gaps between super-pixels (Fig. 2c). Within the
struct features that capture the interactions between modalities. These two TLS regions of interest, resolution enhancement increased the
modality-specific and interaction feature vectors are concatenated to number of gene capture units from 17 spots to 8,613 super-pixels for
produce comprehensive embeddings that encapsulate all modalities. TLS1 and from 12 spots and 9,496 super-pixels for TLS2.

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

a Bladder cancer (Visium V2)

HEVs
b Spot-level TLS score c Super-resolution TLS score
TLS1
1 TLS1 TLS1

1.0
1
2
2

3
3

TLS score
TLS2 TLS2 TLS2
1

1
2
2

3
3
0

d MISO SpatialGlue MUSE

Clustering True-positive HEV False-positive HEV Clustering True-positive HEV False-positive HEV Clustering
0
1
2
3
TLS1 4
5
6
7
8
9
10
11
12
13
TLS2 14
15
16
17

Fig. 2 | Analysis of a 10x Visium bladder cancer spatial transcriptomics d, Shown from left to right are the clustering results for TLS1 and TLS2 provided
dataset. a, H&E-stained histology image of the analyzed tissue section with TLS by MISO, MUSE and SpatialGlue, with masks showing the true-positive and false-
and HEV annotation. b, TLS score across all Visium spots. c, TLS score across all positive HEV super-pixels for MISO and SpatialGlue.
super-pixels following gene expression resolution enhancement using iStar.

Subsequently, we evaluated MISO on the super-pixels contained values across clusters and features for each method with respect to
within TLS1 and TLS2 (Fig. 2d). Cluster 3 identified by MISO corre- each modality. MISO demonstrates superior performance in cluster-
sponded to the annotated HEVs with high sensitivity and specificity for ing based on both RNA and tissue histology data, showing its ability to
both TLS regions. MISO grouped only ten non-HEV super-pixels with the extract low-dimensional embeddings that capture the complexities of
HEV cluster for TLS1 and only six for TLS2, demonstrating its precision both input modalities.
and effectiveness in identifying these fine-grained critical structures.
As a comparison, we also applied SpatialGlue and MUSE to this dataset. Application to gastric cancer 10x Xenium data
Since neither SpatialGlue nor MUSE contains a tissue histology image Encouraged by the above results, we next evaluated a molecular
feature extraction component, we used the tissue histology image imaging-based spatial transcriptomics dataset from a gastric cancer
features extracted by MISO as input for all methods. Figure 2d shows sample generated using 10x Xenium (Fig. 3a). Xenium generates spatial
that SpatialGlue identified the HEV regions with high sensitivity but gene expression data with single-cell resolution for an entire tissue
very low specificity, falsely grouping 289 non-HEV super-pixels with section with the number of profiled capture locations far surpassing
the HEV cluster for TLS1 and 381 for TLS2. MUSE failed to produce a that of 10x Visium and most other sequencing-based spatial transcrip-
cluster resembling the HEV localization, preventing the generation of tomics platforms. Unlike sequencing-based technologies, molecular
true-positive and false-positive HEV masks from its results. The supe- imaging-based spatial transcriptomics platforms cannot measure gene
rior performance of MISO over SpatialGlue and MUSE underscores its expression for the full transcriptome and are instead restricted to a
capability to effectively integrate high-resolution spatial gene expres- preselected gene panel typically consisting of a few hundred genes.
sion and histology imaging features to localize fine-grained tissue For this gastric cancer sample, a total of 377 genes were profiled across
structures with high accuracy. 696,314 cells. Because of the low number of genes measured, effectively
Next, we aimed to quantitatively compare the performance of integrating histology image information becomes even more crucial
different methods. Traditionally, pathologists’ manual annotations for accurately segmenting the tissue section.
serve as the ground truth when assessing clustering accuracy in spatial To evaluate clustering performance, we obtained a pathologist’s
omics data. However, in the context of multimodal spatial omics data, annotation of the tissue section (by J.H.P.), distinguishing poorly
histology represents just one modality in the analysis. Annotations cohesive carcinoma, invasive discohesive tumor cells, lymphoid
derived from a single modality may not fully capture the complex aggregates, mucosa, submucosa and muscle (Fig. 3a). As shown in
tissue structures defined by all modalities. To address this limitation Fig. 3c, MISO accurately localized clusters to all major biological and
in quantitative evaluation, we calculated the intraclass correlation disease-relevant structures identified in the pathologist’s annotation.
coefficient (ICC) for each method–modality cluster combination. MISO effectively distinguished regions of poorly cohesive carci-
The ICC measures the consistency of observations within clusters. noma (cluster 14) from those of discohesive tumor cells (cluster 5).
A higher ICC value indicates increased homogeneity within clusters The poorly cohesive carcinoma cluster was largely contiguous, with
for a given modality, suggesting an improvement in clustering quality. little intermixing of other clusters. In the discohesive tumor region,
The box plots in Extended Data Fig. 1 illustrate the distribution of ICC MISO also identified the areas of healthy muscle and lymphoid

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

a Gastric cancer (Xenium) c MISO: RNA + image e MISO histology image feature 23 (mucosa)
0
Carcinoma oma 1
Lymphoid aggregate rcin cell
s
e ca mor
2
c o hesiv iv e tu 3
a orly he s 4
o s P o c o
Muc sive
dis 5
Inva 6
7
8
osa 9
muc
Sub cle
10
11
Mus 12
Submucosa 13
14

b Mucosa ROI Carcinoma ROI d ERBB2 MISO histology image feature 16 (carcinoma)

Normalized 0 1.0 Intensity 0 1.0

expression

f Colon cancer (Visium HD) g MISO: RNA + image h CEACAM6 i SPP1

Invasive cancer
Inv
asi
ve
ca

Invasive cancer Normal

nc

colonic
er

mucosa

Muscle

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Normalized 0 1.0
Normalized
Invasive cancer 0 1.0
expression expression

Fig. 3 | Analysis of large-scale spatial transcriptomics datasets. a–e, Analysis clusters. f–i, Analysis of a 10x Visium HD CRC spatial transcriptomics dataset.
of a 10x Xenium gastric cancer spatial transcriptomics dataset. a, Pathologist f, Pathologist annotation of tissue section. g, MISO clustering results. h, Spatial
manual annotation. b, Histology image patches of tissue from mucosa and gene expression plot of CEACAM6, a CRC marker gene that shows similar levels
carcinoma regions. c, MISO clustering results. d, Spatial gene expression plot of expression across all annotated invasive cancer regions. i, Spatial gene
of ERBB2, a gastric cancer marker gene that shows similar levels of expression in expression plot of SPP1, a tumor-specific macrophage marker gene that is
the carcinoma and mucosa regions. e, MISO extracted tissue histology colocalized with cluster 4 identified by MISO.
image features that enabled the identification of mucosal and carcinoma

aggregates. Additionally, MISO effectively localized cluster 7 to the generation of increasingly detailed and comprehensive spatial molecu-
annotated lymphoid aggregate regions with high sensitivity. This lar maps, providing deeper insights into complex biological systems.
capability is meaningful because it indicates that MISO can identify
small lymphoid aggregates that are not easily detectable through Application to CRC 10x Visium HD data
direct visualization. MISO can, therefore, greatly reduce the time Building on MISO’s efficacy in analyzing large-scale spatial transcrip-
required for manual annotation in tissue sections with numerous tomics data with single-cell resolution, we next applied it to a colo-
small and detailed structures, enhancing the efficiency and accuracy rectal cancer (CRC) spatial transcriptomics dataset generated using
of pathological analysis. 10x Visium HD21, which provides gene expression data with subcel-
MISO was also able to separate the poorly cohesive carcinoma lular resolution and full transcriptome measurements. This dataset
region from the mucosa, a task of particular intricacy due to the simi- includes expression levels for 18,085 genes across 545,913 bins with
larity in the expression levels of gastric cancer marker genes in these each bin measuring 8 × 8 µm2. For evaluation, we obtained a patholo-
regions. As an example, we included the spatial gene expression plot gist’s annotation of the tissue section (by J.-Y.S.), denoting regions of
of ERBB2, a gastric cancer marker gene20 that shows a similar level of muscle, normal colonic mucosa and invasive carcinoma (Fig. 3f). When
expression in the annotated carcinoma and mucosa regions (Fig. 3d). integrating gene expression and histology image features, MISO suc-
However, despite the similar expression patterns of key marker genes, cessfully separated bins into clusters representing each of these struc-
the mucosa and carcinoma regions exhibit very different histological tures (Fig. 3g). Notably, MISO was able to identify multiple subclasses of
features (Fig. 3b). MISO’s extracted tissue histology image features invasive carcinoma. These invasive carcinoma clusters showed similar
capture these histological differences (Fig. 3e), enabling MISO to clearly expression of CEACAM6 (Fig. 3h), a CRC marker gene associated with
segregate the mucosa and poorly cohesive carcinoma regions into dis- poor overall survival and disease-free survival22,23. If we were to inves-
tinct clusters. Due the large number of cells profiled, neither MUSE nor tigate only the expression of these CRC marker genes, these different
SpatialGlue had the capacity to analyze this tissue section because of cancerous regions identified by MISO might appear equally severe.
memory constraints. MISO’s ability to handle large-scale spatial omics However, bins for one of the invasive carcinoma subclusters
datasets is crucial as technological improvements are enabling the (cluster 4) were colocalized with SPP1+ macrophages (Fig. 3i), a type of

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

a Mouse embryo b
(spatial ATAC–RNA-seq)
Telencephalon

Ventricular zone
Stratum (subpallium)
Stratum (pallium)

1 mm

c MISO MUSE SpatialGlue

0
1
2
3
4
5
6
7
8
1 mm 1 mm 1 mm
9

d SOX2 GAD2 TBR1 e MISO: RNA

RNA

0
1
2
3
4
5
MISO: ATAC
6
7
8
9
ATAC

Normalized
0 1.0
expression

Fig. 4 | Analysis of a spatial ATAC–RNA-seq dataset from a mouse at E13. spatial gene expression plots of marker genes for the VZ, subpallium stratum and
a, H&E-stained histology image of an adjacent tissue section. b, Magnification pallium stratum of the telencephalon. e, Spots plotted according to their RNA or
and annotation of telencephalon. c, Shown from left to right are the clustering ATAC t-SNE coordinates and colored by the MISO clustering results.
results from MISO, MUSE and SpatialGlue. d, Shown from left to right are the

tumor-specific macrophage reported by Oliveira et al.21. The presence of severity is crucial for gaining a deeper understanding of the tumor
SPP1+ macrophages has been associated with CRC progression and worse microenvironment. In contrast, neither MUSE nor SpatialGlue could
clinical outcomes24,25. Unlike CEACAM6 expression, the cancer regions analyze this tissue section due to memory constraints.
on the periphery of the tissue section assigned to clusters 6 and 12 did
not show the same extent of SPP1 colocalization, indicating potentially Application to mouse embryo (E13) spatial ATAC–RNA-seq data
less aggressive cancer cells. In fact, SPP1 was upregulated in cluster 4 To examine the performance of MISO for other omics types, we next
compared to clusters 6 and 12, with a log2 fold change of 1.06 determined analyzed a spatial ATAC–RNA-seq dataset obtained from a mouse at
by differential expression analysis (P value = 1.22 × 10−56). This means that embryonic day 13 (E13; Fig. 4a)5. Focusing on the telencephalon region
MISO was able to differentiate a cluster of more aggressive tumor cells of the forebrain located at the upper left of the tissue (Fig. 4b), MISO
based on its proximity to SPP1+ macrophages. This was achieved because uniquely distinguished spots in the pallium’s stratum from the ven-
of MISO’s approach in extracting hierarchical tissue histology image tricular zone (VZ), a capability not observed with MUSE and SpatialGlue
features that capture both short-range and long-range dependencies (Fig. 4c). MISO’s segmentation of the telencephalon subregions was
across the tissue section, enabling an understanding of the localization validated by comparing the localization of marker gene expression for
of different cancer subclusters relative to SPP1+ macrophages. The ability the VZ, subpallium stratum and pallium stratum (Fig. 4d). To define the
of MISO to make such distinctions among cancerous regions of varying VZ, we visualized expression of the SOX2 gene, which is highly expressed

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

in neural stem cells and neural progenitor cells contained within the VZ MALDI-MSI measure gene and metabolite expression at different spa-
of the telencephalon, but not in postmitotic neurons that have migrated tial resolutions, we enhanced the resolution of both modalities using
into the stratum26. Consequently, visualizing the expression of SOX2 iStar19, enabling alignment of the different omics modalities to the same
allows differentiation between the VZ and the stratum of the different spatial coordinate space. This resolution enhancement also facilitated
subregions in the telencephalon. Spots with high expression of SOX2 a more detailed segmentation of the hippocampus.
closely corresponded to cluster 3 identified by MISO, indicating that MISO was able to accurately localize clusters to biologically rel-
MISO effectively distinguished regions of neural stem cells and neural evant structural components defined by the Allen Brain Atlas (Fig. 5b).
progenitor cells from those containing postmitotic neurons in the For comparison, we also evaluated this dataset using MUSE and Spatial-
telencephalon. Glue. Since MUSE and SpatialGlue can only integrate two modalities,
To localize the subpallium stratum, we examined the expression of we applied MUSE to the transcriptomics and tissue histology image
GAD2, a marker gene for GABAergic neurons that produces the GAD65 data, as it was not designed to take metabolomics data as input, and
protein, responsible for converting glutamate to GABA27. In the mouse SpatialGlue to the transcriptomics and metabolomics data, as it was
telencephalon, GABAergic neurons originate exclusively in the stratum developed for spatial multi-omics analysis. A main region of interest
of the subpallium before starting migration to the pallium to help in the was the pyramidal layer, which contains different subtypes of pyramidal
formation of the cortical plate28. At E13, a majority of GABAergic neurons neurons classified based on areal localization. The pyramidal layer is
have not yet migrated, making GAD2 a suitable marker for the subpallium composed of three main subfields: CA1 stratum pyramidale (CA1sp),
stratum29. We observed that high expression and chromatin accessibility CA2 stratum pyramidale (CA2sp) and CA3 stratum pyramidale (CA3sp).
of GAD2 correspond to cluster 1 identified by MISO, indicating that this While all methods were able to differentiate CA1sp from CA3sp, MISO
cluster likely encompasses GAD2+ GABAergic neurons in the subpallium. was the only method that accurately localized a cluster to the much
To define the pallium stratum, we visualized the expression of smaller CA2sp subfield.
TBR1, which encodes the T-box brain transcription factor 1, crucial for To validate the precision of the MISO-identified subfields, we used
regulating cerebrocortical development30. In the telencephalon of the the MERFISH whole mouse brain atlas33. MERFISH is well-suited for
developing mouse brain starting at E10, TBR1 is expressed only in post- validating these fine-grained subfields because it provides subcellular
mitotic glutamatergic neurons, meaning that neural progenitor cells in spatially resolved gene expression information. Based on the localiza-
the VZ and GABAergic neurons in the subpallium of the telencephalon tion of CA2 and CA3 glutamatergic neurons and the expression patterns
do not express TBR1 (ref. 31). In addition, in the telencephalon, TBR1 is of CA2sp (Amigo2) and CA3sp (Chrm3) pyramidal neuron marker genes,
generally not expressed in regions containing cells that express DLX2, the CA2sp/CA3sp border inferred by MERFISH corresponds precisely
specifically the subpallium31,32. Therefore, within the telencephalon to the border between MISO clusters 11 and 16 (Fig. 5c)34. Furthermore,
regions shown in Fig. 4b, TBR1 serves as a marker gene for cells within MISO was able to identify all three subareas of CA3 (refs. 35,36), offer-
the stratum of the pallium. In Fig. 4d, we see high RNA expression of ing an even more detailed segmentation than can be inferred through
TBR1 restricted to the upper periphery of the telencephalon, closely analysis of the MERFISH gene panel.
corresponding to cluster 6 identified by MISO. There is noticeably
high chromatin accessibility for TBR1 in spots contained within the Application to human tonsil 10x Visium spatial gene and
VZ of the telencephalon, indicating the potential of cells within the VZ protein expression data
to differentiate into TBR1+ glutamatergic neurons that aid in cortical As a second demonstration for MISO’s ability to integrate three modali-
development. By effectively integrating gene expression and chro- ties, we analyzed a human tonsil spatial gene and protein expression
matin accessibility information, MISO can distinguish the VZ from dataset37, comprising spatial transcriptomics, spatial proteomics
the stratum, despite both regions showing high levels of chromatin (35 proteins) and histology imaging data. Figure 5d shows the germi-
accessibility for the TBR1 gene. nal centers identified using the high-resolution histology image and
To further confirm MISO’s precision, we examined the distribution expression of PCNA (germinal centers confirmed by E.E.F.), a protein
of RNA and ATAC data modalities in these areas using t-SNE plots, with known to have heightened expression in germinal centers38. We evalu-
spots colored based on MISO’s cluster assignments (Fig. 4e). Remark- ated MISO’s performance across all available modalities (RNA, protein
ably, cluster 3 (VZ) and cluster 6 (pallium stratum) showed almost com- and tissue histology; Fig. 5e) and compared it with MUSE (RNA and tis-
plete separation in the t-SNE plots for both modalities, underscoring sue histology; Fig. 5f) and SpatialGlue (RNA and protein; Fig. 5g), each
MISO’s ability to discern complex spatial relationships that MUSE and applied to the modalities that these methods were originally designed
SpatialGlue could not. These observations collectively highlight MISO’s for. To quantitatively assess clustering accuracy, we focused on each
ability to integrate complex multimodal spatial omics data, revealing method’s ability to localize clusters to the germinal centers. We desig-
brain structures that are crucial for studying neurodevelopment but nated a spot as a ground truth germinal center spot if it resided within
otherwise obscured in traditional analysis methods. an annotated germinal center region. For each method, we defined a
cluster as a germinal center cluster if more than half of the spots in that
Application to mouse hippocampus spatial transcriptomics cluster were located within a germinal center. Using this classification
and metabolomics data criterion, we calculated the F1 score to evaluate the accuracy of germinal
The previous analyses have been limited to two modalities. However, center localization for all methods (Fig. 5h). MISO, whether utilizing all
as the field progresses, emerging technologies capable of simultane- three modalities or any two-modality combination, consistently out-
ously profiling three or more modalities from the same tissue slice are performed MUSE and SpatialGlue in terms of F1 score. The maximum
being developed6–9. Anticipating the surge of these complex datasets, F1 score (0.90) was reached by MISO when applied to all three modali-
developing computational tools capable of handling such multifaceted ties. The lowest F1 score for MISO (0.86) was observed when using only
data is imperative. In addition, given MISO’s successful segmentation RNA and protein as input, but this performance still surpassed most
of the mouse embryo telencephalon, we sought to assess its ability other methods and was only marginally lower than the highest F1 score
to incorporate more than two modalities to segment finer-grained achieved by MUSE (0.87) with protein and tissue histology image data
structural components of the brain. To do so, we focused on the hip- as input. SpatialGlue achieved its maximum F1 score (0.83) when RNA
pocampus of the coronal section of a mouse brain analyzed using 10x and protein were utilized. The mean MISO F1 score across all combina-
Visium and MALDI mass spectrometry imaging (MALDI-MSI)9, produc- tions of two modalities was 0.87, compared to 0.78 for MUSE and 0.68
ing spatial transcriptomics, spatial metabolomics and H&E histology for SpatialGlue, underscoring the consistently strong performance
imaging data for the same tissue section (Fig. 5a). As 10x Visium and of MISO across various modality pairs. These observations highlight

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

a Mouse brain b
(spatial transcriptomics and
metabolomics)
MISO: RNA + metabolite + image MUSE: RNA + protein SpatialGlue: RNA + metabolite Cluster
0 10
so-1
so-2 1 11
CA1sp
CA1sr-1 2 12
CA1sr-2
3 13
slm 4 14

alv
CA
5 15

e
2s

us
DGmo-1 p

DGmo-2 6 16
DGsg CA3sr
7 17

p-a
DGpo CA2/3sr

8 18

3s
CA3sp-c

CA
DGmo-2 CA3sp-b

9 19
20

Hippocampus c MERFISH classification Amigo2 (MERFISH) Chrm3 (MERFISH)

CA2 glut CA3 glut Normalized

1.0 0
expression

d Tonsil (spatial gene and protein e

expression) MISO: RNA + image + protein MISO: RNA + image MISO: RNA + protein

Cluster
0
1
2
3
4
5
6
7
8
9

Germinal center

f MUSE: RNA + image g SpatialGlue: RNA + protein h F1 score for germinal center localization

0.90 MISO: RNA + image + protein

0.9 0.88
0.86 0.87 0.87 MISO: RNA + image
0.83 MISO: RNA + protein
MISO: protein + image
0.8 MUSE: RNA + image
0.75 MUSE: RNA + protein
0.73
MUSE: protein + image
0.7 SpatialGlue: RNA + image
0.66
SpatialGlue: RNA + protein
SpatialGlue: protein + image
0.6
0.54

0.5

Fig. 5 | Analysis of spatial multi-omics datasets with three modalities. d–h, Clustering results for a human tonsil spatial gene and protein expression
a–c, Clustering results for a coronal mouse brain spatial transcriptomics and dataset. d, Germinal center annotation of the tissue section. e, MISO clustering
metabolomics dataset. a, H&E-stained histology image of an analyzed tissue results when taking all three modalities (that is, RNA, tissue histology and
section, with magnification of hippocampus. b, Shown from left to right are protein), two modalities (RNA and tissue histology) and two modalities (RNA
the clustering results of the super-pixels in the hippocampus from MISO and protein), as input. f, MUSE clustering results when taking RNA and tissue
(RNA, metabolite and tissue histology), MUSE (RNA and tissue histology) and histology data as input. g, SpatialGlue clustering results when taking RNA and
SpatialGlue (RNA and metabolite). c, Shown from left to right are the MERFISH protein data as input. h, F1 score for germinal center localization for all methods
distribution of CA2 and CA3 glutamatergic neurons and spatial gene expression when taking each possible combination of modalities as input.
plots of Amigo2 and Chrm3 in the pyramidal layer of the hippocampus.

MISO’s superior capability in accurately identifying complex spatial Fig. 2 and Supplementary Fig. 2), human breast cancer40 (Extended Data
structures, regardless of the number and type of modalities involved. Fig. 3), zebrafish melanoma41 (Extended Data Fig. 4), mouse olfactory
Such robustness is desired in real studies when multiple modalities bulb42 (Extended Data Fig. 5), mouse coronal brain43 (Extended Data
are incorporated. Fig. 6) and human prostate cancer44 (Extended Data Fig. 7) and spatial
CUT&Tag–RNA-seq data5 (Supplementary Figs. 3 and 4) from mouse
Additional applications coronal brain. We also analyzed additional spatial omics data with
Finally, to showcase MISO’s broad applicability to diverse cancer and three modalities, including spatial gene and protein expression data
healthy tissue types, we applied it to spatial omics data with two modali- from human breast cancer45 (Extended Data Fig. 8) and mouse colon6
ties, including Visium data from mouse anterior brain39 (Extended Data (Supplementary Fig. 5), and spatial transcriptomics and metabolomics

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

data from mouse coronal brain9 (Extended Data Figs. 9 and 10). Some 8. Ben-Chetrit, N. et al. Integration of whole transcriptome spatial
of these analyses involve the integration of immunofluorescence and profiling with protein markers. Nat. Biotechnol. 41, 788–793
brightfield microscopy image data, histology image data with low reso- (2023).
lution or stain artifact, or low RNA quality. In all of these applications, 9. Vicari, M. et al. Spatial multimodal analysis of transcriptomes and
MISO demonstrated superior capability in delineating biologically metabolomes in tissues. Nat. Biotechnol. 42, 1046–1050 (2024).
relevant spatial domains and robustness in handling diverse data types. 10. Ghazanfar, S., Guibentif, C. & Marioni, J. C. Stabilized mosaic
single-cell data integration using unshared features.
Discussion Nat. Biotechnol. 42, 284–292 (2024).
In summary, we have presented MISO, an algorithm capable of accom- 11. Hao, Y. et al. Dictionary learning for integrative, multimodal and
modating all modalities in multimodal spatial omics experiments. scalable single-cell analysis. Nat. Biotechnol. 42, 293–304 (2024).
Distinct from existing methods, MISO’s unique strength lies in its flex- 12. Bao, F. et al. Integrative spatial analysis of cell morphologies and
ibility in handling diverse modalities, its ability to overcome the impact transcriptional states with MUSE. Nat. Biotechnol. 40, 1200–1209
of low-quality modalities, and minimal user-dependent parameter (2022).
tuning. Our comprehensive analyses across 16 datasets—including 13. Long, Y. et al. Deciphering spatial domains from spatial
brain, embryo and colon in mice; melanoma in zebrafish; and bladder multi-omics with SpatialGlue. Nat. Methods 21, 1658–1667 (2024).
cancer, gastric cancer, breast cancer, prostate cancer and tonsil in 14. Jiang, J. et al. METI: deep profiling of tumor ecosystems by
humans—demonstrated that MISO outperformed MUSE and Spatial- integrating cell morphology and spatial transcriptomics.
Glue in precisely identifying known biological structures. Nat. Commun. 15, 7312 (2024).
Currently, MISO is limited in that its feature extraction method 15. Schumacher, T. N. & Thommen, D. S. Tertiary lymphoid structures
is specific to H&E-stained histology images and cannot be applied in cancer. Science 375, eabf9419 (2022).
to other histology or molecular imaging types, including Trichome, 16. Sautès-Fridman, C. et al. Tertiary lymphoid structures in cancers:
immunofluorescence and DAPI. Additionally, because MISO is an prognostic value, regulation, and manipulation for therapeutic
unsupervised algorithm, it requires user interpretation to associate intervention. Front. Immunol. 7, 407 (2016).
identified spatial domains with biological or disease-relevant struc- 17. Di Caro, G. et al. Occurrence of tertiary lymphoid tissue is
tures. MISO also cannot directly quantify the contribution of each associated with T-cell infiltration and predicts better prognosis in
modality or automatically detect low-quality modalities, requiring early-stage colorectal cancers. Clin. Cancer Res. 20, 2147–2158
users to identify these through investigation of the spatial omics and (2014).
imaging data. Despite these limitations, MISO’s unique combination 18. Asrir, A. et al. Tumor-associated high endothelial venules mediate
of versatility, speed, accuracy and scalability positions it as a valuable lymphocyte entry into tumors and predict response to PD-1 plus
tool for modality integration and spatial domain identification in mul- CTLA-4 combination immunotherapy. Cancer Cell 40, 318–334
timodal spatial omics studies. In the future, we plan to extend MISO’s (2022).
capabilities to include automated feature extraction for more histol- 19. Zhang, D. et al. Inferring super-resolution tissue architecture by
ogy and molecular imaging types, enhance the model to quantify each integrating spatial transcriptomics with histology. Nat. Biotechnol.
modality’s contribution in clustering, and broaden its functionality to 42, 1372–1366 (2024).
simultaneously analyze multiple samples. As spatial omics technolo- 20. van Cutsem, E. et al. HER2 screening data from ToGA: targeting
gies continue to evolve, these improvements will support new data HER2 in gastric and gastroesophageal junction cancer.
types that are anticipated in the near future. Gastric Cancer 18, 476–484 (2015).
21. Oliveira, M. F. et al. Characterization of immune cell populations
Online content in the tumor microenvironment of colorectal cancer using high
Any methods, additional references, Nature Portfolio reporting sum- definition spatial profiling. Preprint at bioRxiv https://doi.org/
maries, source data, extended data, supplementary information, 10.1101/2024.06.04.597233 (2024).
acknowledgements, peer review information; details of author contri- 22. Jantscheff, P. et al. Expression of CEACAM6 in resectable
butions and competing interests; and statements of data and code avail- colorectal cancer: a factor of independent prognostic
ability are available at https://doi.org/10.1038/s41592-024-02574-2. significance. Jo. Clin. Oncol. 21, 3638–3646 (2003).
23. Burgos, M. et al. Prognostic value of the immune target
References CEACAM6 in cancer: a meta-analysis. Ther. Adv. Med. Oncol. 14,
1. Vandereyken, K., Sifrim, A., Thienpont, B. & Voet, T. Methods and 17588359211072621 (2022).
applications for single-cell and spatial multi-omics. Nat. Rev. 24. Qi, J. et al. Single-cell and spatial analysis reveal interaction of
Genet. 24, 494–515 (2023). FAP+ fibroblasts and SPP1+ macrophages in colorectal cancer.
2. Chen, K. H., Boettiger, A. N., Moffitt, J. R., Wang, S. & Zhuang, X. Nat. Commun. 13, 1742 (2022).
Spatially resolved, highly multiplexed RNA profiling in single 25. Ozato, Y. et al. Spatial and single-cell transcriptomics decipher
cells. Science 348, aaa6090 (2015). the cellular environment containing HLA-G+ cancer cells and
3. Stickels, R. R. et al. Highly sensitive spatial transcriptomics at SPP1+ macrophages in colorectal cancer. Cell Rep. 42, 111929
near-cellular resolution with Slide-seqV2. Nat. Biotechnol. 39, (2023).
313–319 (2021). 26. Bani-Yaghoub, M. et al. Role of Sox2 in the development of the
4. Eng, C.-H. L. et al. Transcriptome-scale super-resolved imaging in mouse neocortex. Dev. Biol. 295, 52–66 (2006).
tissues by RNA seqFISH+. Nature 568, 235–239 (2019). 27. Koopman, A. D. et al. The association between GAD65
5. Zhang, D. et al. Spatial epigenome–transcriptome co-profiling of antibody levels and incident type 2 diabetes mellitus in
mammalian tissues. Nature 616, 113–122 (2023). an adult population: a meta-analysis. Metabolism 95,
6. Liu, Y. et al. High-plex protein and whole transcriptome 1–7 (2019).
co-mapping at cellular resolution with spatial CITE-seq. 28. Petanjek, Z., Kostovic, I. & Esclapez, M. Primate-specific origins
Nat. Biotechnol. 41, 1405–1409 (2023). and migration of cortical GABAergic neurons. Front. Neuroanat. 3,
7. Liao, S. et al. Integrated spatial transcriptomic and proteomic 26 (2009).
analysis of fresh frozen tissue based on Stereo-seq. Preprint at 29. Kronman, F. N. et al. Developmental mouse brain common
bioRxiv https://doi.org/10.1101/2023.04.28.538364 (2023). coordinate framework. Nat. Commun. 15, 9072 (2024).

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

30. Hsueh, Y. -P., Wang, T. -F., Yang, F. -C. & Sheng, M. Nuclear 39. 10xGenomics. https://www.10xgenomics.com/resources/
translocation and transcription regulation by the membrane- datasets/mouse-brain-serial-section-1-sagittal-anterior-1-
associated guanylate kinase CASK/LIN-2. Nature 404, standard-1-1-0 (2020).
298–302 (2000). 40. 10xGenomics. https://www.10xgenomics.com/resources/
31. Bulfone, A. et al. T-brain-1: a homolog of Brachyury whose datasets/human-breast-cancer-visium-fresh-frozen-whole-
expression defines molecularly distinct domains within the transcriptome-1-standard (2022).
cerebral cortex. Neuron 15, 63–78 (1995). 41. Hunter, M. V., Moncada, R., Weiss, J. M., Yanai, I. & White, R. M.
32. Puelles, L. et al. Pallial and subpallial derivatives in the embryonic Spatially resolved transcriptomics reveals the architecture of the
chick and mouse telencephalon, traced by the expression of the tumor-microenvironment interface. Nat. Commun. 12, 6278 (2021).
genes Dlx-2, Emx-1, Nkx-2.1, Pax-6 and Tbr-1. J. Comp. Neurol. 424, 42. 10xGenomics. https://www.10xgenomics.com/resources/
409–438 (2000). datasets/adult-mouse-olfactory-bulb-1-standard-1 (2022).
33. Zhang, M. et al. Molecularly defined and spatially resolved 43. 10xGenomics. https://www.10xgenomics.com/resources/
cell atlas of the whole mouse brain. Nature 624, 343–354 datasets/mouse-brain-coronal-section-2-ffpe-2-standard (2022).
(2023). 44. Erickson, A. et al. Spatially resolved clonal copy number
34. Hamilton, D., White, C., Rees, C., Wheeler, D. & Ascoli, G. alterations in benign and malignant tissue. Nature 608,
Molecular fingerprinting of principal neurons in the rodent 360–367 (2022).
hippocampus: a neuroinformatics approach. J. Pharm. Biomed. 45. 10xGenomics. https://www.10xgenomics.com/resources/
Anal. 144, 269–278 (2017). datasets/gene-and-protein-expression-library-of-human-breast-
35. Lorente de Nó, R. Studies on the structure of the cerebral cortex. cancer-cytassist-ffpe-2-standard (2023).
II. Continuation of the study of the ammonic system. Journal für
Psychologie und Neurologie 46, 113–177 (1934). Publisher’s note Springer Nature remains neutral with regard to
36. Li, X. G., Somogyi, P., Ylinen, A. & Buzsáki, G. The hippocampal jurisdictional claims in published maps and institutional affiliations.
CA3 network: an in vivo intracellular labeling study. J. Comp.
Neurol. 339, 181–208 (1994). Springer Nature or its licensor (e.g. a society or other partner) holds
37. 10xGenomics. https://www.10xgenomics.com/resources/ exclusive rights to this article under a publishing agreement with
datasets/gene-protein-expression-library-of-human-tonsil- the author(s) or other rightsholder(s); author self-archiving of the
cytassist-ffpe-2-standard (2023). accepted manuscript version of this article is solely governed by the
38. Lipponen, P. K. & Eskelinen, M. J. Cell proliferation of terms of such publishing agreement and applicable law.
transitional cell bladder tumours determined by PCNA/cyclin
immunostaining and its prognostic value. Br. J. Cancer 66, © The Author(s), under exclusive licence to Springer Nature America,
171–176 (1992). Inc. 2025

1
Department of Computational Biomedicine, Cedars-Sinai Medical Center, Los Angeles, CA, USA. 2Statistical Center for Single-Cell and Spatial
Genomics, Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.
3
Department of Pathology, Seoul Metropolitan Government-Seoul National University Boramae Medical Center, Seoul National University College of
Medicine, Seoul, Korea. 4Department of Pathology, College of Medicine, Kyung Hee University Hospital, Kyung Hee University, Seoul, Republic of Korea.
5
Department of Microbiology, Institute for Immunology, and Institute for Diabetes, Obesity and Metabolism, Perelman School of Medicine, University
of Pennsylvania, Philadelphia, PA, USA. 6Faculty of Biology, University of Freiburg, Freiburg, Germany. 7Division of Genetics and Genomics, Department
of Pediatrics, Boston Children’s Hospital, Harvard Medical School, Boston, MA, USA. 8Department of Genitourinary Medical Oncology, The University of
Texas MD Anderson Cancer Center, Houston, TX, USA. 9Department of Genomic Medicine, The University of Texas MD Anderson Cancer Center, Houston,
TX, USA. 10School of Public Health, The University of Texas Health Science Center at Houston, Houston, TX, USA. 11Department of Chemistry, Lewis
Sigler Institute for Integrative Genomics, and Ludwig Institute for Cancer Research, Princeton University, Princeton, NJ, USA. 12Department of Artificial
Intelligence and Informatics, Mayo Clinic, Jacksonville, FL, USA. 13Department of Pathology and Laboratory Medicine, Perelman School of Medicine,
University of Pennsylvania, Philadelphia, PA, USA. 14Epigenetics Institute, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.
15
Department of Pathology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA. 16The University of Texas MD Anderson Cancer Center
UTHealth Graduate School of Biomedical Sciences, Houston, TX, USA. 17Department of Human Genetics, School of Medicine, Emory University, Atlanta,
GA, USA. e-mail: [email protected]; [email protected]; [email protected]

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

Methods and adeptly identify non-convex clusters, a critical capability in

Input data domains like gene expression and tissue histology data where clus-
MISO is designed to accommodate a wide range of omics and tissue ters often exhibit complex shapes and patterns47. The utilization of
histology image data from any multimodal spatial omics experiment spectral clustering in MISO’s feature extraction not only enhances the
as input. When utilizing RNA, protein or chromatin accessibility data algorithm’s robustness but also ensures its applicability to real-world
as input modalities, MISO is capable of automatically handling raw datasets characterized by diverse and nuanced structures.
data and performing all necessary preprocessing steps using Scanpy. To apply a spectral clustering-based feature extraction method, it
For RNA, protein and chromatin accessibility data, MISO takes count is first necessary to transform each input modality into a graph repre-
matrices as input and performs log-normalization. In the case of sentation. To achieve this, for each modality, we construct an adjacency
other types of omics data, such as metabolomics and histone modi- matrix that represents the similarity between each pair of spots with
fications, MISO assumes that the data have been preprocessed. If an respect to that modality. For a given modality m, each element of the
H&E-stained histology image is available, MISO can directly accept the adjacency matrix is constructed as
raw, high-resolution image and conduct image feature extraction. For
2
other image types, such as microscopy and immunofluorescence imag- −‖xm
i
− xmj ‖
2
ing, MISO requires preprocessed image features for each spot in the Am = [exp ( )] ,
2σ2
corresponding omics dataset. MISO performs z-score normalization i, j

for each input modality.

where xm i
and xmj are the modality m input features for spots i and j,
Extraction of H&E-stained histology image features respectively, and σ2 is a scaling parameter that was set to 30 for all
Many multimodal spatial omics technologies have the capability to gen- modalities in all analyses (based on empirical evidence).
erate high-resolution H&E-stained histology images of analyzed tissue Motivated by SpectralNet48, MISO utilizes neural networks for the
sections. However, the pixel-level resolution of the histology image sur- extraction of spectral clustering features. We start by extracting the top
passes that of the corresponding omics data, which typically have spots 128 principal components (PCs) for each modality, giving us an initial
that are a few hundred pixels in diameter. Consequently, it is imperative lower-dimensional representation of the data. For each specific modal-
to extract histology features for each spot in the dataset to facilitate the ity, we then use the corresponding adjacency matrix and spot-level PC
integration of tissue histology and omics features. In our approach, we feature vectors in the training of a deep neural network. This network
aimed to capture both local and global structures of the histology image aims to minimize both reconstruction and spectral clustering loss
in spot-level feature vectors. To accomplish this, we used a hierarchical functions, ensuring a comprehensive understanding and capture of
Vision Transformer (ViT) model pretrained on a substantial dataset of complex patterns within the data. The PC feature vectors for each
over 10,000 whole-slide images46. The model, made available by the modality serve as input for an autoencoder with a bottleneck layer of
Hierarchical Image Pyramid Transformer (HIPT) software, divides the 32 nodes. The matrix of vectors output from the bottleneck layer, Ym,
whole-slide image into sub-images of size 256 × 256 pixels, utilizing each will be used to cluster the spots after the network has been trained.
sub-image as input for a local ViT model to extract low-level image fea- The spectral clustering loss function for the modality m autoencoder
tures. For a given sub-image, this model assigns 384-dimensional patch is given by
embeddings to all 16 × 16-pixel patches via linear projection, which are
2
used to generate a 192-dimensional classification (CLS) token for the Lm = ∑ Am ‖ ym − ym ‖ ,
SC i, j ‖ i j‖ 2
entire 256 × 256-pixel image. The CLS tokens for all 256 × 256-pixel i, j

images within the same 4,096 × 4,096-pixel region are used as input
for a global ViT model to extract high-level image features. The global where ym i
is the output of the bottleneck layer corresponding to modal-
ViT model aggregates information across these CLS tokens, producing ity m for spot i. Defining the spectral clustering loss function in this
embeddings for each 256 × 256-pixel image that capture the long-range way ensures that spots that are similar with respect to the adjacency
dependencies across the tissue section. To construct spot-level histol- matrix will have similar embeddings for that modality. We require
ogy features, we first assign a 576-dimensional embedding to each that Ym is orthonormal so that all spots do not have the same embed-
16 × 16-pixel patch. This embedding comprises the 384-dimensional ding. Each autoencoder is also trained to minimize a reconstruction
embedding from the local ViT model and the 192-dimensional embed- loss, that is,
ding from the global ViT model of the 256 × 256-pixel image containing
2
= ∑ ‖‖zm − zî ‖‖ ,
m
that 16 × 16-pixel patch. This process ensures that the histology embed- Lm
R i 2
dings encompass both the local and global structures of the image. i

Subsequently, all 16 × 16-pixel patches completely contained within a

m
given omics spot are considered, and their embeddings are averaged where zmi
is the modality m PC feature vector for spot i and zî is the
to obtain a 576-dimensional embedding for each spot in the dataset. corresponding output of the decoder layer. The total loss for autoen-
A detailed illustration of the tissue histology image feature extraction coder m is a weighted sum of the spectral clustering and reconstruction
procedure is shown in Supplementary Fig. 6. losses:
m
Extraction of deep modality-specific features Lm = Lm
SC
+ αLR ,
The feature extraction step in MISO relies on a spectral clustering
framework. This approach leverages the distances between spots in where α, set to 1 by default, is a user-defined hyperparameter that
the raw feature space to generate lower-dimensional spot-level embed- determines the amount of weight given to the reconstruction loss
dings, meticulously capturing the graph structure inherent in the data. relative to the spectral clustering loss. A higher value for α means that
The choice of a spectral clustering methodology for this task is driven the extracted features will be more accurate representations of the
by the advantages it offers over alternative clustering methods that original input data, but the embeddings for similar spots may be more
impose rigid assumptions about the input data structure. Spectral distinct. All results provided in this paper were obtained by training the
clustering stands out due to its ability to operate effectively without model with alpha equal to 1. The autoencoders are trained separately
imposing stringent requirements on the underlying data structure. for each modality for 1,000 epochs using the Adam optimizer49 with
This adaptability allows it to navigate through complex input structures a learning rate of 0.001.

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

Cross-modality feature fusion evidence that the image may be of suboptimal quality. On the other
The autoencoders described in the previous step associate each spot hand, for omics data, the evaluation of quality involves the calculation
with a low-dimensional modality-specific vector representation. In of total unique molecular identifier counts across spots in the dataset.
many existing methods for multimodal data integration, embeddings Additionally, assessing the total detected number of genes relative to
for each modality are simply concatenated before utilization in down- other gene expression data from the same tissue type provides valu-
stream analyses. However, this does not account for the interactions able insights into the quality of transcriptomics data. For MALDI-MSI
between the different modalities. In the context of multimodal spatial spatial metabolomics data, we recommend an evaluation of the spatial
omics, interactions between different modalities can have a great localization of a subset of metabolites. In high-quality data, a major-
impact on phenotypic properties that are not evident when consider- ity of the metabolite signal is contained within the tissue as opposed
ing each modality separately. For instance, when considering RNA-seq to the background. In addition, data quality can be assessed using
and ATAC-seq data, interaction features are necessary to uncover the the percentage of detected high mass tissue-specific peaks (above
organization of gene regulatory networks by modeling the relation- a mass-to-charge (m/z) value of 2,000)51. An indication of high data
ships between expression levels of regulated genes and regions of open quality is if 30% of the tissue-specific peaks are above an m/z of 2,000.
chromatin50. For a given pair of modalities m1 and m2, we calculate the Comparison of this percentage with other samples of the same tissue
interaction feature matrix for spot i as type should also provide a better understanding of the quality of the
m m m m
data. For spatial protein sequencing data, the quality can be inferred
y 1 y 2 ⋯ yi,11 yi,32
2
through visualization of the spatial expression patterns of the proteins.
⎛ i,1 i,1 ⎞
= ⎜⎜ ⎟. If there is no spatial variation in the expression levels for the proteins in
m1 ⊗m2 m m2
yi = yi 1 ⊗ yi ⋮ ⋱ ⋮ ⎟
⎜ m1 m2 m1 m2 ⎟
the panel, then the data are likely of low quality. In addition, to include
y y
⎝ i,32 i,1 ⋯ y y
i,32 i,32 ⎠ the protein-specific terms, the protein panel should be reasonably
large (at least 35 proteins). When including multiplex RNA or protein
For each pair of modalities, the interaction feature matrices are imaging data, the most important influence on data quality is cell
flattened for all spots and combined to form a matrix with spot-level segmentation accuracy. This can be evaluated by visualizing the cell
interaction features. Subsequently, PC analysis is applied to each matrix boundaries relative to the locations of cell nuclei using an available
of interaction features, extracting the top 32 PCs to ensure uniform tissue stain (for example, H&E or DAPI). For ATAC-seq, quality can be
dimensionality with modality-specific feature vectors. This step assessed through signal-to-background ratio using transcription start
enhances compatibility for downstream analyses. While our general site enrichment scores.
recommendation is to include all pairs of interactions in clustering, To address low-quality modalities, MISO simplifies the decision-
MISO has a hyperparameter that allows users to specify which modality making process by requiring users to make a binary determination on
interactions they wish to incorporate, offering tailored control over whether to include the modality-specific features for each modality
the clustering process based on specific research needs. The selected in clustering based on its quality. By requiring a binary decision on
interaction feature vectors are concatenated with the modality-specific modality inclusion based on quality, MISO offers versatility across
feature vectors and used as input for the k-means clustering algorithm. multimodal spatial omics data with minimal preprocessing require-
ments. This user-centric design not only enhances the ease of applica-
Scaling to large datasets tion but also ensures broader adaptability, making MISO an accessible
Since the MISO algorithm involves the calculation and storage of an and robust choice for researchers working with diverse types of data
n × n adjacency matrix for a dataset with n spots, it is not well-suited in the multimodal spatial omics domain.
for datasets containing tens or hundreds of thousands of spots, such as
those generated by molecular imaging-based spatial omics technolo- ICC
gies like MERFISH. For these situations, we have introduced a modifi- The ICC is a commonly used metric to evaluate the degree of similari-
cation to the MISO algorithm that facilitates the analysis of large-scale ties for observations within the same cluster. For a given cluster i and
multimodal spatial omics datasets. To effectively handle large datasets, modality k, the ICC is defined as
we construct the adjacency matrix for modality m as
(k) σ2i
−‖xm −xmj ‖
2 ICCi = ,
⎧ exp ( i 2
) , j ∈ KNN(i) σ2i + σ2m
Am
i, j
= 2σ2
⎨
⎩ 0, otherwise. where σ2i is the within-cluster variance for cluster i, and σ2m is the variance
across cluster means of features from modality k. A higher ICC indicates
Specifically, if spot j is one of the k-nearest neighbors of spot i, the more coherent clusters with respect to that modality and, therefore,
jth entry in the spot i adjacency vector is set to the same value as the suggests superior clustering results. For each modality, we computed
previously defined similarity measure. However, if spot j is not among the ICC for all clusters using the top 50 PCs of the preprocessed data
the k-nearest neighbors of spot i, the entry is set to 0. For modality for that modality. For all comparisons of ICC values across different
m, the k-nearest neighbors of a given spot are determined by the dis- methods, P values were obtained using two-sample t-tests.
tances between the preprocessed modality m input feature vectors.
The default value for k is 100. This modification allows us to store Gastric cancer 10x Xenium data generation
the adjacency matrix as a sparse matrix, greatly reducing the storage Formalin-fixed, paraffin-embedded sections of signet-ring cell carci-
requirements for large datasets. noma were cut with 5-μm thickness into corresponding Xenium slides.
Deparaffinization was performed by incubating slides at 60 °C for
Guidance on feature selection 2 h, followed by 10 min in xylene jar 1 and 3 min in xylene jar 2, 100%
MISO necessitates users to designate modalities characterized by ethanol jar 1, 100% ethanol jar 2, 96% ethanol jar 1, 96% ethanol jar 2
low quality to determine which modality-specific feature vectors to and 70% ethanol. After a 20-s water rinse, RNA decrosslinking was per-
include in clustering. In the case of histology imaging data, quality formed at 80 °C for 30 min. The 10x Genomics Human Multi-Tissue and
assessment involves a visual inspection of the image. Indicators of low Cancer gene panel V1 was used to hybridize probes for 377 genes for 18 h
quality include artifacts such as ice crystal residue, uneven staining at 50 °C. Following 10x Genomics protocol no. CG000582, revision E,
and variations in blurriness across the tissue. These elements serve as unhybridized probes were washed away at 37 °C for 30 min, followed by

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

probe ligation for 2 h at 37 °C, and rolling circle amplification at 30 °C (10x Genomics) to align reads, perform spot calling and generate a
for 2 h. After washing, background autofluorescence was quenched, and spatially resolved gene expression matrix.
nuclei where fluorescently stained. Prepared tissue sections were then Alignment of 10x Visium and MALDI-MSI data: The 10x Visium
imaged using the Xenium Analyzer instrument following 10x Genomics and MALDI-MSI measure omics data at different spatial resolutions.
protocol CG000584, revision C. The instrument automatically per- Specifically, 10x Visium measures transcriptomics data at spots that
formed fluorescent probe hybridization, imaging and probe removal as are 55 µm in diameter, whereas MALDI-MSI measures metabolomics
a series of 15 cycles using firmware version 1.7.6.0. The Xenium Analyzer data at spots that are 20 µm in diameter. To obtain data with the same
then automatically processed and analyzed the images by performing spatial resolution, we ran iStar19, a method that enhances the resolu-
image co-registration, probe quality filtering, nuclei and cell segmenta- tion of spatial omics data using histology image information. iStar
tion using software analysis version Xenium 1.7.1.0. predicts molecular expression at super-pixels of 16 × 16 pixels in size
across the entire tissue section. The iStar output for the 10x Visium
Mouse coronal brain spatial transcriptomics and and MALDI-MSI data are thus aligned in the same coordinate space,
metabolomics data generation facilitating their integration.
Visium slide preparation for MALDI imaging mass spectrometry:
Visium (10x Genomics) slides were adapted for imaging mass spec- Pseudocode
trometry by trimming off 1 mm of the slide width using a diamond-tip Number of inputs: X1,…, Xn (matrices of features for each of the
scriber, followed by sanding the trimmed edge with sandpaper while n modalities)
preventing glass debris from covering the fiducial frames and the tis- for i in range(n):
sue capture areas on the slide during sanding. Further clearing of any XiPC = PCA (Xi ) [∶, range (128)]
debris was done by blowing nitrogen gas onto the slide surface. Resizing Ai = calculate_affinity (Xi )
the Visium slide was necessary for fitting it into a slide adaptor (Bruker while iteration < 1,000:
Daltonics) later for MALDI imaging mass spectrometry.   Yi = Encoder (XiPC )
Brain tissue preparation for MALDI imaging mass spectrometry:    XiPĈ = Decoder (Yi )
Fresh brain tissue samples were obtained from 9-month-old mice with-    LR = MSE (XiPC , XiPĈ )
out perfusion. Tissues were immediately snap-frozen in liquid nitrogen   LSC = Spectral_loss(Ai, Yi)
without embedding. Coronal brain sections were cut frozen at 12-µm   L = LR + αLSC
thickness using a cryostat (Leica, CM3050S) at −20 °C, thaw mounted    Update weights to minimize L
on adapted Visium 10x slides and dehydrated in a vacuum chamber for interactions = [Yi ⊗ Y j for i, j ∈ range(n)s.t.i ≠ j]
15 min. N-(1-naphthyl) ethylenediamine dihydrochloride (NEDC) was Y = [Yi for i in range(n)]
used as the MALDI matrix at a concentration of 10 mg ml−1 in a 70:30 interactions = [PCA(Z)[:, range(32)] for Z in interactions]
(vol:vol) methanol:water mixture and sprayed on the tissue sections emb = concatenate (Y, interactions)
using a HTX-TM sprayer (HTX Technologies) under the following spray- clusters = KMeans(emb)
ing conditions: nozzle temperature, 70 °C; matrix solution flow rate,
0.1 ml min−1; nitrogen gas pressure, 10 psi; nozzle velocity, 1,200 mm Reporting summary
min−1; nozzle height, 40 mm; spray spacing, 2 mm; ten passes and 10 s Further information on research design is available in the Nature
of drying time between each pass. Portfolio Reporting Summary linked to this article.
MALDI imaging mass spectrometry: All MALDI imaging mass
spectrometry experiments were done using a MALDI-2 timsTOF fleX References
(Bruker Daltonics) equipped with microGRID technology. Mass calibra- 46. Chen, R. J. et al. Scaling vision transformers to gigapixel images
tion was performed using a Tuning Mix solution (Agilent Technologies, via hierarchical self-supervised learning. In Proc. IEEE/CVF
G1969-85000) before starting the imaging run. The laser Smart Beam Conference on Computer Vision and Pattern Recognition
was set to ‘Custom’ mode while engaging the ‘Beam scan’ to deliver a 16144–16155 (2022).
single burst of 200 laser shots at a frequency of 10 kHz in a pixel area of 47. Ng, A., Jordan, M. & Weiss, Y. On spectral clustering: analysis and
20 × 20 µm2. The raster step size was set to 20 µm in x and y directions. an algorithm. Adv. Neural Inf. Process. Syst. 14, 849–856 (2001).
Acquisitions were recorded in negative ion mode at an m/z range of 48. Shaham, U. et al. Spectralnet: Spectral clustering using deep
100–1,000. Energy difference between the collision cell and the quad- neural networks. Preprint at https://doi.org/10.48550/
rupole was set to 8 eV with a collision RF of 2,000 Vpp. The quadrupole arXiv.1801.01587 (2018).
ion energy was set to 8 eV and the minimum m/z was set to 150, with the 49. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization.
pre-TOF transfer time set to 50 µs. Lock masses of m/z 124.0068 (tau- In Proc. International Conference on Learning Representations
rine), m/z 255.2330 (palmitate), m/z 346.0558 (AMP) and m/z 426.0222 (ICLR, 2015)
(ADP) were used for online calibration. All MALDI imaging experiments 50. Lowe, E. K., Cuomo, C., Voronov, D. & Arnone, M. I. Using
were done without engaging trapped ion mobility separation. ATAC-seq and RNA-seq to increase resolution in GRN
Spatial transcriptomics with 10x Visium: The Visium Spatial Gene connectivity. Methods Cell. Biol. 151, 115–126 (2019).
Expression Reagent Kit (10x Genomics, PN-1000189) was used to cap- 51. Høiem, T. S. et al. An optimized MALDI MSI protocol for spatial
ture spatially resolved transcriptomic data from the tissue sections detection of tryptic peptides in fresh frozen prostate tissue.
according to the manufacturers protocol (CG000239). H&E images Proteomics 22, e2100223 (2022).
were acquired with a Leica Aperio Slidescanner. The optimal tissue
permeabilization time was determined with the Visium Spatial Optimi- Acknowledgements
zation Kit (PN-1000191) according to the manufacturer’s instructions M. Li was partly supported by National Institutes of Health (NIH)
to be 25 min (CG000238). Libraries for sequencing were prepared with grants R01HG013185, R01LM014592, R01EY030192, U19NS135582,
10x genomics library construction kits (PN-1000196). R01HL171595 and U01CA294518. L.W. was partly supported by
Sequencing and data analysis: Prepared libraries were sequenced NIH grants R01CA266280, U01CA264583, U01CA294518 and
on an Illumina platform (NextSeq 2000, Illumina) using the P2 kit U24CA274274; the start-up research fund provided by the University
(100 cycles) to generate high-throughput transcriptomic data. Raw of Texas MD Anderson Cancer Center; The Break Through Cancer
sequencing data were processed using the Space Ranger pipeline Foundation; and the Andrew Sabin Family Foundation. L.W. and J.J

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

were also supported by NIH grant R01 CA254988. T.H.H. was partly X.Q. annotated the hippocampus region and interpreted the results
supported by NIH grants R01CA276690 and U01CA294518, DOD grant of the mouse brain spatial transcriptomics and metabolomics data.
CA190578, the Eric and Wendy Schmidt Foundation’s AI Innovation Y.D. provided input for the spatial CUT&Tag–RNA-seq data analysis.
Award through the Mayo Clinic Foundation, and the Torrey Coast E.B.L. provided input for mouse brain data analysis. E.E.F. confirmed
Foundation. X.Q. was partly supported by NIH grant K99NS135123. J.G. tissue annotation and provided input for interpretation of the Visium
was partly supported by the Doris Duke Clinical Scientist Development HD human colon cancer data. K.C. and M. Li wrote the paper with
Award (2018097), MD Anderson Faculty Scholar Award, the David H. feedback from the other co-authors.
Koch Center for Applied Research of Genitourinary Cancers, Wendy
and Leslie Irvin Barnhart Fund, Joan and Herb Kelleher Charitable Competing interests
Foundation, KCA Advanced Discovery Award, the Williams TNT Fund, M. Li receives research funding from Biogen unrelated to the current
the V Foundation Translational Award, the DOD KCRP Translational manuscript. M. Li and D.Z. are cofounders of OmicPath AI. T.H. is
Research Partnership Award, NIH/NCI R01 CA254988-01A1, NIH/ a cofounder of Kure.ai therapeutics and has received consulting
National Cancer Institute (NCI) R01 CA269489-01A1 and NIH/NCI R01 fees from IQVIA; these affiliations and financial compensations
CA282282-01; as well as in part by the Cancer Center Support Grant to are unrelated to the current paper. The other authors declare no
MDACC (P30 CA016672) from the NCI, by MD Anderson’s Prometheus competing interests.
informatics system and by the Department of Genitourinary Medical
Oncology’s Eckstein and Alexander Laboratories. J.D.R. was supported Additional information
by Ludwig Cancer Research, the Penn Diabetes Research Center grant Extended data is available for this paper at
(P30-DK19525) and the Chan Zuckerberg Initiative DAF (2023-331955), https://doi.org/10.1038/s41592-024-02574-2.
an advised fund of Silicon Valley Community Foundation.
Supplementary information The online version contains supplementary
Author contributions material available at https://doi.org/10.1038/s41592-024-02574-2.
This study was conceived of and led by M. Li and J.H. K.C. designed the
model and algorithm with input from M. Li and J.H., implemented the Correspondence and requests for materials should be addressed to
MISO software and led data analyses. A.S., M. Loth and H.Y. performed Kyle Coleman, Jian Hu or Mingyao Li.
data analyses. D.Z. proposed the histology image feature extraction
approach. T.H.H., J.H.P., J.-Y.S., J.R.C., I.J., M.K. and I.B. generated Peer review information Nature Methods thanks the anonymous
and processed the Xenium gastric cancer data. J.H.P. annotated the reviewers for their contribution to the peer review of this work.
Xenium gastric cancer data. J.-Y.S. annotated the Visium HD colon Peer reviewer reports are available. Primary Handling Editor:
cancer data. L.W., J.G., J.C., A.L. and J.J. generated and processed the Rita Strack, in collaboration with the Nature Methods team.
Visium bladder cancer data. A.L. annotated the Visium bladder cancer
data. C.A.T., J.D.R., N.B., A.J.C. and L.Z.S. generated and processed Reprints and permissions information is available at
the mouse brain spatial transcriptomics and metabolomics data. www.nature.com/reprints.

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

Extended Data Fig. 1 | Intraclass correlation coefficient (ICC) results for all ICC was lower was for the RNA modality in the mouse hippocampus spatial
datasets provided in Figs. 2-5 that were evaluated using MISO, MUSE, and transcriptomics and metabolomics dataset, where the ICC for SpatialGlue
SpatialGlue. The mean ICC for each method and each modality is printed on surpassed that of MISO. The likely cause of this is that, because the RNA data
the corresponding box plot. Test statistics and p-values were obtained using was of low quality, MISO did not use the RNA-specific terms in clustering, and
one-sided t-tests (n = 1250 ICC values for each group). For a vast majority of the only accounted for this modality through the RNAxImage and RNAxMetabolite
modalities across all datasets, the MISO clustering results produced a higher interaction terms. Box plots: center line, median; box limits, upper and lower
ICC compared to the other methods. The only instance in which the MISO quartiles; whiskers, 1.5x interquartile range; points, outliers.

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

Extended Data Fig. 2 | Clustering results for a mouse anterior brain spatial the corresponding box plot. Test statistics and p-values were obtained using one-
transcriptomics dataset. a, Allen Brain Atlas annotation of mouse anterior sided (<,>) or two-sided (≈) t-tests. d, MISO and SpatialGlue RNA ICC distributions
brain. b, Shown from left to right are clustering results from MISO, MUSE, and for all clusters corresponding to the cortical layers in the mouse anterior brain
SpatialGlue, respectively. SpatialGlue is sensitive to weight specified for each data (n = 50 ICC values for each group except SpatialGlue L2/3 and SpatialGlue
modality. WG is the weight for gene expression and WH is the weight for histology. L6, which contain n = 100 ICC values). Test statistics and p-values were obtained
Adjusted Rand Index (ARI) is calculated between SpatialGlue clustering with using one-sided (<,>) or two-sided (≈) t-tests. e, Illustration of image artifact.
different weights. c, RNA and image ICC distributions across all clusters and Box plots: center line, median; box limits, upper and lower quartiles; whiskers,
features for each method in the mouse anterior brain data (n = 1250 ICC values 1.5x interquartile range; points, outliers.
for each group). The mean ICC for each method and each modality is printed on

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

Extended Data Fig. 3 | Clustering results for a human breast cancer spatial coherence with respect to gene expression patterns and the annotated
transcriptomics dataset. a, Pathologist manual annotation of tissue section. histological regions. e, RNA t-SNE plot for the breast cancer Visium dataset
DCIS: Ductal carcinoma in situ. b, Shown from left to right are clustering results with spots colored according to total UMI count. MISO was able to localize
from MISO, MUSE, and SpatialGlue, respectively. Two patches were selected a sub-cluster in the annotated invasive carcinoma region with much lower
to highlight that MISO’s results agree better with histological patterns. c, RNA total UMI counts compared to other sub-clusters in this region. f, SpatialGlue
and image ICC distributions across all clusters and features for each method clustering results when increasing the weight given to histology in the loss
in the breast cancer data (n = 750 ICC values for each group). The mean ICC for function. SpatialGlue was not able to detect the fat region of the tissue section
each method and each modality is printed on the corresponding box plot. Test when making the weight given to histology 10 or 50 times greater than that given
statistics and p-values were obtained using one-sided (<,>) or two-sided (≈) to gene expression. Box plots: center line, median; box limits, upper and lower
t-tests. d, Spots plotted according to their RNA t-SNE coordinates and colored by quartiles; whiskers, 1.5x interquartile range; points, outliers.
the clustering results for each method. The MISO clustering results demonstrate

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

Extended Data Fig. 4 | Clustering results for a zebrafish melanoma spatial artifact. c, RNA and image ICC distributions across all clusters and features for
transcriptomics dataset. a, Blurriness artifact in the H&E-stained histology each method (n = 400 ICC values for each group). The mean ICC for each method
image. b, Clustering results from MISO, MUSE, and SpatialGlue. MISO did not and each modality is printed on the corresponding box plot. Test statistics and
include the image-specific features in clustering because of the low quality of p-values were obtained using one-sided (<,>) or two-sided (≈) t-tests.
the image, but the image features were still accounted for in the RNAximage Box plots: center line, median; box limits, upper and lower quartiles; whiskers,
interaction terms. Clusters in the MUSE results are driven by the blurriness 1.5x interquartile range; points, outliers.

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

Extended Data Fig. 5 | Clustering results for a mouse olfactory bulb spatial tissue section from those in the lower region. c, RNA and image ICC distributions
transcriptomics dataset. a, Clustering results from MISO, MUSE, and across all clusters and features for each method (n = 400 ICC values for each
SpatialGlue. b, H&E-stained histology image of analyzed tissue section with layer group). The mean ICC for each method and each modality is printed on the
annotation. The MISO results align well with the annotation, assigning clusters corresponding box plot. Test statistics and p-values were obtained using one-
to each of the annotated layers. MUSE was not able to accurately localize clusters sided (<,>) or two-sided (≈) t-tests. Box plots: center line, median; box limits,
to the annotated layers, and instead separated spots in the upper region of the upper and lower quartiles; whiskers, 1.5x interquartile range; points, outliers.

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

Extended Data Fig. 6 | Clustering results for a mouse coronal brain spatial modality is printed on the corresponding box plot. Test statistics and p-values
transcriptomics dataset. a, Clustering results from MISO, MUSE, and were obtained using one-sided (<,>) or two-sided (≈) t-tests. Box plots: center
SpatialGlue. b, H&E-stained histology image of analyzed tissue section. c, RNA line, median; box limits, upper and lower quartiles; whiskers, 1.5x interquartile
and image ICC distributions across all clusters and features for each method range; points, outliers.
(n = 1000 ICC values for each group). The mean ICC for each method and each

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

Extended Data Fig. 7 | Clustering results for a human prostate cancer across all cancer spots for MISO (0.51), MUSE (0.44), and SpatialGlue (0.50).
spatial transcriptomics dataset. a, Clustering results from MISO, MUSE, and f, Weighted F1 score for localization of clusters to the annotated clones for MISO
SpatialGlue. b, H&E-stained histology image of analyzed tissue section. c, RNA (0.61), MUSE (0.52), and SpatialGlue (0.59). To calculate F1 score for a given
and image ICC distributions across all clusters and features for each method method, a cluster was assigned to a clone if more than half of the spots from
(n = 750 ICC values for each group). The mean ICC for each method and each that cluster overlapped with the clone annotation. F1 score was weighted by the
modality is printed on the corresponding box plot. Test statistics and p-values number of spots belonging to each clone in the annotation. Box plots: center line,
were obtained using one-sided (<,>) or two-sided (≈) t-tests. d, Clone annotation median; box limits, upper and lower quartiles; whiskers, 1.5x interquartile range;
of cancer spots. e, ARI between the clone annotation and the clustering results points, outliers.

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

Extended Data Fig. 8 | Clustering results for a human breast cancer spatial image, and protein data as input. e, Eosin-stained histology image of analyzed
gene and protein expression dataset. Image features used as input for each tissue section. f, RNA, image, and protein ICC distributions across all clusters
method were extracted from an immunofluorescence image with 3 channels and features for each method when taking each possible combination of
(DAPI, Vimentin, and PCNA) using a pre-trained InceptionV3 model. Patches modalities as input (n = 500 ICC values for each group). For each method, the
corresponding to the omics spots were extracted from the immunofluorescence mean ICC for each modality is printed on the corresponding box plot for its
image, resized to 299×299 pixels, and normalized prior to extracting features top-performing combination of modalities. Test statistics and p-values were
for each patch using InceptionV3. a, Clustering results from MISO, MUSE, and obtained using one-sided (<,>) or two-sided (≈) t-tests when comparing each
SpatialGlue when taking RNA and image data as input. b, Clustering results from method’s top-performing results for a given modality. Box plots: center line,
MISO, MUSE, and SpatialGlue when taking RNA and protein data as input. median; box limits, upper and lower quartiles; whiskers, 1.5x interquartile range;
c, Clustering results from MISO, MUSE, and SpatialGlue when taking protein points, outliers.
and image data as input. d, Clustering results from MISO when taking RNA,

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

Extended Data Fig. 9 | See next page for caption.

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

Extended Data Fig. 9 | Clustering results for a mouse brain spatial MUSE, and SpatialGlue when taking metabolomics and histology image data as
transcriptomics (10x Visium) and metabolomics (MALDI-MSI) dataset, input. d, Clustering results from MISO when taking RNA, histology image, and
which was generated following the protocol described in Vicari et al. [9]. metabolomics data as input. e, Total UMI counts across all spots in the dataset.
To make the super-resolution spatial molecular data inferred by iStar more The UMI counts are low because the tissue section was analyzed using MALDI-
suitable for input to all methods, we merged superpixels obtained from iStar MSI prior to Visium. f, Total metabolite intensities across all spots in the dataset.
to create 4,687 pseudo-spots of size 128×128 pixels, containing paired gene g, H&E-stained histology image of analyzed tissue section. h, RNA, image, and
expression and metabolite information. Because the RNA data is of low quality metabolomics ICC distributions across all clusters and features for each method
(e), the RNA-specific features extracted by MISO were not used to produce any when taking each possible combination of modalities as input (n = 1500 ICC
of the results provided, but RNA was still accounted for in its interactions with values for each group). For each method, the mean ICC for each modality is
metabolomics and image features. For all applicable results, metabolomics printed on the corresponding box plot for its top-performing combination of
data were normalized by total intensity and log transformed. a, Clustering modalities. Test statistics and p-values were obtained using one-sided t-tests
results from MISO, MUSE, and SpatialGlue when taking RNA and histology image when comparing each method’s top-performing results for a given modality.
data as input. b, Clustering results from MISO, MUSE, and SpatialGlue when Box plots: center line, median; box limits, upper and lower quartiles; whiskers,
taking RNA and metabolomics data as input. c, Clustering results from MISO, 1.5x interquartile range; points, outliers.

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

Extended Data Fig. 10 | See next page for caption.

Nature Methods
Article https://doi.org/10.1038/s41592-024-02574-2

Extended Data Fig. 10 | Clustering results for large-scale mouse brain taking RNA and metabomics data as input. c, Clustering results from MISO,
spatial transcriptomics (10x Visium) and metabolomics (MALDI-MSI) MUSE, and SpatialGlue when taking metabolomics and histology image data as
dataset described in Extended Data Fig. 9. Results were obtained using RNA, input. d, Clustering results from MISO when taking RNA, histology image, and
metabolomics, and image features as input, but RNA was only accounted for in metabolomics data as input. e, RNA, image, and metabolomics ICC distributions
the interaction terms due to its low quality. For all applicable results, other than across all clusters and features for each method when taking each possible
the MUSE results obtained when taking metabolomics and image data as input, combination of modalities as input (n = 1500 ICC values for each group). For
metabolomics data were normalized by total intensity and log transformed. each method, the mean ICC for each modality is printed on the corresponding
MUSE was not able to evaluate the combination of metabolomics and image box plot for its top-performing combination of modalities. Test statistics and
data when the metabolomics data were log normalized, so this step was not p-values were obtained using one-sided t-tests when comparing each method’s
utilized to obtain the corresponding results. The dataset used to generate the top-performing results for a given modality. f, Clustering results from MISO
results in (a-e) contains 74,851 pseudo-spots of size 32×32 pixels. The dataset when taking RNA, histology image, and metabolomics data from the dataset
used to generate the results in (f) contains 299,350 pseudo-spots of size 16×16 with pseudo-spots of size 16×16 pixels as input. Test statistics and p-values were
pixels. Due to memory requirements, MISO was the only method that could obtained using one-sided t-tests when comparing each method’s top-performing
evaluate the dataset with pseudo-spots of size 16×16 pixels. a, Clustering results results for a given modality. Box plots: center line, median; box limits, upper and
from MISO, MUSE, and SpatialGlue when taking RNA and histology image lower quartiles; whiskers, 1.5x interquartile range; points, outliers.
data as input. b, Clustering results from MISO, MUSE, and SpatialGlue when

Nature Methods