D E M U X: Data-efficient Multilingual Learning

Simran Khanuja Srinivas Gowriraj Lucio Dery Graham Neubig

Carnegie Mellon University

{skhanuja,sgowrira,ldery,gneubig}@cs.cmu.edu

Abstract fluent in these languages, but this also fails given

crowd-sourcing platforms’ lack of support for the
We consider the task of optimally fine-tuning above languages3 . As an alternative, they decide to
pre-trained multilingual models, given small
use tools that identify best languages for transfer,
arXiv:2311.06379v1 [cs.CL] 10 Nov 2023

amounts of unlabelled target data and an an-

notation budget. In this paper, we introduce
but these either rely on linguistic feature informa-
D E M U X, a framework that prescribes the ex- tion – missing for Dzongkha and Tharu4 (Lin et al.,
act data-points to label from vast amounts of 2019), past model performances – expensive to
unlabelled multilingual data, having unknown obtain (Srinivasan et al., 2022) or don’t support
degrees of overlap with the target set. Unlike multilingual targets (Lin et al., 2019; Kumar et al.,
most prior works, our end-to-end framework 2022). Based on annotator availability, they even-
is language-agnostic, accounts for model rep- tually choose Nepali and Tibetan as optimal trans-
resentations, and supports multilingual target
fer languages, and collect unlabelled corpora from
configurations. Our active learning strategies
rely upon distance and uncertainty measures news articles, social media and online documents.
to select task-specific neighbors that are most Even assuming all the preceding challenges are
informative to label, given a model. D E M U X surmounted, a final question remains unaddressed
outperforms strong baselines in 84% of the test by the traditional pipeline: how do they select the
cases, in the zero-shot setting of disjoint source exact data points to give to annotators for best per-
and target language sets (including multilin- formance in their domain-specific custom model,
gual target pools), across three models and four
under a fixed budget?
tasks. Notably, in low-budget settings (5-100
examples), we observe gains of up to 8-11 F1 In this work, we aim to provide a solution to
points for token-level tasks, and 2-5 F1 for com- the above problem by introducing D E M U X, an
plex tasks. Our code is released here1 . end-to-end framework that replaces the pipelined
approach to multilingual data annotation (Figure
1 Introduction 1). By directly selecting datapoints to annotate,
D E M U X bypasses several stages of the pipeline,
Picture this: Company Y, a healthcare technology
that are barriers for most languages. The alleviation
firm in India, has recently expanded their virtual as-
of needing to identify the target languages itself
sistance services to cover remote locations in Nepal
(Step 1: Figure 1), implies that it can be used for
and Bhutan. Unfortunately, their custom-trained
noisy, unidentifiable, or code-mixed targets.
virtual assistant is struggling with the influx of new
D E M U X makes decisions at the instance level
multilingual data, most of which is in Dzongkha
by using information about the pre-trained multilin-
and Tharu, but unindentifiable by non-native re-
gual language model’s (MultiLM’s) representation
searchers at Y. How can they improve this model?
space. This ensures that the data annotation pro-
Following current approaches, they first attempt to
cess is aware of the model ultimately being utilized.
discern the languages the data belongs to, but com-
Concretely, we draw from the principles of active
mercial language identification systems (LangID)
learning (AL) (Cohn et al., 1996; Settles, 2009)
are incapable of this task2 . Assuming this hurdle
for guidance on model-aware criteria for point se-
is crossed, Company Y then seeks out annotators
lection. AL aims to identify the most informative
1
https://github.com/simran-khanuja/demux points (to a specific model) to label from a stream
2
For instance, this is true of Google Cloud’s LangID as
3
of November 2023: https://developers.google.com/ml-kit/ Given MTurk’s lack of regional support.
4
language/identification/langid-support e.g. in the WALS database https://wals.info/languoid
 Most Current Approaches

Step 1: Language Identification Step 2: Data Selection for Annotation Step 3: Finetuning
Source Data Labelling
Lt1 Models

Target Data
A
Lt2
B
Transfer Language
Lt3
Source Data Labelling C

DeMuX

LEGEND
Models
Target (t)
Source Data
Source
A Labelled
Unlabelled
Target Data B Labelling
Annotator

C No Annotator

Method exists
D
Not addressed

Figure 1: Top: Today, improving the performance of a model on multilingual target data is a three-step process.
First, one would identify the target languages. Next, they would either collect data to label in these languages,
or closely related transfer languages, based on annotator availability. Finally, they would fine-tune a model on
the labelled data. However Step 1 excludes 98% of the world’s languages, Step 2 is constrained by annotators
or linguistic feature information, and Step 3 of factoring in the model to fine-tune, largely remains unaccounted
for. Bottom: With our end-to-end framework D E M U X, we prescribe the exact data to label from a vast pool of
multilingual source data, that provides for best transfer to the target, for a given model.

of unlabelled source data. Through iterations of a combination of the two, which picks uncertain
model training, data acquisition and human annota- points in target points’ local neighborhood (§3).
tion, the goal is to achieve satisfactory performance We experiment with tasks of varying complexity,
on a target test set, labelling only a small fraction categorized based on their label structure: token-
of the data. Past works (Chaudhary et al., 2019; Ku- level (NER and POS), sequence-level (NLI), and
mar et al., 2022; Moniz et al., 2022) have leveraged question answering (QA). We test our strategies
AL in the special case where the same language(s) in a zero-shot setting across three MultiLMs and
constitute the source and target set (Step 2 (upper five target language configurations, for a budget of
branch): Figure 1). However, none so far have 10,000 examples acquired in five AL rounds (§4).
considered the case of source and target languages
having unknown degrees of overlap; a far more We find that our strategies outperform previous
pervasive problem for real-world applications that baselines in most cases, including those with mul-
commonly build classifiers on multi-domain data tilingual target sets. The extent varies, based on
(Dredze and Crammer, 2008). From the AL lens, the budget, the task, the languages and models
this is particularly challenging since conventional (§5). Overall, we observe that the hybrid strategy
strategies of choosing the most uncertain samples performs best for token-level tasks, but picking
(Settles, 2009), could pick distracting examples globally uncertain points gains precedence for NLI
from very dissimilar language distributions (Long- and QA. To test the applicability of D E M U X in
pre et al., 2022). Our strategies are designed to resource constrained settings, we experiment with
deal with this distribution shift by leveraging small lower budgets ranging from 5-1000 examples, ac-
amounts of unlabelled data in target languages. quired in a single AL round. In this setting, we
observe gains of upto 8-11 F1 points for token-
In the rest of the paper, we first describe three level tasks, and 2-5 F1 for complex tasks like NLI
AL strategies based on the principles of a) semantic and QA. For NLI, our strategies surpass the gold
similarity with the target; b) uncertainty; and c) standard of fine-tuning in target languages, while
being zero-shot. unlabelled target pool (McCallum et al., 1998; Set-
tles and Craven, 2008). The source points cho-
2 Notation sen are informative since they are prototypical of
the target data in the representation space (Figure
Assume that we have a set of source languages, 2a). Especially for low degrees of overlap between
Ls = {ls1 . . . lsn }, and a set of target languages, source and target data distributions, this criterion
Lt = {lt1 . . . ltm }. Ls and Lt are assumed to have can ignore uninformative source points. Formally,
unknown degrees of overlap.
Further, let us denote the corpus of unlabelled X
X∗ = argmin X ∈ S b dt (xs )
source data as Xs = {x1s . . . xN s } and the unla-
xs ∈X
belled target data as Xt = {x1t . . . xM
t }.
Our objective is to label a total budget of B Where
data points over K AL rounds from the source 1  
f (x) − f xjt
X
dt (x) =
data. The points to select in each round can then |Xt |
B xjt ∈Xt
be calculated by b = K . Thus considering the
b
super-set S = {X ⊂ Xs |X| = b} of all b-
sized subsets of Xs , our objective is to select some For all task types, we use embeddings of tokens
X∗ ∈ S b according to an appropriate criterion. fed into the final classifier, to represent the whole
sequence. For NLI and QA, this is the [CLS] token
3 Annotation Strategies embedding. For token-level tasks, we compute the
mean of the initial sub-word token embeddings
Based on the broad categorizations of AL methods for each word, as this is the input provided to the
as defined by Zhang et al. (2022), we design three classifier to determine the word-level tag.
annotation strategies that are either representation-
based, information-based, or hybrid. The first 3.2 UNCERTAINTY
picks instances that capture the diversity of the
dataset; the second picks the most uncertain points Uncertainty sampling (Lewis, 1995) improves an-
which are informative to learn a robust decision notation efficiency by choosing points that the
boundary; and the third focuses on optimally com- model would potentially misclassify in the current
bining both criteria. In contrast to the standard AL iteration. Uncertainty measures for each task-
AL setup, there are two added complexities in our type can be found below:
framework: a) source-target domain mismatch; b) Sequence-Level: We use margin-sampling
multiple distributions for each of our target lan- (Scheffer et al., 2001; Schein and Ungar, 2007),
guages. We therefore design our measures to select which selects points having the least difference
samples that are semantically similar (from the between the model’s probabilities for the top-two
perspective of the MultiLM) to the target domain classes. We compute the output probability distri-
(Longpre et al., 2022). bution for all unlabeled samples in Xs and select
All strategies build upon reliable distance and un- samples with the smallest margin. Formally,
certainty measures, whose implementation varies
X
based on the type of task, i.e. whether the task is X∗ = argmin X ∈ S b P∆ (xs )
token-level, sequence-level or question answering. xs ∈X
A detailed visualization of how these are calculated Where
can be found in §A.1. Below, we formally describe
P∆ (x) = pc1 (x) − pc2 (x)
the three strategies, also detailing the motivation
behind our choices.
pc1 (x) and pc2 (x) are the predicted probabilities
3.1 AVERAGE-DIST of the top-two classes for an unlabeled sample x.
AVERAGE-DIST constructs the set X∗ such that it Token-level: For token-level tasks we first com-
minimizes the average distance of points from Xt pute the margin (as described above) for each token
under an embedding function f : X → Rd de- in the sequence. Then, we assign the minimum mar-
fined by the MultiLM. This is a representation- gin across all tokens as the sequence margin score
based strategy that picks points lying close to the and choose construct X∗ with sequences having
 (a) AVERAGE-DIST (b) UNCERTAINTY (c) KNN-UNCERTAINTY

Figure 2: Visualization of datapoints selected using strategies detailed in Section 3, for a three-class sequence
classification task (XNLI). AVERAGE-DIST selects points (dark blue) at a minimum average distance from the
target (pink); UNCERTAINTY selects most uncertain points lying at the decision boundary of two classes, and
KNN-UNCERTAINTY selects uncertain points in the target neighborhood.

the least score. Formally, points (Figure 2a) – even if the model is accurate
X in that region of representation space – resulting in
X∗ = argmin X ∈ S b MARGIN-MIN (xs ) minimal coverage on the target set.
xs ∈X To design a strategy that combines the strengths
Where of both distance and uncertainty, we first measure
|x| how well a target point’s uncertainty correlates with
MARGIN-MIN (x) = min pic1 (x) − pic2 (x)


its neighborhood. We calculate the Pearson’s corre-
i=1
lation coefficient (ρ) (Pearson, 1903) between the
Question Answering: The QA task we investi- uncertainty of a target point in Xt and the average
gate involves extracting the answer span from a rel- uncertainty of its top-k neighbors in Xs . We ob-
evant context for a given question. This is achieved serve a statistically significant ρ value > 0.7, for
by selecting tokens with the highest start and end all tasks. A natural conclusion drawn from this is
probabilities as the boundaries, and predicting to- that decreasing the uncertainty of a target point’s
kens within this range as the answer. Hence, sam- neighborhood would decrease the uncertainty of
ples having the lowest start and end probabilities, the target point itself. Hence, we first select the top-
qualify as most uncertain. Formally, k neighbors for each xt ∈ Xt . Next, we choose the
X most uncertain points from these neighbors until
X∗ = argmin X ∈ S b SUM-PROB (xs ) we reach b data points. Formally, until |X∗ | = b :
xs ∈X X
Where X∗ = argmax k U (xs )
{X ⊂ Nt |X| = b}
xs ∈ X
|x| |x|
SUM-PROB (x) = max log pis (x)+ max log pie (x) Where
i=1 i=1
|Xt |
Above, |x| denotes the sequence length of the k-NEARESTNEIGHBORS(xjt , Xs )
[
Ntk =
unlabeled sample x, and pis (x) and pie (x) represent j=1
the predicted probabilities for the start and end
Above, U (xs ) represents the uncertainty of the
index, respectively.
source point as calculated in §3.2.
3.3 KNN-UNCERTAINTY
4 Experimental Setup
As standalone measures, both distance and uncer-
tainty based criteria have shortcomings. When Our setup design aims to address the following:
there is little overlap between source and target, Q1) Does D E M U X benefit tasks with varying com-
choosing source points based on UNCERTAINTY plexity? Which strategies work well across differ-
alone leads to selecting data that are uninformative ent task types? (§4.1)
to the target. When there is high degrees of overlap Q2) How well does D E M U X perform across a
between source and target, the AVERAGE-DIST met- varied set of target languages? Can it benefit multi-
ric tends to produce a highly concentrated set of lingual target pools as well? (§4.2)
 Task Type Task Dataset Languages (two-letter ISO code)
Part-of-Speech Universal Dependencies tl, af, ru, nl, it, de, es, bg, pt, fr, te, et, el, fi,
Token-level
Tagging (POS) v2.5 (Nivre et al., 2020) hu, mr, kk, hi, tr, eu, id, fa, ur, he, ar, ta, vi,
ko, th, zh, yo, ja
Named Entity WikiAnn (Rahimi et al., nl, pt, bg, it, fr, hu, es, el, vi, fi, et, af, bn, de,
Recognition (NER) 2019) tr, tl, hi, ka, sw, ru, mr, ml, jv, fa, eu, ko, ta,
ms, he, ur, kk, te, my, ar, id, yo, zh, ja, th
Sequence-Level Natural Language XNLI (Conneau et al., es, bg, de, fr, el, vi, ru, zh, tr, th, ar, hi, ur,
Inference (NLI) 2018) sw
Question Answering (QA) TyDiQA (Clark et al., 2020) id, fi, te, ar, ru, sw, bn, ko

Table 1: Tasks and Datastes: D E M U X is applied across tasks of varying complexity, as elucidated in Q1: §4.

Single Target Multi-Target

Dataset
HP MP LP Geo LPP
UDPOS French Turkish Urdu Telugu, Marathi, Urdu Arabic, Hebrew, Japanese, Korean,
Chinese, Persian, Tamil, Vietnamese, Urdu
NER French Turkish Urdu Indonesian, Malay, Arabic, Indonesian, Malay, Hebrew,
Vietnamese Japanese, Kazakh, Malay, Tamil, Telugu,
Thai, Yoruba, Chinese, Urdu
XNLI French Turkish Urdu Bulgarian, Greek, Turkish Arabic, Thai, Swahili, Urdu, Hindi
TyDiQA Finnish Arabic Bengali Bengali, Telugu Swahili, Bengali, Korean

Table 2: Target language configurations. We run five experiments for each model and task, with the language sets
above as targets (details in §4.2). All languages mentioned in Table 1 make up the source set, except the chosen
target languages for a particular configuration.

Q3) How do the benefits of D E M U X vary across From these, we choose languages that have similar-
different MultiLMs? (§4.3) ities with the source set across different linguistic
dimensions (obtained using lang2vec (Littell et al.,
4.1 Task and Dataset Selection
2017)), to study the role of typological similarity
We have three distinct task types, based on the label for different tasks.
format. We remove duplicates from each dataset
Multi-target: Here, we envision two scenarios:
to prevent selecting multiple copies of the same
Geo: Mid-to-low performing languages in geo-
instance. Dataset details can be found in Table 1.
graphical proximity are chosen. From an applica-
4.2 Source and Target Language Selection tion perspective, this would allow one to improve a
We experiment with the zero-shot case of disjoint MultiLM for an entire geographical area.
source and target languages, i.e., the unlabelled LPP: All low-performing languages are pooled,
source pool contains no data from target languages. to test whether we can collectively enhance the
The train and validation splits constitute the unla- MultiLM’s performance across all of them.
belled source or target data, respectively. Evalua-
tion is done on the test split for each target language. 4.3 Model Selection
With Q2) in mind, we experiment with five target We test D E M U X across multiple MultiLMs: XLM-
settings (Table 2): R (Conneau et al., 2019), InfoXLM (Chi et al.,
Single-target: We partition languages into three 2020), and RemBERT (Chung et al., 2020). All
equal tiers based on zero-shot performance post models have a similar number of parameters
fine-tuning on English: high-performing (HP), mid- (∼550M-600M), and support 100+ languages.
performing (MP) and low-performing (LP), and XLM-R is trained on monolingual corpora from
choose one language from each, guided by two CC-100 (Conneau et al., 2019), InfoXLM is trained
factors. First, we select languages that are com- to maximize mutual information between multi-
mon across multiple datasets, to study how data lingual texts, and RemBERT is a deeper model,
selection for the same language varies across tasks. that reallocates input embedding parameters to the
Transformer layers. 1% absolute delta from the best-performing base-
line.
4.4 Baselines How does D E M U X fare on multilingual target
We include a number of baselines to compare our pools? We observe consistent gains given mul-
strategies against: tilingual target pools as well (Geo and LPP). We
1) RANDOM: In each round, a random subset of b believe this is enabled by the language-independent
data points from Xs is selected. design of our strategies, which makes annotation
2) EGALITARIAN: An equal number of randomly decisions at a per-instance level. This has important
selected data points from the unlabeled pool for consequences, since this would enable researchers,
each language, i.e. |xs | = b/|Ls |; ∀xs ∈ Xs is like those at Company Y, to better models for all
chosen. Debnath et al. (2021) demonstrate that this the languages that they care about.
outperforms a diverse set of alternatives. Does the model select data from the same lan-
3) LITMUS: LITMUS (Srinivasan et al., 2022) is a guages across tasks? No! We find that selected
tool to make performance projections for a fine- data distributions vary across tasks for the same
tuned model, but can also be used to generate data target languages. For example, when the target lan-
labeling plans, based on the predictor’s projections. guage is Urdu, D E M U X chooses 70-80% of sam-
We only run this for XLM-R since the tool requires ples from Hindi for NLI and POS, but prioritizes
past fine-tuning performance profiles, and XLM-R Farsi and Arabic (35-45%) for NER. Despite Hindi
is supported by default. and Urdu’s syntactic, genetic, and phonological
4) GOLD: This involves training on data from the similarities as per lang2vec, their differing scripts
target languages itself. Given all other strategies are underscore the significance of script similarity in
zero-shot, we expect GOLD to out-perform them and NER transfer. This also proves that analysing data
help determine an upper bound on performance. selected by D E M U X can offer linguistic insights
into the learned task-specific representations.
4.5 Fine-tuning Details
Method HP MP LP Geo LPP
We first fine-tune all MultiLMs on English (EN-FT)
EN-FT 80.0 79.5 65.6 61.0 45.8
and continue fine-tuning on data selected using GOLD 90.1 92.8 94.5 81.2 73.7
XLM-R

D E M U X, similar to Lauscher et al. (2020); Ku- BASEegal 85.4 87.6 84.0 80.6 62.8
mar et al. (2022). We experiment with a budget D E M U Xknn 87.8 89.2 85.8 82.4 62.3
of 10,000 examples acquired in five AL rounds, ∆base 2.4 1.6 1.8 1.8 -0.5
except for TyDiQA, where our budget is 5,000 ex- EN-FT 80.5 82.8 65.4 64.2 44.8
GOLD 90.0 92.8 94.6 83.5 74.9
InfoXLM

amples (TyDiQA is of the order of 35-40k samples

BASEegal 84.0 87.6 83.2 80.9 63.4
overall across ten languages, and this is to ensure D E M U Xknn 87.4 89.2 85.5 82.2 64.2
fair comparison with our gold strategy). For each ∆base 3.4 1.6 2.2 1.3 0.8
model, we first obtain EN-FT and continually fine-
EN-FT 78.8 80.2 55.7 61.1 48.4
tune using D E M U X. Hyperparameter details are
RemBERT

GOLD 89.4 92.1 93.5 79.8 70.1

given in §A.3, and all results are averaged across BASEegal 84.6 86.8 82.3 79.2 59.8
three seeds: 2, 22, 42. We fine-tune using a fixed D E M U Xknn 87.1 89.0 85.7 79.8 62.1
number of epochs without early stopping, given the ∆base 2.5 2.2 3.4 0.6 2.3
lack of a validation set in our setup (we assume no
Table 3: PAN-X Results (F1): We observe gains across
labelled target data).
all models and KNN-UNCERTAINTY performs best. ∆base
represents the delta from baseline.
5 Results
How does D E M U X perform overall? We present Which strategies work well across different task
results for NER, POS, NLI and QA in Tables 3, types? Our hybrid strategy, which picks uncer-
4, 5 and 6, respectively. In summary, the best- tain points in the local neighborhood of target
performing strategies outperform the best perform- points, performs best for token-level tasks, whereas
ing baselines in 84% of the cases, with variable globally uncertain points maximize performance
gains dependant on the task, model and target lan- for NLI and QA. For NLI, both AVERAGE-DIST
guages. In the remaining cases, the drop is within and UNCERTAINTY outperform baselines, the for-
 Method HP MP LP Geo LPP Method HP MP LP Geo LPP
EN-FT 81.7 75.5 71.5 80.2 62.2 EN-FT 81.8 77.3 69.9 80.1 73.4
XLM-R

GOLD 95.6 81.2 93.2 91.8 88.2 GOLD 81.6 79.5 70.3 81.6 76.0

XLM-R
BASEegal 87.1 79.6 88.4 85.7 68.9 BASEegal 81.6 78.8 73.0 80.9 75.6
D E M U Xknn 87.5 80.1 90.1 86.1 70.9 D E M U Xavg 83.7 79.9 75.3 82.2 77.1
∆base 0.4 0.5 1.7 0.4 2.0 ∆base 2.1 1.1 2.3 1.3 1.5
∆gold 2.1 0.4 5.0 0.6 1.1
EN-FT 79.6 74.0 59.0 73.6 58.2
InfoXLM

GOLD 95.7 81.4 93.3 92.0 88.7 EN-FT 81.9 77.3 68.8 79.8 71.5
BASEegal 88.0 79.4 88.8 86.3 67.8 GOLD 83.6 80.6 73.7 82.4 77.7

InfoXLM
D E M U Xknn 87.8 79.5 90.4 86.0 66.8 BASEegal 83.7 79.8 74.6 81.5 77.3
D E M U Xavg 84.8 80.8 75.9 83.1 77.8
∆base -0.3 0.1 1.6 -0.3 -1.0
∆base 1.1 1.0 1.3 1.6 0.5
EN-FT 72.9 71.1 50.6 66.1 55.7
∆gold
RemBERT

1.2 0.2 2.2 0.7 0.1

GOLD 95.1 80.8 92.3 91.2 86.8
BASEegal 86.9 78.1 85.8 83.8 67.8 EN-FT 83.1 73.9 52.5 77.8 63.0

RemBERT
D E M U Xknn 87.4 77.7 88.2 84.2 68.0 GOLD 81.1 73.3 63.1 76.0 67.5
BASEegal 80.0 75.3 63.9 76.4 67.9
∆base 0.5 -0.3 2.4 0.4 0.2
D E M U Xavg 81.7 76.1 67.6 78.6 70.9

Table 4: UDPOS Results (F1): We observe modest gains ∆base 1.7 0.8 3.7 2.2 3.0
∆gold 0.6 2.8 4.5 2.6 3.4
for a 10k budget, but higher gains for lower budgets (§6)
.
Table 5: XNLI Results (F1): Here we even surpass the
gold standard of in-language finetuning. Details in §5.

including multiple tasks and testing multiple strate-

gies for each, we provide guidance to everyday
practitioners on the best strategy, given a task.
How does performance vary across models? For
all models, the EN-FT performance is similar, ex-
cept for UDPOS, where XLM-R is consistently
better than InfoXLM and RemBERT. We see this
play a role in the consistent gains provided by D E -
M U Xacross languages for XLM-R, as opposed to
RemBERT and InfoXLM. We hypothesize that the
better a model’s representations, the better our mea-
Figure 3: Performance across different AL rounds for
sures of distance and uncertainty, which inherently
XLM-R: Semantic similarity with target matters most
for token-level tasks, but uncertainty gains importance improves data selection.
for NLI and QA. (AVG-DIST picks maximally uncertain
points for NLI due to the nature of the dataset (§5). 6 Further Analysis
What is the minimum budget for which we can
mer proving more effective. On further analysis, observe gains in one AL round? To deploy D E -
we find that this is an artifact of the the nature of the M U X in resource-constrained settings, we test its
dataset which is balanced across three labels, and applicability in low-budget settings, ranging from
is strictly parallel. This makes AVERAGE-DIST se- 5,10,50,100,250,500,1000, acquired using the EN-
lect high-uncertainty points at decision boundaries’ FT model only. As shown in Figure 4, we observe
centroid, as visualized in Figure 2a. Our finding of gains across all budget levels. Notably, we observe
different strategies working well for different tasks gains of up to 8-11 F1 points for token-level tasks,
is consistent with past works. For example, Settles and 2-5 F1 points for NLI and QA, for most lower
and Craven (2008) find information density (identi- budgets (5-100 examples). These gains diminish as
fying semantically similar examples), to work best the budget increases. For complex tasks like NLI
for sequence-labeling tasks; Marti Roman (2022) and QA, semantic similarity with the target holds
discuss how uncertainty-based methods perform importance when the budgets is below 500 exam-
best for QA; and Kumar et al. (2022) also rec- ples, but picking globally uncertain points gains
ommend task-dependant labeling strategies. By precedence for larger budgets.
 UDPOS: Urdu PAN-X: Urdu
90
90 80
Performance

Performance
70
80
60
70 50
40
60
5

10

50

100

250

500

5
1000

10

50

100

250

500

1000
Budget Budget
XNLI: Urdu TyDiQA: Arabic Strategies
75 85 gold
egalitarian
70 knn
80 average
Performance

Performance
65 uncertain
75
60
70
55
5

10

50

100

250

500

5
1000

10

50

100

250

500

1000
Budget Budget

Figure 4: Multiple budgets, one AL round: We experiment with low-budgets acquired using the EN-FT model. We
observe gains of up to 8-11 F1 over baselines for 5-100 examples, with a trend of diminishing gains given larger
budgets. All runs averaged across three seeds (2, 22, 42).

Method HP MP LP Geo LPP maintaining this distribution, performance still de-

EN-FT 78.9 73.2 79.9 80.7 78.5 clines (§A.4), indicating that precise datapoint se-
XLM-R

GOLD 81.2 83.8 83.7 84.7 81.0 lection in identified transfer languages is vital.
BASEegal 79.9 81.7 79.6 81.1 78.7
D E M U Xunc 80.8 82.9 80.3 81.0 77.8
7 Related Work
∆base 0.9 1.2 0.7 -0.1 -0.9
EN-FT 77.6 75.4 82.2 81.9 78.8 Multilingual Fine-tuning: Traditionally models
InfoXLM

GOLD 80.7 85.0 86.3 87.1 81.1

BASEegal 80.5 82.3 82.2 82.3 78.1
were fine-tuned on English, given the availability
D E M U Xunc 81.8 84.1 80.8 82.6 77.8 of labeled data across all tasks. However, signifi-
∆base 1.3 1.8 -1.4 0.3 -0.3 cant transfer gaps were observed across languages
EN-FT 79.7 73.0 82.9 78.0 78.2
(Hu et al., 2020) leading to the emergence of
RemBERT

GOLD 78.4 80.1 86.7 84.4 80.5 two research directions. The first emphasizes the
BASEegal 81.3 78.9 82.8 76.5 75.3 significance of using few-shot target language data
D E M U Xunc 82.7 80.2 80.6 78.0 76.1
(Lauscher et al., 2020) and the development of
∆base 1.4 1.3 -2.2 1.5 0.8
strategies for optimal few-shot selection (Kumar
et al., 2022; Moniz et al., 2022). The second
Table 6: TyDiQA Results (F1): UNCERTAINTY works
best here. Despite TyDiQA being composed of typo- focuses on choosing the best source languages for
logically diverse languages and being extremely small a target, based on linguistic features (Lin et al.,
(35-40k samples), we observe modest gains across mul- 2019) or past model performances (Srinivasan
tiple configs. et al., 2022). Discerning a globally optimal transfer
language however, has been largely ambiguous
(Pelloni et al., 2022) and the language providing
Do the selected datapoints matter or does follow- for highest empirical transfer is at times inex-
ing the language distribution suffice? D E M U X plicable by known linguistic relatedness criteria
not only identifies transfer languages but also se- (Pelloni et al., 2022; Turc et al., 2021). By making
lects specific data for labeling. To evaluate its im- decisions at a data-instance level rather than a
portance, we establish the language distribution language level, D E M U X removes the reliance
of data selected using D E M U X and randomly se- on linguistic features and sidesteps ambiguous
lect datapoints following this distribution. Despite consensus on how MultiLMs learn cross-lingual
relations, while prescribing domain-relevant inference on all of the source data; which can be
instances to label. time-consuming. However, one can run parallel
CPU-inference which greatly reduces latency.
Active learning for NLP: AL has seen wide adop-
tion in NLP, being applied to tasks like text classifi- Aprior Model Selection: We require knowing the
cation (Karlos et al., 2012; Li et al., 2013), named model apriori which might mean a different label-
entity recognition (Shen et al., 2017; Wei et al., ing scheme for different models. This a trade-off
2019; Erdmann et al., 2019), and machine transla- we choose in pursuit of better performance for the
tion (Miura et al., 2016; Zhao et al., 2020), among chosen model, but it may not be the most feasible
others. In the multilingual context, past works (Mo- solution for all users.
niz et al., 2022; Kumar et al., 2022; Chaudhary
et al., 2019) have applied AL to selectively label Refinement to the hybrid approach: Our hy-
data in target languages. However, they do not con- brid strategy picks the most uncertain points in
sider cases with unknown overlap between source the neighborhood of the target. However, its cur-
and target languages. This situation, similar to a rent design prioritizes semantic similarity with the
multi-domain AL setting, is challenging as data target over global uncertainty, since we first pick
selection from the source languages may not prove top-k neighbors and prune this set based on uncer-
beneficial for the target (Longpre et al., 2022). tainty. However, it will be interesting to experiment
with choosing globally uncertain points first and
8 Conclusion then pruning the set based on target similarity. For
In this work, we introduce D E M U X, an end-to- NLI and QA, we observe that globally uncertain
end framework that selects data to label from vast points help for higher budgets but choosing nearest
pools of unlabelled multilingual data, under an an- neighbors helps most for lower budgets. Therefore,
notation budget. D E M U X’s design is language- this alternative may work better for these tasks, and
agnostic, making it viable for cases where source is something we look to explore in future work.
and target data do not overlap. We design three
10 Acknowledgements
strategies drawing from AL principles that encom-
pass semantic similarity with the target, uncertainty, We thanks members of the Neulab and COMEDY,
and a hybrid combination of the two. Our strate- for their invaluable feedback on a draft of this pa-
gies outperform strong baselines for 84% of target per. This work was supported in part by grants from
language configurations (including multilingual tar- the National Science Foundation (No. 2040926),
get sets) in the zero-shot case of disjoint source Google, Two Sigma, Defence Science and Technol-
and target languages, across three models and four ogy Agency (DSTA) Singapore, and DSO National
tasks: NER, UDPOS, NLI and QA. We find that Laboratories Singapore.
semantic similarity with the target mostly benefits
token-level tasks, while picking uncertain points
gains precedence for complex tasks like NLI and References
QA. We further analyse D E M U X’s applicability in
Aditi Chaudhary, Jiateng Xie, Zaid Sheikh, Graham
low-budget settings and observe gains of up to 8-11 Neubig, and Jaime G Carbonell. 2019. A little an-
F1 points for some tasks, with a trend of dimin- notation does a lot of good: A study in bootstrap-
ishing gains for larger budgets. We hope that our ping low-resource named entity recognizers. arXiv
work helps improve the capabilities of MultiLMs preprint arXiv:1908.08983.
for desired languages, in a cost-efficient way. Zewen Chi, Li Dong, Furu Wei, Nan Yang, Sak-
sham Singhal, Wenhui Wang, Xia Song, Xian-Ling
9 Limitations Mao, Heyan Huang, and Ming Zhou. 2020. In-
foxlm: An information-theoretic framework for
With D E M U X’s wider applicability across lan- cross-lingual language model pre-training. arXiv
guages come a few limitations as we detail below: preprint arXiv:2007.07834.

Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin

Inference on source data: D E M U X relies on Johnson, and Sebastian Ruder. 2020. Rethinking
model representations and its output ditribution embedding coupling in pre-trained language models.
for each example. This requires us to run arXiv preprint arXiv:2010.12821.
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan David D Lewis. 1995. A sequential algorithm for train-
Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and ing text classifiers: Corrigendum and additional data.
Jennimaria Palomaki. 2020. Tydi qa: A benchmark In Acm Sigir Forum, volume 29, pages 13–19. ACM
for information-seeking question answering in typo- New York, NY, USA.
logically diverse languages. Transactions of the As-
sociation for Computational Linguistics, 8:454–470. Shoushan Li, Yunxia Xue, Zhongqing Wang, and
Guodong Zhou. 2013. Active learning for cross-
David A Cohn, Zoubin Ghahramani, and Michael I domain sentiment classification. In Twenty-Third
Jordan. 1996. Active learning with statistical models. International Joint Conference on Artificial Intelli-
Journal of artificial intelligence research, 4:129–145. gence. Citeseer.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li,
Vishrav Chaudhary, Guillaume Wenzek, Francisco Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junx-
Guzmán, Edouard Grave, Myle Ott, Luke Zettle- ian He, Zhisong Zhang, Xuezhe Ma, et al. 2019.
moyer, and Veselin Stoyanov. 2019. Unsupervised Choosing transfer languages for cross-lingual learn-
cross-lingual representation learning at scale. arXiv ing. arXiv preprint arXiv:1905.12688.
preprint arXiv:1911.02116.
Patrick Littell, David R Mortensen, Ke Lin, Katherine
Alexis Conneau, Guillaume Lample, Ruty Rinott, Ad- Kairis, Carlisle Turner, and Lori Levin. 2017. Uriel
ina Williams, Samuel R Bowman, Holger Schwenk, and lang2vec: Representing languages as typological,
and Veselin Stoyanov. 2018. Xnli: Evaluating cross- geographical, and phylogenetic vectors. In Proceed-
lingual sentence representations. arXiv preprint ings of the 15th Conference of the European Chap-
arXiv:1809.05053. ter of the Association for Computational Linguistics:
Volume 2, Short Papers, volume 2, pages 8–14.
Arnab Debnath, Navid Rajabi, Fardina Fathmiul Alam,
and Antonios Anastasopoulos. 2021. Towards Shayne Longpre, Julia Reisler, Edward Greg Huang,
more equitable question answering systems: How Yi Lu, Andrew Frank, Nikhil Ramesh, and Chris
much more data do you need? arXiv preprint DuBois. 2022. Active learning over multiple do-
arXiv:2105.14115. mains in natural language tasks. arXiv preprint
Mark Dredze and Koby Crammer. 2008. Online meth- arXiv:2202.00254.
ods for multi-domain learning and adaptation. In
Salvador Marti Roman. 2022. Active learning for ex-
Proceedings of the 2008 Conference on Empirical
tractive question answering.
Methods in Natural Language Processing, pages 689–
697. Andrew McCallum, Kamal Nigam, et al. 1998. Em-
Alexander Erdmann, David Joseph Wrisley, Christo- ploying em and pool-based active learning for text
pher Brown, Sophie Cohen-Bodénès, Micha Elsner, classification. In ICML, volume 98, pages 350–358.
Yukun Feng, Brian Joseph, Béatrice Joyeux-Prunel, Citeseer.
and Marie-Catherine de Marneffe. 2019. Practical,
Akiva Miura, Graham Neubig, Michael Paul, and
efficient, and customizable active learning for named
Satoshi Nakamura. 2016. Selecting syntactic, non-
entity recognition in the digital humanities. Associa-
redundant segments in active learning for machine
tion for Computational Linguistics.
translation. In Proceedings of the 2016 Conference
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- of the North American Chapter of the Association
ham Neubig, Orhan Firat, and Melvin Johnson. for Computational Linguistics: Human Language
2020. Xtreme: A massively multilingual multi-task Technologies, pages 20–29.
benchmark for evaluating cross-lingual generalisa-
tion. In International Conference on Machine Learn- Joel Ruben Antony Moniz, Barun Patra, and Matthew R
ing, pages 4411–4421. PMLR. Gormley. 2022. On efficiently acquiring anno-
tations for multilingual models. arXiv preprint
Stamatis Karlos, Nikos Fazakis, Sotiris Kotsiantis, and arXiv:2204.01016.
Kyriakos Sgarbas. 2012. An empirical study of active
learning for text classification. Advances in Smart Joakim Nivre, Marie-Catherine de Marneffe, Filip
Systems Research, 6(2):1. Ginter, Jan Hajič, Christopher D Manning, Sampo
Pyysalo, Sebastian Schuster, Francis Tyers, and
Shanu Kumar, Sandipan Dandapat, and Monojit Choud- Daniel Zeman. 2020. Universal dependencies v2: An
hury. 2022. " diversity and uncertainty in modera- evergrowing multilingual treebank collection. arXiv
tion" are the key to data selection for multilingual preprint arXiv:2004.10643.
few-shot transfer. arXiv preprint arXiv:2206.15010.
Karl Pearson. 1903. I. mathematical contributions to the
Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and theory of evolution.—xi. on the influence of natural
Goran Glavaš. 2020. From zero to hero: On selection on the variability and correlation of organs.
the limitations of zero-shot cross-lingual transfer Philosophical Transactions of the Royal Society of
with multilingual transformers. arXiv preprint London. Series A, Containing Papers of a Mathemat-
arXiv:2005.00633. ical or Physical Character, 200(321-330):1–66.
Olga Pelloni, Anastassia Shaitarova, and Tanja A Appendix
Samardzic. 2022. Subword evenness (sue) as a pre-
dictor of cross-lingual transfer to low-resource lan- A.1 Distance and Uncertainty Measurements
guages. In Proceedings of the 2022 Conference on
Empirical Methods in Natural Language Processing,
All of our strategies our based on reliable distance
pages 7428–7445. and uncertainty measures. Once these are estab-
lished, it is easy to extend D E M U X to other tasks
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Mas-
and models. An overview of how these are mea-
sively multilingual transfer for ner. arXiv preprint
arXiv:1902.00193. sured for the tasks in our study can be found in
Figure 5. These are formally described in the pa-
Tobias Scheffer, Christian Decomain, and Stefan Wro-
per, in Section 3.
bel. 2001. Active hidden markov models for infor-
mation extraction. In Advances in Intelligent Data
Analysis: 4th International Conference, IDA 2001
A.2 Uncertainty Details
Cascais, Portugal, September 13–15, 2001 Proceed- For token-level tasks, we investigate two strategies.
ings 4, pages 309–318. Springer. First, we employ the Mean Normalized Log Proba-
Andrew I Schein and Lyle H Ungar. 2007. Active learn- bility (MNLP) (Shen et al., 2017) method, which has
ing for logistic regression: an evaluation. been demonstrated as an effective uncertainty mea-
B Settles. 2009. Active learning literature survey (com- sure for Named Entity Recognition (NER). This
puter sciences technical report 1648) university of approach selects instances for which the log proba-
wisconsin-madison. Madison, WI, USA: Jan. bility of model prediction, normalized by sequence
Burr Settles and Mark Craven. 2008. An analysis of length, is the lowest. Formally,
active learning strategies for sequence labeling tasks.
X
X∗ = argmin X ∈ S b MNLP (xs )
In proceedings of the 2008 conference on empirical
methods in natural language processing, pages 1070– xs ∈X
1079. Where
Yanyao Shen, Hyokun Yun, Zachary C Lipton, Yakov |x|
1 X
Kronrod, and Animashree Anandkumar. 2017. Deep MNLP (x) = log pic1 (x)
active learning for named entity recognition. arXiv |x|
i=1
preprint arXiv:1707.05928.
Anirudh Srinivasan, Gauri Kholkar, Rahul Kejriwal,
In this equation, |x| denotes the sequence length
Tanuja Ganu, Sandipan Dandapat, Sunayana Sitaram, of the unlabeled sample x, and pic1 (x) represents
Balakrishnan Santhanam, Somak Aditya, Kalika Bali, the predicted probability of the most probable class
and Monojit Choudhury. 2022. Litmus predictor: An for the ith token in the sequence.
ai assistant for building reliable, high-performing Concurrently, we also explore margin-based un-
and fair multilingual nlp systems. In Proceedings
of the AAAI Conference on Artificial Intelligence, certainty techniques (MARGIN-MIN). For each token
volume 36, pages 13227–13229. in the sequence, we compute the margin as the dif-
ference between the probabilities of the top two
Iulia Turc, Kenton Lee, Jacob Eisenstein, Ming-Wei
Chang, and Kristina Toutanova. 2021. Revisiting the classes. Then, we assign the minimum margin
primacy of english in zero-shot cross-lingual transfer. across all tokens as the sequence margin score and
arXiv preprint arXiv:2106.16171. choose sequences with the smallest margin score.
Qiang Wei, Yukun Chen, Mandana Salimi, Joshua C Formally,
Denny, Qiaozhu Mei, Thomas A Lasko, Qingxia X
Chen, Stephen Wu, Amy Franklin, Trevor Cohen, X∗ = argmin X ∈ S b MARGIN-MIN (xs )
et al. 2019. Cost-aware active learning for named en- xs ∈X
tity recognition in clinical text. Journal of the Ameri-
Where
can Medical Informatics Association, 26(11):1314–
1322. |x|
MARGIN-MIN (x) = min pic1 (x) − pic2 (x)


Zhisong Zhang, Emma Strubell, and Eduard Hovy. 2022. i=1
A survey of active learning for natural language pro-
cessing. arXiv preprint arXiv:2210.10109. We eventually choose the margin based technique
given better performance for both token level tasks.
Yuekai Zhao, Haoran Zhang, Shuchang Zhou, and Zhi-
hua Zhang. 2020. Active learning approaches to A.3 Fine-tuning Details
enhancing neural machine translation. In Findings
of the Association for Computational Linguistics: As mentioned in the text, we first fine-tune each
EMNLP 2020, pages 1796–1806. model on EN using the hyperparameters mentioned
 Distance Distance Distance
Source CLS Source AVG{ } Source CLS
L2 Distance L2 L2 Distance
Target CLS Target AVG{ } Target CLS

Uncertainty Uncertainty Uncertainty

SORTED PROB. SORTED PROB.

LOG-SOFTMAX
Linear Layer + LOG-SOFTMAX
–

LOG-SOFTMAX
…
CLS CLF 0.5 – 0.4 …
CLF – … Start MAX{ }
MARGIN End MAX{ }
0.1
– …
… MIN{[MARGINS]} +

Model
CLS A1 A1 A2 A2 A2 A3 … SEP B1 B2 B2 B3 B3 B3 …

Multilingual Pre-trained Model

CLS A1 A1 A2 A2 A2 A3 … SEP B1 B2 B2 B3 B3 B3 …

Input

Sequence Level Tasks Token Level Tasks Question Answering

[CLS] Sentence A [SEP] Sentence B [CLS] Sentence A [CLS] Question [SEP] Context

Figure 5: An overview of how distance and uncertainty are measured in our setup. A1 , A2 , A3 denote three words
in Sentence A that are tokenized into 2, 3, and 1 subword, respectively.

Single Target Multi-Target

Dataset
HP MP LP Geo LP Pool
UDPOS fr:4938 tr:984 ur:545 te:130, mr:46, ur:545 ar:906, he:484, ja:511, ko:3014, zh:2689,
fa:590, ta:80, vi:794, ur:545
NER fr:9300 tr:9497 ur:944 id:7525, my:100, vi:8577 ar:9319,id:7525, my:100, he:9538, ja:9641,
kk:910, ms:761, ta:965, te:939, th:9293, yo:93,
zh:9406, ur:944
XNLI fr:2490 tr:2490 ur:2490 bg:2490, el:2490, tr:2490 ar:2490, th:2490 ,sw:2490, ur:2490, hi:2490
TyDiQA fi:1371 ar:2961 bn:478 bn:478, te:1113 sw:551, bn:478, ko:325

Table 7: Number of unlabelled target examples used in each configuration. This is the size of the validation set.

in the paper. Hyperparameters are in Table 8 and

we report average results across three seeds: 2, 22,
Model Dataset LR Epochs 42. For UDPOS, we include all languages except
NER 2e-5 10
Tagalog, Thai, Yoruba and Kazakh, because they
UDPOS 2e-5 10 do not have training data for the task5 .
XLM-R
XNLI 5e-6 10
TyDiQA 1e-5 3 A.4 Detailed Results
NER 2e-5 10 The detailed results for the first ablation study
UDPOS 2e-5 10
InfoXLM where we test D E M U X for multiple budgets in
XNLI 5e-6 10
TyDiQA 1e-5 3 one AL round, can be found in Table 11. Results
NER 8e-6 10 for the second ablation, where we fine-tune models
UDPOS 8e-6 10 on randomly selected data that follows the same
RemBERT
XNLI 8e-6 10
TyDiQA 1e-5 3 language distribution as D E M U X, can be found in
Table 10.
Table 8: Hyperparameter Details.
5
https://huggingface.co/datasets/xtreme/blob/main/xtreme.py
#L914
 Single Target Multi-Target
Dataset
HP MP LP Geo LP Pool
fr tr ur te, mr, ur ar, he, ja, ko, zh, fa, ta,
UDPOS
vi, ur
it: 3434 et:799, fa:598, de:5095, eu:84, ar:93, bg:68, hi:830
ja:890, vi:694 hi:3323, mr:92, et:3197,eu:84, fr:65,
nl:556 hi:3323, ja:890
fr tr ur id, my, vi ar, id, my, he, ja, kk, ms,
NER
ta, te, th, yo, zh, ur
it: 572 he: 4582; mr: fa: 4007, ms: bn:1000, he:572, bn:62, ko:4699,
1015, nl: 4393 79, ta: 788, vi: ko:469, ms:79 ml:4306
62
fr tr ur bg, el, tr ar, th, sw, ur, hi
XNLI
ur:781, zh:6250 ru:781, ur:6250 hi:6250 ru:781, ur:6250 bg:6250, de:781, fr:781,
vi:781
fi ar bn bn, te sw, bn, ko
TyDiQA
ar:185, bn:119, bn:14, fi:2742, ar:740 ar:46, fi:2742, id:570, -
id:142, ko:2, ko:20, sw:4, ko:325, ru:5, sw:8
ru:1, sw:17, te:2
te:4450

Table 9: LITMUS prescribed annotation budget: LITMUS prescribes how many samples to select from each
language. We select a random sample of data following the prescribed annotation.

AL Round
Dataset Strategy
1 2 3 4 5
SR 81.1 81.9 84.1 85.1 84.0
PAN-X DEMUX 83.2 83.1 84.1 85.8 85.2
∆ 2.0 1.2 0.0 0.7 1.2
SR 89.9 89.3 89.5 90.0 89.8
UDPOS DEMUX 89.7 89.8 89.9 89.5 90.1
∆ -0.2 0.5 0.5 -0.4 0.3
SR 73.3 73.8 73.8 73.8 73.9
XNLI DEMUX 74.5 74.7 75.5 75.3 75.3
∆ 1.2 0.9 1.6 1.5 1.4
SR 80.6 81.5 81.5 82.0 81.7
TyDiQA DEMUX 82.8 83.2 83.2 83.5 83.8
∆ 2.2 1.7 1.8 1.5 2.1

Table 10: Detailed results: SR stands for SAME-RATIO.

Same data distribution across languages but a random
subset of datapoints selected.
 Strategy
Dataset Budget
GOLD EGAL KNN-UNC AVG-DIST UNC
5 77.1 70.9 79.8 81.8 65.7
10 80.1 70.7 81.8 82.3 65.4
UDPOS 50 83.5 72.4 83.2 83.8 71.1
100 86.5 74.5 86.0 85.8 75.1
250 89.6 77.7 88.2 88.2 77.4
500 91.1 79.1 89.3 89.3 79.7
1000 92.2 81.5 89.3 89.5 82.2
5 64.3 54 64.9 40.2 40.8
10 71.8 52.7 58.4 44.2 47.9
50 77.1 61.6 74.5 65.4 46.8
PAN-X 100 79.2 67.6 80.5 70.2 37
250 82.9 76.7 82.4 76.7 61.1
500 85.7 80.1 84.5 82.5 73.6
1000 87.7 80.6 83.5 81.7 79.1
5 64.3 63.6 69 57.1 67.6
10 70.6 68.1 70.5 70.8 72.5
50 73.1 73.3 73.2 72.6 72.2
XNLI 100 72.9 71.2 72.7 71.1 73.8
250 71.5 72.4 72.4 72.9 73.9
500 73.3 72.8 72.3 72.4 72.6
1000 73.9 72.3 72.3 73.7 72.3
74.5 73.3 73.4 72.9 74.2 74.2
74.1 73.3 74.0 73.8 74.4 74.4
75.5 75.5 77.1 73.0 75.3 75.3
TyDiQA 77.6 74.6 80.2 73.7 77.8 77.8
80.6 74.9 80.6 76.8 78.8 78.8
82.2 78.1 81.1 78.7 81.6 81.6
82.8 79.9 80.4 80.8 81.5 81.5

Table 11: Detailed results: Multiple budgets, one AL round