│3

PISA 2021 Mathematics: A Broadened Perspective

1. The PISA 2021 Mathematics Strategic Advisory Group was established in March
2017 to provide overall direction as an input to subsequent framework development.
The group’s final report, below, proposes that the PISA Mathematics framework should
be significantly updated, through the introduction of six underpinning mathematical
concepts, four new content areas and a number of relevant 21st century skills. It ends with
a set of design principles to guide framework and item construction. The report was
discussed and supported by the PISA Strategic Development Group (SDG) in October
2017.
2. This document was presented to the PGB at its 44th meeting.
3. The Group’s members are as follows:

Name Country Title Field

SCHMIDT Bill (Chair) USA University Distinguished Professor, Michigan State University Statistics
GOOS Marilyn Australia Professor of Education, University of Queensland Algebra
WOLFE Richard Canada Professor Emeritus, OISE/U. Toronto Statistics
JUKK Hannes Estonia Lecturer, Institute of Maths and Statistics, University of Tartu Maths education
Associate Professor and Head, Maths and Maths education, National Institute of
GUAN Tay Eng Singapore Discrete maths
Education
Foundational
MARCIANAK Zbigniew Poland Professor, Mathematical Institute, University of Warsaw
maths
FERRINI-MUNDY Joan USA Deputy Director, National Science Foundation Calculus
McCALLUM William USA University Distinguished Professor of Mathematics, University of Arizona Geometry

4. Charles Fadel (Founder, Center for Curriculum Redesign and Chair, OECD BIAC
education group) has been retained to advise the Group on the role of mathematics in
emerging industries and sectors.
5. For most countries mathematical competencies are an expected outcome
of schooling. This has been true for a long time. Mathematical competencies initially
encompassed basic arithmetic skills, including adding, subtracting, multiplying,
and dividing whole numbers, decimals, and fractions; computing percentages;
and computing the area and volume of simple geometric shapes. In recent times,
the digitisation of many aspects of life, the ubiquity of data for making personal decisions
involving health and investments, as well as major societal decisions to address areas
such as climate change, taxation, governmental debt, population growth, spread
of pandemic diseases and the global economy, have reshaped what it means to be
mathematically competent and prepared to be a thoughtful, engaged, and reflective
citizen.
6. These critical issues as well as others that are facing societies throughout
the world all have a quantitative component to them. Understanding them, as well as
addressing them, at least in part, requires thinking mathematically. Such thinking is not
driven by the basic computational procedures described above, but by mathematical
and statistical reasoning, and it demands a reconsideration of what it means for all

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

4│

students to be competent in mathematics. Mathematical competencies become the new

mathematical literacy. They go beyond problem solving, to a deeper level, that
of mathematical reasoning, which provides the intellectual acumen behind problem
solving.
7. Today’s countries face new challenges in all areas of life, stemming from
the rapid deployment of computers and robots. For example, the vast majority of students
comprising the university freshman class of fall 2017 have always considered phones to
be mobile hand-held devices capable of sharing voice, texts, and images and accessing
the internet—capabilities seen as science fiction by many of their parents and certainly by
all of their grandparents (Beloit College, 2017). The recognition of the growing
contextual discontinuity between the last century and the future has prompted
a discussion around the development of 21st century skills in students (Ananiadou &
Claro, 2009; Fadel, Bialik & Trilling, 2015; NRC, 2012; Reimers & Chung, 2016).
8. It is this discontinuity that drives the need for education reform and the challenge
of achieving it. Periodically, educators, policy makers, and other stakeholders revisit
public education standards and policies. In the course of these deliberations new or
revised responses to two general questions are generated: 1) What do students need to
learn, and 2) Which students need to learn what? The most used argument in defence
of common mathematics education for all students is its usefulness in various practical
situations. However, this argument alone gets weaker with time – a lot of simple activities
have been automated. Not so long ago waiters in restaurants would multiply and add on
paper to calculate the price to be paid. Today they just press buttons on hand-held
devices. Not so long ago we were using printed timetables to plan travel – it required
a good understanding of the time axis and inequalities. Today we just make a direct
internet inquiry.
9. As to the question of “what to teach”, many misunderstandings arise from
the way mathematics is conceived. Some see mathematics as no more than a useful toolbox.
A clear trace of this approach can be found in school curricula in many countries. These
are often confined to a list of mathematics topics or procedures, with students asked to
practice a selected few, in predictable situations. This perspective on mathematics is far
too narrow for today’s world. It overlooks key features of mathematics that are growing in
importance.
10. Ultimately the answer to the two questions is that every student should learn
(and be given the opportunity to learn) to think mathematically, using mathematical
and statistical reasoning in conjunction with a small set of fundamental mathematics
concepts which themselves are not taught explicitly but are made manifest and reinforced
throughout a student’s mathematics learning experiences. This equips students with
a conceptual framework by which to address the quantitative dimensions of life in
the 21st century. This is the new mathematical literacy which includes problem solving
but goes beyond it to becoming mathematically competent.

Defining Mathematical Literacy

11. The PISA 2012 definition of mathematical literacy was as follows: An

individual’s capacity to formulate, employ and interpret mathematics in a variety
of contexts. It includes reasoning mathematically and using mathematical concepts,
procedures, facts and tools to describe, explain and predict phenomena. It assists
individuals to recognise the role that mathematics plays in the world and to make

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

 │5

the well-founded judgments and decisions needed by constructive, engaged and reflective
citizens (p 25, OECD, 2013a).
12. The Assessment and Analytical Framework document clarified the definition by
suggesting it reflects “A view of students as active problem solvers”. The document
further stated that “The focus of the language in the definition of mathematical literacy is
on active engagement in mathematics, and is intended to encompass reasoning
mathematically and using mathematical concepts, procedures, facts and tools in
describing, explaining and predicting phenomena”. It is important to note that
the definition not only focuses on the use of mathematics to solve real-world problems,
but also identifies mathematical reasoning as a central aspect of mathematical literacy.
13. The definition is represented pictorially in Figure 1. Further elaborated in
the model are the categories of mathematics content knowledge which students must
draw on both to formulate the problem by transforming the real world situation into
mathematical terms but also to solve the mathematics problem once formulated. Those
categories of mathematics content include: quantity, uncertainty and data, change
and relationships, and space and shape. Also specified in Figure 1 are the three contexts
PISA uses to define real-world situations: personal, societal and scientific.

Figure 1. A Model of Mathematical Literacy in Practice

14. Given this definition, in order for students to be mathematically literate they must
be able first to use their mathematics content knowledge to recognise the mathematical
nature of a situation (problem) encountered in the real world and then to formulate it in
mathematical terms. This transformation – from an ambiguous, messy, real-world
situation to a well-defined mathematics problem – is, perhaps, the critical component
of what it means to be mathematically literate. In pursuit of the “broadened perspective”
of mathematical literacy referred to in its title, this paper focuses on the delineation
of the mathematical competencies needed in this transformation process. Once
the transformation is successfully made, the resulting mathematics problem merely needs
to be solved using the mathematics concepts, algorithms and procedures taught in
schools. The final component in the PISA definition requires the student to evaluate

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

6│

the mathematics solution by interpreting the results within the original real-world
situation.
15. It is mathematical reasoning that provides the competencies needed to transform
the messy, real-world into the world of mathematics. Mathematical procedures can then
be used to solve the problem and arrive at an answer.

Evidence on the Role of Mathematical Reasoning in Problem Solving

16. In 2012, PISA incorporated measures of opportunity to learn (OTL) in the student
questionnaire to indicate the extent to which students had studied mathematics. This was
labelled formal mathematics OTL. Another set of items had students indicate how often
they were exposed to applied real-world problems in their classroom instruction and tests.
This was termed applied OTL.
17. Given the applied, real-world orientation of the PISA assessment, it was
hypothesised that applied OTL would be related to the PISA mathematics literacy test to
a greater extent than formal mathematics OTL. In fact both were statistically significant
in relation to the overall PISA score, as well as to the seven sub-scores, in most countries.
The surprising result, however, was that formal mathematics OTL demonstrated this
statistically significant relationship in every country, whereas applied OTL was
significant in fewer countries – only 79% of the 62 countries/economies that participated
in 2012 (see Table 1). In addition, the effect sizes were larger for formal mathematics
OTL than for applied OTL.

Table 1. Percentage of PISA Countries with Statistically Significant Relationships of OTL

to PISA Performance

Effect Literacy Change Quantity Space Data Employ Formulate Interpret

Applied -2.19* -2.52 -2.38 -2.02 -2.35 -2.12 -2.45 -2.49
Within-
Math 79% 73% 76% 76% 76% 79% 70% 76%
School
Formal 53.15 57.19 52.23 55.87 49.45 51.88 57.37 51.16
Level
Math 100% 100% 100% 100% 100% 100% 100% 100%

Applied -8.54 -8.32 -10.8 -7.67 -8.7 -6.61 -8.98 -10.17

Between-
Math 36% 42% 36% 33% 52% 42% 39% 45%
School
Formal 95.16 102.92 94.7 102.21 91.62 96.35 101.01 92.98
Level
Math 97% 100% 97% 97% 97% 97% 97% 97%

Either Level

Applied
79% 73% 76% 79% 82% 85% 76% 82%
Math
OTL
Formal
100% 100% 100% 100% 100% 100% 100% 100%
Math

Note: Average estimated coefficient for those countries with a significant relationship

18. This result prompts the question: why the strikingly strong and consistent
relationship of formal mathematics OTL to PISA mathematics literacy? Is it simply due
to studying more topics and procedures or is it something else? Again, it is striking that
this relationship is even stronger than that demonstrated by applied OTL although this

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

 │7

weaker relationship was still significant in most countries. In addition, applied OTL had
a non-linear relationship with PISA mathematics literacy: after a certain point it was
negatively related to the PISA score (see Figure 2). The linear relationship for formal
mathematics OTL is more straightforwardly interpretable – more OTL is related to
a greater literacy score.

Figure 2. The non-linear relationship of Applied Mathematics to performance illustrated for

four countries
PISA Mathematics Literacy vs. Applied Mathematics
550

530

510
Mathematics Literacy

490

470
Netherlands

Canada
450
France

United Kingdom

430
0 0.5 1 1.5 2 2.5 3
Applied Mathematics Index

19. The fact that students with greater formal mathematics OTL, i.e. opportunities to
learn advanced mathematics such as complex numbers and trigonometry, do better on
the PISA mathematics literacy test supports the hypothesis that mathematical reasoning is
a critical component of mathematical problem solving.
20. This hypothesis is especially reasonable given that the PISA 2012 mathematics
literacy items do not require knowledge of advanced mathematics such as complex
numbers or trigonometry.
21. In other words, it is not the specifics of the advanced mathematics topics studied
that are needed to solve the problems on the mathematics literacy test, but rather
the increased practice in applying mathematics and reasoning with mathematics.
22. The fact that more mathematics usually means more advanced mathematics, given
the hierarchical nature of the discipline, may not be the critical issue. What may well be
the critical issue is the greater opportunity to develop the way of thinking logically
and reasoning mathematically that is provided by the continuing study of mathematics.
This does not imply that this is the only way to acquire such reasoning. Rather, it is
important to understand what it is about mathematics that is central to the development
of mathematical reasoning and to incorporate this into the definition of mathematical
literacy, into mathematics instruction, and into the PISA assessment.

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

8│

The Broader Role of Mathematical Reasoning

23. Mathematical reasoning and solving real-world problems overlap in the core
activity of formulating the problem: transforming the messy nature of real-world
problems into tightly framed and well-defined mathematics. But there is an aspect to
mathematical reasoning which goes beyond solving real-world problems; it is also a way
of evaluating and making arguments, interpretations and inferences related to important
public policy debates that are, by their quantitative nature, best understood
mathematically and statistically.
24. We have argued that mathematical literacy comprises two related aspects:
mathematical reasoning, which plays a key role in being able to apply mathematics to
solve real-world problems. But mathematical reasoning goes beyond solving problems in
the traditional sense of the word to include making informed judgements about those
important family or societal issues which can be addressed mathematically. It is here
where mathematical reasoning contributes to the development of a select set
of 21st century skills.
25. In recognition of the concatenation of the above ideas together with the PISA
2012 model (Figure 1), we have titled this report “PISA 2021 Mathematics: A Broadened
Perspective.” We are proposing to broaden the 2012 model in three ways:
 By identifying four areas of emphasis – one under each of the four content
categories identified in Figure 1
 By elevating the importance of mathematical reasoning both for the role it plays
in problem solving, especially in the formulate stage of the model,
and the broader role of being an informed citizen around those important societal
issues involving quantitative information. We propose six fundamental concepts
that are crosscutting across all of mathematics and provide a foundation for
reasoning mathematically.
 By picking out those 21st century skills most closely related to the six
fundamental concepts.
26. Taken together, they allow us to go beyond problem solving in measuring
mathematical literacy (see Figure 3).

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

 │9

Figure 3. Broadened Model of Mathematical Literacy

Note: The mathematical content category topics listed in parentheses are subtopics from each of the content
categories that should receive greater emphasis given their relevance to important societal issues
and the nature of the new economy.

27. The three proposals as summarised in Figure 3 show how the 2021 assessment
builds directly from the original work done in the PISA 2012 study. Comparing Figure 3
with the original model (Figure 1) shows the increased emphasis on mathematical
reasoning so as to deepen the assessment of conceptual understanding as it relates to
mathematical literacy. Problem solving retains its place as an important aspect
of mathematical literacy but the new model goes beyond it to an even more foundational
aspect of mathematical literacy – that of mathematical and statistical reasoning.
28. The detail of the three proposals that follow in the next section should not be
taken as un-related to each other. Brought together they create not only a new vision for
the 2021 PISA assessment but a way for schooling to support their development. They
should neither be taught nor tested separately but in an integrated fashion. They become
the three entwined pillars supporting mathematical literacy. Mathematical reasoning
(Proposal I) and the six supporting concepts provide the means of addressing problems or
broader issues that can be addressed mathematically and increasingly in today’s complex
world those problems and issues will come from the four mathematics areas listed in
Proposal III. Ideally the reasoning and conceptual understanding of the six concepts as

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

10 │

applied in those areas will contribute to the development of a related set of 21st century
skills (Proposal II).

Proposal I: Mathematical Reasoning: A Challenging Opportunity for PISA

29. Mathematics is a science about objects and notions which are completely defined,
independent of their origin or nature. Once we isolate them in a particular context, they
become entities which can be analysed and transformed in ways using ‘mathematical
reasoning’ to obtain 100% sure and timeless conclusions. What is also important is that
those conclusions are impartial, without any need for validation by some authority. On
the other hand, statistics is a science about reasoning with uncertainty or put another way
statistics is the search for certainty in the midst of uncertainty.
30. The ability to reason logically and to present arguments in honest and convincing
ways is a skill which is becoming increasingly important in today’s world. This kind
of reasoning is useful far beyond mathematics, but it can be learned and practiced most
effectively within mathematics, just because it has the advantage of a fully-defined
context, which creates a comfortable training environment and under the assumed axioms
the experience of objective truth in a platonic sense.
31. Mathematical reasoning has two aspects, both important in today’s world. One is
deduction from clear assumptions, which is a characteristic feature of ‘pure’ mathematics.
The usefulness of this ability has been stressed above.
32. Another important dimension is probabilistic reasoning. At the logical level,
there is nowadays constant confusion in the minds of individuals between the possible
and the probable, leading many to fall prey to conspiracy theories or fake news. At
the more computational level, today’s world is increasingly complex and its multiple
dimensions are represented by terabytes of data. Making sense of these data is one
of the biggest challenges t h at humanity will face in the future. Our students should be
familiarised with the nature of such data and making decisions in the context
of variation.
33. The power of mathematics, from its very beginnings, lies in the ability
of reducing complex contexts to sets of simple basic principles. Euclid’s ‘Elements’
constituted the first spectacular success in this field; he was able to reduce all known
ancient geometry to conclusions from five simple assertions. Today’s mathematics
theories are no less successful (including the studies on chaos). Good mathematics
education should build the attitude for hunting for those ‘prime principles’ in well
designed, yet quite complicated contexts.
34. It is our contention that the use of mathematical reasoning, supported by a small
number of key concepts that undergird the specific content, skills, and algorithms
of school mathematics but also provide a structure in which those specifics are best
understood, is the core of mathematical literacy. It is these fundamental concepts that
provide the structure and support for mathematical and statistical reasoning. The six
fundamental concepts are as follows:

1. Number systems and their algebraic properties

35. Counting is one of the most basic and the oldest of human abstract activities.
The basic notion of Quantity may be the most pervasive and essential mathematical
aspect of engaging with, and functioning in, the world (OECD, 2015, p. 18). At the most

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

 │ 11

basic level it deals with the useful ability to compare cardinalities of sets of objects.
The ability to count usually involves rather small sets - in most languages, only a small
subset of numbers have names. When we assess larger sets, we engage in more complex
operations of estimating, rounding and applying orders of magnitude. Counting is very
closely related to another fundamental operation of classifying things, where the ordinal
aspect of numbers emerges. Quantification of attributes of objects, relationships,
situations and entities in the world is one of the most basic ways of conceptualising
the surrounding world (OECD, 2015).
36. Beyond counting, number is fundamental to measurement, which some would
argue is an essential practice in using mathematics to solve problems about our world. As
Lord Kelvin once claimed: “When you can measure what you are speaking about
and express it in numbers, you know something about it; but when you cannot measure it,
when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory
kind.” (Fey, On the Shoulders of Giants, National Academies, 1989).
37. This fundamental concept includes the basic concept of number, nested number
systems (e.g., whole numbers to integers to rationals to reals), the arithmetic of numbers,
and the algebraic properties that the systems enjoy. In particular, it is useful to understand
how progressively more expansive systems of numbers enable the solution
of progressively more complex equations. To use quantification efficiently, one has to be
able to apply not just numbers, but the number systems. Numbers themselves are
of limited relevance; what makes them into a powerful tool are the operations that we can
perform with them. As such, a good understanding of the operations of numbers is
the foundation of mathematical reasoning.
38. It is important to understand matters of representation (as symbols involving
numerals, as points on a number line, as geometric quantities, and by special symbols
such as pi and e) and how to move between them; .the ways in which these
representations are affected by number systems; the ways in which algebraic properties
of these systems are relevant and matter for operating within the systems;
and the significance of the additive and multiplicative identities, associativity,
commutativity, and the distributive property of multiplication over addition. Algebraic
principles undergird the place value system, allowing for economical expression
of numbers and efficient approaches to operations on them. They are also central to
number-line based operations with numbers, including work with additive inverses that
are central to addition and subtraction of first integers and then reals.
39. The centrality of number as a key concept in all the other mathematical areas
under consideration here and to mathematical reasoning itself, is undeniable. Students’
grasp of the algebraic principles and properties first experienced through work with
numbers is fundamental to their understanding of the concepts of secondary school
algebra, along with their ability to become fluent in the manipulations of algebraic
expressions necessary for solving equations, setting up models, graphing functions,
and coding and making spreadsheet formulas. Algebra provides generalisations
of the arithmetic in the number systems. And in today’s data-intensive world, facility with
interpretation of patterns of numbers, comparison of patterns, and other numerical skills
are evolving in importance.

2. Mathematics as a system based on abstraction and symbolic representation

40. The fundamental ideas of mathematics have arisen from human experience in
the world, in order to give that experience coherence, order, and predictability. Many

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

12 │

mathematics objects model reality, or at least reflect aspects of reality in some way.
However, the essence of abstraction in mathematics is that it is a self-contained system,
and mathematics objects derive their meaning from within that system. Abstraction
involves deliberately and selectively attending to structural similarities between
mathematics objects, and constructing relationships between those objects based on those
similarities. In school mathematics, abstraction forms relationships between concrete
objects, symbolic representations and operations including algorithms and mental models.
41. For example, children begin to develop the concept of “circle” by experiencing
specific objects that lead them to an informal understanding of circles as being perfectly
round. They might draw circles to represent these objects, noticing similarities between
the drawings to generalise about “roundness” even though the circles are of different
sizes. “Circle” becomes an abstract mathematics object only when it is defined as
the locus of points equidistant from a fixed point.
42. Students use representations – whether symbolic, graphical, numerical, or
geometric – to organise and communicate their mathematical thinking. Representations
can condense mathematical meanings and processes into efficient algorithms.
Representations are also a core element of mathematical modelling, allowing students to
abstract a simplified or idealised formulation of a real world problem.

3. The structure of mathematics and its regularities

43. When elementary students see
5 + (3 + 8)
some see a string of symbols indicating a computation to be performed in a certain order
according to the rules of order of operations; others see a number added to the sum of two
other numbers. The latter group are seeing structure; and because of that they don’t need
to be told about order, since if you want to add a number to a sum you first have to
compute the sum.
44. Seeing structure continues to be important as students move to higher grades.
A student who sees
𝑓(𝑥) = 5 + (𝑥 − 3)2
as saying that f(x) is the sum of 5 and a square which is zero when x = 3 understands that
the minimum of f is 5.
45. Structure is intimately related to symbolic representation. The use of symbols is
powerful, but only if they retain meaning for the symboliser, rather than becoming
meaningless objects to be rearranged on a page. Seeing structure is a way of finding
and remembering the meaning of an abstract representation. Being able to see structure is
an important conceptual aid to purely procedural knowledge.
46. What is the relationship between mathematical structure and reasoning? As
the examples above illustrate, seeing structure in abstract mathematical objects is a way
of replacing parsing rules, which can be performed by a computer, with conceptual
images of those objects that make their properties clear. An object held in the mind in
such a way is subject to reasoning at a higher level than pure symbolic manipulation.
47. A robust sense of mathematical structure also supports modelling. When
the objects under study are not abstract mathematical objects, but rather objects from
the real world to be modelled by mathematics, then mathematical structure can guide

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

 │ 13

the modelling. Students can also impose structure on non-mathematical objects in order to
make them subject to mathematical analysis. An irregular shape can be approximated by
simpler shapes whose area is known. A geometric pattern can be understood by
hypothesising translational, rotational, or reflectional symmetry and abstractly extending
the pattern into all of space. Statistical analysis is often a matter of imposing a structure
on a set of data, for example by assuming it comes from a normal distribution.

4. Functional relationships between quantities

48. Students in elementary school encounter problems where they must find specific
quantities. For example, how fast do you have to drive to get from Tucson to Phoenix,
a distance of 180 km, in 1 hour and 40 minutes? Such problems have a specific answer: to
drive 180 km in 1 hour and 40 minutes you must drive at 108 km per hour.
49. At some point students start to consider situations where quantities are variable,
that is, where they can take on a range of values. For example, what is the relation
between the distance driven, d, in miles, and time spent driving, t, in hours, if you drive at
a constant speed of 108 km per hour? Such questions are the beginnings of thinking about
functional relationships. In this case the relationship, expressed by the equation d = 108t,
is a proportional relationship, the fundamental example and perhaps the most important
for general knowledge.
50. Relationships between quantities can be expressed with equations, graphs, tables,
or verbal descriptions. An important step in learning is to extract from these the notion
of a function itself, as an abstract object of which these are representations. The essential
elements of the concept are a domain, from which inputs are selected, a codomain, in
which outputs lie, and a process for producing outputs from inputs.
51. Explicitly noting the domain and codomain allows for many different topics to be
brought under the function concept. A parametric curve is a function whose domain is
a subset of the real numbers and whose codomain is two- or three-dimensional space.
Arithmetic operations can be viewed as functions whose domain is the set of ordered
pairs of numbers. Geometric formulas for circumference, area, surface area, and volume
can be viewed as functions whose domain is the set of geometric objects. Geometric
transformations, such as translations, rotations, reflections, and dilations, can be viewed
as functions from space to itself.
52. The more formal definition of a function as a set of ordered pairs is both
problematic and useful in school mathematics. It is problematic because it removes
the dynamic aspect of students’ conceptualisation of function: function as process, or
mapping, or coordination of two varying quantities. These conceptualisations are useful
in many common uses of functions in science, society, and everyday life. On the other
hand, the ordered-pair definition emphasises the invariance of the function as an object in
its own right, independent of different methods of computing its outputs from its inputs.
Thus different forms for the expression of a quadratic function, say
𝑓 (𝑥) = (𝑥 − 1)(𝑥 − 3) and 𝑓(𝑥) = (𝑥 − 2)2 − 1,
throw light on different properties of the same object: the one shows its zeros, the other
its minimum value.
53. The two views of function—the naïve view as a process and the more abstract
view as an object—can be reconciled in the graph of the function. As a set of ordered
pairs it is a manifestation of the object. But reading a graph, coordinating the values on

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

14 │

the axes, also has a dynamic or process aspect. And the graph of a function is
an important tool for exploring the notion of a rate of change. The graph provides a visual
tool for understanding a function as a relationship between covarying quantities.

5. Mathematical modelling as a lens onto the real world (e.g. those arising in
the physical, biological, social, economic, and behavioural sciences)
54. Models represent an ideal conceptualisation of a scientific phenomenon. They are
in that sense abstractions of reality. A model may present a conceptualisation that is
understood to be an approximation or working hypothesis concerning the object
phenomenon or it may be an intentional simplification. Mathematical models are
formulated in mathematical language and use a wide variety of mathematical tools
and results (e.g., from arithmetic, algebra, geometry, etc.). As such, they are used as ways
of precisely defining the conceptualisation or theory of a phenomenon, for analysing
and evaluating data (does the model fit the data?), and for making predictions. Models
can be operated—that is, made to run over time or with varying inputs, thus producing
a simulation. When this is done, it can be possible to make predictions, study
consequences, and evaluate the adequacy and accuracy of the models.

6. Variance as the heart of statistics

55. One of the aesthetically rewarding aspects of our world is its variability. Living
things as well as non-living things vary with respect to many characteristics. However, as
a result of that typically large variation, it is difficult to make generalisations in such
a world without characterising in some way to what extent that generalisation holds. In
statistics accounting for variability is one, if not the central, defining element around
which the discipline is based. In today’s world people often deal with these types
of situations by merely ignoring the variation and as a result suggesting sweeping
generalisations which are often misleading, if not wrong, and as a result very dangerous.
Bias in the social science sense is usually created by not accounting for the variability in
the trait under discussion.
56. Statistics is essentially about accounting for or modelling variability as measured
by the variance or in the case of multiple variables the covariance matrix. This provides
a probabilistic environment in which to understand various phenomena as well as to make
critical decisions. Statistics is in many ways a search for patterns in a highly variable
context: trying to find the signal defining “truth” in the midst of a great deal of random
noise. “Truth” is set in quotes as it is not the Platonic truth that mathematics can deliver
but an estimate of truth set in a probabilistic context, accompanied by an estimate
of the error contained in the process. Ultimately, the decision maker is left with
the dilemma of never knowing for certain what the truth is. The estimate in the end is
a set of plausible values.

Proposal II: Application of Mathematical Reasoning in the Real-World: How 21st

Century Skills Fit into PISA 2021

57. There is increased interest worldwide in what are called 21st century skills
and their possible inclusion in educational systems. The OECD itself has put out
a publication focusing on such skills and has sponsored a research project entitled
The Future of Education and Skills: An OECD 2030 Framework in which some 25
countries are involved in a cross-national study of curriculum including the incorporation

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

 │ 15

of such skills. The project has as its central focus what the curriculum might look like in
the future, focusing initially on mathematics.
58. Over the past 15 years or so a number of publications have sought to bring clarity
to the discussion and consideration of 21st century skills. One of the more recent reports
was produced by the National Research Council (2012) of the United States. While most
of these publications focus on the question of what schools need to teach students to
know and to do, the NRC report makes a connection between the two questions.
The report notes that what many are now referring to as 21st century skills are not
something new in the learning enterprise. What may well be different is “society’s desire
that all students attain levels of mastery – across multiple areas of skill and knowledge –
that were previously unnecessary for individual success in education and the workplace”.
59. Lists of 21st century skills that students need to be taught have been based, at least
to some extent, on reviews of educational and psychological studies around learning.
Such skills have been discussed in the literature using terms such as “deeper learning,”
“college and career ready,” “higher order thinking skills,” “new basic skills,” or “next
generation learning” (NRC, 2012). Consistent with the viewpoint of earlier reviews
the NRC emphasised the conception that 21st century skills are not general skills that are
simply applied to various tasks in different contexts but rather “dimensions of expertise”
that are intertwined with and specific to a particular domain of knowledge.
60. Consequently, to underscore this view more completely the authors of the report
prefer to “use the term ‘competencies’ rather than ‘skills’”. The rationale for this move is
made clear in the earlier work of Anadiadou and Claro (2009) in which they refer to
a skill as a component of competence. Competence is then defined as “the ability to apply
learning outcomes in a defined context” which involves functional knowledge as well as
the application of interpersonal, social, and ethical values. Anadiadou and Claro (OECD,
2009) adopt the formal definition of competence developed by Rychen and Salganik that
distinguishes competence and skill: “A competence is more than just knowledge or skills.
It involves the ability to meet complex demands, by drawing on and mobilising
psychosocial resources (including skills and attitudes) in a particular context” (Rychen
& Salganik, OECD, 2003).
61. Anadiadou and Claro develop a framework for 21st century skills that has two
dimensions: 1) information, i.e. knowing how to acquire, interpret, and apply appropriate
information; and 2) communication which includes the ability to assess and navigate
the ethical, social, and interpersonal contexts of the workplace, home, and society.
A framework developed by the Partnership for 21st Century Learning (2009) has three
dimensions, as elaborated and further developed by Fadel, Bialik & Trilling (ref, 2015):
1. learning and innovation skills such as thinking creatively, working
collaboratively, and reasoning effectively;
2. information, media, and technology skills which include accessing, using,
and managing information using technology; and
3. life and career skills such as flexibility and adaptability, taking initiative,
and working effectively with people of diverse backgrounds.

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

16 │

62. Table 2 is adapted from Table 2-2 in the NRC (2012) report. It summarises
and categorises the various terms that have been used to refer to 21st century skills
according to three dimensions:
1. cognitive including knowledge domains and critical thinking,
2. intrapersonal including values, ethics, and self-management, and
3. interpersonal, considered a cluster of the two competencies
teamwork/collaboration and leadership.

Table 2. Terms for 21st century skills

Dimensions of
Clusters Labels/terms referring to 21st Century Skills
Competencies
Cognitive Cognitive Processes and Strategies Critical thinking, problem solving, analysis, reasoning/argumentation,
interpretation, decision making, adaptive learning, executive function
Knowledge Information literacy, information and communications technology literacy, oral
and written communication, active listening
Creativity Creativity, innovation
Intrapersonal Intellectual Openness Flexibility, adaptability, artistic and cultural appreciation, personal and social
responsibility, appreciation for diversity, intellectual interest and curiosity
Work Ethic/ Conscientiousness Initiative, self-direction, responsibility, perseverance, productivity, grit, self-
regulation, ethics, integrity, citizenship, career orientation
Positive Core Self-Evaluation Self-monitoring, self-evaluation, self-reinforcement, physical and
psychological health
Interpersonal Teamwork and Collaboration Communication, collaboration, teamwork, cooperation, coordination,
empathy, trust, service orientation, conflict resolution, negotiation
Leadership Leadership, responsibility, assertive communication, self-presentation, social
influence with others

Note: Adapted from Table 2-2, NRC, 2012

63. Anadiadou and Claro, reporting results from their survey of how countries are
including 21st century skills in their curricula, conclude that most countries “integrate
the development of 21st century skills and competencies in a cross-curricular way”.
Indeed reviews of 21st century skills typically envision embedding these across
the academic content of the curriculum but offer little insight into what this might look
like in any one area. The NRC report is an exception as it elaborates how the list
of 21st century skills they considered may be expressed through the recent Common Core
State Standards for English Language Arts, Mathematics, and the Next Generation
Science Standards in the US. Table 3 is an adaptation of Figure 5-2 that identifies
the overlap seen between learning expectations in the Common Core State Standards for
Mathematics and the 21st Century Skills.

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

 │ 17

Table 3. Overlap Between Learning Expectations in the Common Core State Standards for
Mathematics and 21st Century Skills

Areas of Overlap
Constructing and evaluating evidence-based arguments
Non-routine problem solving
Disciplinary discourse
Systems thinking
Critical thinking
Motivation; persistence
Identity
Self-development
Self-regulation, executive functioning
Collaboration/teamwork

Note: Adapted from Figure 5-2, NRC, 2012

64. The consensus of the various reports is that 21st century skills should be
incorporated in some fashion into the current curriculum, i.e. mathematics, science,
history, physical education, art, language, etc. One proposed method of incorporation
envisions the curriculum as a matrix with the rows defined by the subject matters
(mathematics, language, etc.) and the columns defined by the 21st century skills. Each
of the cells would then be an opportunity for an infusion of the skills into that subject
matter.
65. The advisory group felt this approach to be arbitrary and unrealistic. Not all skills
would naturally occur in all subject matters. By natural we mean that the skill is already
present or imbedded in the nature of the discipline and its canons of inquiry. This would
likely not be true for all skills in a single subject matter.
66. On the other hand, a strong case can be made for the infusion of specific
21st Century skills into specific disciplines. For example, it will become increasingly
important to teach students at school how to make reasonable arguments and be sure that
they are right. The arguments they make should be strong enough to withstand criticism,
and yet, whenever possible, avoid referring to authorities (e.g. ‘Google says so’). This is
part of the fundamental competence to make independent judgements and take
responsibility for them (OECD, 2005). In the social context it is not enough to be right;
one must be able and ready to present arguments and to defend them. Learning
mathematics, with its 100% clarity of contexts, is a perfect opportunity to practice
and develop the ability for this kind of argumentation.
67. Similarly, in the context of the ‘post-factual’ era, it is urgent to equip students
with tools that they can use to defend themselves from lies. Quite often some fluency in
logical reasoning is sufficient; a lie usually hides some hidden contradiction.
The alertness of young minds towards possible contradictions can be developed most easily
in good classes of mathematics.
68. Using the logic of finding the union between generic 21st century skills a related
but subject-matter specific skill that is a natural part of the instruction related to that
subject matter, the advisory group identified eight 21st century skills for inclusion in
the mathematics curriculum and, as such, in the PISA 2021 assessment framework. They
are:

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

18 │

 Critical thinking
 Creativity
 Research and inquiry
 Self-direction, initiative, and persistence
 Information use
 Systems thinking
 Communication
 Reflection
69. There always needs to be a context for the use of the 21st century skills. For
PISA 2021, the context is mathematics. We therefore propose testing the eight
21st century skills through an item format that allows recording the students’ use
of mathematical reasoning when solving real-world problems. It is important however to
ensure that the item format does not become too burdensome; for example, in having to
learn how to navigate the computer when responding to the computer-based items. Such
a testing design will require a complex system of scoring that reflects information on
the process the student used. Additionally it will require a different and more
sophisticated approach to item writing. In effect the items and their scoring rubrics must
require that the student demonstrate use of and be evaluated by the appropriate
21st century skill in addressing the problem posed.

Proposal III: Mathematics Content Areas for Emphasis in PISA 2021

70. Four areas of mathematics are proposed for special emphasis in the PISA 2021
assessment. The topics are not outside the domains identified in the PISA 2021
framework, i.e., space and shape, changing and relationships, uncertainty and data
and quantity, but are sub-areas of these. In the work of Fadel et al. (“Recommendations
for PISA Maths 2021”, (2017)) the topics are represented not only as commonly
encountered situations in adult life in general but as the types of mathematics needed in
the emerging new areas of the economy such as high-tech manufacturing. The four are:
 Computer simulations
 Exponential growth
 Conditional decision making
 Geometric Approximation
71. What follows is a brief description of each of the four areas together with example
assessment items.

Computer Simulations
The use of standard mathematical algorithms, together with the computer, to solve
complex quantitative problems
72. Both in mathematics and statistics there are problems that are not so easily
addressed because the required mathematics are complex or involve a large number
of factors all operating in the same system. Increasingly in today’s world such problems
are being approached using computer simulations driven by algorithmic mathematics.
A good example is the use of such simulations towards helping individuals plan their
retirement so as to have enough money on which to live and accomplish their goals.
The number of factors to consider is very large. They include income, age of retirement,
expected expenses, investment earnings, stock market values, and proposed age at death,

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

 │ 19

the values of which require a set of assumptions to be made by the individual

the computer program. Changing any of those assumptions individually or collectively
provides different results which can then be aggregated statistically to provide an overall
estimate of how achievable retirement is, given the goals. Users of such a simulation
program need to understand at some level how this is done so as to interpret the results as
to the implied impact of their assumptions

Exponential Growth
The growth of a system in which the amount being added to is proportional to the amount
currently present
73. Understanding the dangers of flu pandemics and bacterial outbreaks, as well as
the threat of climate change, demand that people think not only in terms of linear
relationships but recognise that such phenomena need to be represented mathematically
with non-linear (exponential) models. Linear relationships are common and are easy to
recognise and understand but to assume linearity can be dangerous. A good example
of linearity and one probably used by everyone is estimating the distance travelled in
various amounts of time while traveling at a given speed. Such an application provides
a reasonable estimate as long as the speed stays relatively constant. But with flu
epidemics, for example, such a linear approach would grossly underestimate the number
of people sick in 5 days after the initial outbreak. Here is where a basic understanding
of exponential growth and how rapidly infections can spread given that the rate of change
increases from day to day is critical. The recent spread of the Zika infection is
an important example of exponential growth; recognising it as such helped medical
personnel to understand the inherent threat and the need for fast action.

Conditional Decision Making

Decision-making in a probabilistic environment, using all the relevant information
74. Going to the doctor often ends up requiring a decision about what to do next. Do I
take the medicine? Given my age, the doctor says that I should live another 20 years.
The question arises – should I take a low dosage aspirin every day? This is a decision for
which certainty is not possible. However, the decision making can be improved by asking
the doctor how long I should live and not have a heart attack if I have a particular blood
pressure, heart rate, cholesterol-level and body fat ratio.
75. This is a question answerable best in terms of conditional probability. Such
a probability or even an estimate of that probability takes into account the additional
information such as the patient’s current heath as indicated by the blood and other tests. If
there is a relationship between blood pressure and cholesterol levels and the other test
results in terms of the chances of having a future heart attack, the predicated conditional
mean might for example change the life expectancy estimate of 20 years to only 5 years,
impacting decisively on the patient’s view of whether to take a daily aspirin. Essentially
the conditional mean provides a non-technical approach to regression analysis.
76. Variables which are dichotomies or polychotomous in nature must be analysed
differently to continuous variables such as blood pressure and age. Questions such as do
people with blue eyes also tend to have blonde hair, or how many ways can 20 people be
organised into the 11 positions on soccer team are examples of situations where knowing
the basic rules of combinatorics would help in making decisions. In the case
of the cross-classification of categorical variables, such as blue eyes/or not vs. blonde

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

20 │

hair/or not, understanding the different types of percentages that can be computed
and what each of them means is critical to understanding the phenotypic genetics.
Combinatorics (Into how many different orders can 5 objects be placed? Into how many
combinations can 16 objects be placed, taking 2 at a time?) also help to understand
commonly occurring situations. Game theory, a more formal approach, can be applied to
decision making between different categorical options such as winning vs. losing, using
many of these same approaches together with conditional probability.

Geometric Approximation
Shapes that do not follow typical patterns of evenness or symmetry
77. Today’s world is full of shapes that do not follow typical patterns of evenness or
symmetry. For example, architectural designers often deploy traditional geometric objects
(lines, angles, squares, triangles and circles) in non-traditional ways, in order to enhance
the use of space and aesthetic beauty. The concept has entered our lives through
the design of our homes, the art we buy, the urban design of our cities and the places in
which we work, just to name a few.
78. Because simple formulas do not deal with irregularity, it has become more
difficult to understand what we see and find the area or volume of the resulting structures.
For example, finding the needed amount of carpeting in a building in which
the apartments have acute angles together with narrow curves demands a different
approach than would be the case with a typically rectangular room. Finding the shortest
distance between two points also becomes less obvious.

Some Directions for the Contractors

79. In this document the Special Advisory Group has laid out a vision of a broadened
definition of mathematical literacy. It builds directly on the 2012 framework and retains
a central focus on the problem solving model as illustrated in Figure 1. It, however,
includes a second central focus, that of mathematical reasoning. These two aspects
of mathematical literacy are what need to be built into the PISA 2021 assessment.
Mathematical reasoning, the more basic or fundamental of the two as well as the more
general, is likely present but not explicit in the problem solving aspect, especially in
the first stage of the model – formulating the messy somewhat ill-defined “real-world”
problem mathematically.
80. The task of developing a test specific framework or blueprint for the 2021
assessment must not only continue to incorporate problem solving but now must also
include the separate aspect of mathematical reasoning, focusing on the six fundamental
concepts supporting it. In fleshing this out into the test blueprint, attention should also be
given to their relationship to the eight 21st century skills.
81. In effect, the test design has 6 factors (see Figure 4 which is a reformatted
version of Figure 3 that facilitates this discussion by more clearly delineating the factors
and their levels). Each factor is listed below with the number of aspects (levels) included
in parentheses:
 Mathematical reasoning and problem solving foci (2)
 Real-world contexts (4)
 Mathematics content categories and emphases (4)
 Fundamental concepts supporting mathematical reasoning (6)

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

 │ 21

 21st century skills specifically relevant to mathematics (8)

 Processes in problem solving (4)
82. There is some implicit nesting among these factors: The processes in problem
solving are aspects of the main problem solving focus. The fundamental concepts
supporting mathematical reasoning are aspects of the mathematical reasoning focus. It is
also clear that mathematical reasoning is important in problem solving but that
mathematical reasoning is present in situations other than problem solving, such as in
evaluating arguments, interpretations, and inferences.
83. This test design is obviously too big as the number of potential cells vastly
exceeds the number of items that can be included on a PISA test. The contractor needs to
look at ways to address this, for example by building a partial replicate of the full design
that limits the number of needed items. This must be examined carefully looking at
the alias structure of different partial replicates to determine the necessary assumptions
regarding various interaction terms. Important substantive considerations are essential in
making those assumptions as well as in making sure the desired subtest scores from
a substantive perspective are estimable. Clearly, part of that effort will need to include
choosing areas of emphasis and collapsing across aspects within a factor as well.

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

22 │

Figure 4. The Broadened Model of Mathematical Literacy (expanded)

Reformatted version of Figure 3

84. We respect the professionalism of the contractor and feel the task of working this
out as well as other technical issues is their responsibility and not ours. But we also feel
that there are special concerns we have as mathematicians, statisticians and mathematics
educators that derive from the content itself and are critical to the success of measuring
mathematical reasoning within and outside of how it interacts with problem solving.
Mathematical reasoning may be more fundamental as we have argued but it will also be
more difficult to measure in terms of the response data needed and also as to how we map
those data into meaningful score points.

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

 │ 23

85. With that in mind, we offer several special content relevant concerns, issues
and suggested approaches in the form of guidelines for the contractor to take into
consideration:
 In the final analysis the test design must represent mathematical literacy including
both aspects – mathematical reasoning and problem solving – as articulated in
and contextualised in Figures 3 and 4. In other words, the test design must
generalise to the population of tasks. Weighting will need to be specified
determining the relative contribution of various factors.
 Given that mathematical reasoning is a sub-score of mathematical literacy, as is
problem solving, we recommend the distribution of score points be allocated as
60% problem solving and 40% mathematical reasoning (or 67/33) leaving
a substantial overlap with 2012. Ultimately consideration of a 50/50 distribution
would be warranted but for continuity with the 2012 we recommend the former
for 2021.
 Some factors such as “real-world contexts” and “mathematics content categories”
could be considered not as creating marginal sub-scores but enter the design
randomly in a balanced fashion across the fixed factors for which sub-scores will
be developed. Essentially in 2012 that was what was done with the real-world
context. The four content areas seem to add little and be mostly redundant with
the total score but it is likely that the OECD will want to continue them. We have
suggested subtopics that seem to be more interesting and relevant than the general
content areas.
 Three mathematical reasoning sub-scores are proposed by the Special Advisory
Group including: mathematical modelling (5), the structure of mathematics (3),
and variance (6).
 For the 21st century factor, two sub-scores are proposed: communication (6)
and persistence (4).
 This creates a mathematical literacy score; a mathematical reasoning score, as
well as three sub-scores including the structure of mathematics, variance
and mathematical modelling. Also included is a 21st century skills score and two
sub-scores representing communication and persistence. In addition there would
be a problem solving score and three sub-scores related to the problem solving
model: formulate, employ and interpret/evaluate – a total of 12 scores. If the four
content category sub-scores are developed as they were in 2012 that would
produce 16 scores. Even 12 scores appears to be too large a number.
 New scoring approaches need to be developed around partial credit. The kind
of items that need to be developed to measure mathematical reasoning call for
open-ended responses more than choosing from multiple options. We propose
including partial credit for the correct recognition or setting up of a problem
and not just the final correct answer as well as credit for trying to understand
an argument or solve a problem even if totally wrong in the approach. Here
the mere attempt at a solution would include credit.
 The contractor also needs to be prepared for large amounts of qualitative data
resulting from such item types. This is not just for purposes of scoring the items
but such data become part of the results that need to be summarised for report
writing. This will take a substantial amount of work.
 The scores derived here will not typically be unidimensional. Items may be
needed to contribute to multiple sub-scores. Multidimensional scaling should also
be considered.

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

24 │

86. All of the above proposed guidelines essentially focus on one issue that
the Strategic Advisory Group wants the PISA2021 test framework and corresponding
assessment to have – content validity. The measurement validity and relevance of the test
and its reporting is what is most important. We recognise that the type of measurement
we propose is difficult, in fact very difficult. As some of the members of the Group are
psychometricians, we know what the arguments against such an approach will be
regarding reliability and other technical issues, but the committee feels those arguments
need to be challenges, not barriers. What we want to measure is important and when we
do it, it will likely have a strong positive impact on mathematics instruction across the
world. We know what we want to measure, i.e. mainly the underlying reasoning
and problem-solving abilities; what we need are the means by which to do it. This is
the challenge that faces PISA and the contractors.

Some Item Types for Consideration

87. The following 10 items are included as examples of the types of items that will
be needed. There is no particular order in the presentation of these items. Although not
written explicitly for the purposes of this proposal, they represent the range of what needs
to be developed.

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

 │ 25

Figure 5. Sample Item 1

A group of children are learning how to multiply 2-digit numbers in different ways.
In the examples below they are calculating 47 x 36.
Explain: 1) how the calculation is done in each representation
2) the connections between all three representations
Representation 1

Representation 2
X 40 7
6 240 42 282
30 1200 210 1410
1692

Representation 3
Th H T O
4 7
Th = Thousands
X 3 6
H = Hundreds
4 2
T = Tens 2 4
O = Ones 2 1
1 2
1 6 9 2

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

26 │

Figure 6. Sample Item 2

OAK HORNBEAM MAPLE

Estimating the surface area of tree leaves.
a) Above are drawings of leaves from three trees. Which leaf has the greatest surface area? How
would you measure the areas to check which is largest? Explain what problems there are in
your measurement procedure.
b) These are just particular leaves from the three kinds of trees. The actual leaves on a tree vary
somewhat in size from one to the next. Suppose you had a sample of, say, 100 leaves from
each tree. How would you estimate the average surface area of leaves of each type.
Note. It might be tiresome to apply the method you suggested in part a) so many times in part b)!

Figure 7. Sample Item 3

There are three stores in the town of Gluck which sell CDs. The prices vary across the stores
ranging from $3 to $10 depending on the particular CD. In store A, the prices range from $4 - $8. In
store B the cost of the same CDs range from $3 - $9. The third store sells all its CDs for $5 -$7.
1) You want to buy several different CDs and are not sure how much the ones you want cost. You
only have time to go to one store. Which store would you pick – A, B, or C – and why?
2) Assume you just want to buy several CDs and don’t really care about specific ones, which store
would you pick and why?

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

 │ 27

Figure 8. Sample Item 4

A fatal hit and run accident occurred on a dark and stormy night in Gotham City. A suspect driving
a damaged red truck was arrested. A witness to the accident has given evidence that the hit and run
vehicle was a blue truck. The following facts are available to you as the judge.

i) The witness is truthful.

ii) The witness was tested earlier with a blue truck and a red truck in similar weather
conditions. She was able to correctly identify a flue truck 8 out of 10 times and correctly
identify a red truck 6 out of 10 times.

1. What is the probability that the witness saw a blue truck when the truck was actually red?

2. What is the probability that the witness saw a blue truck when the truck was actually blue?

The following information is available to you.

iii) There are 1000 red trucks and 50 blue trucks in Gotham City.

iv) 50% of the red trucks and 80% of the blue trucks were on the road at the time of the
accident.

v) Red trucks are generally more visible than blue trucks.

vi) All trucks in Gotham City are either red or blue.

vii) The suspect knew the victim.

3. Incorporate the additional information above which you think is relevant. What would you as
judge rule concerning the suspect? Explain and support your conclusion.

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

28 │

Figure 9. Sample Item 5

Suppose it is known that over the 100 year history of the town that each day there is a 1 in 5
chance that the brown bears that live in the forest near the city come to the garbage dump
where people can observe them. You notice on your cellphone that the city has announced
that they were there yesterday so you decide not to go to observe for the next 4 days.
1) Is that a wise decision; why is it or isn’t it?
2) How would you time your trip if you wanted to watch the bears?

Figure 10. Sample Item 6

Shopping at the new store in town includes a 43% discount on all items. The state you live
in has a 7% sales tax. You want to buy many things but only have a total of $52 hours to
spend. How can you decide what to buy?
1) Describe in words what you could buy.
2) Make up an algorithm by which to determine what you can buy.
3) Write a decision model to help you.

Figure 11. Sample Item 7

This diagram defines a simple machine for producing strings of letters. You start with
position “A” and write down that letter. Then you go to any position connected with an
arrow to your current position. Write down the letter you find there. You continue from
position to position, but if you reach position “E”, you stop.
a) What is the shortest string you can obtain?
b) What is the longest string?
c) Can you describe the set of all strings that can be obtained?

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

 │ 29

Figure 12. Sample Item 8

 Fine Needle Aspiration Cytology is a biopsy method for testing women for breast cancer. A biopsy
is a tissue sample taken from the body so that it can be examined more closely.

 No medical test is 100% accurate – sometimes a positive result is returned for a healthy person (a
false positive result) or a negative result for someone who actually has the disease (false negative).

 The incidence of women biopsied who actually have breast cancer is 30%. The FNA biopsy has a
2% probability of giving a false positive result and a 14% probability of giving a false negative
result.

 If 100,000 women were biopsied for breast cancer, how many of these women actually have breast
cancer? How many would receive a false negative biopsy result? How many would correctly
receive a positive biopsy result?

 How many of the 100,000 biopsied women don’t have breast cancer? How many of these women
would receive a false positive biopsy result?

 What is the probability that a woman with a negative biopsy result actually has breast cancer?

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

30 │

Figure 13. Sample Item 9

Population density varies greatly between countries, and also within countries, because much of the
Earth’s land surface is not suitable for human habitation. Population density is measured by the
number of people divided by land area in square kilometers.

Country Population (2016) Land area (km2)

New Zealand 4.7 million 68,918
Ireland 4.7 million 263,884
New Zealand and Ireland are two countries with the same population but different land areas.
1) Calculate the population densities for the two countries. (NZ 17.8 people/km2, Ireland 68.2
people/km2).
Because it can be difficult to image what a square kilometer looks like, Liam and Naomi are thinking
of different ways to visualise population density.
2) Liam imagines that all the people in Ireland join hands and stretch themselves out in a straight
line.
a) How long would this line be? Write out an explanation for this calculation that one of your
classmates would understand.
b) How does the length of this line compare to something in the real world that is familiar to
you? [e.g., distance between two towns, length of a football field]
3) Naomi imagines that all the people in New Zealand want to stand close together on football
fields measuring 100m by 70m.
a) How many people could stand close together on one football field? Write out an
explanation for this calculation that one of your classmates would understand.
b) How many football fields would it take to hold the entire population of New Zealand?
4) Which of these two approaches – Liam’s or Naomi’s – do you think is better for understanding
and visualising population density? Write a letter to Liam and Naomi to explain your decision to
them.
5) Singapore has a population of 5.7 million people and a land area of 719 km2. Use your preferred
method described in (4) to represent the population density of Singapore and compare this to the
population densities of both New Zealand and Ireland.

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

 │ 31

Figure 14. Sample Item 10

A designer uses a computer to create a tile pattern as follows. A square of side 1 undergoes a
transformation f that consists of 3 concurrent sub-transformations:

1. An enlargement of factor 0.5, followed by an anti-clockwise rotation of 90, followed by a

translation 0.5 units to the right and 0.5 units up.
2. An enlargement of factor 0.5, followed by a reflection about the y-axis, followed by a
translation 0.5 units to the right.
3. X.

The resulting shape is made to undergo the transformation f again.

This is repeated 20 times.

The figures below show the original square and the results of the first 3 iterations:

(0) (1)

(2) (3)
(A) (B)
a) Describe the transformation X.
b) Which of the following would be the shape after 20 iterations? Explain your choice.

(C) (D)

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

32 │

Final Thoughts

88. The three suggestions made for broadening the PISA 2021 Assessment
Framework fit well with the original 2012 PISA framework. That was the advisory
group’s intention. The original 2012 framework focuses on problem solving and outlines
the steps involved in that process. Also contained in the 2012 framework are four broad
areas of mathematics. Proposal III adds emphasis to certain topics in each of those four
original domains of content, reflecting areas that are more commonly found and salient in
today’s world. Proposal I focuses on identifying six fundamental concepts that transcend
and provide structure, to the definition of mathematics reasoning which is proposed as a
second aspect of mathematical literacy in addition to problem solving.
89. Finally, Proposal II adds the 21st century skills which are an abstraction and
generalisation of many of the mathematics processes used in the solution of problems as
defined by the original 2012 model. Students using those skills in solving the problems
are using the kinds of mathematical processes that generalise beyond mathematics to life
more broadly. This is not artificial as these processes are inherent in and a major part of
mathematics. They answer the age old question of, when am I going to use this math in
my real life anyway? This is what is meant by quantitative literacy, the ability to apply
those mathematical concepts and processes to different real-life situations. Perhaps, even
in situations in which they have no idea of the traditional mathematics used to solve it.
The question is whether the student can adapt using his/her understanding of the six
elements supporting mathematical reasoning in Proposal I to find a solution to the
problem. Generalised, to the extent possible, those become 21st century skills. Whether
these skills are generalisable is still an open research question while some earlier studies
would suggest not. Measuring them, demands that we be able to record the process by
which students attempt those solutions and that demands a non-traditional item format
that could be nicely supported given that the 2021 test will be computer based.

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

 │ 33

References

Ananiadou, K., and Claro, M. (2009), "21st century skills and competences for new millennium
learners in OECD countries", EDU Working Papers, No. 41, OECD Publishing, Paris.
Beloit College. (2017). Mindset List for the Class of 2021. Retrieved August 22, 2017, from
https://www.beloit.edu/mindset/2021/.
Blum, W., and Niss, M. (1991), "Applied mathematics problem solving, modelling,
applications, and links to other subjects—state, trends and issues in mathematics
instruction", Educational Studies in Mathematics, Vol. 22/1, pp. 37-68.
Center for Curriculum Redesign: “21st Century Mathematics” Conference (Stockholm, 2013)
and Report: “Recommendations for PISA Maths 2021” (2017).
CNBC.com (2017, Published 8:41 AM ET Tue, 15 Aug 2017), "High-tech US plants offer
jobs even as the laid-off struggle", retrieved from
https://www.cnbc.com/2017/08/15/high-tech-us-plants-offer-jobs-even-as-the-laid-off-
struggle.html.
Donoghue, E. F. (2003), "Algebra and Geometry Textbooks in Twentieth-Century America",
in G. M. A. Stanic & J. Kilpatrick (Eds.), A History of School Mathematics, Vol. 1,
National Council of Teachers of Mathematics, Reston, VA, pp. 329-398.
Fadel, C., Bialik, M., and Trilling, B. (2015), Four-Dimensional Education: the competencies
learners need to succeed, Boston Massachusetts USA, by Center for Curriculum
Redesign.
Kliebard, H. M., and Franklin, B. M. (2003), "The Ascendance of Practical and Vocational
Mathematics, 1893-1945: Academic Mathematics under Siege", in G. M. A. Stanic and
J. Kilpatrick (Eds.), A History of School Mathematics, Vol. 1, National Council of
Teachers of Mathematics, Reston, VA, pp. 399-440.
National Numeracy, (2015, February 16), retrieved from
https://www.nationalnumeracy.org.uk/what-numeracy.
National Research Council (2012), "Education for Life and Work: Developing Transferable
Knowledge and Skills in the 21st Century", in J. W. Pellegrino, M. L. Hilton, Board on
Testing and Assessment and Board on Science Education, and Division of Behavioral
and Social Sciences and Education (Eds.), The National Academies Press, Washington,
DC, pp. 242.
OECD (2005), Definition and Selection of Key Competencies – Executive Summary, retrieved
from http://deseco.ch/bfs/deseco/en/index/02.html.
OECD (2013a), PISA 2012 Assessment and Analytical Framework: Mathematics, Reading,
Science, Problem Solving and Financial Literacy, OECD Publishing, Paris, pp. 264.
OECD (2016 onwards), The Future of Education and Skills: An OECD 2030 Framework.
Partnership for 21st Century Skills (2009), P21 Framework Definitions. Tucson, AZ.
Reimers, F. M., & Chung, C. K. (2016), Teaching and Learning for the Twenty-First Century,
Harvard Education Press, Cambridge, MA.

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE

34 │

Rychen, D. S., & Salganik, L. H. (Eds.) (2003), Key Competencies for a Successful Life and
Well-functioning Society. Hogrefe & Huber, Cambridge, MA.
Schmidt, W. H., Cogan, L. S., & Guo, S. (submitted), "The Role That Mathematics Plays in
College- and Career-Readiness: Evidence from PISA", Journal of Curriculum Studies.

PISA 2021 MATHEMATICS: A BROADENED PERSPECTIVE