PRINCIPLES OF HIGH

QUALITY ASSESSMENT
PAUL JOHN B. ONGCOY, PHD
 INTENDED LEARNING OUTCOMES

• Enumerate the principles of high quality assessment.

• Explain factors that makes assessment valid, reliable,
fair, ethical, practical and efficient
• Illustrate various principles of high quality
assessment.
PRINCIPLES OF HIGH QUALITY
ASSESSMENT
• Clear and appropriate learning targets
• Alignment of assessment methods and learning targets
• Validity
• Reliability/precision
• Fairness
• Positive consequences
• Alignment
• Practicality and efficiency
WHY HIGH QUALITY ASSESSMENT?

•provide results that verify and

promote targeted student learning
and motivation.
•inform instructional decision making
 CLEAR AND APPROPRIATE LEARNING TARGETS

• Sound assessment begins with clear and appropriate learning

targets
• Clear targets mean that both students and teachers understand
the nature of learning that is expected, and what student
proficiencies will result.
• Appropriate targets are those that are reasonable and aligned with
student characteristics, instruction, and standards
ALIGNMENT OF ASSESSMENT METHODS AND
LEARNING TARGETS

• Match appropriate assessment methods with

learning targets
• Types of Assessment Methods – selected-response,
constructed response, performance, essay, oral
questioning, teacher observations, self-assessment
VALIDITY

• a characteristic that refers to the appropriateness of

the inferences, uses, and consequences that result
from the assessment. It is concerned with the
soundness, trustworthiness, or legitimacy of the
claims or inferences that are made on the basis of
obtained scores.
 VALIDITY
• Latin word validus meaning strong
• It is seemed valid if it measures what it is supposed to.
• Accuracy of the inferences teachers make about students based on
the information gathered from an assessment (McMillan, 2007; Fives
& DiDonato-Barnes, 2013)
• Decisions include those about instruction and classroom climate
(Russell & Airasian, 2012)
 VALIDITY

• An assessment is valid if it measures a student’s actual

knowledge and performance with respect to intended
learning outcomes and not something else.
• It is a representative of the area of learning or content of
the curricular aim being assessed (McMillan, 2007; Popham,
2011)
TWO SOURCES OF INFORMATION USED TO
ESTABLISH VALIDITY

• Content – related evidence

• Criterion – related evidence
• Construct – related evidence
CONTENT-RELATED VALIDITY

• Extent to which the test covers the entire domain of

the content
• Table of Specifications (TOS)
 CRITERION – RELATED EVIDENCE
• Degree to which test scores agree with an external criterion
• Relationship between an assessment and another measure of the
same trait (McMillan, 2007)
• Three Types (Nitko & Brookhart, 2011): Achievement test scores;
ratings, grades and other numerical judgements made by the teacher;
and career data
• Concurrent Validity and Predictive Validity
CRITERION – RELATED EVIDENCE

• Predictive validity – power or usefulness of test

scores to predict future performance
• Concurrent Validity – estimate of a student’s
current performance in relation to a previously
validated or established measure
CONSTRUCT – RELATED EVIDENCE

• Construct – an individual characteristic that explains

some aspect of behavior (Miller, Linn & Gronlund, 2009)
• Quality of the instrument used
• Assessment is a meaningful measure of an unobservable
trait (McMillan, 2007)
 UNIFIED CONCEPT OF VALIDITY

• Integrates considerations of content, criteria and consequences

into a construct framework for the empirical testing of rational
hypotheses about score meaning and theoretically relevant
relationship (Messick, 1995, p741)
• External validity – how score properties and interpretations
generalize to and across population groups, contexts and tasks
 VALIDITY OF ASSESSMENT METHODS (MOSKALL, 2003)

• The selected performance should reflect a valued activity

• The completion of performance assessments should provide a valuable learning
experience.
• The statement of goals and objectives should be clearly aligned with the
measurable outcomes of the performance activity.
• The task should not examine extraneous or unintended variables.
• Performance assessments should be fair and free from bias.
 THREATS TO VALIDITY (MILLER, LINN & GRONLUND,
2009)
• Unclear test directions
• Poorly Constructed test items
• Complicated vocabulary and
sentence structure • Inappropriate test items for
outcomes being measured
• Ambiguous statements
• Short test
• Inadequate time limits
• Improper arrangement of items
• Inappropriate level of difficulty of
test items • Identifiable pattern of answers
TABLE OF SPECIFICATIONS

•is a two-way grid that shows the

content and types of learning targets
represented in your assessment.
 RELIABILITY
• Reproducibility and consistency in methods and criteria
• An assessment is reliable if it produce the same results if given to an
examinee on two occasions.
• For a test to be valid, it has to be reliable.
• Expressed in correlation coefficient
• A high reliability coefficient – similar test is readministered to the
same group of student – comparable test result
 RELIABILITY/PRECISION

• combination of measurement error, as previously

described, and instructional decision making (Parkes, 2013)
• estimating the influence of various sources of error within
the context of decision making about student learning. The
primary focus is on the decision.
TYPES OF RELIABILITY

• Internal Reliability – Consistency of results

across items within a test
• External Reliability – Extent to which a measure
varies from one use to another
SOURCES OF RELIABILITY

• Stability – Test-retest reliability – Correlation of

scores obtained from two administration of the
same test over a period of time.
• Equivalence – Parallel forms – two different
versions of an assessment tool with parallel items
are administered to the same group of individual
 SOURCES OF RELIABILITY

• Internal Consistency – a student who has mastery learning will

get all or most of the items correctly while a student who
knows a little or nothing about the subject matter will get all
or most of the items wrong
 INTERNAL CONSISTENCY

• Split-half method – Correlating the results of two halves

• Cronbach alpha
• Kuder – Richardson (KR) 20/21 – dichotomous items
• <0.5 the reliability is low; 0.5 – 0.8 the reliability if moderate;
>0.5 the reliability is high (Salvucci, et al., 1997)
 SOURCES OF RELIABILITY

• Scorer or Rater Consistency – Inter-rater reliability

• There should be a should variation of products to be judged,
scoring criteria are clear and raters are knowledgeable or trained
on how to use the observation instrument (McMillan, 2007)
• Spearman’s rho (Ordinal data)
• Cohen’s Kappa (Nominal and Discrete data)
SOURCES OF RELIABILITY

• Decision Consistency – how consistent the

classification decisions are rather than how consistent
the scores are (Nitko & Brookhart, 2011)
• Who will receive a passing mark or failing mark
 MEASUREMENT ERRORS

•Classical Test Theory – X = T + E

• X – observation
• T – measured score
• E – some measurement error
RELIABILITY OF ASSESSMENT METHODS

• Between a well-constructed objective tests and

performance assessment, the former has better
reliability (Miller, Linn & Gronlund, 20019; Harris, 1997)
• Performance assessment is said to have low reliability
because of judgmental scoring.
• Limited sampling of course content
 WAYS TO IMPROVE RELIABILITY OF ASSESSMENT
RESULTS (NITKO & BROOKHART, 2011)

• Lengthen the assessment procedure by providing more time,

more questions and more observation whenever is practical
• Broaden the scope of the procedure by assessing all the
significant aspects of the largest learning performance
• Improve objectivity by using a systematic and more formal
procedure for scoring student performance
WAYS TO IMPROVE RELIABILITY OF ASSESSMENT
RESULTS (NITKO & BROOKHART, 2011)

• Use multiple markers by employing inter-rater

reliability
• Combine results from several assessments especially
when making crucial educational decisions
• Provide sufficient time to students in completing the
assessment procedure
 WAYS TO IMPROVE RELIABILITY OF ASSESSMENT
RESULTS (NITKO & BROOKHART, 2011)
• Teach students how to perform their best by providing practice and
training to students and motivating them
• Match the assessment difficulty to the students’ ability levels by
providing tasks that are neither too easy nor too difficult, and tailoring
the assessment to each student’s ability level when possible
• Differentiate among students by selecting assessment tasks that
distinguish or discriminate the best from the least able students
PRACTICALITY AND EFFICIENCY

• Practical means “useful”

• Can be used to improve classroom
instruction and for outcomes purposes
• Pertains to judicious use of classroom time
FAMILIARITY WITH THE METHOD

• A rule of thumb is that simple learning targets require

simple assessment, complex learning targets demand
complex assessment (Nitko & Brookhart, 2011)
• It is critical that teachers learn the strengths and
weaknesses of each method of assessment, how they
are developed, administered and marked.
TIME REQUIRED

• A desirable assessment is short yet able to provide

valid and reliable results
• Assessment should allow students to respond readily
but not hastily
• Time is a matter of choice
EASE IN ADMINISTRATION

• Easy to administer
• Instruction must be clear and complete
EASE OF SCORING

• It is more practical to use rating scales and checklists

rather than writing extended individualized evaluations
for performance assessments (McMilla, 2007).
EASE OF SCORING

•Students are given a score that reflects

their knowledge, skills or performance.
•Meaningless if such score is not
interpreted.
 COST

• Classroom tests are generally inexpensive compared to

national or high-stakes tests.
• One chooses the most economical assessment.
• High quality assessments focus on deep learning which is
essential if students are to develop the skills they need for a
knowledge society (Darling-Hammond & Adamson, 2013)
 ETHICS

• Assessment is more than just a technical activity – it is a

human activity (Russel & Airasian, 2012)
• Teachers’ assessment have important long-term and short-
term consequences for students; thus teachers have an
ethical responsibility to make decisions using the most valid
and reliable information possible (Russel & Airasian, 2012)
 FAIRNESS
• Validity and reliability are aspects of fairness
• Students’ knowledge of learning targets and assessments
• Opportunity to learn
• Prerequisite knowledge and skills
• Avoiding student stereotyping
• Avoiding bias in assessment tasks and procedures
• Accommodating special needs (McMillan, 2007; Russel and Airasian, 2012)
FAIRNESS
• A fair assessment is one that provides all students an
equal opportunity to demonstrate achievement. This
is achieved with transparency about learning
expectations, clear criteria for judging student
performance, and the absence of bias (Tierney, 2013).
STUDENTS’ KNOWLEDGE OF LEARNING TARGETS
AND ASSESSMENTS

• Transparency – disclosure of information to

students about assessment
• Unannounced quiz
• Test-taking skills
OPPORTUNITY TO LEARN

• Fair assessments are aligned with instruction

that provides adequate time and
opportunities for all students to learn
(McMillan, 2007)
PREREQUISITE KNOWLEDGE AND SKILLS

• Teacher must identify early on the prerequisite

skills necessary for completing an assessment.
• Clinics or reinforced tutorials
• Remediation
AVOID STEREOTYPING

• Stereotype – generalization of a group or people based

on inconclusive observation of small sample of this
group
• Teachers should avoid terms and examples that may be
offensive to students’ gender, race religion, culture or
nationality.
THEORY OF STEREOTYPE THREAT

• People who are challenged in areas they deem

important like intellectual ability, their fear of
confirming negative stereotypes can cause them to
falter in their actual test performance (Steele &
Aronson, 1995)
AVOIDING BIAS IN ASSESSMENT TASKS AND
PROCEDURES

• Assessment must be free from bias.

• Two forms of assessment bias (Popham, 2011):
offensiveness and unfair penalization
OFFENSIVENESS

• Test-takers get distressed, upset or distracted about

how an individual or a particular group is portrayed in
the test
UNIFAIR PENALIZATION

• Harms students performance due to test content, not

because items are offensive but rather, the contents
caters to some particular groups from the same
economic classm race, gender, etc leaving other groups
at a loss or disadvantage
 ACCOMMODATING SPECIAL NEED

• Teachers need to be sensitive to the needs of the students.

• RA 7277 (An Act Providing for the Rehabilitation, Self-Development
and Self-Reliance of Disabled Person and their Integration into the
Mainstream of Society and other Purposes) Section 12
• Access to quality education
• CHED CMO 09 2013 “Enhanced Policies and Guidelines on Student
Affairs and Services Section 32
RELEVANCE

• Assessment should reflect the knowledge and skills that

are most important for students to learn.
• Assessment should support every student’s
opportunity to learn things that are important.
• Assessment should tell teachers and individual students
something that they do not already know. (Killen, 2000)
 ETHICAL ISSUES

• Asking students to answer sensitive questions

• Grades and reports from invalid and unreliable test instruments
• Confidentiality of results
• Deception in regards to the purpose and use of the assessment
• Temptation to assist student in answering tests or responding to
surveys
POSITIVE CONSEQUENCES

• High-quality assessments have positive consequences

for yourself, instruction, and your students.
• Positive Consequences for Students
• Positive Consequences for Teachers and
Instruction
ALIGNMENT WITH STANDARDS

•Degree of agreement among standards,

tests, curriculum and instruction
•what is taught is about the same as what
is tested (instructional validity)
PRACTICALITY AND EFFICIENCY

•Teacher familiarity with the method

•Time required
•Complexity of Administration
•Ease of Scoring
•Ease of Interpretation
•Cost
ACTIVITY 2.

•Enumerate and explain each principle of

high quality assessment, then illustrate it by
providing an example for each principle.
•Explain “All reliable assessments are valid
but not all valid assessments are reliable.”