Lecture 9 Designing

Open-ended assessment
questions (speaking and
writing)
[email protected]
 Objectives
• Interpret /Evaluate the appropriateness of
language assessment instruments for the
purpose of assessment-for-learning
• Distinguish major performance-based
assessment instruments
• Understand key principles in designing major
performance-based language assessment
instruments
• Evaluate major performance-based
assessment instruments
 Main contents
1) Open-ended questions
2) Performance-based assessment
instruments/questions
3) Essays – a category of open-ended
questions
 Checking your reading:
• Can you differentiate ‘open-ended
questions’ and ‘performance-based
assessment’?
• When we want to assess student’s recall
or recognition skills, can we use essay
items? If yes, should we use?
• Open ended

Performance- Essay
based
OPEN-ENDED QUESTIONS
 Open-ended assessment tasks
• allow test takers to relatively freely respond
in whatever way they choose.
• are suitable for assessing language
proficiency in authentic situations that call
for interactive, dynamic language use.
• are often used to assess the skills of
advanced level learners.
 Open-ended assessment tasks
• suitable for testing speaking and writing
skills because they require language
production.
• used to assess higher order skills and a
combination of skills.
• require more demanding scoring.
 Features
• Guidelines for item construction
- Appropriateness
- Understandability
- Feasibility
 Guideline Questions for item crafting

• Is the task appropriate with respect to

instructional objectives and instructional
activities?
• Is the task understandable with respect to
expected performance and assessment
standards?
• Is the task feasible with respect to topic,
level of difficulty, and time?
 Appropriate
In terms of…
• learning objectives
• learning activities
• students’ expectations of performance
standards
• actual situations or simulations in which
students use L2
 Guideline for Appropriateness
Objectives
• elicit the kinds and range of language skills in the
instructional objectives (What)
• Design the language skills so that they assess the
students’ performance according to the expected
standard (how high)
Conditions
• Consider the actual performance demands (how real)
• Prepare the students for the task in the above areas
(What, how high, how real)
• Example of an inappropriate task for a final-
year research method course
• List the 7-step path in the process to making “research
decisions.” List them in their correct progressive order.
• Problems:
– The question does not require students to use complete sentences or more
than one sentence.
– all the responses they give may be identical. Students will basically repeat
the 7-step path word for word. The question does not allow for original
responses or response patterns.
– Although the question does require a list of steps, it does not require
systematic and complex thinking about the ideas.
• A better version
• Explain in what ways a person’s failure to
apply the steps of the seven-step path for
making research decisions will impact his
or her ability to make research decisions.
Provide an example that illustrates this
impact.
• Example of inappropriate task for 7-
graders in public schools in Vietnam (in
terms of learning objectives and
performance standards):
– Write a job application letter
– Compose a play and act out with a friend
 Tasks of different appropriateness

Example 1 Example 2
• What are the major • Given their advantages and
limitations, should an essay
advantages and question be used to assess
limitations of essay students’ abilities to create a
questions? solution to a problem? In
answering this question, provide
brief explanations of the major
advantages and limitations of
essay questions. Clearly state
whether or not you think an essay
question should be used and
explain the reasoning for your
judgment.
Example 1 Example B
• assesses recall of factual • requires more of
knowledge students.
• requires students to recall
facts, to make an
evaluative judgment, and
to explain the reasoning
for the judgment
• requires more
complicated thinking
 Understandable
In terms of…
• task instruction (simple, straightforward,
clear)
• standards of performance (scoring criteria,
weighting, etc.)
 Guideline for Understandability
Language issues
• Clarify the task demands: What
• Clarify the standards of performance (How) and
evaluation (How to assess)
 Feasible
• Topic
• Difficulty
• Time
 Topic
• Can elicit the kind(s) of skills you want to
assess
• Is authentic
• Is realistic
• Is interesting
• Is fair
• Example of an infeasible topic for general
students:
– The advantages of Buddhism for young
people
– The aeronautical science
– Working in large cities
 Difficulty
• The form of the tasks should be of
appropriate difficulty, neither too difficult
nor easy.
• Too easy/difficult tasks cannot distinguish
students who have made more or less
progress.
 Time
• Students should be given enough time to
do the task to be able to demonstrate their
full achievement.
• Test performance under speeded
conditions is not usually the best indicator
of maximum performance capabilities.
Inappropriate time allowance
• Create a writing portfolios from nothing in
one week
• Write a 250 word essay in 15 minutes for
ULIS students
PERFORMANCE-BASED
ASSESSMENT
• Definition
• Components
• Varieties
• Issues in performance assessment
• Evaluating performance tasks
 Definition
• Require students to apply knowledge and skills
from several areas
• Present a hands-on task (experience-sharing)
• Use clearly defined criteria to evaluate
 Components
• Performance task
• Rubrics of scoring
 Performance task
• Requires a student to directly demonstrate
her achievement by producing an
extended written or spoken answer (in
language or content-based courses)
– by engaging in group or individual activities
– by creating a specific product.
• can assess both product and process.
 Rubrics for scoring
• a coherent set of rules you use to assess
the quality of a student's performance.
– Example: a rating scale, a checklist
• A scale: reflects the quality levels of
performance.
 Types of performance tasks
• Structured (paper and pen tasks)
• Naturally Occurring
• Longer-term projects (individual or group)
• Portfolios (best work, growth/learning
process)
• Demonstrations
• Experiments
• Oral Presentations and Dramatizations
• Simulations
 A checklist for judging the quality of performance tasks.
1. Does the task focus on an important aspect of the unit's learning targets? Yes No

2. Does the task match your assessment plan in terms of performance, emphasis, and number of points Yes No
(marks)?
3. Does the task require a student to actually do something (i.e. a performance) rather than only write Yes No
about how to do it or simply recall or copy information?
4. Do you allow enough time so all of your students can complete the task under your conditions? Yes No

5. If the task is an open-response task, do your wording and directions make it clear to students that Yes No
they may use a variety of approaches and strategies ,that you will accept more than one answer as
correct, and that they need to fully elaborate their response?

6. If the task is intended to be authentic or realistic, do you present a situation that your level of Yes No
students will recognize as coming from the real world?
7. If the task requires using resources and locating information outside of the classroom, will all of your Yes No
students have fair and equal access to the expected resources?

8. Do you directions and other wording: Yes No

a. define the task that is appropriate to the educational maturity of your students ?
b. lead all student ,including those from diverse cultural and ethnic backgrounds, to interpret the task
requirements in the way you intend?
c. make clear the purpose or goal the task?
d. make clear the length or degree of elaboration of the responses to the task ?
e. make clear the bases on which you will evaluate the responses to the task ?
9. Are the drawings, graphs, diagrams, charts, manipulatives, and other task materials clearly drawn, Yes No
properly constructed, appropriate to the intended performance; and in good working order?

10. Do you need to modify or adapt the task to accommodate students with disabilities? Yes No
• Which assessment qualities have been in
focus?
• Compare/contrast this list and the list for
close-ended assessment tasks and make
some comments
ESSAYS
• Varieties
• Usefulness
• Scoring
• Improving essay items
 Varieties
• Restricted response
• Extended response
 Restricted response
• restrict or limit what student answer
– the content of the answer
– the form of the written response.
• narrows the focus to a specific and well-defined
performance.
• could be used to assess more than recall or
recognition (analyzing, problem solving).
• may test student’ ability to interpret materials.
• Are subjectively scored
• Examples in a CLIL course
– Describe two situations that demonstrate the
application of the law of supply and
demand. Do not use those examples
discussed in class.
– State the main differences between the
Vietnam War and previous wars in which the
United States has participated.
 Extended response
• allow students to be free
– to express their own ideas & interrelationships among
their idea
– To organize arguments
– to organize their answers.
• Can be used to assess the subject matter, general
writing skill or a combination of skills.
• Are more subjective and difficult to score
• Example
• Evaluate the qualities of the provided
assessment plan. Provide evidence for
your arguments where possible.
• Write a research essay to explain how
people’s practices in the modern world are
affected by globalization. Include both
personal and empirical evidence in your
essay.
 Usefulness
• Opportunity to display their abilities to write
about, to organize, to express, and to explain
interrelationships among ideas.
• Students will study what they will be tested on,
yet their study strategies will depend on the level
of the thinking skills, not the exam format.
• Can assess in-depth learning.
 Discussion
How can we improve the scoring reliability
of extended response?

How can we improve the limitation of the

essay items that they cannot cover a wide
range of knowledge content?
 Other issues
• Use of teacher’s time: issues of professional
ethics and abuse of responsibility.
• Factors influencing reliability and validity:
- inconsistent standards
- rater drift
- changes in topic and prompt
- reader biases: halo effect, carryover effect
Checklist for judging the quality of essay items
1) Does the item test an important aspect of this unit’s learning target? Yes No
2) Does the item match the specifications of your test plan in terms of required Yes No
performance, emphasis, and number of points?
3) Does the item require students to apply their knowledge and skill to a new or novel Yes No
situation?
4) When viewed in relation to other items on the test, does this item contribute to Yes No
covering the range of content and behavior specified in your test plan?
5) Is the item focused? Does it define a task with specific directions, rather than leave Yes No
the assignment so broad that virtually any response can satisfy the question?

6) Is the task defined by the item within the level of complexity that is appropriate for Yes No
the educational maturity of the students?
7) To get a good mark on the item, is the student required to demonstrate more than Yes No
recall of facts, ideas, lists, definitions, generalization, etc.?
8) Is the item worded in a way that leads all students to interpret the assignment in the Yes No
way you intended?
9) Does the wording of the item make clear to the students the following: Yes No
- Magnitude of length of the required writing?
- Purpose for which they are writing?
- Amount of time to be devoted to answering this item?
- Basis on which their answers will be evaluated?
10) If the essay item asks students to state and support their opinions on controversial Yes No
matters, does the wording of the item make it clear the students’ assessment will
be based on the logic and evidence supporting their arguments, rather than the
actual position taken or opinion stated?
• Compare/contrast this essay question
checklist and the list for open-ended task
(A, U, F)
• Practice exercises: Performance-based
assessment + Exercise 1, 2 Essays
• In groups: Practice evaluating
performance-based assessment tasks in a
certain review unit in the new English text
book