Scribd
Upload
11 views

Lesson 12 Evaluation

Uploaded by

dylanzhang777
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
11 views

Lesson 12 Evaluation

Uploaded by

dylanzhang777
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 23

Lesson 12

Implementation &
Evaluation in
Instructional Design
Lijia Lin
University of Macau
ADDIE
 Analysis
 Design
 Development
 Implementation
 Evaluation
Implementation
 To execute the instruction, to put it into practice
 Preparation for students
 Preparation for teachers

Preparation for Students


 Schedule: number of students, location, etc.
 Pre-course Communication: location, date, equipment, etc.
 Tracking and maintaining student records (e.g., exams,
attendance, surveys)
Preparation for Teachers
 Provide teacher/instructor guide
 Provide trainings for teachers/instructors
 Logistical issues (e.g., location, technology)
In Reality
 Something unexpected is very likely to happen.
 Teacher does not follow the teacher guide.
 Technology does not work.
 Students are discussing with each other while it should be
individual work.
 ……
 It is very common.
Implementation
 Implementation usually involves data collection.
 Quantitative: test, survey, etc.
 Qualitative: survey, interview, observation, etc.
 Data are used for evaluation.
Evaluation
 Evaluation frameworks
 Two types of evaluation
Evaluation Frameworks
 Design principles
 Practice-oriented: CRAP principle
 Research-based: modality principle, redundancy principle, etc.
 Taxonomy of learning
 Bloom’s Taxonomy
 Gagné’s Classification Targeting objectives and assessment
 Sullivan & Higgins’s Taxonomy
Bloom’s Taxonomy
 3 domains: cognitive, affective, & psychomotor
 Cognitive domain has been a major focus.
 Knowledge, comprehension, application, analysis, synthesis, evaluation
Gagné’s Classification of Learning Outcomes
 Intellectual skills
 Verbal information
 Cognitive strategies
 Motor skills
 Attitudes
Other Evaluation Frameworks
1. Kirkpatrick’s model
2. SAMR model
3. Context-Input-Process-Output (CIPO) model
1. Kirkpatrick’s
Developed by DonFour-Level
Kirkpatrick in 1950s Evaluation

Model
 Level 1: Reaction
 Level 2: Learning
 Level 3: Behavior
 Level 4: Results
2. SAMR Model
 Dr. Ruben Puentedura
(2015)
 Technology can play
different roles in
learning tasks.
 Can guide teaching
practice & evaluation
An Example Based on SARM Model
 To write an essay on paper
Substitution  To write an essay on a computer/tablet
Augmentation  To write an essay on Google doc to let other
classmates comment on it
Modification  To write a blog post by doing research online,
shared editing online, & publishing online
Redefinition  To do research online, video conferencing with
experts, creating podcasts or YouTube videos
Context-Input-Process-Output (CIPO)
Model

Context

Input Process Output

Stufflebeam DL, Zhang G. The CIPO Evaluation Model: How to Evaluate for Improvement and Accountability. 1st ed. New York:
Guilford Publications; 2017.
Context-Input-Process-Output Model
 Context evaluation
 Needs, obstacles or resources for meeting the needs…
 Input evaluation
 Feasibility and cost
 Process evaluation
 Do/did the program go well?
 Output evaluation
 Outcome, results
Two Types of Evaluation
 Formative evaluation / formative assessment
形成性评估
 Summative evaluation / summative assessment
终结性评估
 When will the evaluation be administered?
Formative Evaluation
 According to Cronbach (1975) and Scriven (1973)
 The collection of data and information during the
development of instruction that can be used to
improve the effectiveness of the instruction

Cronbach, L. J. (1975). Course improvement through evaluation. In D. A. M. Payne, R. F. (Ed.), Educational and psychological
measurement: Contributions to theory and practice (2nd ed., pp. 243-256). Morristown, N.J.: General Learning Press.
Scriven, M. (1973). The methodology of evaluation. In B. R. Worthen & J. R. Sanders (Eds.), Educational evaluation: Theory and
practice (pp. 60-106). Belmont, California: Wadsworth Publishing Company, Inc.
When to conduct formative
evaluation?
 After the first draft of instruction is developed
 Could be earlier in the design and development
process
Data Sources
Phases of Formative Evaluation
 Review by specialists/experts
 The designer works with individual learners (1 to 1)
to obtain data.
 A small-group evaluation
 Field trial with larger samples

Dick, W., Carey, L., & Carey, J. O. (2015). The systematic design of instruction.
Rating Scale
 Can be used by experts or target learners
Criteria Rating
1: Not aligned 2: Partial alignment 3: Mostly aligned 4:
Alignment with Learning Objectives Fully aligned

Clarity of Content 1: Unclear 2: Somewhat clear 3: Clear 4: Very clear

1: Minimal engagement 2: Some interactivity 3: Engaging


Engagement and Interactivity 4: Highly interactive

1: Ineffective 2: Needs improvement 3: Adequate 4:


Assessment Effectiveness Effective

Media and Visuals 1: Poor quality 2: Fair 3: Good 4: Excellent

User Experience (UX) 1: Frustrating 2: Usable 3: Pleasant 4: Exceptional


Formative Evaluation Framework

 An integrated framework from The


Systematic Design of Instruction
(2015) by Dick, Carey, & Carey
Note
 Formative evaluation can be very informal.
 Make decisions on your own.
 Nothing is perfect. You need to decide when to
move on to the next stage.
Summative Evaluation
 Data collected to determine the effectiveness of the
final version of the instructional program/material
 Final judgment, high stakes 高風險

Phases of Summative Evaluation


 Expert judgement
 Field trial
ADDIE
A step-by-step instructional design procedure
 Analysis
 Design
 Development
 Implementation
 Evaluation

You might also like