WP2 1 Evaluation Strategy Design » History » Version 9

Steve Welburn, 2012-08-08 04:08 PM

1 1 Steve Welburn
h1. WP2.1 Evaluation Strategy Design
2 1 Steve Welburn
3 2 Steve Welburn
In order to evaluate the (training) materials, it is necessary to:
4 2 Steve Welburn
* identify specific (learning) objectives which they aim to meet
5 2 Steve Welburn
* evaluate whether the materials meet those objectives
6 2 Steve Welburn
additionally, we need to 
7 2 Steve Welburn
* identify the overall purposes of the materials
8 2 Steve Welburn
* evaluate whether the cumulative objectives satisfy these overall purposes
9 1 Steve Welburn
10 1 Steve Welburn
h2. Methods of Evaluation
11 1 Steve Welburn
12 9 Steve Welburn
Design review
13 1 Steve Welburn
14 9 Steve Welburn
Pre-course (evaluation of materials)
15 9 Steve Welburn
* "Cloze Test":http://en.wikipedia.org/wiki/Cloze_test to assess readability
16 9 Steve Welburn
* I-Tech Instructional Design & Materials Evaluation "Form and Guidelines":http://www.go2itech.org/HTML/TT06/toolkit/evaluation/forms.html
17 9 Steve Welburn
* Focus Groups ("I-Tech guide":http://www.go2itech.org/HTML/TT06/toolkit/evaluation/forms.html) e.g. some of C4DM
18 9 Steve Welburn
* Internal use of materials i.e. C4DM
19 9 Steve Welburn
20 1 Steve Welburn
On-going evaluation of course 
21 1 Steve Welburn
22 5 Steve Welburn
* Informal / Formal Review e.g.:
23 9 Steve Welburn
** Questionnaire to see how easy it is to find relevant material / test users knowledge
24 5 Steve Welburn
** Focus Groups ("I-Tech guide":http://www.go2itech.org/HTML/TT06/toolkit/evaluation/forms.html)
25 1 Steve Welburn
* In-course / formative evaluation (see "I-Tech":http://www.go2itech.org/HTML/TT06/toolkit/evaluation/dev_eval.html)
26 1 Steve Welburn
** Assessment of level of knowledge within training group
27 1 Steve Welburn
** Checking progress with participants
28 1 Steve Welburn
** Trainer assessment - self assessment and from other trainers if possible
29 5 Steve Welburn
** Pre- and Post-course questionnaires - assess change in answers (true/false + multiple-choice)
30 5 Steve Welburn
* Post-course / summative evaluation
31 5 Steve Welburn
** Debriefing of trainer (did it work ? to time ? did it engage people ?)
32 5 Steve Welburn
** Questionnaire for participants "Sample training evaluation forms":http://www.go2itech.org/HTML/TT06/toolkit/evaluation/forms.html
33 4 Steve Welburn
** Medium-term review of usefulness of course content / adoption of techniques (e.g. 2-3 months after course)
34 5 Steve Welburn
35 5 Steve Welburn
"Kirkpatrick's Four Level Evaluation Model":http://www.nwlink.com/~donclark/hrd/isd/kirkpatrick.html
36 4 Steve Welburn
* Reaction - to the course ("motivation" may be more appropriate)
37 4 Steve Welburn
** Pacing, was it enjoyable
38 4 Steve Welburn
* Learning - from the course
39 4 Steve Welburn
** Did the facts get across ?
40 4 Steve Welburn
* Behavior - changes after the course
41 4 Steve Welburn
** Did participants actually manage their data better
42 4 Steve Welburn
*** during research ? 
43 4 Steve Welburn
*** at the end of research ?
44 4 Steve Welburn
** Have data management plans been produced for grant proposals ?
45 4 Steve Welburn
* Results or Impact - to a wider community
46 2 Steve Welburn
** Did they publish data ?
47 2 Steve Welburn
** Was any data loss avoided ?
48 1 Steve Welburn
49 2 Steve Welburn
Review content for:
50 2 Steve Welburn
* reading level
51 2 Steve Welburn
* correctness
52 1 Steve Welburn
* organization
53 1 Steve Welburn
* ease of use
54 1 Steve Welburn
55 1 Steve Welburn
based on the target audience
56 5 Steve Welburn
57 2 Steve Welburn
h2. Tools and links
58 1 Steve Welburn
59 7 Steve Welburn
"Bristol Online Surveys":http://www.survey.bris.ac.uk/
60 7 Steve Welburn
61 7 Steve Welburn
"I-Tech Training Toolkit":http://www.go2itech.org/HTML/TT06/toolkit/evaluation/index.html 
62 1 Steve Welburn
63 6 Steve Welburn
Instructional System Design approach to training
64 1 Steve Welburn
* "Analysis, Design, Development, Implement, Evaluate (ADDIE)":http://www.nwlink.com/~donclark/hrd/sat6.html
65 1 Steve Welburn
66 6 Steve Welburn
Free Managemnt Library - "Evaluating Training and Results (ROI of Training)":http://managementhelp.org/training/systematic/ROI-evaluating-training.htm
67 8 Steve Welburn
68 8 Steve Welburn
Lingualinks "Implement A Literacy Program - Evaluating Training":http://www.sil.org/lingualinks/literacy/ImplementALiteracyProgram/EvaluatingTraining.htm
69 8 Steve Welburn
70 9 Steve Welburn
"Training Works!...":http://www.reproline.jhu.edu/english/6read/6training/Tngworks/index.htm ...what you need to know about managing, designing, delivering, and evaluating group-based training