Formative Evaluation Plan
Project Title
EdTech Certificate
Project Manager
Brouke Reynolds
Clients
Dr. Clif Mims & Dr. Jennifer Townes
Sponsor
Dr. Trey Martindale
Date
October 17, 2016
Revision History
Revision Date
Revised By
Description of
Change
Overview
The purpose of the formative evaluation process is to review the four courses designed for the
IDT EdTech certificate. The courses have been redesigned and will be facilitated and
managed via Google Classroom, a new LMS for the University of Memphis. The newly
redesigned classes are a hybrid of original material in the four courses, as well as new
material to reflect updated technologies and to utilize G Suites. The four courses are each
seven weeks long, and the evaluators will determine if the assignments meet that time
requirement as well as the validity of the instruction and assessments presented. The
evaluation will be conducted in three phases: SME Review, One-to-One Evaluation, and
Small Group Evaluation.
SME Review
The subject matter expert (SME) will have expert knowledge in the four courses and will be
able to determine the validity of the course material based on her experience teaching the
courses. The SME will be provided a checklist with the criteria for each of the modules for
the four courses. The SME will offer revisions and changes that will be made before the next
phase of the process. Each subsequent phase by the one-to-one and small group evaluators
will be updated based on results of the data collection, before a final product is presented
during the showcase and to our clients.
Dr. Jennifer Townes will act as our SME for this project. She has taught several of the courses
in the past, is part of the IDT department, and is one of our clients. She has expert knowledge
of what the expectations are for these courses. We want her to analyze our instructional
goals, accuracy and validity of content, clarity and appropriateness of module instructions,
sequencing of instruction, ease of using new LMS, and rigor of projects.
Dr. Townes will have access to all four classes which will be built in the Google Classroom
LMS. She will have access to assignments in the same environment that the students will
experience. The SME will review all learning modules and provide us with feedback so
revisions may be made before the next phase of the project. The checklist used by the SME
may be accessed here.
One-to-One Evaluation
The second phase of the evaluation is the one-to-one evaluation which will be completed by
past instructors and/or students of the courses. During one-to-one evaluation, data will be
collected via an interview and an attitude survey. The purpose of the One-To-One evaluation
is to test the courses with two learners with some previous experience of the courses either as
instructors or as students. The one-to-one participants were recommended by the client.
Kristy Cogner and Fair Josey will act as evaluators for this phase. The participants will
provide feedback to improve the four courses before the next phase of testing. The courses
will be revised based on problems the participants discovered before the Small Group
Testing.
The participants will have access to the courses via Google Classroom. Data will be collected
via an email interview and an attitude survey. Because of the distance constraint, the
participants will not be directly observed by a member of the team, therefore observation
notes are not possible.
Small Group Evaluation
The final phase of the formative evaluation will be the small group evaluation, which will
also be completed by past students, instructors, and/or experts in the field. Data will be
collected by a checklist, interviews, and an attitude survey. The purpose of the small group
evaluation is to test the courses with four learners that do not have direct experience with the
courses, but are experienced in the field of IDT. The small group participants were
recommended by the clients. Drew Polly, Gretchen Thomas, Rick West, and Tracy
McAllister will act as our small group evaluators. The evaluators will move through the
courses after the recommended changes from the SME and one-to-one evaluations have been
completed. The participants will provide feedback, and adjustments will be made based on
the feedback. This will be the last phase before our project is presented in the showcase.
The participants will have access to the courses via Google Classroom. Data will be collected
via an email attitude survey. Because of the distance constraint, the participants will not be
directly observed by a member of the team, therefore observation notes are not possible. Due
to the nature of the four courses, a pre and post test will not be administered.
Resources
Evaluation
Participants
Unit
Computer
Access
Forms
SME
1 client
Four Google
Classroom courses
Personal
Computer
Checklist
One-to-One
2 former
students/instructors
Four Google
Classroom courses
Personal
Computer
Participant
Interview
Attitude Survey
Small Group
Evaluation
4 field experts
Four Google
Classroom courses
Personal
Computer
Checklist
Interview
Attitude Survey
Evaluation Schedule
Weekly Evaluation Schedule
Evaluation
Type
Person
Evaluation
Begins
Location
Ends
SME
Dr. Jennifer
Townes
October 31
November 4
Personal Office
One-To-One
Fair Josey
November 7
November 11
Personal Office
One-To-One
Kristy Conger
November 7
November 11
Personal Office
Small Group
Tracy McAllister
November 14
November 18
Personal Office
Small Group
Rick West
November 14
November 18
Personal Office
Small Group
Drew Polly
November 14
November 18
Personal Office
Small Group
Gretchen Thomas
November 14
November 18
Personal Office
Data Analysis
The checklists, interviews, and surveys will undergo a qualitative analysis and organized
categorically to determine changes to the flow and verbiage of the instructional units. All data
will be organized in a spreadsheet for interpretation.
Reference
Dick, W., Carey, L., & Carey, J. O. (2005). The systematic design of instruction. Boston:
Pearson/Allyn & Bacon.
Instruments
SME Checklist
IDT 7061
Clarity
Appropriate
Placement in course
Clarity
Appropriate
Placement in course
Module 1
Module 2
Module 3
Module 4
Modul 5
Module 6
Module 7
IDT 7062
Module 1
Module 2
Module 3
Module 4
Modul 5
Module 6
Module 7
IDT 7063
Clarity
Appropriate
Placement in course
Clarity
Appropriate
Placement in course
Module 1
Module 2
Module 3
Module 4
Modul 5
Module 6
Module 7
IDT 7064
Module 1
Module 2
Module 3
Module 4
Modul 5
Module 6
Module 7
Interview Protocol
Participant Name:
Date:
Type of observation: Interview
1. Was the course easy to navigate?
2. What is your opinion of the format of the course?
3. Was the content easy to understand?
4. Would the projects be easy to complete based on the instruction provided?
5. What projects are unnecessary, if any?
6. Is there any additional content that you would like to see added?
7. Do you have any additional comments or feedback?
Attitude Survey
Client Sign Off
Clif Mims
10/23/16
_______________________________________ ___________________
Dr. Mims, Client
Date
_______________________________________ ____________________
Dr. Townes, Client
Date
_______________________________________ ____________________
Dr. Martindale, Project Sponsor
Date
_______________________________________ ____________________
Brouke Reynolds, Project Manager
Date
Formative Evaluation Report
Results
SME Review
The SME received access to the course on Monday, October 31st as originally planned. Prior to
delivering the materials, it was decided that we would focus on finalizing 7061, 7062, and 7063,
and that we would deliver 7064 by Thursday of the review week. Many of her comments were
about Google Classroom, how the class would be setup within the LMS, and how it compared to
eCourseware. We unfortunately do not have much control over how Google Classroom formats
different things, and it will be very different from eCourseware in the fact that it is a stream feed
and not a website. The revisions after the SME review are listed below:
Revisions
Every team member reviewed their modules to apply the Heading Styles, spacing,
naming scheme, etc. to make all course design consistent with each other.
Every team member reviewed over all of the objectives in the modules to make sure that
they were measurable and were applicable to the assessment.
BadgeSchool submission directions were added to each assignment. This was added for
all assignments, regardless if there was a badge or not for that particular module.
All video links were turned into clickable images that were hyperlinked to YouTube
videos. This broke up the content with purposeful videos to give students more
instruction.
In 7061.3, the instruction over writing objectives involved color to identify the different
parts of the objectives. It was suggested that they be rewritten using initials to represent
the sections of the objectives. We adjusted the instruction to reflect her suggestion.
In 7062.2, the spreadsheet template for the Skills Map was no longer needed as the
students are to now write the skills map using one of the Web 2.0 tools. It was removed
from the module and Google Classroom.
In 7064, there was some confusion on the layout inside of Google Classroom and the
timeline of the course. This course is different from the other courses because it only has
two modules and the rest of the course is the unit design project. The checkpoints were
renamed to be more clear. A schedule was added to the Orientation Modules, and in the
actual class, due dates would be added to all assignment posts.
Recommendations
We opted not to include any rubrics for the modules to allow for the professor to have
autonomy on how they would like to grade the courses that they teach.
Through viewing Google Classroom and the documents in different browsers and screen
resolutions, it was noted that the spacing and page breaks appear differently depending on
which browser you are using. It is recommended that students and teachers use Google
Chrome to access the classroom and documents for the best viewing experience.
We recommend using the topic tags to help organize posts. In the actual course, it
would be beneficial to sort posts by individual modules because there will potentially be
multiple posts per module. In our case, we typically only had one post for each module.
Because Google Classroom works as a stream and not a static website, posts will
continuously move and be pushed down the page as more posts are added. Labeling all
posts with the topics will help with organization.
One-to-One
In the evaluation plan, we had originally planned for only two evaluators. Based on a
recommendation from the client, we sent the materials to six evaluators during the one-to-one
phase. At the end of the phase, only five evaluators had completed the interview questions, and
only four evaluators completed the attitude survey. Evaluators commented that the courses were
easy to navigate, the modules were formatted well, the content was easy to understand, and the
assessments were appropriate for graduate level work. The revisions and recommendations of the
one-to-one evaluation phase are listed below:
Revisions
Every team member reviewed their modules for grammar and spelling.
Instructional content was reviewed for extraneous text to shorten documents. This was
done to address concerns that the instructional documents were too long and required
scrolling.
More reflection components were added using the Visible Thinking strategies that
students learn about during the courses. Students also complete multiple peer reviews of
each other's lesson plans.
7061.1 - Broken link from UGA website was fixed with correct link.
A Google Classroom orientation document was added to give students an overview of
Google Classroom and G Suite. Professors will need training before using Google
Classroom.
A BadgeSchool orientation document was also added to give students an overview of
BadgeSchool, badges, how to upload their evidence into BadgeSchool, and what they can
do with their badges after earning them.
The submission directions of each assignment were rewritten to better reflect the use of
Google Classroom and G Suite.
Recommendations
One evaluator suggested adding a portfolio aspect to the program. This was specifically
eliminated from the program by the request of the client.
When assignments are added by the professor, they will be able to add a due date to each
assignment. This will compile all due dates into the Google Calendar. For the purposes of
our design, we did not have exact dates for each assignment. A suggested general course
outline is included in the orientation module.
The MTVT lesson plans are used as a midterm and/or final to have students apply
the previous instruction in a classroom setting.
Personalization Principle - Two evaluators mentioned the casual, personal writing style.
The team discussed this thoroughly and cited the Personalization Principle as justification
for keeping this style of writing. The use of conversational style writing helps students
learn and retain information in an e-Learning setting.
Small Group
We sent the materials to the four evaluators originally planned for the small group evaluation. At
the end of the phase, none of the evaluators had completed the interview questions. They were
given additional time and asked to have it complete by the following Tuesday. All four
evaluators were able to give feedback by the second deadline, but this was after the submission
deadline for the course. Evaluators were only given the attitude survey to complete. One
evaluator rated the modules with an all 1 rating, the lowest score, and did not leave any
additional feedback. We did ask the evaluators to clarify the reasoning for their ratings so that we
could improve the modules accordingly. The evaluator responded that ratings were a mistake,
and that they had meant to rate them with an all 5 rating, the highest score. All evaluators rated
all survey questions with either a four or five rating.
During this phase, we also received informal feedback from Dr. Mims, one of our clients. This
feedback was not solicited, and did not follow any of the interview or attitude survey questions.
The feedback was very helpful in making specific changes that the evaluators missed because
they are not as familiar with the expectations of the certificate or the wants of the client. A
majority of the revisions for this phase came from the clients feedback.
The revisions of the small group evaluation phase are listed below:
Revisions
Every team member was to review their modules for Visible Thinking. It is a proper
noun and should therefore be capitalized. Several times it was also referred to as Visual
Thinking. These are two separate things.
Every team member was to adjust the footer of their modules to a size 10 and gray font to
better distinguish the footers from the content.
BadgeSchool directions were reformatted to distinguish them separately from the
assignment and submission.
7062.6 - The assignment was originally supposed to be a paper over a type of Adaptive
Technology. The EdTech program is supposed to be all project based assignments. This
was rewritten to be a screencast presentation.
There is an announcement post that introduces the SIP and Unit Design that was
identified by Planning Ahead labels. There is later an assignment post where students
would actually turn in these assignments.
Recommendations/Conclusions
Based on the evaluations and revisions, it is our recommendation that the modules are ready for
implementation. It is our recommendation that each course is a prerequisite for the next.
Professors will need some training and time to become familiar with the new Google Classroom
LMS. It should be noted that the courses were not tested with actual Educational Technology
students, but with former students, professors, and experts in the field.
