Instructional Design Standards for

Distance Learning

Edited by
Anthony A. Piña
Sullivan University
 Association for Educational Communications
and Technology
320 West 8th Street, Suite 101
Bloomington, Indiana, 47404-3745

ISBN 978-0-9970755-2-6 (trade paperback)

ISBN 978-0-9970755-3-3 (online PDF)

Copyright © 2017 Association for Educational Communications

and Technology
All rights reserved
 3

Table of Contents

Foreword
Phillip Harris...................................................................................................... 4

Introduction
Anthony A. Piña ............................................................................................... 7

The Standards ................................................................................................ 11

Purpose
Michael Molenda ............................................................................................. 14

Assumptions
Michael Simonson ........................................................................................... 27

Instructional Sequences
Wilhelmina C. Savenye and Yi-Chun Hong .................................................... 33

Activities
Gary R. Morrison ............................................................................................. 47

Resources
Anthony A. Piña .............................................................................................. 54

Application
Peggy A. Ertmer, Jennifer C. Richardson and Judith Lewandowski................ 62

Assessment
Steven M. Ross and Jennifer R. Morrison ....................................................... 75

Reflection
Monica W. Tracy ............................................................................................. 81

Independent Learning
David W. Price, Saul Carliner and Yuan Chen ................................................ 94

Evaluation
Anthony A. Piña ............................................................................................ 107

Sample Rubrics ............................................................................................ 117

 4

Foreword
Phillip Harris
Association for Educational Communications and
Technology

In 1995, when Utah Gov. Mike Leavitt proposed the

idea of an online university to the Western Governors
Association, distance learning was not new, and even
online learning was already taking place in a few settings.
Since then, with the exception of a few growing pains,
online learning has done nothing but expand. In the spring
of 2017, when Purdue University acquired what remained
of the for-profit Kaplan University, with an eye to folding it
into a nonprofit institution, Purdue President Mitch Daniels
claimed in an official statement, “None of us knows how
fast or in what direction online higher education will
evolve, but we know its role will grow.”

While that comment will no doubt continue to prove

true from an institutional standpoint, the question of the
quality of the online learning that takes place is still
unanswered. One central reason that question remains apt is
 5

that no research-based standards for the design of online

learning exist. There are standards for content, and there
are delivery standards as well. But there were no design
standards for online learning.

The version of the standards presented here marks

the beginning of what we hope will be an extended
conversation. The standards began as a series of
discussions between AECT staff and members about the
large number of universities now developing online courses
without the guidance of any research-based standards for
designing online learning. Working from the extensive
research literature on instructional design, two AECT
members drafted the standards, and an edited version was
approved by AECT’s Executive Committee and shared
with selected members of the Division of Distance
Learning. Once all comments had been addressed, they
were presented to the full Board of Directors, and they too
approved them.

At this point, the authors of each of the chapters

included in this publication were invited to flesh out the
rationale for each standard. We approached Dr. Anthony
Piña, Past-President of AECT’s Division of Distance
 6

Learning, to arrange for peer review of the chapters, serve

as general editor and put them together in this publication.

We present them here as a way to start the conversation.

Phillip Harris
Executive Director, AECT
Bloomington, IN
 7

Introduction
Anthony A. Piña
Sullivan University

Throughout my nearly three decades as a member

of the Association for Educational Communications and
Technology (AECT), I have heard the lamentations of my
colleagues that principles of systematic instructional design
and other principles that we hold dear have not made the
impact into education at large that we have hoped.
However, in both my current role as Associate Provost for
Instruction and Online Learning at my institution and as a
peer reviewer for the Southern Association of Colleges and
Schools Commission on Colleges (SACSCOC), my
experience is quite different.

In my efforts to affect “bottom-up” change at my

institution, we established standards for online master
courses that emphasized learning objectives as measurable
student outcomes, assessment activities aligned with
objectives, and instructional activities designed to bridge
the gaps between objectives and assessments. I empowered
my team of instructional designers to be able to enforce the
standards for online master courses as they worked with
our faculty subject matter experts. In other words, we
pursued a course that is very familiar to those involved in
AECT.

Our goal, of course, was to use the “bottom up”

approach to influence our campus design culture--
particularly for on-campus (i.e. face-to-face) courses. As
with many institutions, our culture for many years had been
 8

that on-campus courses were designed and developed by

individual faculty members with little oversight and
sometimes “in a vacuum.” As a result, we had more
inconsistent quality among our on-campus courses, that we
did our online courses. Changing culture was a slow and
laborious process—we met with opposition in some areas
and were welcomed with open arms in others.

However, during the past few years, there has been

a change that has facilitated greatly our efforts to diffuse
instructional design across our entire institution. The
change has come “top-down” in the form of calls from our
regional and programmatic accrediting agencies and our
state and federal regulators to realign our focus and
reporting to student learning outcomes, assessment of
outcomes and continuous improvement. Note the following
items from the SACSCOC Proposed Revisions to the
Principles of Accreditation:

The institution identifies expected outcomes,

assesses the extent to which it achieves these
outcomes, and provides evidence of seeking
improvement based on the analysis of the
results in the areas below: (a) student
learning outcomes for each of its educational
programs; (b) student learning outcomes for
collegiate-level general education
competencies of its undergraduate degree
programs; (c) academic and student services
that support student success (SACSCOC,
2017).

The new “top-down” emphasis on

measurable outcomes and assessment has been
music to the ears of my instructional designers and
me. As we prepared for our reaffirmation of
 9

institutional accreditation, we saw the previous

resistance to institution-wide adoption of
instructional design principles quickly evaporate.

We had won after all!

I wish to thank Gary Morrison who got the

instructional design standards project started and who
invited me to subject chapters on resources and evaluation.
I also wish to express my gratitude to Phil Harris who
invited me to take over and run with this project.

Each of the chapters in this book underwent blind

peer review by experienced professionals in instructional
design and online education, including colleagues from the
Division of Distance Learning of the Association for
Educational Communications and Technology (AECT). I
wish to express gratitude to Cassandra Black, Linda
Campion, Jeffrey Corkran, Diane Curtis, Kathleen Decker,
Yvonne Earnshaw, Peggy Muller, Megan Murtaugh,
Ayesha Sadaf, Barry Sanford and Tony Stewart.

Finally--and especially--I wish to thank my

colleagues Saul Carliner, Yuan Chen, Peggy Ertmer, Yi-
Chun Hong, Judith Lewandowski, Michael Molenda, Gary
Morrison, Jennifer Morrison, David Price, Jennifer
Richardson, Steven Ross, Michael Simonson and Monica
Tracy. They have given freely of the fruits of their
experience and expertise and we are the beneficiaries of it
in this volume. It is a special treat for me to have
Wilhelmina Savenye as a contributor to this work. Willi
mentored me as a doctoral student and she encouraged me
to continue to pursue scholarly activities as my career led
me into leadership and administration.
 10

It is our hope that the AECT Instructional Design

Standards for Distance Learning becomes a useful tool for
our members, their institutions and others involved in
educating learners at a distance.

Anthony Piña
Past President, AECT Division of Distance Learning
Louisville, KY
 11

The Standards
Purpose. Effective course design begins with a clearly
articulated purpose. This is the standard to which all other
standards must align. Purpose may be thought of as two-
dimensional: institution or instructor and student. The
design should include both the purpose of the course as
envisioned by the institution or instructor and the purpose
as viewed by the student. As the purpose is articulated
through goals and objectives, collaboration between
instructor and student will set a firmer foundation than can
be achieved through a one-dimensional purpose statement.

Assumptions. Course design must take into account

assumptions that shape the purpose and subsequent course
development. Most assumptions are based on students’
prior knowledge and established understandings and skills.
Articulating these content assumptions provides a starting
point for new learning. Assumptions in the case of online
learning also encompass students’ ability to use delivery
technology.

Sequence. Learning opportunities must be sequenced in a

manner that promotes efficient knowledge acquisition
consistent with the prior-knowledge assumptions. Various
models of sequencing—linear, spiral, scaffold, etc.—
should be considered, and the course design should
incorporate those strategies best suited to the content within
the constraints of online delivery.

Activities. Learning is achieved through activities both

passive (reading, listening, viewing) and active
(experimenting, rehearsing, trying). Activities should be
chosen that best suit the content, students’ levels of
 12

knowledge, experience, and ability, and online delivery

constraints, particularly accommodating synchronous,
asynchronous, and mixed course participation. Student self-
selected or self-developed learning activities should be
incorporated along with instructor-selected and instructor-
developed activities, consistent with a two-dimensional
purpose.

Resources. A range of resources should be articulated to

foster deep learning and extend course-centered
experiences and activities. Resources should be multimodal
to accommodate students’ interests, understandings, and
capacities, consistent with course content and technological
accessibility. Resources should allow students to go beyond
the constraints of the formal course structure to engage in
self-directed, extended learning.

Application. Consistent with providing for active learning,

students should have integral opportunities within the
course design to apply new learning. Effective course
design incorporates opportunities to practice newly
acquired understandings and skills, both independently and
collaboratively. Online collaborative application
opportunities should be developed using social media, and
offline collegial groups also should be structured whenever
physical proximity of students affords this opportunity.

Assessment. Regardless of the model of sequencing

learning opportunities, the sequence should include points
of assessment for purposes of feedback and review, with
instances of re-teaching as necessary for students to acquire
full understanding. Formative assessment, whether formal,
informal, or incidental, allows teachers and students to give
feedback to one another and to review the operationalized
design in order to revise the course design based on
 13

students’ input with regard to knowledge acquisition and

effective use of new understandings and skills.

Reflection. Effective course design must include

opportunities for reflection as an extension of the
Feedback/Review/Reteach standard. Reflection involves
both instructor self-reflection and student self-reflection
related to achievement of the purposes that have been
articulated as the basis for the course. Such reflection is
intended to deepen the learning experience and may serve
as reiteration of purpose at key points during the course.

Independent Learning. Effective course design

incorporates opportunities for independent learning, both
instructor- and self-directed. Online course development,
particularly in the asynchronous mode, should epitomize
independent learning, which should include opportunities
for feedback, review, and reflection—all of which should
resonate with the purpose.

Evaluation. Course evaluation must be purpose-driven.

Alignment with the purpose should be threefold: a) based
on acquisition of new knowledge, understandings, and
skills; b) based on instructor self-evaluation; and c) based
on student self-evaluation. Multidimensional evaluation
offers a fully articulated basis for judging the success of the
course and the students as well as providing information
that can help shape future iterations of the course.
Purpose 14

Purpose
Michael Molenda
Indiana University

Effective course design begins with a clearly

articulated purpose. This is the standard to
which all other standards must align.
Purpose may be thought of as two-
dimensional: institution or instructor and
student. The design should include both the
purpose of the course as envisioned by the
institution or instructor and the purpose as
viewed by the student. As the purpose is
articulated through goals and objectives,
collaboration between instructor and student
will set a firmer foundation than can be
achieved through a one-dimensional purpose
statement.

Background

“Purpose”--usually referred to as goals or

objectives--has been the keystone of curriculum
development and instructional development since the birth
of modern curriculum development theory. Early in the 20th
century. Franklin Bobbitt and Werrett Charters were
colleagues in educational administration at the University
of Chicago, and both were influenced by the scientific
revolution being wrought in academia, business, and
government. In the early 1920s, both Bobbitt (1924) and
Charters (1925) published major books on curriculum
development. Both advocated strongly for the use of
Purpose 15

objectives as the foundation stones of school curricula,

arguing that useful objectives could be derived through
systematic analysis of society’s industrial needs and social
requirements. What’s more, the form of the objectives
statements they supported is not substantially different
from what is being recommended today.

Ralph Tyler, who studied under Charters at

University of Chicago and later worked with him at Ohio
State University, applied the scientific approach to the
improvement of instruction at the university. In the 1940s,
Tyler became the spokesman for the scientific approach to
curriculum development, exemplified in his concise classic,
Basic Principles of Curriculum and Instruction, first
published locally in 1949, and then republished as a mass-
market paperback in 1969. In it, Tyler enunciated the
logical framework that still informs today’s educational
practice:

The rationale developed here begins with

identifying four fundamental questions
which must be answered in developing any
curriculum and plan of instruction. These
are:

1. What educational purposes should

the school seek to attain?
2. What educational experiences can be
provided that are likely to attain
these purposes?
3. How can these educational
experiences be effectively
organized?
4. How can we determine whether
these purposes are being attained?
(Tyler, 1969, p. 1)
Purpose 16

These four questions are congruent with the four

steps in curriculum construction proposed by Charters in
1925: “selecting objectives, dividing them into ideals and
activities, analyzing them to the limits of working units,
and collecting methods of achievement” (p. 101). Bobbit,
Charters, and Tyler all propose that objectives are the
“stake in the ground” around which other
curricular/instructional decisions revolve. Indeed, now,
nine decades after Charters and six decades after Tyler,
educators have come to a firm consensus that instructional
quality rests on the mutual alignment of objectives,
learning activities, and evaluation activities.

Objectives drew heightened attention in the early

1960s as such behaviorist technologies as programmed
instruction gained popularity in education and training.
Behaviorist technology depends on clear and precise
specification of desired terminal behaviors, so advocates
devoted entire monographs to just this topic, exemplified
by Robert Mager (1962a, 1962b). By the time of Ivor
Davies’ writing in 1976, there was enough field experience
and research to fill a book with advice on the subject
(Davies, 1976).

Attention to objectives has not waned in the

succeeding generations. If anything, the centrality of
objectives has become even more widely accepted. For
example, in their guide for corporate trainers, Stolovitch
and Keeps (2002) condense their advice into five steps that
clearly focus on objectives: 1) tell the learners the rationale
for the lesson, 2) tell them the objectives of the lesson, 3)
create learning activities that lead to attaining the
objectives, 4) evaluate learner performance, and 5) provide
feedback on how well they mastered the objectives.
Purpose 17

Relevance to Instructional Development

Jere Brophy probably makes the clearest, most
succinct case for the centrality of objectives for
instructional designers:

The key to making learning experiences

worthwhile is to focus your planning on
major instructional goals, phrased in terms
of desired student outcomes—the
knowledge, skills, attitudes, values, and
dispositions that you want to develop in
your students. Goals, not content coverage
or learning processes, provide the rationale
for curriculum and instruction. (Brophy,
2010, p. 33)

Although it may seem so to educational technology

professionals, the necessity of clearly stating objectives is
not intuitively obvious to most teachers. Indeed, according
to Clark and Peterson (1986), “Teachers typically plan by
concentrating on the content they will teach and the
activities their students will do, without giving much
thought to the goals that provide the rationale for the
content and activities in the first place.” In short, objectives
communicate to learners what is critical and what is
peripheral and clarify what is expected; they guide the
choice of learning activities; they facilitate the selection of
materials; and they provide the keys to the development of
evaluation activities and items
Purpose 18

Relevance to Distance Learning

For several reasons, clear enunciation of purposes,
goals, and objectives may be even more crucial to distance
education than to face-to-face instruction. First, given that
distance learners are often seeking specific credentials or
job skills, it is consistent with “truth in advertising” to be
explicit about what learning outcomes they can expect from
a distance education course or module.

Second, in distance education, instructors usually

have little basis for evaluating student performance other
than through their work products (papers, projects,
examination answers). In the face-to-face classroom,
instructors have a lot of other information about a student’s
attendance, attentiveness, responsiveness, contribution to
group discussion, and so forth. Hence, it is even more
important to be clear about exactly how the distant student
will be evaluated. Explicit objectives will also help hold the
instructor accountable for providing the resources, learning
experiences, and feedback necessary for students to reach
the stated learning outcomes. In pre-packaged modules
delivered online--as opposed to conventional classroom
teaching--it is much more difficult to go back and fill in
with extra resources or exercises when instructors begin to
sense that students are struggling.

Third, in distance education, learner autonomy and

learner control loom as larger issues than in traditional
instruction (Shearer, 2003). If the distance learning
environment is too structured and rigid, the life demands
experienced by learners may leave them feeling forced to
drop out. If he environment is too loosely structured,
distance learners may struggle to find their footing. Clear
statements of objectives contribute to a predictable and
controllable environment.
Purpose 19

Empirical Support for the Value of

Objectives

Ideally, one would like research to answer the

question: Does stating objectives makes the rest of the
development process faster or easier for the developer, or
make the final product more successful? Conducting a
study to answer this question with any level of generality
would require samples, procedures, and controls that are
difficult to achieve. One study that involved eight teacher-
designers and their fourth-grade classes compared student
achievement on a science unit (Sullivan, Lievens,
Villalpando, and Watkins, 1986). Half the teacher-
designers created lessons based on specific instructional
objectives. Their students performed significantly better,
both on the post-test and on attitude toward the topic,
indicating that designers guided by specific objectives
create more effective and more appealing lessons.

Value for Designers

In addition, there have been some large-scale

studies and research syntheses that cast some light on the
value of objectives to designers. First, do experienced
instructional developers actually make the effort to write
specific objectives? A survey of training professionals
revealed that 94% “regularly” wrote out specific objectives
(and 82% “always” did so), a higher percentage than any
other step in the instructional design (ID) process.

Next, if designers do state objectives, are those

objectives appropriate? And are they reflected in the
learning activities and evaluation items created in the
design process? A large-scale evaluation study was
conducted with the U.S. Navy’s extremely detailed ID
procedures, which are designed to be used by instructors
Purpose 20

untrained in ID (Taylor and Ellis, 1991). After evaluating

100 courses, they found that: 1) most objectives were
appropriately stated, at least in terms of the “action”
component; 2) nearly half the instructional activities failed
to provide appropriate practice; and 3) nearly half of all test
items failed to align with objectives. While this study does
not answer the question about the utility of objectives, it
does cast some light on the pitfalls of trying to improve
instructional quality by developing ever more detailed
design procedures.

One of the early literature surveys on instructional

development (McCombs, 1986) highlighted the problem of
“overproceduralization.” That is, when following a
systematic procedure, it is easy for users to fall into the
practice of using the procedure as a recipe with steps to be
done in a routine, unthinking manner (for example, filling
in objectives statements after the lesson is completed),
rather than viewing instructional development as a
complex, creative process. Shrock (1985) also identified
similar misunderstandings of ID procedures among college
faculty members who were novices to ID.

Value for Learners

As meta-analyses of educational research have

become more and more common, a clear consensus has
emerged that specifying learning goals or objectives does
have a measurable impact on student achievement. For
example, in their synthesis, Beesley and Apthorp (2010)
conclude, “All studies produced positive effects for
objective setting with an overall effect of g = 0.31” (p.
109), which Hattie (2009) would rate “medium” on the
scale of low to high impact. An earlier synthesis of research
found that “goal setting on intended outcomes” achieved an
even higher effect size of 0.40 (Walberg, 1999, p. 80).
Purpose 21

One of the earliest meta-analyses reached a similar

conclusion. In their study of techniques in science teaching,
Wise and Okey (1983) found an effect size of .57 (p. 430),
which would rate as “medium” impact on Hattie’s scale.
This was for the variable they term “focusing,” by which
they mean “where something occurs to alert students to the
objectives or intent of instruction. Focusing techniques may
be employed before, during, or after instruction” (p. 421).
Interestingly, Hattie’s own synthesis of meta-analytic
studies (2009) arrived at virtually the same effect size for
what he terms “goals,” a subset of “learning intentions.” In
all of these studies the authors are referring to instructional
strategies in which learners are informed of the specific
intent of their lessons.

Why Objectives Are Valuable

First, as pointed out by Wise and Okey, objectives
help students narrow their focus to that which is most
important. Second, goals are motivational. Contemporary
theories of motivation emphasize facilitation rather than
control of behavior (Pintrich & Schunk, 2002). The aim is
to encourage learners accept the goals and to believe in
their ability to achieve them. Third, objectives are
indispensable to effective practice-and-feedback. Indeed,
there are few instructional strategies proven to be more
powerful than providing informational feedback as students
practice new skills. Hattie’s synthesis of meta-analyses on
feedback (Hattie, 2009) yields an effect size of 0.73,
placing this strategy in the range of “high” impact. Since
feedback means information about progress toward an
objective, it is axiomatic that there must be explicit
objectives to start with.
Purpose 22

Features of Well-Stated Objectives

Specificity

Dean, Hubble, Pitler and Stone (2012) counseled

course designers to “Set learning objectives that are
specific but not restrictive” (p. 5). Marzano, Pickering, and
Pollock (2005) agree and delve further into whether
objectives can be “too specific.” Their meta-analysis of
research on objectives concludes that highly specific
behavioral objectives have an effect size of only 0.12—
very low impact. They speculate that using highly detailed
and technical language may inhibit students from
internalizing and personalizing such objectives.

Meaningfulness

One of the truisms of modern educational

philosophy is that learners will invest more effort in pursuit
of goals that are relevant to them. This principle is well
supported by current meta-analyses of educational research,
which indicates that instruction should “Engage students in
setting personal learning objectives” (Dean et al., 2012, p.
9).

According to motivation theory, goals are more

likely to be activated if they are salient to the individual
(Pintrich and Schunk, 2002). Salience may have many
dimensions: seeing the connection between today’s lesson
and past and future lessons, perceiving the goal to be within
learners’ capability, compatible with their cultural
background, and useful in their future life, among others.
Objectives statements that are personally relevant are more
likely to stimulate learner effort. Learners’ “ownership” of
objectives seems to increase their commitment to achieving
those objectives. The importance of commitment in
Purpose 23

achieving goals is well established in the literature of

organizational development (Locke and Latham, 1990).

One way to make learner commitment explicit is to

employ learning contracts, in which the learners agree to
create some learning product, such as an essay, in return for
a passing grade or other reward controlled by the instructor.
The reward could be some form of public recognition, as
advocated by Marzano (2007) under the rubric of a
“celebration” marking progress toward a larger goal.

Difficulty

This principle was first established in laboratory

studies of job performance by Locke and Latham (1984)
and later verified in field studies. They found that “people
who were assigned difficult goals performed better than did
those who were assigned moderately difficult or easy
goals” (p. 10). Hattie (2009) found a high positive
correlation between goal difficulty and student performance
and speculated that difficult goals “lead to a clearer notion
of success and direct the student’s attention to relevant
behavior or outcomes” (p. 164). This finding is reflected in
a 2007 review of research on feedback, which shows that
feedback is most helpful when “goals are specific and
challenging but task complexity is low” (Hattie and
Timperley, pp. 85-86).

Summary
Although we lack abundant research evidence that
specifying objectives ensures an efficient and effective
instructional development process, we have ample evidence
that the provision of goals and objectives to learners
enhances achievement. In the end, the strongest argument
for objectives is the compelling logic that one can hardly
Purpose 24

begin to create learning materials or assessment measures

without first having carefully prescribed the intended
outcomes of the lesson, whether face-to-face or at a
distance.

References
Beesley, A. D., & Apthorp, H. S. (Eds.) (2010). Classroom
instruction that works, 2nd ed., research report.
Denver, CO: McREL Inc.
Bobbitt, F. (1924). How to make a curriculum. Boston:
Houghton Mifflin.
Brophy, J. (2010). Motivating students to learn (3rd ed.).
New York: Routledge.
Charters, W. W. (1923; rpt. 1925). Curriculum
construction. New York: Macmillan.
Clark, C., & Peterson, P. (1986). Teachers’ thought
processes. In M. C. Wittrock (Ed.), Handbook of
research on teaching (3rd ed., pp. 255-296). New
York: Macmillan.
Davies, I. K. (1976). Objectives in curriculum design.
London: McGraw-Hill.
Dean, C. B., Hubble, E. R., Pitler, H., & Stone, B. J.
(2012). Classroom instruction that works:
Research-based strategies for increasing student
achievement (2nd ed.). Alexandria, VA: Association
for Supervision and Curriculum Development; and
Denver, CO: McREL.
Hattie, J. (2009). Visible learning. London and New York:
Routledge.
Hattie, J., & Timperley, H. (2007). The power of feedback.
Review of educational research, 77(1), 81-112.
Locke, E. A., & Latham, G. P. (1984). Goal setting: A
motivational technique that works. Englewood
Cliffs, NJ: Prentice-Hall.
Purpose 25

Locke, E. A., & Latham, G. P. (1990). A theory of goal

setting and task performance. Englewood Cliffs,
NJ: Prentice-Hall.
McCombs, B. L. (1986). The instructional systems
development (ISD) model: A review of those
factors critical to its successful implementation.
Educational communication and technology
journal, 31(4), 187-199.
Mager, R. F. (1962a). Preparing objectives for
programmed instruction. San Francisco: Fearon.
Mager, R. F. (1962b). Preparing instructional objectives.
Belmont, CA: Fearon
Marzano, R. J. (2007). The art and science of teaching.
Alexandria, VA: Association for Supervision and
Curriculum Development.
Marzano, R. J., Pickering, D. J., & Pollock, J. E. (2005).
Classroom instruction that works: Research-based
strategies for increasing student achievement.
Upper Saddle River, NJ: Pearson Education.
Pintrich, P. R., & Schunk, D. H. (2002). Motivation in
education: Theory, research, and applications (2nd
ed.). Upper Saddle River, NJ: Pearson Education.
Shearer, R. (2003). Instructional design in distance
education: An overview. In M. G. Moore and W. G.
Anderson (Eds.), Handbook of distance education.
Mahwah, NJ: Lawrence Erlbaum Associates.
Shrock, S. A. (1985). Faculty perceptions of instructional
development and the success/failure of an
instructional development program: A naturalistic
study. Educational communications and technology
journal, 33(1), 16-25.
Stolovitch, H. D., & Keeps, E. J. (2002). Telling ain’t
training. Alexandria, VA: American Society for
Training & Development.
Sullivan, H. J., Lievens, R. C., Villalpando, E. M.,
Marquez, C., & Watkins, G. (1986). Supplementing
Purpose 26

traditional instruction with objectives-based

instructional development. Journal of instructional
development, 9(2), 29-32.
Taylor, B., & Ellis, J. (1991). An evaluation of instructional
systems development in the Navy. Educational
technology research and development, 39(1), 93-
103.
Tyler, R. W. (1949; rpt. paper 1969). Basic principles of
curriculum and instruction. Chicago: University of
Chicago Press.
Walberg, H. J. (1999). Productive teaching. In H. C.
Waxman and H. J. Walberg (Eds.), New directions
for teaching practice and research. Berkeley, CA:
McCutchan.
Wedman, J., & Tessmer, M. (1993). Instructional
designers’ decisions and priorities: A survey of
design practice. Performance improvement
quarterly, 6(2), 43-57.
Wise, K. C., & Okey, J. R. (1983). A meta-analysis of the
effects of various science teaching strategies on
achievement. Journal of research in science
teaching, 20(5), 419-435.
Assumptions 27

Assumptions
Michael Simonson
Nova Southeastern University

Course design must take into account

assumptions that shape the purpose and
subsequent course development. Most
assumptions are based on students’ prior
knowledge and established understandings
and skills. Articulating these content
assumptions provides a starting point for new
learning. Assumptions in the case of online
learning also encompass students’ ability to
use delivery technology.

Background
An assumption is a position taken until something is
proven. Stated another way, assumptions are ideas without
evidence--things taken for granted. Assumptions seem to be
the antithesis of the systems approach and systematic
design of instruction. After all, the systems approach is
based on a premise that aspects of any system should be
correctly defined, clearly understood, and well-
documented. Distance education programs are usually
planned using the systems approach (Dick, Carey & Carey,
2015; Simonson, Smaldino & Zvacek, 2015).

Instructional designers may not like to talk about

assumptions or discuss what they take for granted.
However, in the privacy of the design studio, everyone who
plans instruction, both face-to-face and online, makes
Assumptions 28

assumptions and takes things for granted. But let us look at

the types of assumptions that form the basis for designing
instruction, with an emphasis on online instruction. There
are seven broad areas where assumptions guide
instructional design, at least initially:

1. Learners
2. Structure
3. Communication
4. Technology
5. Interaction
6. Literacy (visual and verbal)
7. Learning

Assumptions About Learners

Assumptions about learners are those things that are
taken for granted about them. Standards for online
instruction often begin with the designers listing the
characteristics of the target audience for the instruction,
such as: prerequisite competencies, access to resources, and
level of self-motivation. The massive body of research
provided by those advocating for mastery learning during
the 1950s and 1960s provides guidance about learner
assumptions (Saettler, 2004). One assumption, later
supported by research, was the idea that any
communication between a student and an instructor must be
based on what both have in common: language,
background, interests, motivation, and so on. All models of
communication require assumptions about what the sender
of a message and the receiver of the message have in
common (Simonson, 1984).
Assumptions 29

Assumptions About Course Structure

One of the first decisions that online designers must
make regards time: individualized or personalized
instruction holds learning outcomes constant and allows for
variations in time (i.e., students have as much time as they
need to meet a course’s learning outcomes). Most often,
however, distance education is time bound (the 15-week
semester or the eight-week term, for example). Once the
time issue is resolved, then the structure of the learning
experience is decided. At this point, decisions about
learning theory must be made. Will the course be built on
behaviorism, constructivism, or combinations of theories?
A comprehensive review of online courses and programs
shows that behaviorism-based course structures dominate
(Simonson, Smaldino & Zvacek, 2015). Courses organized
around weeks or units/modules/topics are most common.
Decisions about structure are usually decisions made
according to assumptions, that is, without direct evidence
of their efficacy.

Assumptions About Communication

Early on, the designer makes decisions about how

communication between the instructor and students should
occur and whether this communication should be
asynchronous or synchronous (Orellana, Hudgins, &
Simonson, 2009). There is evidence available to help
designers make these decisions, but prescriptive evidence is
largely lacking, and the assumptions about communication
are generally made based on prior experience or personal
preferences. The evidence does support one trend: novice
distance educators design their courses with considerable
live communication, while more experienced distance
educators opt for asynchronous communication (Simonson,
Smaldino & Zvacek, 2015).
Assumptions 30

Assumptions About Technology

It is hard to imagine online instruction without
instructional technology, so one immediate assumption is
that online course design and delivery must be instructional
technology-based. The types of communication technology
and instructional technology to be used grow out of a
critical set of decisions that are made early in the design
process (Dick, Carey & Carey, 2015).

Assumptions About Interaction

The U.S. Department of Education states that

distance education must provide for regular and substantive
interaction (U.S. Department of Education, 2017). Both
words – regular and substantive – are vague and open to
interpretation. In other words, assumptions must be made.
Recently, some--including MOOC supporters--have
advocated the idea that interaction is no longer necessary
and is an outdated concept. Others, however, think that the
entire distance education experience should begin with
provisions for interaction (Simonson, 2015). The designer
of online instruction must make decisions about how
interaction is to take place, and standards for online
education should have clear guidelines about interaction.

Assumptions About Visual and Verbal

Literacy
Dwyer’s research on relevant cues (Moore &
Dwyer, 1994) and Dale’s realism theory (Dale, 1946)
indicate that decisions about text, pictures, video, and
graphics are critical in any instructional design activity.
The individual or team who creates online instruction needs
to make many decisions about literacy and how the selected
Assumptions 31

medium supports the delivery and understanding of

content. It is clear that Clark (2012) was correct: media do
not directly influence achievement. But it is also a basic
assumption that, without employing some medium of
communication, it is nearly impossible to communicate at a
distance. Thus designers may make assumptions related to
the literacy—both visual and verbal--of all learners
involved in distance education.

Assumptions About Learning

Actually, this is the most straightforward category
of assumptions that affect the standards for the design and
delivery of online learning. It is clear that 90% of any
content area can be successfully learned by 90% of any
group of learners, given enough time (Saettler, 2004).
However, the six areas of assumptions listed previously all
support the assumption that online students will learn. This
assumes that the assumptions are correct.

Summary
An initial design step, and eventually a design
standard for online instruction, should be for the project
manager to list the assumptions that are at the foundation of
the instructional design plan. Assumptions can be
organized into the categories listed here and presented as
part of the design plan. Finally, designers of online
instruction can, indeed must, make assumptions. Scientists
interested in distance education should conduct research on
those assumptions so they become standards--expectations
that must be met. Standards without such research will
remain assumptions only--ideas without evidence.
Assumptions 32

References
Clark, R. (2012). Learning from media. (3rd ed.) Charlotte,
NC: Information Age Publishing.
Dale, E. (1946). Audio-visual methods for teaching. New
York: Dryden Press.
Dick, W., Carey, L., & Carey, J. (2015). The systematic
design of instruction, (8th Ed.). Boston, MA:
Pearson.
Moore, D., & Dwyer, F. (1994). Visual literacy: A
spectrum of visual learning. Englewood Cliffs, NJ:
Educational Technology Publications.
Orellana, A., Hudgins, T., & Simonson, M. (2009). The
perfect online courses: Best practices for designing
and teaching. Charlotte, NC: Information Age
Publishing.
Saettler, P. (2004). The evolution of American educational
technology. Charlotte, NC: Information Age
Publishing.
Simonson, M., Smaldino, S., & Zvacek, S. (2015).
Teaching and learning at a distance: Foundations
of Distance Education, (6th Ed.). Charlotte, NC:
Information Age Publishing.
Simonson, M. (2015). Regular and substantive. Distance
Education, 12(4), 68-67.
Simonson, M. (1984). Media planning and production.
Columbus, OH: Merrill.
U.S. Department of Education (2017). Accreditation in the
United States. Retrieved from
https://www2.ed.gov/admins/finaid/accred/
accreditation_pg12.html.
Sequencing 33

Sequence
Wilhelmina C. Savenye and Yi-Chun
Hong
Arizona State University

Learning opportunities must be sequenced in

a manner that promotes efficient knowledge
acquisition consistent with the prior-
knowledge assumptions. Various models of
sequencing—linear, spiral, scaffold, etc.—
should be considered, and the course design
should incorporate those strategies best
suited to the content within the constraints of
online delivery.

Background
When designing instruction, for any type of
delivery, but especially for online learning, how to
sequence instruction may seem to be a deceptively simple
step. However, sequencing instruction for effective student
learning involves a very important set of decisions that are
made at several levels and stages of design. Morrison,
Ross, Kalman and Kemp (2013) define sequencing as “the
efficient ordering of content in such a way as to help the
learner achieve the objectives in an efficient and effective
manner” (p. 122). Instructors or designers may typically
begin developing instruction by using the sequence
provided in textbooks or other available materials;
however, after one or two times using these materials, they
often find that the sequence does not work for their
particular learners.
Sequencing 34

At the outset let us suggest that a simple approach

to sequencing might be used when time is short, when
instruction will be offered only once or only for few
learners, or when risks and costs of failed instruction are
low. In these cases, sequencing may be based upon: 1)
knowledge about the learners and 2) going from easy to
more difficult or simple to more complex. However, this
simplistic approach is not optimal in many cases,
particularly for complex learning or large-scale, high-
impact, or extremely important instruction. In such cases, a
deeper view of developing instructional sequences is
needed.

Keller and Suzuki (2004) remind us of the critical

importance of motivation to learn in E-learning design,
noting, “First, a lesson must gain and sustain a learner’s
attention” (p. 231). In the ARCS model for motivation,
Keller (2010) recommends strategies for not only gaining
attention, but for supporting learners’ perceptions of the
relevance of the instruction, as well as their confidence and
satisfaction in learning the material. Keller and Suzuki and
others (cf. Sullivan & Higgins, 1983) have suggested that a
motivator addressing at least one aspect of ARCS, based on
knowledge of the learners, be included early on in any
instructional lesson. Motivators are often included in a
lesson introduction; however, they may be included at any
point in instructional sequences.

Where Sequencing Fits in the Design of

Online Learning

We argue that it is best to design instruction, but

especially online learning materials, using a systematic
Sequencing 35

approach overall. There are many models for designing

instruction. For instance, as early as 1980, Andrews and
Goodson reviewed over 60 instructional design (ID)
models. More recently, Branch and Kopcha (2014) have
indicated that there are now hundreds of ID models. If we
look to simplify, one model, called ADDIE, aids designers
by suggesting five major steps to designing learning
materials: analyze, design, develop, implement, and
evaluate (Branch, 2009).

Several widely-used instructional design models

enable us to look more deeply at what goes into developing
instruction. For instance, Dick, Carey & Carey (2005)
suggest that the instructional designer begins by identifying
instructional goals, which is followed by conducting an
instructional analysis and identifying entry behaviors. A
critical next step is writing learning/performance
objectives. In this model for designing instruction,
developing tests to measure student performance on the
objectives usually comes next, followed by designing the
instructional strategy, developing and selecting
instructional materials, and conducting formative
evaluation. Based on the results of the formative
evaluation, revisions are made to the materials before
implementing the learning materials. For Dick, Carey, and
Carey, sequencing of instruction would take place when
writing objectives, when developing strategies, and when
developing the instructional material.

Morrison et al. (2013) have developed a more

circular ID model, in which the early steps are somewhat
different. These authors suggest that an instructional project
begins by identifying an instructional need by conducting a
needs assessment. They then recommend analyzing the
characteristics of the target learners. Analyzing the learners
cannot be shortchanged, as the lesson or unit may otherwise
Sequencing 36

fail to meet the needs of those for whom it is intended.

Many decisions in designing the instruction rest upon the
knowledge of the learners’ age, performance needs,
backgrounds, interests, and, most important, levels and
types of prior knowledge and skills related to the content.
Smith and Ragan (2004) would add that the “context” –
that is, where and how the instruction will be delivered –
should also be analyzed early on.

The next crucial step, upon which the foundation of

the instruction rests, is writing a set of student learning
objectives (which may also be called performance
objectives, learning outcomes, etc.). For Morrison et al.
(2013) sequencing of content is best done right after
developing objectives, though again, their model is iterative
and circular, indicating that steps often can be completed in
different orders and that decisions made early on are
subject to change throughout the design process.

Levels of Sequences in Instruction

There are several levels of sequences to consider in
designing learning materials. Gagne, Wager, Golas and
Keller (2005) note that sequencing of instructional content
needs to be done at several levels, depending on the length
of the instruction. These authors provide guidance for
making sequencing decisions at the following levels:

• Course
• Topic or unit
• Lesson
• Lesson component or objective

Gagne et al. (2005) remind us that what makes up a

course can vary in many different learning contexts. For
example, in a university setting a course may be completed
Sequencing 37

in 5 weeks, 8 weeks, 10 weeks, or 15 weeks. However, in

other adult learning settings, such as in training, a course
might be completed in a few hours. Typically, however, a
course would consist of several lessons, which each would
be designed to teach learners to achieve several objectives.

Designers may use tools to help plan and sequence

a large course or set of courses at a macroscopic level. One
common planning tool is a “scope and sequence matrix”
(Gagne et al., 2005, p. 175). This is a table in which the
course planners develop content topics on one axis with
key objectives on the other.

Another tool that can be used at many levels of

sequencing is the Instructional Curriculum Map (ICM)
(Gagne et al., 2005, p. 178). An ICM may be developed to
show the basic units or topics in a course and how they lead
to subsequent topics, ultimately leading to the end-of-
course culminating unit or skill. After the overall topics or
units in a course have been identified, a detailed ICM can
be developed for each unit in the course, showing the key
objectives and how they lead to the other key objectives in
a unit. ICMs can also be developed for each level (some
call this lesson mapping).

Approaches to Sequencing

Not surprisingly, there are several types of

approaches that designers may apply when planning how to
sequence instruction. A tour of these approaches may make
planning a course or unit easier. Posner and Strike (1976)
suggest that methods for sequencing learning content fall
into five categories, though they also note that these
categories may certainly overlap. These are:
Sequencing 38

• Learner-related
• World-related
• Concept-related
• Inquiry-related
• Utilization-related.

Learner-Related

Learner-related sequencing is highly learner-

centered and rests upon psychology of learning theories.
Based on the knowledge about the learners gained by doing
the learner analysis mentioned above, designers may
employ one of several methods. They may identify
“empirical prerequisites,” typically related to skills that
need to be learned before other skills. Knowledge of the
learner leads the designer to build sequences from what is
familiar to the learner to that which is unfamiliar. Another
method would move from less to more difficult, again,
from the perspective of the target learners. Sequencing
based on the interests of the learners, with more interesting
first, is another method, as is sequencing based on the
learners’ developmental levels. Finally, particularly useful
for teaching attitudes is to develop instruction to help
learners increasingly internalize the attitude or position.

World-Related

World-related sequencing involves analyzing the

actual relationships that occur in the world and ordering
material accordingly. Depending upon the instructional
problem, such sequencing might be based on space/spatial
relations, time/temporal relations, or physical
characteristics or features of the phenomena to be taught.
Sequencing 39

Concept-Related

According to Posner and Strike (1976), concept-

related sequencing is a type of logical ordering of content
particularly adaptable to teaching concepts and
propositions. Concept-related sequencing includes four
subtypes or principles for ordering instructional content.
One is based on class relations; Morrison et al. (2013)
suggest that characteristics of a class be taught first, for
example. In contrast, sequencing using propositional
relations involves teaching examples first and then the
proposition. A third method for concept-related sequencing,
according to Posner and Strike (1976), is by level of
sophistication of the concepts, that is simpler to more
complex or concrete to more abstract. Finally, sequencing
based on logical prerequisites is used when a concept must
be understood that is a prerequisite of another concept.

Inquiry-Related

Inquiry-related sequencing is employed when a

more discovery-oriented approach to instruction is desired
or when the nature of the learning involves “generating,
discovering, or verifying knowledge” (Posner & Strike,
1976, p. 676).

Utilization-Related

Utilization-related sequencing is often used for

organizing content for three contexts: social, personal, and
career (Posner & Strike, 1976). For these contexts,
sequencing content can be done according to procedures
that need to be learned, such as steps in a training process,
or according to how frequently the content to be learned
would be used. That might involve teaching content that is
most important or most frequently used first.
Sequencing 40

Learning Hierarchies Approach

Gagne et al. (2005) emphasize a learning
hierarchies approach to sequencing learning content,
though they acknowledge that there are many ways to
conduct sequencing. Designers first analyze the major
course, unit, and lesson objectives and classify the
objectives into “domains of learning.” These include:
verbal information, such as facts or bodies of knowledge;
cognitive strategies, which may be thought of as skills
involved in learning how to learn or remember; motor
skills; attitudes and intellectual skills. For example, because
intellectual skills usually build upon each other,
discriminations need to be learned before concepts,
concepts before rules, and rules before principles or higher-
order rules.

Learning hierarchies are especially applicable when

designers determine that many of the course objectives
represent intellectual skills, as these clearly build upon one
other. Designers analyze what are the prerequisite skills
learners need before learning the entry skills in the course;
they then may build curriculum maps, as described above,
that show which concrete and abstract concepts must be
learned before students may learn to apply rules that
involve those concepts and which rules and principles must
be learned before students can learn to solve problems
involving those rules. Gagne et al. (2005) and Smith and
Ragan (2004) provide extensive guidance, based on a great
body of research on learning, for strategies for developing
instruction to teach all these domains of learning outcomes.

Knowledge-Based Sequencing
According to Gagne et al. (2005), knowledge-based
sequencing leads us to consider design of more technology-
Sequencing 41

based instruction, especially if designed for self-instruction

via online technologies. A designer analyzes the optimum
path for a learner through the instruction, developing not
just guidance and content, but also ways for the student to
practice skills, apply knowledge, and receive individualized
feedback. Though not typically employed by online
instructors for groups, a designer could build a module for
a course that would include alternative paths through the
instruction, based either on learners’ choices or on their
performance.

Spiral Sequencing
The types of sequencing of content described above
might be considered somewhat linear. A very different
approach to sequencing is based on the idea of spiral
sequencing of content. One might visualize a spiral with
instruction that begins at the base and moves upward
through topics. The topics are revisited throughout the
course, at deeper and deeper levels. Gagne et al. (2005)
contend that spiral sequencing is particularly common in
language courses and in vocational courses. In language
courses, they suggest that objectives involving vocabulary,
pronunciation, and grammatical rules are learned at
increasingly more complex levels. They add that spiral
sequencing in language courses allows for many
opportunities for learners to practice their language skills.

Elaboration Theory
Like spiral sequencing, elaboration theory organizes
instruction from simpler to more complex knowledge.
Reigeluth (1987) uses an analogy of a camera zoom lens to
illustrate the elaborative sequence. When provided with a
photo, viewers start with a wide-angle view, which allows
them to see the major components of a picture and the
Sequencing 42

interrelationships among those components. After obtaining

a broad view of a picture, viewers then zoom in to focus on
the details of a specific component, followed by a zoom out
to a wide-angle view. Viewers then continue this pattern
until they obtain both the whole picture and the details of
components. Applying this notion, the learning of complex
knowledge can be supported by designing instruction to
proceed from the broadest, most general, and most
inclusive idea toward narrower, more precise, and less
inclusive ideas in order to assist learners to obtain both the
breadth and depth of knowledge.

Reigeluth (1999) further identifies three types of

elaborative sequence for teaching different types of
content: conceptual elaboration, theoretical elaboration, and
procedural elaboration. The conceptual elaboration
sequence aims to teach topics with interrelated concepts,
while the theoretical elaboration sequence is intended for
organizing a set of interrelated principles. The interrelated
concepts and principles can be structured into a concept
map. Based on the concept map, designers then use the top
down approach to choose the more general, superordinate
concept or principle to be taught first and then gradually
progress toward more detailed, subordinate concepts or
principles. In some cases, the focus of the instruction rests
on the procedural knowledge (e.g., how to plan a trip). The
designers then can apply the procedural elaboration
sequence and to teach the simplest version of a task (e.g.,
plan a day trip) and then gradually add other components to
prepare learners for more complex version of the task.

Online Settings

Online learning has various forms, including but not

limited to Web-based learning, e-learning, distance
learning, and computer-assisted learning. No matter which
Sequencing 43

form of online learning is developed, applying sequencing

strategies is arguably at least as important in online courses,
as in traditional, face-to-face courses. Online learning
requires additional planning up-front because of the distinct
natures of online learning environments and the
affordances of technologies. Technology affordance,
according to Gagne et al. (2005), refers to the functions of
technology that increase the potential for enhanced
learning. Some technologies afford instant and ubiquitous
access to learning resources and materials. Some enable
learners to connect with a geographically diverse student
body. Other aspects of technologies afford learners access
to learning materials and activities at their own pace. Ally
(2008) identifies that one type of interaction in which
students engage in an online learning environment is that of
learner-content interaction, which provides opportunities
for learners to navigate on their own from the first to the
last learning episodes.

Echoing the value of learner control, Alessi and

Trollip (2001) offer several concrete recommendations.
Designers should carefully design the opportunities for
learners to control their learning speed, moving forward,
pausing, and even moving backward as needed; the
opportunities for accessing all topics or partial topics on the
learning menu; and the opportunities to skip learning
materials when necessary. With the affordances of
technology, designers and instructors are able to design
completely personalized learning environments; however,
this may considerably increase development time and cost.

No matter what approach a designer/instructor uses

to develop the sequence of content and learning objectives,
it is always wise to make a bit of time to conduct a
formative evaluation of the instruction. For instructors, this
may be the first time they teach a course or unit, and for
Sequencing 44

designers, this may be a small-scale tryout with a group of

sample learners, before the course is fully implemented.
Data can be easily collected regarding student mastery of
the learning objectives, as well as their attitudes toward the
instruction, with revisions being made before the next
iteration of the course.

Summary
Many approaches to sequencing instructional
content are available. However, a designer or instructor will
have the most impact on learners by employing a
systematic approach to designing instruction. Sequences
based on applying the data from a needs assessment, task
analysis, and learner analysis are most effective, whether
the setting is face-to-face or at a distance In addition,
following the development and implementation of the
learning materials with careful evaluation and revision adds
value and power to the instruction.

References

Alessi, S. M., & Trollip, S. R. (2001). Multimedia for

learning: Methods and development. Needham
Heights, MA: Allyn & Bacon.
Ally, M. (2008). Foundations of educational theory for
online learning. In T. Anderson (Ed.), The theory
and practice of online learning (pp. 15-44).
Edmonton, AB: AU Press.
Andrews, D. H., & Goodson, L. A. (1980). A comparative
analysis of models of instructional design. Journal
of Instructional Development, 3(4), 2-16.
http://www.jstor.org/stable/30220623.
Branch, R. M. (2009). Instructional Design: The ADDIE
approach. New York, NY: Springer.
Sequencing 45

Branch, R. M., & Kopcha, T. J. (2014). Instructional design

models. In J. M. Spector et al. (Eds.). Handbook of
research on educational communications and
technology (4th ed.). (pp. 77-87). New York, NY:
Springer.
Dick, W., Carey, l., & Carey, J. (2005). The systematic
design of instruction (6th ed.). Boston, MA: Allyn
& Bacon.
Gagne, R. M., Wager, W. W., Golas, K. & Keller, J. M.
(2005). Principles of instructional design (5th ed.).
Belmont, CA: Wadsworth.
Keller, J. M. (2010). Motivational Design for Learning and
Performance: The ARCS Model Approach. New
York, NY: Springer.
Keller, J., & Suzuki, K. (2004). Learner motivation and E-
learning design: A multinationally validated
process. Journal of Educational Media, 29(3), 229-
239.
http://dx.doi.org/10.1080/1358165042000283084
Morrison, G. R., Ross, S. M., Kalman, H. K., & Kemp, J.
E. (2013). Designing effective instruction (7th ed.).
Hoboken, NJ: Wiley.
Posner, G. J., & Strike, K. A. (1976). A categorization
scheme for principles of sequencing content.
Review of Educational Research, 46 (4), 665-690.
Reigeluth, C. M. (1987). Lesson blueprints based on the
elaboration theory of instruction. In C. M. Reigeluth
(Ed.), Instructional theories in action: Lessons
illustrating selected theories and models (pp. 245-
288). Hillsdale, NJ: Lawrence Erlbaum Associates.
Reigeluth, C. M. (1999). The elaboration theory: Guidance
for scope and sequence decisions. In C. M.
Reigeluth (Ed.), Instructional-design theories and
models: A new paradigm of instructional theory
(Vol. 2, pp. 425-453). Mahwah, NJ: Lawrence
Erlbaum Associates.
Sequencing 46

Smith, P. L., & Ragan, T. J. (2004). Instructional design

(3rd ed.). Hoboken, NJ: Wiley.
Sullivan, H., & Higgins, N. (1983). Teaching for
Competence. New York, NY: Teachers College Press.
Activities 47

Activities
Gary R. Morrison
Old Dominion University

Learning is achieved through activities both

passive (reading, listening, viewing) and
active (experimenting, rehearsing, trying).
Activities should be chosen that best suit the
content, students’ levels of knowledge,
experience, and ability, and online delivery
constraints, particularly accommodating
synchronous, asynchronous, and mixed
course participation. Student self-selected or
self-developed learning activities should be
incorporated along with instructor-selected
and instructor-developed activities,
consistent with a two-dimensional purpose.

Background

Instruction is different from information. Libraries

are full of information, and instruction uses information.
However, instruction carefully structures the information in
a manner to make it comprehensible to the learner, engages
the learner with the content, and includes instructional
strategies that help the learner make the information
meaningful and relate it to other knowledge. Both
classroom instruction and learning in an online setting
make use of a wide array of activities and strategies.

Early research by Craik and Lockhart (1972) found

that individuals who process information at a deeper level
Activities 48

recalled more information than those processing

information at a superficial level. Instructional strategies
are designed to create a deeper level of processing that
leads to increased learning and longer-term retention. This
section focuses on the use of instructional strategies or
activities instructional designers employ.

Defining Instructional Strategies

The field of instructional design has a long history
of using empirically-based instructional strategies. One of
the earliest descriptions was Susan Markle’s 1969 book,
which was used in the design of programmed instruction.
Some of the strategies discussed in the book are still in use
today. Instructional strategies can range from process, such
as rehearsal and practice, to mnemonics, to teaching more
complex skills, such as interpersonal communication
(Bandura 1977). Each of these strategies is based on
empirical research that supports its effectiveness and
provides guidelines for its use. For example, one would not
use Bandura’s social learning theory to teach a concept or
EG-Rule (discovery method) to teach a fact. That is,
specific types of content (e.g., facts, concepts, principles, or
problem solving) require different strategies. Based on a
quick review of some early examples of programmed
instruction, one might conclude that rehearsal and practice
was the one-size-fits-all strategy. However, instructional
intervention research over the last 70 years has identified a
wide array of instructional strategies for designing effective
instruction (Hsieh et al., 2005).

Conceptualizing Instructional Strategies

Rigney (1978) identified two distinct classes of
instructional strategies. First is the embedded instructional
strategy that becomes part of the content. Embedded
Activities 49

strategies require the learner to manipulate data to develop

understanding. For example, when teaching gas laws, the
learner might use a computer simulation to collect data and
then plot the data to discover, for example, a relationship
between temperature and pressure. More generic, and
applicable with a variety of content, are detached strategies.
For example, a generative strategy, such as “Describe what
you just read in your own words,” is easily used with a
range of content and so would be described as a detached
instructional strategy. Wittrock (2010) and others
(Grabowski, 2004; Jonassen, 1988; Mayer, 2010) view
learning as a generative process. Learning is the process of
attending to stimuli and then giving meaning to the stimuli
using one’s prior knowledge and experiences. That is, the
learner relates the new information to old information to
create meaningful information that is resistant to forgetting.
Wittwer and Renkl (2010) state that an instructional
explanation or strategy has two essential parts: a goal-
oriented instructional explanation that may include
elaborations, and a deeper engagement with the content,
initiated by an instructional strategy.

Empirical Support for Instructional

Strategies
Instructional design, educational psychology, and
psychology have a rich history of conducting empirical
research that compares one or more instructional strategies
for teaching the same content to determine effectiveness.
Levin and O’Donnell (1999) label this type of research as
educational intervention research in which the researcher
creates an intervention, applies the intervention, and
observes changes in human behavior. They trace some of
the earliest intervention research to Thorndike (1910).
Early examples of intervention research in our field include
a study by Whelden (1954) comparing the use of guided
Activities 50

practice and a study by Peterson and Schramm (1954)

comparing the effectiveness of eight different types of
graphs to teach percentages.

The following is a brief summary of intervention

research on the more common types of content for which
an instructional designer may design a strategy. Facts are
one of the simplest forms of content to teach. Strategies for
teaching facts include mnemonics (Rummel, Levin &
Woodward, 2003) and elaborative interrogation
(Woloshyn, Paivio & Pressley, 1994). Concepts are
categories we use to simplify the facts of the world. Markle
(1969, 1975) proposed an elaborate strategy based on
examples and non-examples for teaching categories.
Tennyson and Cocchiarella (1986) proposed a simpler
approach using a definition, one best example, and then
practice using examples and non-examples. Another
category of knowledge is the principles or rules that explain
a relationship between concepts. Markle (1969) proposed
that a rule be stated and then followed by the presentation
of one or more examples, or alternatively, that a series of
examples be presented and the learner prompted to discover
the rule. Variations include having the learner develop an
argument explaining why something happens (Jonassen &
Hung, 2006) or why the rule works (Wiley & Voss, 1999).
Procedures can be described as either cognitive (e.g.,
solving a math problem) or psychomotor (e.g., drilling a
hole). One example strategy for teaching cognitive
procedures is worked examples (Atkinson & Renkl, 2007;
Sweller & Cooper, 1985; Van Gog, Paas & Van
Merriënboer, 2006). Psychomotor skills can be taught by
modeling the behavior and then having the learner develop
a mental model before practicing the skill (Bandura, 1977;
Bandura & Jeffery, 1973).
Activities 51

Summary
Selection of instructional strategies to teach content
should be based on intervention research, as described by
Levin and O’Donnell (1999). When designers make use of
verified instructional strategies, the design is more likely to
produce consistent results for all learners.

References
Atkinson, R. K., & Renkl, A. (2007). Interactive example-
based learning environments: Using interactive
elements to encourage effective processing of
worked examples. Educational Psychological
Review, 19 (3), 375-386. doi: DOI 10.1007/s10648-
007-9055-2
Bandura, A. (1977). Social learning theory. Englewood
Cliffs, NJ: Prentice-Hall.
Bandura, A., & Jeffery, R. W. (1973). Role of symbolic
coding and rehearsal prcesses in observational
learning. Journal of Personality and Social
Psychology, 26(1), 122-130.
Craik, F. I. M., & Lockhart, R. S. (1972). Levels of
processing: A framework for memory research.
Journal of Verbal Learning and Verbal Behavior,
11, 671-684.
Grabowski, B. J. (2004). Generative learning contributions
to the design of instruction and learning. In D. J.
Jonassen (Ed.), Handbook of educational
communications and technology (2nd ed., pp. 719-
743). Hillsdale, NJ: Lawrence Erlbaum Associates.
Hsieh, P.et al. (2005). Is educational intervention research
on the decline? Journal of Educational Psychology,
97(4), 523-529.
Jonassen, D. H., & Hung, W. (2006). Learning to
troubleshoot: A new theory-based design
Activities 52

architecture. Educational Psychology Review,

18(1), 77-114.
Jonassen, D. J. (1988). Integrating learning strategies into
courseware to facilitate deeper process. In D. J.
Jonassen (Ed.), Instructional designs for
microcomputer courseware (pp. 151-181).
Hillsdale, NJ: Lawrence Erlbaum Associates.
Levin, J. R., & O'Donnell, A. M. (1999). What to do about
educational research's credibility gap? Issues in
Education: Contributions from Educational
Psychology, 5(2), 177-229.
Markle, S. (1969). Good frames and bad: A grammar of
frame writing. New York: Wiley.
Markle, S. (1975). They teach concepts, don't they?
American Educational Research Journal, 4(6), 3-9.
Mayer, R. E. (2010). Merlin C. Wittrock's Enduring
Contributions to the Science of Learning.
Educational Psychologist, 45(1), 46-50. doi:
10.1080/00461520903433547
Peterson, L. V., & Schramm, W. (1954). How accurately
are different kinds of graphs read? Audiovisual
communication review, 2(3), 178-189. doi:
10.1007/bf02713334
Rigney, J. W. (1978). Learning strategies: A theoretical
perspective. In H. Rummel, N., J. R. Levin, and M.
M. Woodward (2003). Do pictorial mnemonic text-
learning aids give students something worth writing
about? Journal of Educational Psychology, 95(2),
327-334.
Sweller, J., & Cooper, G. (1985). The use of worked
examples as a substitute for problem solving in
algebra. Cognition and Instruction, 2(1), 59-89.
Tennyson, R. D., & Cocchiarella, M. J. (1986). An
empirically based instructional design theory foF.
&*r teaching concepts. Review of Educational
Research, 56(1), 40-71.
Activities 53

Thorndike, E. L. (1910). The contribution of psychology to

education. Journal of Educational Research, 1(1),
5-12.
van Gog, T., Paas, F., & van Merriënboer, & J. J. G.
(2006). Effects of process-oriented worked
examples on troubleshooting transfer performance.
Learning and Instruction, 16(2), 154-164.
Whelden, C. H. (1954). Using a film to teach a home
nursing procedure. Audiovisual communication
review, 2(4), 277-281. doi: 10.1007/bf02713294
Wiley, J., & Voss, J. F. (1999). Consturcting argument for
multiple sources: Tasks that promote
understandiung and not just memory for text.
Journal of Educational Research, 91(2), 301-311.
Wittrock, M. C. (2010). Learning as a generative process.
Educational Psychologist, 45(1), 40-45.
doi:10.1080/00461520903433554
Wittwer, J., & Renkl, A. (2010). How effective are
instructional explanations in example-based
learning? A meta-analytic review. Educational
Psychology Review, 22(4), 393-409.
Woloshyn, V. E., Paivio, A., & Pressley, M. (1994). Use of
elaborative interrogation to help students acquire
information consistent with prior knowledge and
information inconsistent with prior knowledge.
Journal of Educational Psychology, 86(1), 79-90.
Resources 54

Resources
Anthony A. Piña
Sullivan University

A range of resources should be articulated to

foster deep learning and extend course-
centered experiences and activities.
Resources should be multimodal to
accommodate students’ interests,
understandings, and capacities, consistent
with course content and technological
accessibility. Resources should allow
students to go beyond the constraints of the
formal course structure to engage in self-
directed, extended learning.

Background
The terms instructional resources and instructional
materials are often used interchangeably in the literature
(Koszalka, Russ-Eft & Reiser, 2013; Polly, 2011). For the
purpose of this chapter, instructional resources will be
defined as encompassing the broad range of materials,
media, and services available to deliver, enhance, and
facilitate instruction and learning online. Resources for
online courses include textbooks and other reading
materials and common components found within learning
management systems, such as course announcements,
discussion forums, pages, documents, presentations,
tutorials, assessments, feedback mechanisms and grade
books (Piña, 2017). Resources also include items delivered
Resources 55

outside the learning management system, such as links to

websites, search engines, library databases, and mobile
apps. The information contained in these resources consists
of individual or combined instances of text, audio, still
images, and video/motion images (Meyer, 2014).

Morrison and Anglin (2011) have emphasized the

critical nature of resources to online course design: “When
there is a separation of the learner and instructor in both
time and location, the traditional sources of instructional
support disappear, and there is a great need for reliance on
well-designed instructional materials” (p. 243).

Multimodality

Head, Lockee, and Oliver (2002) propose that the

quality of online courses is dependent upon the teaching
methods used and upon the attributes of the instructional
resources, including the ability “to deliver information via
specific sensory modalities, such as auditory or visual
channels of communication” (p. 263). Dual coding – the
idea that humans process visual and auditory information
using separate cognitive channels (Paivio, 1986, 1991) –
together with the recognition that the brain can process a
limited amount of information in each of these channels
(Miller, 1956; Moore, Burton & Myers, 2004), suggests
that the design of online courses should include resources
that are multimodal in nature. Richard Meyer has posited
several research-based principles to maximize learners’
abilities to process and code information (Meyer, 2005;
Meyer 2014):

• Coherence Principle. People learn better when

extraneous words, graphics, and audio that are not
directly relevant are excluded rather than included.
Resources 56

• Modality Principle. People learn better from

animation and audio narration than from animation
and on-screen text.
• Multimedia Principle. Retention is improved
through words and corresponding still or motion
graphics/video than from words alone.
• Spatial Contiguity Principle. People learn better
when corresponding words and graphics are
presented near, rather than far, from each other on
the screen.
• Temporal Contiguity Principle. People learn better
when corresponding words and graphics are
presented at the same time.
• Redundancy Principle. People learn better when
visual information is accompanied by audio
narration alone—not with audio narration plus
onscreen text.

Self-Directed Extended Learning

Mobile devices, such as smartphones and tablets,
allow learners to access, manipulate, create, and share
information outside the confines of the physical classroom
or learning management system (Shepard, 2011). Morrison
and Anglin (2011) have noted that research supporting self-
directed learning has identified a number of strategies that
contribute to positive learning effects and that online course
design should incorporate resources that promote self-
directed, personalized learning. Jung and Latchem (2011)
advocate for the use of wikis, online website creation tools,
Google apps, and various websites to create extended
learning spaces. These allow learners to acquire
knowledge, apply knowledge, and construct new and
original knowledge (McTighe & March, 2015). The design
of online courses need not begin and end within the
confines of the learning management system (Piña, 2017).
Resources 57

Empirical Support for Instructional

Resources

Clark and Meyer (2011) have stated, “Based on

cognitive theory and research evidence, we recommend that
e-learning courses include words and graphics rather than
words alone. … By graphics we mean static illustrations
such as drawings, graphs, charts, maps, or photos, and
dynamic graphics such as animation or video” (p. 70). In a
two-part study involving 93 undergraduate students,
Brunye, Taylor, and Rapp (2008) found that procedural
learning was enhanced when instructional materials were
presented in a multimedia format versus text alone, even
when the multimedia was as minimal as a single simple
image.

Richard Meyer and his associates have established

each of their multimedia learning principles governing the
use of instructional resources on empirical evidence (Clark
and Meyer, 2011; Meyer, 2005). Summarizing eleven
studies in which the use of onscreen text and graphics
together was compared to onscreen text alone, Clark and
Meyer (2011) observed:

[P]eople who learned from words and

graphics produced between 55 percent to
121 percent more correct solutions to
transfer problems than people who learned
from words alone. Across all studies, a
median percentage gain of 89 percent was
achieved with a median effect size of 1.50.
pp. 80-81.

The contiguity principle was affirmed by eight

studies in which students receiving integrated presentations
generated 60% more correct solutions than those who
Resources 58

received separated presentations. The overall effect size

was 1.60.

Meyer reviewed 21 experimental comparisons that

confirmed the modality principle by comparing graphics
and text with graphics and audio narration. Learners
performed better in the latter treatment, with a median
effect size of .97. Seven different studies looking at the
effects of redundant onscreen text found support for the
redundancy principle with effect sizes ranging from .65 to
1.0.

Finally, Clark and Meyer (2011) state that, “the

coherence principle is important because it is commonly
violated, is straightforward to apply, and can have a strong
impact on learning” (p. 151). To test the principle, two
versions of a narrated presentation – one with background
music and sound effects relevant to the presentation topic
were delivered to students. Those who received the
narration presentation without the added audio performed
an average of 104% better, with a median effect size of
1.66. The message: keep lessons uncluttered.

To determine the effectiveness of self-directed

learning (SDL) in improving learning outcomes in health
professionals, Murad et al. analyzed 59 studies involving
8,011 learners. Results indicated a moderate median effect
size of .45 indicating acquisition of knowledge was grater
in SDL environments than using traditional teaching
methods (Murad et al., 2010).

In a survey of 1,429 self-directed learners who

subscribed to MIT’s OpenCourseWare e-newsletter, Curtis
Bonk and his associates found that SDL was occurring at
home, work, and school, but also in libraries, cafes,
airports, cars, subways, and trains. Thirty-five percent of
Resources 59

respondents stated that they engaged in self-directed

learning “anywhere with a mobile device” (Bonk et. al, p.
353). Respondents used a number of online resources,
including Wikipedia, MIT OpenCourseWare, YouTube,
TED, Khan Academy, How Stuff Works, and MOOC sites
Coursera, Udacity, and EdX. Eighty-eight percent of
respondents indicated that they had gained new knowledge
as a result of their SDL activities, and 72% rated the
freedom to learn as the most significant factor to their
success (Bonk, Lee, Kou, Xu & Sheu, 2015).

Summary
This chapter has provided a mere sample of the
evidence of the benefits of instructional resources that are
multimodal and that extend learning beyond the formal
course structure. The wise and judicious use of resources in
the instructional design of online courses is a key
component of online course quality. If fact, it would not be
a stretch to say that, without instructional resources, there
would not be any online education.

References

Bonk, C. J., Lee, M., Kou, X., Xu, S., & Sheu, F. (2015).
Understanding the self-directed online learning
preferences, goals, achievements, and challenges of
MIT OpenCourseWare subscribers. Journal of
Educational Technology & Society 18(2), 349-368.
Brunye, T. T., Taylor, H. A., & Rapp, D. N. (2008).
Repetition and dual coding in procedural
multimedia presentations. Applied Cognitive
Psychology, 22(7), 877-895.
Clark, R. C., & Mayer, R. E. (2011). E-Learning and the
science of instruction: Proven guidelines for
Resources 60

consumers and designers of multimedia learning

(3rd Ed). San Francisco, CA: Pfeiffer.
Head, J. T., Lockee, B. B., & Oliver, K. M. (2002).
Method, media and mode: clarifying the discussion
of distance education effectiveness. Quarterly
Review of Distance Education 3(3), 261-269.
Jung, I., & Latchem, C. (2011). A model for e-education:
Extended teaching spaces and extended learning
spaces. British Journal of Educational Technology
42(1), 6–18.
Koszalka, T. A., Russ-Eft, D. F., & Reiser, R. A. (2013).
Instructional designer competencies: The standards
(4th Ed.). Charlotte, N.C.: Information Age
Publishing.
McTighe, J., & March, T. (2015). Choosing apps by design.
Educational Leadership 72(8), 36-41.
Miller, G. A. (1956). The magical number seven, plus or
minus two: Some limits on our capacity for
processing information. Psychological Review,
63(2), 81-97.
Meyer, R. E. (2005). Multimedia strategies. In H. F.
O’Neill (Ed.), What works in distance learning:
Guidelines (7-24). Charlotte, NC: Information Age
Publishing.
Meyer, R. E. (2014). The Cambridge handbook of
multimedia learning. New York, NY: Cambridge
University Press.
Moore, D. M., Burton, J. K., & Myers, R. J. (2004).
Multiple-channel communication: The theoretical
and research foundations of multimedia. In D. H.
Jonassen (Ed.) Handbook of research on
educational communications and technology (851-
875). Mahwah, NJ: Lawrence Erlbaum Associates.
Morrison, G. R., & Anglin, G. J. (2011). An analysis of
success and failures: Focusing on learner-content
interactions for the next generation of distance
Resources 61

education. In L. Moller and J. B. Huett (Eds.), The

next generation of distance education:
Unconstrained learning (235-250). New York, NY:
Springer.
Murad, M. H., Coto-Yglesias, F., Varkey, P., Prokop, L. J,
& Murad, A. L. (2010). The effectiveness of self-
directed learning in health professions education: A
systematic review. Medical Education 44(11),
1057-1068.
Paivio, A. (1986). Mental representations: A dual coding
approach. New York: Oxford University Press.
Paivio, A. (1991). Dual coding theory: Retrospect and
current status. Canadian Journal of Psychology 45,
255-287.
Piña, A. A. (2017). An educational leader’s view of
learning management systems. In A. A. Piña, V. L.
Lowell, and B. R. Harris (Eds.). Leading and
managing e-learning: What the e-learning leader
needs to know. New York, NY: Springer
Polly, D. (2011). Teachers' learning while constructing
technology-based instructional resources. British
Journal of Educational Technology 42(6), 950-961.
Shepard, M. (2011). Creating a culture of digital
collaboration in online learning. In L. Moller and J.
B. Huett (Eds.) The next generation of distance
education: Unconstrained learning (127-138). New
York, NY: Springer.
Application 62

Application
Peggy A. Ertmer, Jennifer C.
Richardson
and Judith Lewandowski
Purdue University

Consistent with providing for active learning,

students should have integral opportunities
within the course design to apply new
learning. Effective course design
incorporates opportunities to practice newly
acquired understandings and skills, both
independently and collaboratively. Online
collaborative application opportunities
should be developed using social media, and
offline collegial groups also should be
structured whenever physical proximity of
students affords this opportunity.

Background

It is generally acknowledged that students learn best

when they are actively engaged in the learning process
(Chickering & Gamson, 1987; Dewey, 1938). Wirkala and
Kuhn (2011) state their conclusion: “Students show better
long-term retention and ability to apply new material if the
instructional method is one that actively engages them and
enables them to put new ideas to use” (p. 1180). Although
there are many ways in which to engage students in the
learning process, one of the most common is asking
students to apply what they have learned to solve new
Application 63

problems (Bransford, Brown & Cocking, 2000; Kolodner,

1992). In general, application activities should engage
learners in problem situations that practicing professionals
encounter (Stepich, Ertmer & Lane, 2001).

For many years, professional educators have

recognized the problem of inert knowledge (Whitehead,
1929), which refers to the domain knowledge acquired by
students during their studies but which remains inaccessible
when needed to solve authentic problems. Although
graduates might perform reasonably well in familiar
situations, their performances quickly deteriorate as
situations become more complex. Efforts to address this
problem have focused on strategies designed to develop the
kind of active knowledge students need to “think like a
professional” in order to solve authentic problems (Quinn,
1994; Rowland, Para & Basnet, 1995).

Currently, a variety of learning and instructional

models incorporate learner engagement as a way to
increase knowledge, develop critical thinking skills,
support transfer of knowledge, and increase motivation.
Instructional approaches, such as problem-based learning
(Barrows, 1986), project-based learning (Krajcik &
Blumenfeld, 2006), case-based learning (Ertmer, Quinn &
Glazewski, 2014), and authentic apprenticeship learning
(Toohey, Ryan & Hughes, 1996) are some of the more
commonly used approaches that require learners to apply
previously acquired conceptual and domain knowledge. For
the sake of brevity, we include all of these approaches
under the umbrella term of experiential learning (Hawkins
& Weiss, 2004).
Application 64

Experiential Learning
Experiential learning dates back to Dewey (1938),
who explained that people learn through a series of primary
and secondary experiences. The primary experience
comprises the activity in which learners directly engage,
while the secondary experience encompasses the reflection
and/or feedback that enables learners to process the initial
experience. Contemporary researchers suggest that
experiential learning is a method of “learning by doing”
and is “fundamental to meaningful learning” (Lewis &
Williams, 1994, p. 5). In formal education, experiential
learning links “academic knowledge and practical skills”
(Ruhaneu, 2005, p. 34). This approach is especially
important when considering adult learners who have
accumulated years of prior learning and work experiences.
Given that adults often enroll in graduate programs to make
themselves more employable, they are especially eager to
engage in experiences that develop real-world skills that
employers seek. Experiential learning “stresses practical
application of knowledge to real world situations”
(Hawkins & Weiss, 2004, p. 3).

Experiential learning supports a participative,

learner-centered approach, which emphasizes “direct
engagement, rich learning events and the construction of
meaning by learners” (Andersen, Boud & Cohen, 2000, p.
225). Benefits include a greater potential for meaningful
learning (Lewis & Williams, 1994), recognition of what
learners bring to the experience (Andersen et al., 2000),
increased self-efficacy and learner motivation (Driscoll,
2005), and the development of self-directed learners (Linn,
Howard & Miller, 2004) who engage as full partners in the
learning process and assume primary responsibility for
their learning (O’Banion, 1997).
Application 65

Application in Online Learning

It is imperative that online students have
opportunities to apply what they are learning, as adult
learners tend to prefer learning opportunities that reflect the
nature of their professional roles (Cercone, 2008). This
requires learner engagement in both individual and
collaborative experiences (MacNeill, Telner, Sparaggis-
Agaliotis & Hanna, 2014). Developing effective
experiential learning, especially in the online environment,
entails planning for, and providing, relevant opportunities
for learners to explore how their learning experiences can
be applied to real-world situations. As such, instructors
need to "be aware of the effect of the environment and to
utilize all aspects of it to create as worthwhile an
experience as possible” (Lindsey & Berger, 2009, p. 122).
Experiential learning effectiveness requires a “dynamic
match” between the learner and the task (Sims, 1983).
Instructors facilitate this match by carefully considering the
learning goal, the context of the experience, the needs of
their learners, and the learners’ prior course experiences.
With this level of preparation and instructor engagement,
the effectiveness of application experiences can be
maximized.

A common element of many higher education

programs is a practicum experience--an on-site, supervised
experience that utilizes knowledge and skills gained from
the program of study. Such experiences are sometimes
referred to as a placement, internship, externship or
apprenticeship program. According to Toohey, Ryan and
Hughes (1996), the purpose of a practicum is to give
students insight into the real world, help integrate them into
the workplace, support the development of professional and
interpersonal skills, link theory to practice, and even
enhance employment possibilities. Katula and Threnhauser
Application 66

(1999) have noted that this form of experiential learning

had been standard practice in many programs for decades.
For example, a year-long practicum is required for clinical
psychologists attending accredited institutions (Hatcher &
Lassiter, 2007), a minimum of one practicum (student
teaching) experience is required of U.S. teachers, the
majority of business schools require a formal internship,
and the medical field requires formal residency programs
for the purpose of providing a guided application of skills
(Toohey et al., 1996).

For online programs, a practicum opportunity can

also be incorporated but may require an extra level of
organization (Donovan, Porter & Stellar, 2010). For
example, the relationship between the student, the
practicum supervisor, and the course facilitator needs to be
clearly established with a clear communication pattern.
Participants will also benefit from having a highly
organized set of documents to guide the process: a written
proposal complete with goals and benchmarks, a Gantt
Chart to standardize the timeline, regular reflections, and a
final project to showcase not only the work completed, but
also the lessons learned (Higgins, 2009).

Application, in and of itself, is not enough to build

professional knowledge and expertise. Students also must
be able to reflect on and articulate what those experiences
mean and to index those experiences in ways that promote
ready access and retrieval (Kolodner, 1992; Schön, 1983).
According to Shulman (1996), “We do not learn from
experience; we learn by thinking about our experience” (p.
208).
Application 67

Empirical Support for Application

Paul and Mukhopadhyay (2005) incorporated
experiential learning, guided by Chickering and Gamson’s
(1987) learner-centered principles, into an international
business education program as a means to improve
students’ problem-solving skills. Five types of class
activities were included: cases, projects (e.g., mock
business negotiations), in-class exercises (e.g.,
demonstrations), guest speakers (e.g., interactive sessions
with business executives who shared real-life experiences),
and video cases. Data were collected from two classes (n =
81), and students were asked to compare the experiential
classes to other courses they had taken within the program.
Students reported that the case write-up, case discussion
questions, role-playing in-class exercises, and use of guest
speakers helped them reach a higher level of understanding
and made learning more fun, effective, efficient, and easier.
The researchers also reported that learners “perceived that
their skills, such as analytical, problem-solving, creative
and critical thinking, improved” (p. 18).

Strobel and van Barneveld (2009) conducted a

meta-synthesis of meta-analyses of problem-based learning
(PBL) research, using quantitative findings and the
narrative descriptions from eight meta-analyses. More
specifically, their focus was on findings that assessed
effectiveness of PBL versus traditional approaches. Results
indicated that PBL instruction was more effective for long-
term retention and resulted in high student and teacher
satisfaction, while traditional approaches were favored for
short-term retention, such as that associated with
standardized board exams. Performance- or skill-based
assessments, including observations by supervisors as well
as students’ case analyses, also favored a PBL approach.
Finally, assessments that covered both knowledge and skill
Application 68

(including oral examinations) favored PBL approaches.

The authors concluded, “PBL is significantly more
effective than traditional instruction to train competent and
skilled practitioners and to promote long-term retention of
knowledge and skills acquired during the learning
experience or training session” (p. 55).

Newmann, Bryk, and Nagaoka (2001) examined the

impact of assignments that required “authentic intellectual
work” on K-12 students’ academic achievement (n = 1,800
3rd-graders; 1,700 6th-graders; 1,400 8th-graders) in the
Chicago Public Schools. Authentic intellectual work
involves “original application of knowledge and skills,
rather than just routine use of facts and procedures. It also
entails disciplined inquiry into details of a particular
problem and results in a product or presentation that has
meaning or value beyond school” (p. 14). The researchers
contrasted 1) didactic assignments, which required students
to learn facts, definitions, and algorithms and typically to
restate them in the same form they were learned with 2)
interactive assignments, which required students to
formulate problems, to organize knowledge and
experiences in new ways, and to express themselves using
elaborated statements both orally and in writing. Outcome
measures included the math and reading subtests on the
Iowa Test of Basic Skills (ITBS) and the reading, math, and
writing subtests of the Illinois Goals and Assessment
Program (IGAP). Results demonstrated a consistent
positive relationship between student exposure to
interactive assignments, as described above, and students’
learning gains on the ITBS regardless of socioeconomic
status, gender, and prior achievement, with effect sizes of
.43, .64, and .52 on the IGAP reading, math, and writing
tests.
Application 69

Service-learning opportunities represent another

avenue for applying skills learned in academic courses.
Maddrell (2015) defined service learning as “an
educational approach that combines community service,
academic coursework, and work-based experience” and
“has been shown to positively affect academic
achievement, as well as personal and social outcomes” (p.
216). For example, Yorio and Ye (2012) conducted a meta-
analysis (n = 40) of service-learning studies with a focus on
social, personal, and cognitive learning outcomes in a
college or university setting. Their study found that service
learning positively impacts students’ cognitive
development, understanding of social issues, and personal
insights. Similarly, Novak, Markey, and Allen (2007)
conducted a meta-analysis (n = 9) of studies comparing
service-learning course options with non-service-learning
course options for undergraduates. Results showed that
service learning improved academic understanding, skills
learned, and the ability to apply knowledge and to reframe
complex social issues.

Tiwari, Lai, So & Yuen (2006) compared the effects

of problem based learning to those of lecturing on
development of students’ critical thinking skills. The
undergraduate nursing students (n = 79) were randomly
assigned to either a PBL or lecture-based version of the
same course. Critical thinking was measured by the
California Critical Thinking Disposition Inventory
(CCTDI), and data were collected four times over three
years. Initially, a significant difference was not found
between the two groups; however, over time, the students
from the PBL version of the course showed significantly
greater improvements in critical thinking skills. Students
enrolled in the PBL-based course showed significant
differences in the development of critical thinking
Application 70

dispositions, outperforming those enrolled in the lecture-

based course.

Summary

The evidence provided in this chapter indicate that

there are many ways to incorporate effective application
activities within online courses. As online courses are
planned and developed, instructional designers should
consider the most relevant application experiences and the
most appropriate ways to include them in online courses.

References
Andersen, L., Boud, D. & Cohen, R. (2000). Experience-
based learning. In G. Foley, Understanding adult
education and training, (2nd ed.). Sydney, Australia:
Allen & Unwin.
Barrows, H. S. (1986). A taxonomy of problem‐based
learning methods. Medical Education, 20, 481–486.
http://dx.doi.org/10.1111/j.1365-
2923.1986.tb01386.x
Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.)
(2000). How people learn: Brain, mind, experience,
and school. Washington, DC: National Academy
Press.
Chickering, A. W., & Gamson, Z. F. (1987, March). Seven
principles for good practice in undergraduate
education. American Association for Higher
Education Bulletin.
Cercone, K. (2008). Characteristics of adult learners with
implications for online learning design. AACE
Journal, 16(2), 137-159.
Dewey, J. (1938). Experience and education. New York,
NY: Collier.
Application 71

Donovan, T., Porter, R. & Stellar, J. (2010). Experiencing

success: Some strategies for planning the program.
New Directions for Teaching and Learning, 124,
89-94.
Driscoll, M. P. (2005). Psychology of learning for
instruction (3rd ed.). Boston, MA: Allyn & Bacon.
Ertmer, P. A., Quinn, J. A., & Glazewski, K. D. (2014).
The ID CaseBook: Case studies in instructional
design (4th ed.). Upper Saddle River, NJ: Pearson.
Hatcher, R., & Lassiter K. (2007). Initial training in
professional psychology: The practicum
competencies outline. Training and Education in
Professional Psychology, 1(1), 49-63.
Hawkins, D. E., & Weiss, B. L. (2004). Experiential
education in graduate tourism studies: An
international consulting practicum. Journal of
Teaching in Travel and Tourism, 4(3), 1-29.
Higgins, P. (2009). Into the big wide world: Sustainable
experiential education for the 21st century. Journal
of Experiential Education, 32(1), 44-60.
Katula, R. A., & Threnhauser, E. (1999). Experiential
education in the undergraduate curriculum,
Communication Education, 48(3), 238-255.
Kolodner, J. L. (1992). An introduction to case-based
reasoning. Artificial Intelligence Review, 6(1), 3-34.
Krajcik, J. S., & Blumenfeld, P. C. (2006). Project-based
learning. In R. K. Sawyer (Ed.), The Cambridge
handbook of the learning sciences (pp. 317-333).
New York, NY: Cambridge University Press.
Lewis, L. H., & Williams, C. J. (1994). Experiential
learning: Past and present. In J. Lewis and R. A.
Caffarella (Eds.), Experiential learning: A new
approach (pp. 5-16). San Francisco: Jossey-Bass.
Lindsey, L., & Berger, N. (2009). Experiential approach to
instruction. In C. M. Reigeluth and A. A. Carr-
Chellman (Eds.), Instructional-design theories and
Application 72

models: Building a common knowledge base

(Volume III, pp. 117-142). New York, NY: Taylor
and Francis.
Linn, P. L., Howard, A. & Miller, E. (Eds.) (2004). The
handbook for research in cooperative education
and internships. Mahwah, NJ: Erlbaum.
MacNeill, H., Telner, D., Sparaggis-Agaliotis, A., &
Hanna, E. (2014). All for one and one for all:
Understanding health professionals’ experience in
individual versus collaborative online learning.
Journal of Continuing Education in the Health
Professions, 34(2), 102–111.
DOI: 10.1002/chp.21226
Maddrell, J. A. (2015). Designing authentic educational
experiences through virtual service learning. In B.
Hokanson, G. Clinton, and M. W. Tracey (Eds.),
The Design of learning experiences (pp. 215–229).
New York, NY: Springer.
Newmann, F. M., Bryk, A. S., & Nagaoka, J. K. (2001).
Authentic intellectual work and standardized tests:
Conflict or coexistence? Chicago, IL: Consortium
on Chicago School Research.
Novak, J. M., Markey, V. & Allen, M. (2007). Evaluating
cognitive outcomes of service learning in higher
education: A meta-analysis. Communication
Research Reports, 24(2), 149–157. DOI:
10.1080/08824090701304881
O’Banion, T. (1997). A learning college for the 21st
century. Washington, DC: American Association of
Community Colleges.
Paul, P., & Mukhopadhyay, K. (2005) Experiential learning
in international business education. Journal of
Teaching in International Business, 16(2), 7-25.
DOI:10.1300/J066v16n02_02
Quinn, J. (1994). Connecting education and practice in an
instructional design graduate program. Educational
Application 73

Technology Research and Development, 42(3), 71–

82.
Rowland, G., Parra, M. L. & Basnet, K. (1995). Educating
instructional designers: Different methods for
different outcomes. In B. B. Seels (Ed.),
Instructional design fundamentals: A
reconsideration (pp. 223–236). Englewood Cliffs,
NJ: Educational Technology Publications.
Ruhaneu, L. (2005). Bridging the divide between theory
and practice. Experiential learning approaches for
tourism and hospitality management education.
Journal of Teaching in Travel and Tourism, 5(4),
33-51.
Stepich, D. A., Ertmer, P.A., & Lane, M. M. (2001).
Problem-solving in a case-based course: Strategies
for facilitating coached expertise. Educational
Technology Research and Development, 49(3), 53-
69.
Schön, D. A. (1983). The reflective practitioner: How
professionals think in action. New York, NY: Basic
Books.
Shulman, L. S. (1996). Just in case: Reflections on learning
from experience. In J. A. Colbert, P. Desberg, and
K. Trimble (Eds.), The case for education:
Contemporary approaches for using case methods
(pp. 197–217). Boston: Allyn & Bacon.
Sims, R. (1983). Kolb’s experiential learning theory: A
framework for assessing person-job interaction. The
Academy of Management Review, 8(3), 501-508.
Strobel, J., & van Barneveld, A. (2009). When is PBL more
effective? A meta-synthesis of meta-analyses
comparing PBL to conventional classrooms.
Interdisciplinary Journal of Problem-Based
Learning, 3(1). dx.doi.org/10.7771/1541-5015.1046
Tiwari, A., Lai, P., So, M., & Yuen, K. (2006). A
comparison of the effects of problem-based learning
Application 74

and lecturing on the development of students'

critical thinking. Medical Education, 40(6), 547-
554. dx.doi.org/10.1111/j.1365-2929.2006.02481.x
Toohey, S., Ryan, G., & Hughes, C. (1996). Assessing the
practicum. Assessment and Evaluation in Higher
Education, 21, 215-229.
Whitehead, A. N. (1929). The aims of education and other
essays. New York, NY: Macmillan.
Wirkala, C., & Kuhn, D. (2011). Problem-based learning in
K-12 education: Is it effective and how does it
achieve its effects? American Educational Research
Journal, 48(5), 1157–1186.
Yorio, P. L., & Ye, F. (2012). A meta-analysis on the
effects of service-learning on the social, personal,
and cognitive outcomes of learning. Academy of
Management Learning & Education, 11(1), 9–27.
Assessment 75

Assessment
Steven M. Ross and Jennifer R.
Morrison
Johns Hopkins University

Regardless of the model of sequencing

learning opportunities, the sequence should
include points of assessment for purposes of
feedback and review, with instances of re-
teaching as necessary for students to acquire
full understanding. Formative assessment,
whether formal, informal, or incidental,
allows teachers and students to give feedback
to one another and to review the
operationalized design in order to revise the
course design based on students’ input with
regard to knowledge acquisition and effective
use of new understandings and skills.

Background

Assessment is often used interchangeably in the

literature--and particularly by practitioners--with processes
such as testing, measurement, and evaluation. In reality, all
have similar functions, and separating them often boils
down to finer-grained uses and sequencing in lessons. For
instructional designers, assessment uses various types of
measurement, and sometimes formal testing, to collect data
regarding learner activities, progress, and attainment of
objectives in a course or lesson. Sequenced, intermittent
assessment is encouraged so that instructors can address
Assessment 76

learner needs for re-teaching and review or for adapting

content to be easier or more advanced. At the same time,
the instructional designer obtains feedback about what
aspects of the lesson are working well and what needs to be
changed. When instruction moves to an asynchronous
online setting, opportunities for feedback are, if anything,
even more crucial because the number of channels by
which students and instructors can communicate is reduced.

Formative Assessment

Formative assessment takes place while instruction

is still going on. In contrast, summative assessment is used
by evaluators as a basis for judging the effectiveness of
completed instruction. Arguably, if one conceives of
instruction as always subject to revision and improvement,
then all assessment can be seen as formative to some
degree (Morrison, Ross, Kalman & Kemp, 2013).
According to Sadler (1989), for everyone involved in
instruction:

“Feedback is a key element in formative

assessment, and is usually defined in terms
of information about how successfully
something has been or is being done. Few
physical, intellectual or social skills can be
acquired satisfactorily simply through being
told about them. Most require practice in a
supportive environment which incorporates
feedback loops.” (p.120)

For teachers, feedback guides decisions about how

effective both the content and delivery of instruction has
been for addressing student needs. Accordingly, decisions
can be made regarding the pace and difficulty of the
instruction and the possible provision of extra practice or
Assessment 77

remediation. For students, feedback provides information

about the strength, weaknesses, and success of their
performance, and, in turn, guidance for subsequent learning
activities (e.g., rereading, seeking help, progressing to the
next unit). For designers, feedback can reveal lesson
completion times for different types of learners, reactions
by teachers and students to varied design components, the
degree to which individual learning objectives are being
achieved, and which parts of the instruction are working as
planned and which need refinement. Simply put, without
assessment, there would be no systematic or reliable
feedback to guide lesson delivery (teachers), lesson usage
(learners), and lesson development and continuous
refinement (instructional designers).

Assessment and Learning

Assessment can take many forms and serve
different audiences; however, its fundamental purpose in
instructional design is to improve the quality of learning as
students “engage in the problems and discourse of a given
area and are given encouragement, response, and feedback
on what they do, as appropriate, with a view to them
becoming more effective in their learning” (Boud, 1990,
p.103). Effective learning is facilitated as students use
metacognitive strategies to predict their performance on
various tasks and to monitor their current levels of mastery
and understanding (Bransford, Brown, and Cocking, 1999).
Teaching practices congruent with a metacognitive
approach provide ongoing assessments that help both
teachers and students monitor progress and make
instructional decisions accordingly (Bransford et al., p. 24).

According to Van Gog, Sluijsmans, Brinke and

Prins (2010), if assessments during instruction are aligned
with the outcomes desired, learners receive guidance on
Assessment 78

their progress and level of mastery. That is, they “can work
on a task, assess their performance on that task, decide
which aspects of their performance need improvement, and
select a next task to work on that will help them improve
these performance aspects” (p.313). Without ongoing
assessment, learners and instructors would operate in a
vacuum, not knowing the degree to which lesson objectives
are being achieved in time to make any adjustments. The
instructional design literature consistently promotes
assessment as an essential design component (Dick &
Carey, 2001; Gagne, 1989; Morrison et al., 2013).

Empirical Support for Assessment

The earliest research on assessment dates back more
than a century to the classic studies by E. L. Thorndike
(1913) in support of the “Law of Exercise.” Thorndike
demonstrated that practice in the absence of knowledge of
results (i.e., feedback) leads to no improvement toward
mastery. Numerous studies have since been conducted on
the use of various assessment-feedback strategies with
different curricula and types of learners. To synthesize the
research evidence, Bangert-Drowns, Kulik, Kulik &
Morgan (1991) conducted a meta-analysis that examined
the effects of feedback in “test-like events” present in text-
based and technology-based instruction. Their analysis
included 58 effect sizes from 40 studies, most of which
were published between 1960 and 1990. Feedback had a
moderate, significantly positive effect on achievement
(effect size = 0.26). The authors found that one important
mediating variable was the type of feedback. Effects were
lower for feedback that merely indicated correctness of
response as compared with feedback that informed the
learner of the correct answer. According to Bangert-
Drowns, et al., “Feedback’s primary importance is in
correcting errors” (p. 232), and it “is most effective under
Assessment 79

conditions that encourage learners’ mindful reception” (p.

233).

Given that feedback occurs as a natural product of

assessment, what about the direct effects of testing itself?
In a second meta-analysis of the same studies, Bangert-
Drowns, Kulik, and Kulik (1991) examined the effects of
frequent classroom testing. Results indicated that students
who took at least one test during a 15-week period scored
roughly 0.5 standard deviations higher (a strong effect) on
criterion examinations than those who did not take a test. In
addition, improved criterion performance was associated
with more frequent testing, although testing more
frequently than once every two weeks did not produce an
additional benefit.

References

Bangert-Drowns, R. L., Kulik, J. S., & Kulik, C. (1991).

Effects of frequent classroom testing. Journal of
Educational Research, 85(2), 89-99.
Bangert-Drowns, R. L., Kulik, J. S., Kulik, C., & Morgan,
M. (1991). The instructional effect of feedback in
test-like events. Review of Educational Research,
61(2), 213-238.
Boud, D. (1990). Assessment and the promotion of
academic values. Studies in Higher Education,
15(1), 101–111.
doi:10.1080/03075079012331377621.
Bransford, J. D., Brown, A. L., & Cocking, R. R. (1999).
How people learn: Brain, mind, experience, and
schools. Washington, D.C.: National Academy
Press.
Dick, W., Carey, L., & Carey, J. (2001) The systematic
design of instruction (5th ed.). New York: Longman.
Assessment 80

Gagne, R. (1985). The conditions of learning (4th ed.). New

York: Holt, Rinehart, and Winston.
Kulhavy, R. W., & Stock, W. A. (1989). Feedback in
written instruction: The place of response certitude.
Educational Psychology Review, 1(4), 279-308.
Kulhavy, R. W., & Wager, W. (1993). Feedback in
programmed instruction: Historical context and
implications for practice. In J.V. Dempsey and G.
C. Sales (Eds.), Interactive instruction and
feedback. Englewood Cliffs, NJ: Educational
Technology Publications.
Morrison, G. R., Ross, S, M, Kalman, H. K., & Kemp, J. E.
(2013). Designing effective instruction (7th ed.).
Hoboken, NJ: Wiley.
Sadler, D. R. (1989). Formative assessment and the design
of instructional systems. Instructional Science,
18(2), 119–144. doi:10.1007/BF00117714.
Sutton, R. (2010). Making formative assessment the way
the school does business: The impact and
implications of formative assessment for teachers,
students and school leaders. In A. Hargreaves et al.
(Eds.). Second international handbook of
educational change, Springer international
handbooks of education 23, DOI 10.1007/978-90-
481-2660-6_49.
Thorndike, E. L. (1913). Educational psychology: The
psychology of learning. New York: Teachers
College Press.
Van Gog, T., Sluijsmans, D., Brinke, D., & Prins, F.
(2010). Formative assessment in an online learning
environment to support flexible one-the-job
learning in complex professional domains.
Educational Technology Research & Development,
58(3), 311-324.
Reflection 81

Reflection
Monica W. Tracey
Wayne State University

Effective course design must include

opportunities for reflection as an extension of
the Feedback/Review/Reteach standard.
Reflection involves both instructor self-
reflection and student self-reflection related
to achievement of the purposes that have
been articulated as the basis for the course.
Such reflection is intended to deepen the
learning experience and may serve as
reiteration of purpose at key points during
the course.

Background
Instructional design, once approached in a
systematic step-by-step manner, is now understood to be a
human-centered activity requiring new ways of discovering
and understanding a problem or challenge. While the tools
of the traditional model of design must be in the designer’s
toolbox, designers do not typically follow the step-by-step
approach prescribed in these models (Kirschner, Carr, van
Merriënboer & Sloep, 2002).

Design relies on the designer’s judgment, which

emerges from the accumulated episodes in an individual’s
history of design choices and their impact. These episodes
are called design precedents (Tracey & Boling, 2013). No
Reflection 82

single method of designing instruction can address every

situation effectively. The designer must be prepared to
frame a challenge or problem, generate a range of ideas
based on previously encountered solutions or precedents,
and interact with the end users to determine the best design
solution. This method of design is messy and chaotic,
requiring designers to continuously embrace uncertainty
and reflect on their beliefs and precedents in order to arrive
at a successful design solution.

Reflection-in-Action
A specific type of reflective practice, reflection-in-
action emphasizes that unique and uncertain situations can
be understood through attempts to reflect upon them while
they are still in progress-- rather than in the past (Schön,
1983). Designers focus on problems or challenges by
having a reflective conversation of the situation and then
drawing on a repertoire of precedents to design solutions.
Such reflection leads to new discoveries that lead, in turn,
to additional reflections-in-action. Reflection expands
designers’ repertoires of precedents, which further develops
their capacity to visualize and understand the world and
increases their ability to be innovative in design.

Successful online courses require designers to

reflect on their unique design constraints, including the
limitations of learning management systems (LMS), the
lack of traditional communication methods between
instructor and learner; and the challenge of facilitating
effective learner-to-learner collaboration—each of which
may impede innovative design strategies. Considering a
wide range of possibilities during online course design
enables designers to refine the design and determine the
appropriate choices for the delivery environment and for
the needs of online users.
Reflection 83

Reflection and Student Learning

Student reflection and self-assessment, incorporated
through instructional strategies, is particularly useful in
online course design. LMS tools promoting student
engagement and reflection include discussion boards,
reflection journals, and group discussions. Incorporating
these strategies into the online learning environment can
encourage student reflection on identified topics, expand
thought processes, and integrate content into specific
situations. Reflections also encourage growth through self-
examination. Replacing “in class” discussion with online
guided reflection questions, provides students with the
opportunity to use reflection to explore course topics while
developing their content knowledge and deepening the
learning experience. Guided reflections also provide
instructors with necessary feedback on student learning and
may serve as an indicator of the need for remediation.

Empirical Support for Reflection

Reflection is generally identified as the personal

and internal construction of knowledge through recursive
observations and interpretations of one’s experiences or
beliefs. The use of reflective thinking and writing as a
pedagogical tool has a long tradition in the practice of
education. As early as 1910, John Dewey (1991) introduced
the idea of reflection as a facilitator of learning. Journal
writing in particular has been researched and implemented
as a space for documenting reflection on experiences,
beliefs, and knowledge (Pavlovich, Collins & Jones, 2009).

Dewey’s original conception of reflection

emphasized an open, holistic space for learners to engage in
reflection, free from the imposition of outside structure,
while other scholars have emphasized the value of prompts
Reflection 84

as scaffolding to support novices in acquiring reflective

skills. Contemporary research suggests that students,
particularly when they are novices in a field, may benefit
from scaffolding to guide the reflective process, including
prompts that encourage them to draw connections between
course content and personal experiences (Whipp, 2003). At
the same time, it is crucial to align scaffolding with student
development in order to ensure that prompts or other
structuring activities continually push students to achieve
greater complexity as their knowledge base grows and their
reflective abilities mature (Ada, 2010).

Whipp (2003) found meaningful improvements in

the levels of reflection among teacher education students
after increasing the amount of scaffolding provided to
students in an online course. Techniques that were found to
be most effective in this study included tailored and general
questions related to social, political, and moral issues, as
well as prompts to draw connections between course
readings and student experiences.

Reflective writing gives students the space to

explore the stories they tell themselves about themselves,
their experiences, and their beliefs (Bourner, 2003; Davis,
2006; Henderson, Napan & Monteiro, 2004; Luehmann,
2007). There is also an empirical basis for using reflective
writing as a learning activity to engage students in
examining their beliefs and integrating personal
experiences into a framework of professional knowledge
and identity (Hutchison & Tracey, 2014; Tillema, 2000;
Tracey & Hutchinson, 2013; Tracey & Hutchinson, 2015;
Tracey, Hutchinson & Grzeybk, 2014). In these instances,
reflection was seen as a method to define and refine one’s
beliefs, values, and conceptual perspectives (Atkins &
Murphy, 1993; Hong and Choi, 2011; Langley & Brown,
2010). As such, reflection becomes a crucial tool for the
Reflection 85

formation of professional identity, which materializes in

part from continuing, dynamic narratives and
reinterpretations of relevant experiences in support of
conceptions of the professional self (Luehmann, 2007).

In order to explore the ideas of professional

development through reflection, Tracey and Hutchinson
(2013) conducted a preliminary study examining how
graduate students in instructional design use reflection to
build their identity as instructional designers within a
design-thinking framework. The subjects included 40
instructional technology graduate students across two
semesters of a foundational online course in instructional
systems design. As part of the course requirements,
students were required to maintain a reflection journal,
which was shared throughout the course with the instructor
for feedback and assessment via Google documents. In
addition, the online course included a case study
component, which gave students hands-on experiences in
developing instructional design plans. Because novice
students may benefit from scaffolding in order to better
understand concepts, the researchers used structured
reflection in response to assigned topics or questions to
spur narratives and interpretations. Lin et al. (1999)
describe such prompts as providing “learners with a means
of externalizing mental activities that are usually covert”
(p. 49). While this study was preliminary in nature, it
represents an important step in exploring how students can
use reflective practice to develop the foundations of their
professional identity. The data indicated that scaffolding
via prompts and feedback can support students in moving
from unproductive to productive reflection and can lead to
development of an emerging professional identity. It is also
important to remember that scaffolding practices need to
take account of student progress and evolve to continue to
Reflection 86

challenge students to improve the depth of their reflection

(Ada, 2010).

The results of this preliminary study provided the

impetus to move forward with a more detailed study, with
the primary goal of developing a more sophisticated and
nuanced understanding how graduate students in ID use
reflection in relation to design precedents and professional
identity. The secondary goal is to establish a more rigorous
methodological framework for conducting this type of
qualitative research.

For this follow-up study, the researchers used the

Reflection Evaluation for Learners’ Enhanced
Competencies Tool (REFLCT), which was initially
developed as a rubric to assess the reflective writing of
medical students (Wald, Borkan, Taylor, Anthony & Reis,
2012). The REFLECT tool was designed with clear criteria
for placing a response on the reflection spectrum, offering
guidelines across multiple areas of potential reflection.
After several design iterations, the final version of
REFLECT achieved an ICC of 0.632 and a Cronbach alpha
of 0.774.

The REFLECT rubric divides the reflective writing

spectrum into four categories: 1) habitual action is
associated with short responses typically characterized by
basic, impersonal fact reporting and omission of important
aspects of the response; 2) thoughtful action is more
detailed and elaborate, but still remains on the factual level
without moving into meaningful reflection; 3) reflection is
viewed as writing that demonstrates effort to move beyond
description to incorporate exploration, questioning,
analysis, or some other form of meaning-making; and 4)
critical reflection represents a thorough and thoughtful
critical approach to any reflection domain (Wald et al.,
Reflection 87

2012). REFLECT applies these categories across six

domains of reflection: writing spectrum, which addresses
the overall reflective quality; presence, which addresses
authorial voice; description of conflict, which concerns the
level of detail and insight in the description of a
precipitating event or issue; emotion, which is related to the
inclusion and exploration of emotion and emotional insight;
analysis, which attends to the quality of meaning-making in
the response; and attention to assignment, an optional
category that addresses how well the response aligns with
the writing prompt or task.

The primary goal for this research study was to

conduct a descriptive analysis of how first-year graduate
students use reflection to explore topics related to their
development of a professional identity. Using the
REFLECT rubric, 70% of all assessments were considered
either reflection or critical reflection, clearly indicating that
graduate students are able to respond to prompts covering
design concepts, experiences, and identity attributes in
ways that demonstrate the ability to examine, integrate, and
analyze their beliefs, knowledge, and experiences.

Reflective writing has been used in medical

education to measure student learning and development. In
a study of reflective learning in medical students, formative
feedback was found to be a crucial factor in both the
development of reflective skills and in student engagement.
Formal grading, on the other hand, a clearly summative
evaluation, was not found to be an effective factor in
fostering reflection (Vivekananda-Schmidt, Marshall,
Stark, McKendree, Sandars, & Smithson, 2011). Peer
feedback to support deeper reflection is an alternative also
worthy of exploration; some studies have supported its use
in fostering reflection (Hall & Davison, 2007; Maor, 2003;
Vivivekananda-Schmidt et al., 2011), but there are also
Reflection 88

indications that peer feedback may be associated with

reduced reflective quality when compared with private
reflective assignments (Xie, Ke & Sharma, 2008).

One of the key benefits of reflection is its

connection to life-long learning. In other words, reflective
learning assists students in acquiring the metacognitive
tools to construct knowledge and engage in critical analysis
of their own thinking, actions and experiences long after
they leave a formal learning environment (Ada, 2010;
Blaschke & Brindley, 2011; Bourner, 2003; Lin, Hmelo,
Kinzer & Secules, 1999).

Reflection as it relates to professional practice has

also been the topic of research. Schön (1983) was the first
to connect reflection with professional practice through
“reflection-in-action” and “reflection-on action.” The
former refers to the ongoing internal dialogue individuals
have while confronting and attempting to solve a problem,
while the latter refers to the individual’s construction and
revision of personal narratives around past experiences and
beliefs as they encounter new experiences or gain new
knowledge. McAlpine and Weston (2000) introduced a
third category, “reflection-for-action,” focusing on using
past experiences to help shape future actions. Each of these
reflection activities are relevant to instructional designers,
as the goal of design is to solve instructional problems.

Research indicates that reflection-in-action is most

effective when considering four aspects of design activity:
designer, process, content, and context (Tracey & Baaki,
2014). The first of these is the most straightforward.
Process refers to looking at design in two different ways:
through rational problem solving and through reflective
practice (Cross, 2011; Dorst, 2008; Schön, 1983). A
specific aspect of context is how designers draw from a
Reflection 89

repertoire of precedents inside and outside of the project

(Brown, 2009; Cross, 2011; Dorst, 2008; Schön, 1983).

Baaki and Tracey (2014) researched reflection-in-

action during design activity by studying designers from
four design fields. Using criterion sampling, participants: 1)
were involved in their own, real short-term project lasting
between 37 to 87 days; 2) had at least five years of design
experience; 3), were individually responsible for at least
75% of the design work, and 4) were engaged in a non-
routine, non-procedural design project. This last criterion
was chosen because a non-routine design is one that lacks a
well-formed approach to a solution (Snider, Culley &
Dekoninck, 2013). The study engaged each of the eight
participants for a relatively long period of time (average of
64 days) through the life of a design project. Participants
completed weekly reflection journals, participated in
interview meetings, shared design artifacts, and reviewed
design milestones.

Summary

Research on reflection indicates that it is a useful

and an effective activity for designers while designing
online courses and as an instructional strategy for students
taking these courses. Reflection supports innovative course
design, student content and knowledge acquisition, and
student and designer professional identity development.
Finally, reflective activities such as journal writing and the
use of reflective prompts support instructor feedback in an
online environment.

References

Ada, W. W. (2010). Computer supported collaborative

learning and critical reflection: A case study of
Reflection 90

fashion consumerism. Interdisciplinary Journal of

E-Learning and Learning Objects, 6, 87-102.
Atkins, S., & Murphy, K. (1993). Reflection: A review of
the literature. Journal of Advanced Nursing, 18(8),
1188–1192.
Baaki, J., & Tracey, M. W. (2015). Repertoire of
precedents: designers kindling fatwood during
reflection-in-action. In Hokanson, B., G. Clinton,
and M. Tracey, (Eds.), The design of learning
experience: Creating the future of educational
technology (pp.155-166). New York: Springer.
Blaschke, L.M., & Brindley, J. (2011). Establishing a
foundation for reflective practice: A case study of
learning journal use. European Journal of Open,
Distance, and E-Learning (EURODL), Special
Issue. Retrieved from
http://www.eurodl.org/materials/special/2011/Blasc
hke_Brindley.pdf.
Bourner, T. (2003). Assessing reflective learning.
Education + Training, 45(5), 267-272.
Brown, T. (2009). Change by Design. New York: Harper
Business. London: Berg Publishers.
Davis, E. A. (2006). Characterizing productive reflection
among preservice elementary teachers: Seeing what
matters. Teaching and Teacher Education, 22(3),
281–301.
Dewey, J. (1991). How we think: A restatement of the
relation of reflective thinking to the educational
process. Lexington, MA: Heath.
Dorst, K. (2008). Design research: A revolution-waiting-to-
happen. Design Studies, 29, 4-11. doi:
10.1016/j.destud.2007.12.001
Hall, H., & Davison, B. (2007). Social software as support
in hybrid learning environments: the value of the
blog as a tool for reflective learning and peer
Reflection 91

support. Library and Information Science Research,

29(2), 163–187.
Henderson, K., Napan, K. & Monteiro, S. (2004).
Encouraging reflective learning: An online
challenge. In Beyond the comfort zone: Proceedings
of the 21st ASCILITE Conference (pp. 357-364).
Hong, Y., & Choi, I. (2011). Three dimensions of reflective
thinking in solving design problems: A conceptual
model. Educational Technology Research and
Development, 59(5), 687–710.
Hutchinson, A., & Tracey, M. W. (2015). Documenting
design ideas: How graduate ID students reflect on
idea generation. Instructional Science, 43(5), 527-
544.
Kirschner, P., Carr, C., van Merriënboer, J., & Sloep, P.
(2002). How experts designers design. Performance
Improvement Quarterly, 15(4), 86-104.
Lai, G., & Calandra, B. (2010). Examining the effects of
computer-based scaffolds on novice teachers’
reflective journal writing. Educational Technology
Research and Development, 58(4), 421-437.
Langley, M., & Brown, S. (2010). Perceptions of the use of
reflective journals in online graduate nursing
education. Nursing Education Perspectives, 31(1),
12-17.
Lin, X., Hmelo, C., Kinzer, C., & Secules, T. (1999).
Designing technology to support reflection.
Educational Technology, Research and
Development, 47(3), 43–62.
Luehmann, A.L. (2007). Identity development as a lens to
science teacher preparation. Science Education,
91(5), 822-839. doi: 10.1002/sce
Maor, D. (2003). The teacher's role in developing
interaction and reflection in an online learning
community. Educational Media International, 40(1-
2), 127-37. doi:10.1080/0952398032000092170
Reflection 92

McAlpine, L., & Weston, C. (2000). Reflection: Issues

related to improving professors’ teaching and
students’ learning. Instructional Science, 28(5),
363–385.
Pavlovich, K., Collins, E., & Jones, G. (2009). Developing
students’ skills in reflective practice: Design and
assessment. Journal of Management Education,
22(1), 37–58.
Schön, D. (1983). The Reflective Practitioner: How
professionals think in action. New York: Basic
Books Inc.
Snider, C.M., Culley, S. J., & Dekoninck, E. A. (2013).
Analysing creative behavior in the later design
process. Mechanical Engineering, 34, 543-574. doi:
10.1016/j.destud.2013.03.001
Tillema, H. H. (2000). Belief change towards self-directed
learning in student teachers: Immersion in practice
or reflection on action. Teaching and Teacher
Education, 16(5-6), 575–591.
Tracey, M. W., & Baaki, J. (2014) Design, designers and
reflection-in-action. In B. Hokanson and A.
Gibbons (Eds.), Design in educational technology:
Design thinking, design process, and the design
studio. (pp.1-13). New York: Springer.
Tracey, M. W., & Boling, E. (2013). Preparing
instructional designers and educational
technologists: Traditional and emerging
perspectives. In M. Spector et al. (Eds.), Handbook
of Research on Educational Communications and
Technology (4th ed.). (pp.653-660). New York:
Springer.
Tracey, M. W., & Hutchinson, A. (2013). Developing
designer identity through reflection. Educational
Technology, 53(3), 28-32.
Reflection 93

Tracey, M. W., & Hutchinson, A. (2015). Uncertainty,

reflection, and designer identity development.
Design Studies, 42, 86-109.
Tracey, M. W., Hutchinson, A. & Quinn-Grzebyk, T.
(2014). Instructional designers as reflective
practitioners: Developing professional identity
through reflection. Educational Technology
Research & Development, 62(3), 315-334.
Vivekananda-Schmidt, P., Marshall, M., Stark, P.,
McKendree, J., Sandars, J., & Smithson, S. (2011).
Lessons from medical students' perceptions of
learning reflective skills: A multi-institutional
study. Medical Teacher, 33(10), 846-850.
doi:10.3109/0142159X.2011.577120
Wald, H. S., Borkan, J. M., Taylor, J. S., Anthony, D., &
Reis, S.P. (2012). Fostering and evaluating
reflective capacity in medical education:
Developing the REFLECT rubric for assessing
reflective writing. Academic Medicine, 87(1), 41–
50.
Whipp, J. (2003). Scaffolding critical reflection in online
discussions: Helping prospective teachers think
deeply about field experiences in urban schools.
Journal of Teacher Education, 54(4), 321–334.
Xie, Y., Ke, F. & Sharma, P. (2008). The effect of peer
feedback for blogging on college students’
reflective learning processes. The Internet and
Higher Education, 11(1), 18–25.
Independent Learning 94

Independent Learning
David W. Price, Saul Carliner and
Yuan Chen
Concordia University

Effective course design incorporates

opportunities for independent learning, both
instructor- and self-directed. Online course
development, particularly in the
asynchronous mode, should epitomize
independent learning, which should include
opportunities for feedback, review, and
reflection—all of which should resonate with
the purpose.

Background

Independent learning can be seen from two broad

perspectives. The first perspective is as a general pedagogy,
which Gunasekara (2008) describes as a pedagogy of
questioning rather than a pedagogy of delivering answers.
Independent learning can also be seen from the perspective
of individual “ownership” of the learning process, which
includes making informed choices about seeking guidance
or collaborating with others, as independent learning does
not mean learning in isolation (Field, Duffy & Huggins,
2014; Meyer, Haywood, Sachdev & Faraday, 2008;
Mitchell, Zutshi & Weaver, 2010).
Independent Learning 95

General Independent Learning Skills

Dependent and independent learners differ from one

another in some fundamental ways. Dependent learners
expect others not only to choose their goals and criteria, but
also to regulate their learning experience (Rogers, 2012). In
contrast, independent learners rely on self-regulation by
trying a variety of strategies before asking for help, by
seeking clarifications as needed, by acting on feedback, and
by reflecting on their own progress rather than continually
seeking approval (Murdoch & Wilson, 2006).

While essential to independent learning, self-

regulation alone will not guarantee effective independent
learning, whether online or in person. It merely
demonstrates an individual’s ability to conform to someone
else’s values and goals (Rogers, 2012). For instance, high
achievers in traditional schools may be good self-regulators
as they study according to assigned content and criteria, but
if they lack ownership of their learning, they can feel lost,
angry, or cheated when deprived of lectures and clear
“correct” answers and are expected to choose their own
values, criteria, and content; to assess themselves against
their own values and criteria; or to decide what is valuable
and what they should change (Gunasekara, 2008; Higher
Education Academy, 2014; Knobbs & Grayson, 2012;
Rogers, 2012; Warring, 2013). Ownership also affects
motivation and satisfaction. Learners who negotiate a topic
area and select a supervisor can be more satisfied with
independent learning than those who feel forced into
independent learning, are assigned a topic, and are given a
supervisor who does not provide regular contact and
support (Hunt, Scicluna & McNeil, 2011).
Independent Learning 96

Importance for Distance Learning

Independent learning can perform an essential role

in distance learning, regardless of whether a particular
program is designed for synchronous or asynchronous
study. If a program is designed for synchronous study, such
as a live virtual classroom, television, radio, or similar
approach, students may attend scheduled learning sessions;
however, they may lack opportunities for face-to-face
interaction with the instructor. While newer options for
face-to-face digital contact exist, these require some
planning and training in advance. Some synchronous
instructors compensate for limited interaction by engaging
students in discussion boards and one-on-one meetings--but
many do not. In such situations, students have little direct
interaction with instructors and often feel invisible or
anonymous. As a result, students persist in their studies
only as a result of their own initiative. Independent learning
skills are crucial to maintaining that initiative.

Asynchronous instruction, such as self-study e-

learning, workbooks, and the tutorial-style study in many
programs, proceeds entirely at the initiative and pace of the
student. In the case of self-study programs, students require
independent study skills to schedule their own learning,
engage regularly with material, and persist through the
prescribed course of study. In the case of tutorials, students
usually need to propose their own content and learning
goals, which they finalize in consultation with the
supervising faculty member. Online course development,
particularly in the asynchronous mode, should include
multiple opportunities for feedback, review, and reflection.
Independent Learning 97

Skills for Independent Distance Learning

Regardless of the type of independent learning

experience in distance education, two skills are essential for
independent learning: acting with autonomy and acting
with agency.

Acting with Autonomy

As previously noted, personal ownership of the

learning experience is central to independent learning. Also
central to independent learning is self-regulation (Meyer et
al., 2008), which refers to the ability of a student to
monitor, direct, and manage behaviors so that the student
successfully completes the assigned learning. Self-
regulated learners engage in a cyclical process of setting
goals, choosing strategies, and measuring the ability to
meet goals with those strategies. Self-regulation includes
committing to a goal (Higher Education Academy, 2014),
and applying a process to manage intrinsic motivation,
confidence, and emotion (Meyer et al., 2008; Murdoch and
Wilson, 2006), to manage time and pacing (Meyer et al.,
2008), and to reflect on progress to adapt to and overcome
obstacles to achieve the goal (Marshall, 2008; Meyer et al.,
2008; Mitchell et al., 2010; Murdoch & Wilson, 2006).

Some will combine ownership and self-regulation

under the broader concept of autonomy (Mitchell et al.,
2010). Autonomy relates to control or the ability to make
informed choices about a learning experience based on
personal needs (Broad, 2006). Autonomy starts with
deciding which objectives are worth pursuing and
continues with choosing content on which to focus, the
learning process to use (Higher Education Academy, 2014;
Mitchell et al., 2010), and the criteria for determining
success (Rogers, 2012). Autonomous learners apply their
Independent Learning 98

own creativity during learning and draw on assistance as

needed to produce a uniquely individual learning
experience (Jones & Dexter, 2014; Mitchell et al., 2010).

Acting with Agency

Learners face three types of situations with varying

expectations for control: 1) dependent learning, in which
they rely on external sources such as an instructor for
direction and motivation; 2) accredited learning, in which
the learning objectives and evaluations are externally
defined but flexibility exists in the way that learning
proceeds; and 3) independent learning, in which they rely
on themselves for most direction and motivation.
Independent learners actively seek ways to influence or
interpret an experience in a way that it furthers personal
values and needs, even if they have limited control over the
situation (Field et al., 2014).

Impact of Autonomy and Agency

As a result of their autonomy and agency,

independent learners show higher motivation, confidence,
and awareness of limitations. They also use more learning
strategies, have higher standards and performance, and
form their own opinions, which extends their learning
(Meyer et al., 2008). In assignments, independent learners
recognize their assumptions, exclude redundant
information, and address ethical issues (Gunasekara, 2008).
Provided with learning strategies, independent learners can
use their own content to construct personally motivated
learning experiences (Bishop, 2006). Independent learners
are conscious of how they learn, enabling them to better
control their learning experience outside of school, to
question and improve practice in the workplace, to share
their developing knowledge with their professional
Independent Learning 99

community (Hunt et al., 2011; Jones & Dexter, 2014;

Marshall, 2008) and to compensate for a lack of workplace
training to meet professional needs (Jones & Dexter, 2014).

Designing for Independent Learning

Research suggests some general guidance in

designing and facilitating distance education that promotes
the development of independent learning skills. Many of
these strategies are intended to promote autonomy and
agency, especially among students who have not yet had
opportunities to develop these skills.

Preparing Learners for Independent Learning

Developing the skills to learn independently

requires explicit modeling, practice, and feedback during a
learning experience (Silver-Pacuilla, 2008). If independent
learning is simply assumed to be within the skill set of the
students, then dependent learners may face frustration and
blame the course or the facilitator for low contact time, lack
of explicit specifications of what to do and how to do it,
and not explaining how to use limited summative feedback
to improve future performance (Rogers, 2012). Expecting
dependent learners to seek help with generic study skills
and apply what they learn would be expecting them to act
like independent learners to solve their own dependent
learning problems (Field et al., 2014). Dependent learners
need to have their expectations reset (Higher Education
Academy, 2014) and, when entering university, may
require assistance with study skills, information literacy
skills, and reflection (Field et al., 2014; Marshall, 2008).
Independent Learning 100

Preparing Learners to Use Technology

Technology is no panacea for the challenges of

independent learning. To the extent it fosters ownership
and self-regulation (Meyer et al., 2008), technology can
support independent learning by providing ways for
learners to easily access resources, quickly measure their
progress, and communicate with peers and facilitators to
gather feedback (Meyer et al., 2008). But just as the skills
needed for independent learning are not initially possessed
by all learners, designers and instructors should not assume
that students already have the skills to effectively use
technology to support their learning. Some allowances need
to be made for students to grow accustomed to the features
of their online learning environment.

Designing for Learning Experiences

Dependent learning experiences, such as lectures,

exclude learners from choosing content and setting pace,
performing analysis and synthesis of content, or
constructing explanations, all of which are done by the
instructor (Mitchell et al., 2010). As a result, learners may
retreat to merely doing only what is required to complete an
instructor’s assessments (Mitchell et al., 2010) rather than
engaging in the deep learning the instructor hopes to spark.
Certain experiential teaching strategies – such as case
studies, site visits, speakers, games, and simulations – do
not on their own create independent learning. They must
integrate autonomy and agency into their designs,
particularly through incorporating aspects of the “messy”
real world (Gunasekara, 2008).

More fundamentally, when designing independent

learning experiences, designers and instructors can shift
away from organizing and teaching content (Bishop, 2006;
Independent Learning 101

Meyer et al., 2008) and focus on teaching and supporting

strategies for learning (Bishop, 2006). To promote
independent learning, designs should help learners set their
own objectives and assessments, choose and structure their
own activities, and evaluate themselves (Marshall, 2008);
choose sources of feedback; and even change the structure
of their learning environment (Meyer et al., 2008).

In disciplinary materials, designs can demonstrate

disciplinary experts explicitly modeling independent
learning strategies such as motivation, planning,
monitoring, pacing, and evaluation (Meyer et al., 2008;
Mitchell et al., 2010), as well as critical strategies such as
testing ideas and adapting them based on empirical
evidence (Johnson and Harreld, 2012; Marshall, 2008).
When they do so, experts can explain the rationales for
their chosen strategies, think aloud as they use various
strategies, and reflect on the results. The learning
experience can include opportunities for learners to make
use of similar processes (Field et al., 2014; Marshall, 2008)
to build their confidence (Johnson & Harreld, 2012).

One other issue poses a challenge to designing

learning experiences that foster independent learning in
online settings. Some formal curricula, from which the
distance education programs emerge, primarily focus on
delivering content. That makes a shift to focusing on
learning strategies difficult (Mitchell et al., 2010), because
the philosophy of the curriculum conflicts with a revised
approach to learning.

Integrating Student Interaction

Although they are independent, learning

experiences need not be solitary. Students can work in
dyads and groups to learn content and, in the process, work
Independent Learning 102

through challenges about determining what to learn, how to

structure the learning experience, and how to assess
success. Regardless of whether such choices succeed or
fail, instructors can encourage learners to reflect on the
process critically to consider the effectiveness of their
group work strategies (Mitchell et al., 2010) and how to
improve them.

Providing Learners with Feedback

Feedback is essential both for independent learning

and distance education, providing a link between the
instructor and the learner and minimizing the anonymity of
the experience. Formative feedback entails providing
ongoing evaluations of work to help learners revise their
approaches to learning to achieve improvements, while
summative feedback means evaluating work when it is
completed and not providing any opportunity to revise
(Issa, Issa, and Kommers, 2014; Mitchell et al., 2010). In
the case of formative feedback, the focus shifts from
content to a mixed focus on content and learning processes
(Mitchell et al., 2010; Stern, 2009).

Formative feedback is central to independent

learning partly because it is a key component of self-
regulation. Formative feedback allows learners to measure
their success and adjust their approaches accordingly,
which can strengthen the learning process, improve
academic outcomes, and increase satisfaction (Issa et al.,
2014). Formative feedback might relate to content, the
independent learning process, or both. Content-focused
feedback primarily assists learners with better mimicking
the instructor’s values and goals, while process-focused
feedback assists learners with discovering how they can
adjust their strategies to improve their learning (Marshall,
2008). Without feedback, learners persist with their current
Independent Learning 103

approaches, perhaps unaware that doing so might not

enable them to improve (Kirschner, Sweller & Clark,
2006).

Preparing Facilitators for Change

When designing independent learning experiences,

instructional designers and facilitators rely on learner
autonomy to allow for more focused work with particular
groups as they need it (Meyer et al., 2008). However, this
autonomy among learners is often not fully developed,
which poses challenges to both instructors and learners.

One such challenge is that learner autonomy may

require more time to develop and test than is required for
simple delivery of content. Moreover, because of the focus
on process over content, both learners and facilitators may
not recognize the value of independent learning and it may
be particularly challenging to implement these strategies in
large classes (McLinden & Edwards, 2011; Mitchell et al.,
2010). Facilitators might feel uncomfortable with these
learning experiences due to having less control in “messier”
independent learning environments and may fear negative
impacts on teaching evaluations (Gunasekara, 2008).

Summary

Independent learning is a skill—both for learners

and for learning designers. Learners skilled in independent
learning can identify personal values and needs and make
informed choices about their goals and the strategies
required to meet them. They can engage in self-regulation
as they try strategies, evaluate progress, adapt to feedback,
and persist to overcome challenges. Finally, they can
critically reflect on sources of knowledge and their own
performance to determine how to adjust their goals and
Independent Learning 104

strategies or even their values in response to differing

contexts and perspectives (Hunt, Scicluna & McNeil, 2011;
Marshall, 2008; Mitchell et al., 2010). Those who design
distance education would be well-served by become fluent
in design for independent learning.

References

Bishop, G. (2006). True independent learning—an

andragogical approach: Giving control to the learner
over choice of material and design of the study
session. Language Learning Journal, 33(1), 40–46.
Broad, J. (2006). Interpretations of independent learning in
further education. Journal of Further and Higher
Education, 30(2), 119–143.
http://doi.org/10.1080/03098770600617521.
Field, R. M., Duffy, J., & Huggins, A. (2014). Independent
learning skills, Self-determination theory and
psychological well-being: Strategies for supporting
the first year university experience. Retrieved from
http://eprints.qut.edu.au/73556/.
Gunasekara, C. S. (2008). Fostering independent learning
and critical thinking in management higher
education using an information literacy framework.
Journal of Information Literacy, 2(2), 74–85.
Hunt, J. E., Scicluna, H., & McNeil, H. P. (2011).
Development and evaluation of a mandatory
research experience in a medical education
program: The Independent Learning Project at
UNSW. Medical Science Educator, 21(1), 78–85.
Issa, T., Issa, T., &. Kommers, P. (2014). Feedback and
learning support that fosters students’ independent
learning: An Australian case study. International
Journal of Learning, 19, 29–39.
Jones, W. M., & Dexter, S. (2014). How teachers learn: the
roles of formal, informal, and independent learning.
Independent Learning 105

Educational Technology Research and

Development, 62(3), 367–384.
http://doi.org/10.1007/s11423-014-9337-6
Johnson, B. A., & Harreld, D. J. (2012). Nurturing
independent learning in the undergraduate student
in History: A faculty–student mentoring experience.
Mentoring & Tutoring: Partnership in Learning,
20(3), 361–378.
http://doi.org/10.1080/13611267.2012.701962.
Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why
minimal guidance during instruction does not work:
An analysis of the failure of constructivist,
discovery, problem-based, experiential, and inquiry-
based teaching. Educational psychologist, 41(2),
75-86.
Knobbs, C. G., & Grayson, D. J. (2012). An approach to
developing independent learning and non-technical
skills amongst final year mining engineering
students. European Journal of Engineering
Education, 37(3), 307–320.
http://doi.org/10.1080/03043797.2012.684673.
Marshall, G. (2008). Promoting independent learning by
curriculum design and assessment in a taught
postgraduate MRI programme. Radiography, 14(3),
238–245. http://doi.org/10.1016/j.radi.2006.11.001.
McLinden, M., & Edwards, C. (2011). Developing a
culture of enquiry-based, independent learning in a
research-led institution: Findings from a survey of
pedagogic practice. International Journal for
Academic Development, 16(2), 147–162.
http://doi.org/10.1080/1360144X.2011.568699.
Meyer, B., Haywood, N., Sachdev, D., & Faraday, S.
(2008). What is independent learning and what are
the benefits for students? London: Department for
Children, Schools and Families Research Report
051. Retrieved from
Independent Learning 106

http://www.curee.co.uk/files/publication/[site-
timestamp]/Whatisindependentlearningandwhataret
hebenefits.pdf
Mitchell, M., Zutshi, S., & Weaver, D. (2010). An
investigation into unit design for developing
undergraduate independent learning abilities. In
Proceedings of the 33rd Higher Education Research
and Development Society of Australasia Annual
International Conference (HERDSA 2010).
Retrieved from
http://researchbank.swinburne.edu.au/vital/access/se
rvices/Download/swin:18164/SOURCE2.
Murdoch, K., & Wilson, J. (2006). Student independent
learning. Education Quarterly. Retrieved from
http://lnnz2.vivid.net.nz/shared/professionalReading
/JWGOALART.pdf.
Rogers, C. (2012). Transition, self-regulation, independent
learning and goal theory. Psychology of Education
Review, 36(2), 26–31.
Silver-Pacuilla, H. (2008). Investigating the language and
literacy skills required for independent online
learning. National Institute for Literacy. Retrieved
from http://eric.ed.gov/?id=ED505199.
Stern, T. (2009). The role of formative assessment in
enhancing independent learning and reflective
teaching: some results of the Austrian IMST-
Project. Retrieved from
https://www.researchgate.net/publication/23992262
9_The_Role_of_Formative_Assessment_in_Enhanc
ing_Independent_Learning_and_Reflective_Teachi
ng_Some_Results_of_the_Austrian_IMST-Project.
Warring, S. (2013). A model of independent learning
applied to the online context. Quarterly Review of
Distance Education, 14(1), 25.
Evaluation
107

Evaluation
Anthony A. Piña
Sullivan University

Course evaluation must be purpose-driven.

Alignment with the purpose should be
threefold: a) based on acquisition of new
knowledge, understandings, and skills; b)
based on instructor self-evaluation; and c)
based on student self-evaluation.
Multidimensional evaluation offers a fully
articulated basis for judging the success of
the course and the students as well as
providing information that can help shape
future iterations of the course.

Background
Historically, the assessment of online learners and
the design, development and evaluation of online programs
and courses has been, by and large, a local concern of
institutions and their faculty. However, in recent years, the
issue of quality assurance in distance learning has been a
growing focus of accreditation bodies, government
regulatory agencies, and other interested parties. Once
considered an afterthought by instructors and
administrators, evaluation is increasingly becoming a
central activity for educational institutions and for those
who design distance learning programs and courses.
Evaluation
108

For the purpose of establishing instructional design

standards for distance learning, assessment is distinguished
from evaluation, in that the former focuses upon
determining whether learners have achieved a desired
learning outcome, while the latter implies making a value
judgment regarding the quality of a distance learning
course. Evaluation, assessment and research all involve
gathering data, interpreting data, and making decisions,
however, unlike research, the purpose of evaluation is “not
to prove but to improve” (Madaus, Scriven & Stufflebeam,
2012, p. 118).

Purpose-Driven
Just as with the assessment of distance learners,
evaluation of online courses can be done for formative or
summative purposes (see Ross and Morrison, 2017 in this
volume). Both formative and summative evaluation
activities are undertaken to determine and address the
quality of distance learning courses and both can be used as
part of the continuous improvement process of systematic
instructional design, albeit they are conducted at different
stages of the process (Dick, Carey & Carey, 2015).

Formative Evaluation

Formative evaluation activities for distance learning

courses are conducted while the development of the course
is still in progress. Formative evaluation can occur during
different stages of the course’s development and can
involve instructional designers, subject matter experts,
faculty who may teach the courses, and students who are
members of the course’s target population.
Evaluation
109

Many useful standards and rubrics have been

devised to facilitate the evaluation of distance learning
courses. They can be utilized to: 1) describe the
characteristics or features of high-quality courses; 2)
identify specific strengths and weaknesses of a given
course; and 3) provide guidance for how to improve
courses before they are deployed to students. Popular
standards and rubrics include Quality Matters (Maryland
Online, 2017), California State University’s Quality Online
Learning and Teaching Instrument (California State
University, 2017), iNACOL’s National Standards
(iNACOL, 2012), Blackboard’s Exemplary Course Project
(Blackboard, 2017), Illinois Online Network’s Quality
Online Course Initiative (Illinois Online Network, 2012)
and the Quality Scorecard (Shelton, 2010). Links to each of
these resources are found in the corresponding citations
within the References section.

Summative Evaluation

“Formative course evaluation determines design

flaws that may hinder a learner’s acquisition of the desired
problem solving skill” (Merrill 2013, p. 376). Summative
evaluation activities are undertaken to judge the
effectiveness of the course to aid learners in the acquisition
of knowledge and skills once the design and development
process is complete (Dick, Carey and Carey, 2015).
Simonson (2007) notes that stakeholders want assurance
that “the time and effort required to move to distance
delivery of instruction produces a valuable educational
experience’ (vii) and advocates for the use of Kirkpatrick’s
Four-Level Model of evaluation for distance learning
courses (Kirkpatrick & Kirkpatrick, 2016):
Evaluation
110

• Level 1 (Reaction): The degree to which

participants find the training favorable,
engaging and relevant to their jobs.
• Level 2 (Learning): The degree to which
participants acquire the intended knowledge,
skills, attitude, confidence
and commitment based on their participation in
the training.
• Level 3 (Behavior): The degree to which
participants apply what they learned during
training when they are back on the job.
• Level 4 (Results): The degree to which targeted
outcomes occur as a result of the training and
the support and accountability package.

Chang and Chen (2014) reported the results of a

mixed-method study conducted with Kirkpatrick’s Four-
Level model. Results from 194 questionnaires and ten
semi-structured interviews indicated that Kirkpatrick’s
model was an effective summative evaluation process for
gauging learning effectiveness, return on expectations and
return on investment. While the Kirkpatrick’s Four-Level
Model has been a stalwart in evaluation for nearly six
decades, some are concerned about its dominance.
Abernathy (1999) has warned that when those who conduct
evaluations “regard the four-level approach as a universal
framework for all evaluations, they tend not to examine
whether the approach itself is shaping their questions and
their results. The simplicity and common sense of
Kirkpatrick's model imply that conducting an evaluation is
a standardized, prepackaged process” (p. 20). As a result,
alternative evaluation models may not be considered.
Evaluation
111

Multidimensional Evaluation
In a study investigating how faculty teaching online
courses were being evaluated at different institutions, Píña
and Bohn (2016) found that most institutions were using a
single measure of evaluation data: end-of-course student
evaluations. For those institutions that included the
additional method of performing an “observation” of the
online course, the majority used a rubric or other
observation instrument based on the Quality Matters rubric-
-which is used primarily to evaluate the course design, not
to evaluate the activities of an instructor who may not have
designed the course. Evaluations at the former institutions
suffered from a limited and incomplete data set, while the
latter institutions suffered from using instruments that
yielded the wrong data.

To be most effective, the evaluation of distance

education courses should not be limited to a single measure
and must be measuring the right things. Fortunately,
learning management systems and student information
systems are making “big data” information, including
student demographics, past academic performance,
retention/attrition rates, records of email and phone
contacts with the institution, help desk records, and student
activity within the LMS, more readily available.
Unfortunately, the ability to access and utilize this data—
without experiencing cognitive overload—is still under
development.

Acquisition of New Knowledge and Skills

Faculty, instructional designers and administrators

tend to be most interested in whether students acquire new
knowledge and skills provided by an institution’s distance
Evaluation
112

learning courses (i.e. Kirkpatrick’s Level 2). Increasingly,

legislators and the general public are insisting for evidence
of student knowledge and skill acquisition to justify
educational expenditures.
Common indicators of student learning outcomes
include mid-term and final examination scores, samples of
student writing, portfolios, reflection assignments and final
grades. Accrediting agencies often request for evidence that
course-level outcomes are linked to program-level
outcomes. The latest generation of learning management
systems (Blackboard, Canvas, Desire2Learn, etc.) all
contain the ability to perform item analyses of individual
test items and to map assignments to specified learning
objectives at the course or program level. As these tools
become more intuitive and their use more widespread,
judgements about the effectiveness of a given course—and
how to improve its effectiveness--can be based on more
robust and precise data.

Student Self-Evaluation

Accrediting agencies are frequently requiring both

direct and indirect measures of student success. Exam
scores and related data satisfy the requirements for direct
measures, while student satisfaction surveys are often
utilized as examples of indirect measures. These can
include 3rd-party nationally administered surveys, such as
the Ruffalo Noel-Levitz Priorities Survey for Online
Learners (Noel-Levitz, 2014) or locally-produced
instruments. In a review of studies on student course
evaluation surveys, Tobin, Mandernach and Taylor (2015)
reported that answers on end-of-course student surveys
tended to focus on students’ view about their instructors,
rather than about their courses.
Evaluation
113

Evaluation by students should include items that

allow students to examine their own understanding,
motivation and learning strategies that they employ in order
to become self-regulated learners (Harris & Piña, 2014).
The evaluation should have separate and distinct sections
for evaluating the instructor and evaluating the course.
Items relevant to course design may include asking
whether: 1) the course interface is inviting and
professional; 2) the course navigation is intuitive; 3)
readings, instructional activities and media employed in the
course facilitates the attainment of the learning objectives;
4) the course provided sufficient student-content, student-
instructor and student-student interaction (Piña & Baird,
2014).

Instructor Self-Evaluation

As online learning continues to grow in prevalence

within institutions and the number of online courses
increase, it is becoming increasingly common that an
instructor teaching a given distance learning course may
not have been involved in the development of that course
(Piña & Bohn, 2016). Instructors, along with their students,
engage most closely and intimately with distance education
courses and are in a position to provide highly relevant and
valuable data for course evaluation. Many of the same
questions asked of students about their courses can be
asked of instructors, as the goal is the same: to be able to
make a judgement regarding whether the course design
facilitates or hinders learning and instruction and where
improvements may be made (Piña & Baird, 2014).
Evaluation
114

Summary
Berk (2013) points out that there is an extensive
literature base for studies of student evaluation of face-to-
face courses, but that little attention has been given to the
evaluation of online or blended/hybrid courses. An even
greater deficit exist for studies into instructor evaluation of
distance education courses. Given the present and future
prominence of distance learning to students, faculty,
institutions, accrediting agencies, regulators and others, the
need for empirically-validated standards to guide that
future, it will be critical for instructional design and
distance education scholars to devote attention to
increasing the evaluation knowledge base.

References
Abernathy, D. J. (1999). Thinking outside the evaluation
box. Training and Development 53(2), 18-23.
Berk, R. (2013). Face-to-face versus online course
evaluations: A "consumer's guide" to seven
strategies. MERLOT Journal of Online Learning
and Teaching 9(1), 140-148.
Blackboard (2017). Exemplary course program rubric.
Retrieved from
http://www.blackboard.com/resources/catalyst-
awards/bb_exemplary_course_rubric_apr2017.pdf
California State University (2017). Quality online learning
and teaching (QOLT) instrument. Retrieved from
https://docs.google.com/document/d/1h-
vcRbvjwhchpLbPVPnSAf9YgOEktFHMO6w5bFP
8zbA/edit
Chang, N., & Chen, L. (2014). Evaluating the learning
effectiveness of an online information literacy class
based on the Kirkpatrick framework. Libri:
Evaluation
115

International Journal of Libraries & Information

Services 64(3), 211-223.
Dick, W., Carey, L., & Carey, J. (2015). The systematic
design of instruction, (8th Ed.). Boston, MA:
Pearson.
Harris, B. R., & Piña, A. A. (2014). Incorporating self-
regulated learning strategies in online courses. In A.
A. Piña & A. P. Mizell (Eds.). Real-life distance
education: Case studies in practice (3-20).
Charlotte, NC: Information Age Publishing.
Illinois Online Network (2012). Rubric: Quality online
course initiative. Retrieved from
http://www.ion.uillinois.edu/initiatives/qoci/index.a
sp.
iNACOL (2011). Version 2: National standards for quality
online courses. Viena, VA: International
Association for K-12 Online Learning.
Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016).
Kirkpatrick’s four levels of training evaluation.
Alexandria, VA: ATD Press.
Madaus, G. F., Scriven, M., & Stufflebeam, D. D. (2012).
Evaluation models: Viewpoints on educational and
human services evaluation. New York, NY:
Springer Science and Business Media.
Maryland Online (2017). Non-annotated standards from
the QM higher education rubric (5th Ed.). Retrieved
from
https://www.qualitymatters.org/sites/default/files/P
DFs/StandardsfromtheQMHigherEducationRubric.p
df
Merrill, M. D. (2013). First principles of instruction:
Identifying and designing effective, efficient and
engaging instruction. San Francisco, CA: Pfeiffer.
Noel-Levitz (2014). 2014-15 national online learners
priorities report. Coralville, IA: Noel-Levitz.
Retrieved from www.noellevitz.com/Benchmark.
Evaluation
116

Piña, A. A., & Baird, C. (2014, June). Evaluating distance

education evaluation. Presented at the annual
Distance Learning Administration Conference,
Jekyll Island, GA.
Piña, A. A., & Bohn, L. (2016). Assessing online faculty.
In A. A. Piña, & J. B. Huett (Eds.) Beyond the
online course: Leadership perspectives on e-
learning (317-330). Charlotte, NC: Information Age
Publishing
Ross, S. M., & Morrison, J. R. (2017). Assessment. In A.
A. Piña (Ed.) AECT instructional design standards
for distance learning. Bloomington, IN: Association
for Educational Communications and Technology.
Shelton, K. (2011). A review of paradigms for evaluating
the quality of online education programs. Online
Journal of Distance Learning Administration 14(1).
Simonson, M. (2007) Evaluation and distance education:
Five steps. Quarterly Review of Distance Education
8(3). vii-ix.
Tobin, T. J., Mandernach, B. J., & Taylor, A. H. (2015).
Evaluating online teaching: Implementing best
practices. New Jersey: John Wiley and Sons, Inc.
Sample Rubrics
117

Design Standards: Online Learning Courses

Sample Rubrics
Following are sample rubrics to evaluate the extent to
which each of the following design standards has been met.
1. Purpose. Effective course design begins with a clearly
articulated purpose. This is the standard to which all other
standards must align. Purpose may be thought of as two-
dimensional: institution or instructor and student. The
design should include both the purpose of the course as
envisioned by the institution or instructor and the purpose
as viewed by the student. As the purpose is articulated
through goals and objectives, collaboration between
instructor and student will set a firmer foundation than can
be achieved through a one-dimensional purpose statement.

Purpose is stated.  Yes  No (If no, evaluation cannot be

made.)
COMPONENT ADVANCED ADEQUATE INADEQUATE

Purpose Statement Statement Statement is

statement is incorporates recognizes generally
multidimensional. multiple multiple applicable
viewpoints and viewpoints and but does not
clearly is generally adequately
articulates applicable to the address one
purpose as institution, the or more
specifically instructor, and viewpoints
applicable to the student. among the
the institution, institution,
the instructor, the
and the instructor,
student. and the
student.
Sample Rubrics
118

Purpose Statement is Statement is a Statement is

statement made through generalized not reflective
incorporates collaboration reflection of of
collaboration. between the instructor and collaboration.
instructor and student views.
the student.
Goals and Statement Statement Goals and
objectives are includes includes basic objectives are
articulated. comprehensive goals and missing or
elaboration objectives that only partially
through are developed.
specific goals comprehensive
and objectives and at least
that are partially
coherent and detailed.
fully
articulated.
Purpose is Statement Statement Statement
aligned with aligns fully generally aligns either does
external with external with external not fully align
requirements. requirements, requirements with external
such as state with at least requirements,
or federal partial one-to- or there is
standards, and one little or no
alignment is correspondence. evidence that
detailed and such
specific. requirements
have been
considered.

2. Assumptions. Course design must take into account

assumptions that shape the purpose and subsequent course
development. Most assumptions are based on students’
prior knowledge and established understandings and skills;
others may be derived from programmatic outcomes, such
as curricular expectations or institutional requirements.
Articulating these content assumptions provides a starting
point for new learning. Assumptions in the case of online
Sample Rubrics
119

learning also encompass students’ ability to use delivery

technology.

Assumptions  Yes  No (If no, evaluation cannot be

are stated. made.)
COMPONENT ADVANCED ADEQUATE INADEQUATE

Students’ prior Students’ prior Students’ prior Students’ prior

knowledge knowledge is knowledge is knowledge is
assessed in assessed in assumed rather
detail and such general terms than assessed.
information is and such
used as a information is
primary factor used to help
to shape course shape course
design. design.

Curricular Curricular Curricular Curricular

expectations expectations expectations expectations
are clearly are generally are unstated or
articulated and stated and used non-specific.
incorporated to shape the
into the course course design.
design.
Institutional Institutional Institutional Institutional
requirements requirements requirements requirements
are clearly are generally are unstated or
articulated and stated and used non-specific.
incorporated to shape the
into the course course design.
design.
Sample Rubrics
120

Technology Students’ Students’ Students’

skills ability to use ability to use ability to use
required required required
technology is technology is technology is
assessed and basically assumed rather
such assessed and than assessed.
information is used to help
a factor in shape course
course design. design.

3. Sequence. Learning opportunities must be sequenced in

a manner that promotes efficient knowledge acquisition
consistent with the prior-knowledge assumptions. Various
models of sequencing—linear, spiral, scaffold, etc.—
should be considered, and the course design should
incorporate those strategies best suited to the content within
the constraints of online delivery.

Sequence is  Yes  No (If no, evaluation cannot be

stated. made.)
COMPONENT ADVANCED ADEQUATE INADEQUATE

Sequence is Students’ prior Students’ prior Students’ prior

consistent with knowledge knowledge knowledge is
prior assessment is assessment is not a major
knowledge. fully used in general factor in
incorporated terms to determining
into the sequence the sequence of
learning learning learning
opportunities opportunities. opportunities.
sequence.
Sample Rubrics
121

Sequence is Various models The sequencing The sequencing

varied in of sequencing model is model is based
accordance are chosen, chosen based on factors other
with learning based on the on the than the
needs. student’s student’s student’s
learning needs. learning needs learning needs.
but is relatively
static.
Sequence The sequence Sequencing Sequencing is
compliments models are well generally determined
content. matched to compliments independent
content for content. from content.
optimal
learning.

Sequence Sequencing is Online delivery Online delivery

optimizes determined in constraints are constraints are
delivery. order to taken into not well
provide a best consideration matched to
fit within when choosing chosen
online delivery sequence. sequence.
constraints.

4. Activities. Learning is achieved through activities both

passive (reading, listening, viewing) and active
(experimenting, rehearsing, applying). Activities should be
chosen that best suit the content, students’ levels of
knowledge, experience, and ability, and online delivery
constraints, particularly accommodating synchronous,
asynchronous, and mixed course participation. Student self-
selected or self-developed learning activities should be
incorporated along with instructor-selected and instructor-
developed activities, consistent with a two-dimensional
purpose.
Sample Rubrics
122

Activities are  Yes  No (If no, evaluation cannot be

stated. made.)
COMPONENT ADVANCED ADEQUATE INADEQUATE

Activities are Activities combine a Activities Activities tend to be

varied. variety of passive combine mostly limited to
and active forms of some forms either active or
engagement. of active and passive
passive engagement, not
engagement. both.

Activities are Activities are Activities Activities tend to be

matched to chosen based on generally suit predetermined,
knowledge, the student’s the student’s rather than
experience, specific level of level of specifically related
and ability. knowledge, knowledge, to the student’s
experience, and experience, knowledge,
ability. and ability. experience, or
ability.

Activities are Activities balance The student’s Instructor-

self-selected self-selected/self- self- selected/instructor-
or self- developed options selected/self- developed activities
developed by and instructor- developed dominate, with
the student. selected/instructor- activities are little
developed options. given accommodation for
consideration the student’s self-
and included selected/self-
whenever developed
possible. activities.

Activities Activities are highly Online Online delivery

match online adaptable and delivery constraints do not
delivery provide for constraints accommodate both
constraints. synchronous, are taken into synchronous and
asynchronous, and consideration asynchronous
mixed delivery. when activities.
choosing
activities, and
synchronous
Sample Rubrics
123

and
asynchronous
activities are
included
whenever
possible.

5. Resources. A range of resources should be articulated to

foster deep learning and extend course-centered
experiences and activities. Resources should be multimodal
to accommodate students’ interests, understandings, and
capacities, consistent with course content and technological
accessibility. Resources should allow students to go beyond
the constraints of the formal course structure to engage in
self-directed, extended learning.

Resources are  Yes  No (If no, evaluation cannot be

stated. made.)
COMPONENT ADVANCED ADEQUATE INADEQUATE

Resources Resources offer Resources are Resources tend

foster deep multiple, rich varied and to be limited to
learning. avenues to provide course-
deepen avenues to centered
understanding deepen and content.
and extend extend course
learning content
beyond course learning.
content.
Sample Rubrics
124

Resources are Resources are Resources Resources are

multimodal. based on the generally suit general, rather
student’s the student’s than
specific level of specifically
understandings knowledge, related to the
and capacities experience, and student’s
of knowledge, ability. knowledge,
experience, and experience, or
ability. ability.

Resources are Resources fully Resources Resources do

consistent with take into generally not fully take
technological account recognize into account
accessibility. technological limits of technological
accessibility to technological accessibility,
ensure that the accessibility making some
student can use and ensure that resources
the resources the student can difficult or
both within the fully use the impossible for
course resources. the student to
structure and use.
independently.

Resources Resources are Resources are Resources are

encourage self- consistent with consistent with consistent with
directed course content course content course content
learning. and provide and at least but may be
avenues for the some offer difficult or
student to ways the impossible for
engage in self- student can the student to
directed, extend learning use in
extended through self- independent
learning. direction. learning.

6. Application. Consistent with providing for active

learning, students should have integral opportunities within
the course design to apply new learning. Effective course
design incorporates opportunities to practice newly
acquired understandings and skills, both independently and
Sample Rubrics
125

collaboratively, and to incorporate feedback. Online

collaborative application opportunities should be developed
using social and conferencing media, and offline collegial
groups also should be structured whenever physical
proximity of students affords this opportunity.

Application is  Yes  No (If no, evaluation cannot be

stated. made.)
COMPONENT ADVANCED ADEQUATE INADEQUATE

Application is Application Application Application

integral to the offers multiple, provides varied tends to be
course design. rich opportunities limited or
opportunities to deepen and isolated from
to deepen extend course course content.
understanding content
through learning
practice of through
newly acquired practice.
skills and
knowledge.

Application Application Application Application is

provides for provides many offers multiple limited and
collaborative opportunities opportunities includes few
and and encourages for opportunities
independent the student to independent for either
learning. work with and collaboration
others and collaborative or self-directed
independently practice of new learning.
to practice new skills and
skills and knowledge.
knowledge.
Sample Rubrics
126

Application Application Application Application

includes includes rich incorporates includes only
feedback. feedback from instructor and limited
the instructor peer feedback. feedback.
and multiple
student peers.

Application Application is Application Application

incorporates enriched incorporates includes few if
collaboration through collegial any
outside the multiple interaction, opportunities
course setting. opportunities both face to for collegial
for the student face and collaboration
to interact with through outside the
peers outside electronic class setting.
the course communication.
setting, using
face-to-face as
well as
electronic
modes of
communication.

7. Assessment. Regardless of the model of sequencing

learning opportunities, the sequence should include points
of assessment for purposes of feedback and review, with
instances of review as necessary for students to acquire full
understanding. Formative assessment, whether formal,
informal, or incidental, allows teachers and students to give
feedback to one another and to review the operationalized
design in order to revise the course design based on
students’ input with regard to knowledge acquisition and
effective use of new understandings and skills.
Sample Rubrics
127

Assessment is  Yes  No (If no, evaluation cannot be

stated. made.)
COMPONENT ADVANCED ADEQUATE INADEQUATE

Assessment is Assessment is Assessment Assessment is

formative. an integral part provides for limited or tends
of the learning logical points of to be
sequence to feedback and summative
ensure that the review over the rather than
student’s learning formative.
acquisition of sequence.
knowledge and
skills is
optimal.
Assessment is Assessment Assessment Assessment
formal, provides incorporates tends to be
informal, and multiple both formal one-
incidental. opportunities and informal dimensional,
for formal and review and either formal or
informal allows for informal rather
review as well incidental than both.
as encouraging review when
incidental the need arises.
review
whenever the
need arises.
Assessment Assessment is Assessment is Assessment is
fosters review key to used to review limited to the
of operational reviewing both not only the student’s
design. the student’s student’s learning.
learning and learning but
the operational also the
design of the operational
course, which is design of the
flexible and course.
subject to
adjustment.
Sample Rubrics
128

Assessment Assessment is Assessment Assessment is

makes use of largely driven incorporates largely
student input. by student the student’s instructor-
input in order input in the directed or
to ensure revision of instructor-
optimal course design determined.
learning as needed.
through
operational
redesign of the
course on an
ongoing basis.

8. Reflection. Effective course design must include opportunities

for reflection as an extension of the Feedback/Review/Reteach
standard. Reflection involves both instructor self-reflection and
student self-reflection related to achievement of the purposes that
have been articulated as the basis for the course. Such reflection
is intended to deepen the learning experience and may serve as
reiteration of purpose at key points during the course.

Reflection is  Yes  No (If no, evaluation cannot be

stated. made.)
COMPONENT ADVANCED ADEQUATE INADEQUATE

Reflection is an Reflection is Reflection is Reflection

integral part of integrated into included at seems to be an
the operational the course regular after-thought, if
design. design so that it intervals in the it is included at
occurs course design. all.
naturally at
significant
intervals as
well as
spontaneously
when the need
arises.
Sample Rubrics
129

Reflection Reflection Reflection Reflection may

extends provides a actively extend the
feedback and regular means extends feedback and
review. of extending feedback and review
feedback and review activities but
review activities. that does not
activities and seem to be its
contributes to central
reshaping the purpose.
operational
design.

Reflection Reflection Reflection Reflection,

includes both offers multiple incorporates when it occurs,
instructor and opportunities opportunities is limited.
student self- for instructor for instructor
reflection. and student and student
self-reflection, self-reflection.
both shared
and individual.

Reflection Reflection is Reflection is Reflection only

deepens regularly consciously serendipitously
learning. employed as a used to deepen deepens
means of significant learning.
deepening learning
learning at all experiences.
stages.

9. Independent Learning. Effective course design

incorporates opportunities for independent learning, both
instructor- and self-directed. Online course development,
particularly in the asynchronous mode, should epitomize
independent learning, which should include opportunities
Sample Rubrics
130

for feedback, review, and reflection—all of which should

resonate with the purpose.

Independent  Yes  No (If no, evaluation cannot be

learning is made.)
stated.

COMPONENT ADVANCED ADEQUATE INADEQUATE

Independent Independent Independent Independent

learning is learning is as learning learning occurs
incorporated important in opportunities or is
into the the operational are regularly encouraged
operational design as occurring in the only
design. structured operational serendipitously
learning. design. or occasionally.

Independent Independent Independent Independent

learning learning, learning learning is
includes through parallels the unstructured.
feedback, feedback, operational
review, and review, and design in terms
reflection. reflection, helps of feedback,
to direct or review, and
redirect the reflection.
course’s
operational
design.
Sample Rubrics
131

Independent Independent Independent Independent

learning is learning is learning is learning, if it
included in incorporated in encouraged in occurs, tends to
both both both happen only
synchronous synchronous synchronous during either
and and and synchronous or
asynchronous asynchronous asynchronous asynchronous
activities. activities but is activities. activities but
particularly not both.
emphasized in
asynchronous
activities.

Independent Independent Independent Independent

learning is both learning is learning learning, if it
instructor- and equally valid includes both occurs, is either
self-directed. and essential instructor- and instructor-
whether self-directed directed or self-
instructor- or learning directed but
self-directed. activities. not both.

10. Evaluation. Course evaluation must be purpose-driven.

Alignment with the purpose should be threefold: a) based
on acquisition of new knowledge, understandings, and
skills; b) based on instructor self-evaluation; and c) based
on student self-evaluation. Multidimensional evaluation
offers a fully articulated basis for judging the success of the
course and the students as well as providing information
that can help shape future iterations of the course.
Sample Rubrics
132

Evaluation is  Yes  No (If no, evaluation cannot be

stated. made.)
COMPONENT ADVANCED ADEQUATE INADEQUATE

Evaluation is Evaluation is Evaluation is Evaluation is

purpose-driven. fully aligned with aligned with the only somewhat
the stated course related to the
purpose(s) of the purpose(s). stated
course and based purpose(s) of
on multiple the course.
factors;
evaluation is
used to shape
future iterations
of the course.

Evaluation is Evaluation Evaluation is Evaluation does

based on incorporates multidimensional not fully
student multiple factors and fully takes incorporate an
acquisition of to judge the into account the accounting of
new success of the student’s the student’s
knowledge, student’s acquisition of acquisition of
understandings, acquisition of new knowledge, new
and skills new knowledge, understandings, knowledge,
understandings, and skills. understandings,
and skills. and skills.
Sample Rubrics
133

Evaluation is Evaluation is Evaluation Evaluation does

based on based on the incorporates the not include or
instructor self- instructor’s self- instructor’s self- only partially
evaluation. evaluation as a evaluation of the considers
co-equal element course and its instructor self-
in the operational evaluation.
multidimensional design.
evaluation of the
course and its
design.

Evaluation is Evaluation is Evaluation Evaluation does

based on based on the incorporates the not include or
student self- student’s self- student’s self- only partially
evaluation. evaluation as a evaluation of the considers
co-equal element course and its student self-
in the operational evaluation.
multidimensional design.
evaluation of the
course and its
design.