The initial design of an online class provides a
prompt for instructors to examine deliberate
learning assessment strategies. For example, an
instructor may contemplate questions such as “How
will I know when students are confused about a
topic?” or “Is there a way that I can monitor the
readiness of students to advance to more complex
concepts?” “How will I document the achievement
of student learning outcomes for regional or
national accreditation agencies?”
An effective online instructor orchestrates a
number of interactive learning and assessment
activities to guide and document the learning
process. The purpose of this paper is to discuss
the essential elements for the design and use of
formative and summative online assessments
including discussion postings, assignments, SCORM
modules, and proctored and non-proctored tests or
quizzes. Issues associated with online test
security and “cheating” will also be discussed.
The paper is designed to provide an overview about
design and use of assessment strategies for
instructors who are novices to online learning.
Matching Assessment Techniques to Learning
Objectives
Learning activities and assessment are connected
very closely in well-designed online courses. The
first step in making connections is to identify
the desired course objectives. The syllabus
should clearly state information about course
learning objectives, learning methods, and how
learning assessments will be used. Typically,
instructors use Bloom's (1956) taxonomy to write
objectives for learning gains in three domains:
cognitive (what the learner should know),
psychomotor (what skills the learner should be
able to do), and affective (how the learner feels
or modifies his/her attitudes). Instructors
further specify a desired level of learning for
each domain. Within the cognitive domain, in
higher education, the higher order skills of
application, analysis, synthesis, and evaluation
are preferred for course learning objectives.
New faculty members may be familiar with the more
recent work of Anderson and Krathwohl (2001;
Krathwohl, 2002). The most obvious difference in
the two cognitive taxonomies is the change of
nomenclature of categories. Bloom (1956) used
nouns to describe the categories; whereas,
Anderson & Krathwohl used verbs which denoted an
active cognitive processes required to learn.
Anderson and Krathwohl also repositioned the last
two categories.
After writing or analyzing course level learning
objectives, the next step is to identify
individual lessons (learning modules) that
constitute the course. Learning objectives should
be specified for each lesson using the same
process of identifying the learning domain(s) and
levels of learning as course objectives. The
lesson objectives should clearly map back to the
course objectives and support achievement of
course objectives (See Fig. 1). Course objectives
should clearly map to program objectives and
program objectives to the overall
college/university general education objectives
(See Fig. 2).
Figure 1. The Educational Triangle
A deeper examination of the work reveals how the
taxonomy acts on the various levels of the
knowledge dimension—factual, conceptual,
procedural and metacognitive. This discussion is
beyond the scope of this article. Table 1 shows a
matrix with the types of learning assessment
techniques and the associated Bloom’s taxonomy
level (Thede & Sewell, 2009) and the more recent
Anderson & Krathwohl (2001) taxonomy level. Wilson
(2006) provides a more detailed discussion with an
associated graphic depicting changes that Anderson
& Krathwohl made to Bloom’s Taxonomy with the use
of verbs instead of nouns, and the switch of order
for evaluation (evaluate) and synthesis (create)
at
http://www.uwsp.edu/education/lwilson/curric/newtaxonomy.htm.
Learning Assessment Techniques
Learner assessment techniques are sometimes known
as classroom assessment techniques (CATs). Long
before technology was commonly used in teaching
and learning management systems (LMS), Angelo and
Cross (1993) described classroom assessment as “an
approach to help teachers find out what students
are learning in the classroom and how well they
are learning it” (p. 4). The most effective
assessments serve as motivational tools, promote
learning, and can be graded using explicit
criteria and standards (rubrics) (Walvoord &
Anderson, 1998). Angelo and Cross discussed CATS
specifically as a formative assessment tool.
Today, classroom-learning assessments are regarded
as both formative (developmental) and summative
(final); both types are appropriate to online
learning.
Figure 2.
Relationship of Lesson Objectives to Course,
Program, and Overall College/University Objectives
Exemplary Assessments
Exemplary assessments, whether classified as
formative or summative, are meaningful,
motivational, engaging, and should guide the
student in the learning process (Huba & Freed,
1999; Walvoord & Anderson, 1998). Huba and Freed
(1999) identified eight characteristics of
exemplar assessments (See Table 2). Design of
assessments can be time intensive, so the
instructor should give careful consideration to
make sure that the assessment techniques are
exemplary and guide the student to meet the
intended learning outcomes.
Table 1.
Learning Assessment Techniques and Associated
Taxonomy Levels
|
Learning Assessment Techniques |
Bloom’s |
Anderson & Krathwohl
|
|
Virtual labs, computer simulation models,
case studies, multiple choice questions |
Analysis
Synthesis
Evaluation |
Creating
Evaluating
Analyzing |
|
Interactive tutorials, simulations,
instructional
games, case studies |
Application |
Applying |
|
Simulations, animations, tutorials |
Comprehension |
Understanding |
|
Flash cards, games, quizzes |
Knowledge |
Remembering |
Table
2.
Characteristics of Exemplary Learning Assessments
|
Characteristics of an Exemplary Learning
Assessments |
|
Authentic – reflects real life experiences
Challenging – stimulates the learner to apply
knowledge
Coherent – serves as a guide for the student
to achieve the learning goal
Engaging – attracts the learner’s interest
Respectful – sensitive to the individual
learner’s beliefs and values
Responsive – includes a feedback mechanism to
assist the student in the learning process
Rigorous – requires applied understanding of
learning to achieve a successful outcome
Valid – provides information that is useful to
meet the intended learning outcomes |
Formative Assessments
Formative assessments entail sampling student
learning and providing feedback to guide the
learning process. Formative assessments can be
anonymous surveys or they can be individual or
group learning activities. In all instances,
feedback rather than grading is the ultimate
goal. Formative assessments allow the instructor
the opportunity to modify the teaching plan and
learning experience in order to meet the learning
outcomes.
Selection of Learning Assessments.
Because formative assessments give instructors
information about progress toward learning
objectives, assessments should be tightly
connected to the objectives. For example, an
appropriate formative assessment of students’
knowledge of acid-base balance might be the
one-minute paper or the muddiest point because
these techniques are used to assess knowledge.
Classroom opinion polls and self-confidence
surveys are more useful to assess affective
learning: values, attitudes, and self-awareness.
Case studies are often used to assess critical
thinking by asking students to solve ill-defined
problems reflecting the real world and without a
clearly identified correct answer (Huba & Freed,
1999). Finally, the instructor may want to assess
the learner’s response to instruction. Email,
feedback forms, and reflection discussions are
techniques to obtain feedback useful for refining
the instruction process.
Most popular LMSs provide numerous features for
the design and re-use of formative assessments.
For example, e-mail and the discussion boards can
be used for the one-minute paper, the muddiest
point, one-sentence summary, and reflection
postings. Quizzing and SCORM (shareable content
object reference model) modules, (discussed later)
are useful for self-tests and game designed
learning assessments.
Individual versus Group Learning Assessments.
The decision to make learning assessments
individual versus group depends upon the learning
objectives and the class size. In all instances,
the instructor must consider the class size when
designing assessment techniques. When class size
is large, group activities allow learning
collaboration where students discuss and have the
opportunity to appreciate other’s perspectives.
Most online formative assessment techniques are
adaptable for use with groups. A feature in some
LMSs allows the instructor to assign a learning
activity to a group and to provide a group grade,
which is recorded in each student’s gradebook.
Reflection and self-assessments.
Narrative postings typically prompt learners to
use critical thinking and reflection. For
learners, the ability to recognize which concepts
they understand and which concepts they have a
tenuous grasp is an important skill (Kayler &
Weller, 2007). Reflection and self-assessment can
take many different forms in online classes. When
the one-minute paper is used, learners are asked
to summarize the most important aspect of learning
and to reflect upon questions regarding their
understanding. The muddiest point simply focuses
on learning that was unclear after a particular
lesson or group of lessons. (Angelo & Cross,
1993) Information from the one-minute paper and
the muddiest point guides the instructor to assess
whether or not students achieved the lesson
learning objectives and to provide clarification,
as required. Both the one-minute paper and the
muddiest point assessments could be designed as
either individual or group postings.
A one-sentence summary asks students to write a
sentence that answers the questions “who, when,
where, how, and why?” (Angelo & Cross, 1993, p.
183). The one-sentence summary provides
information about how students synthesize and
summarize large amounts of information into
one-sentence. The instructor could also use
reflective writing as a learning assessment. Fink
(2003) notes that reflective writing calls upon
learners to assess learning critically using
questions such as: “What am I learning? Of what
value is this, to me? How did I learn best, most
comfortably, with difficulty? What else do I need
to learn? “(117).
Self-tests.
Certain formative learning activities, such as
multiple-choice self-tests, provide feedback to
the learner using automated scoring. The
instructor could require the learner to complete a
reading assignment and then take an associated
short, timed, self-test (10-15 questions) with a
minimum competency expectation. If mastery is the
desired outcome, the instructor may choose to
provide two or more opportunities to take the
self-test. Instructors should be aware that the
score for the “self-test” is not recorded in the
gradebook of some LMSs. A graded and recorded
formative test can be helpful to both the
instructor and the learner. The instructor can
use item analysis to identify areas that are not
clear to learners in addition to noting
performance of learners. Item analysis allows the
instructor to provide remediation before moving
learners to new concepts. Many LMSs contain tools
that can send automated messages from the
instructor to students based on the score of the
self-test. The learner benefits from receiving
timely feedback on answers and knowing that the
instructor has reviewed the score and provided
additional suggestions based on their scores.
SCORM modules.
SCORM (shareable content object reference model)
is a robust feature that provides common
specification and standards for interactive
learning activities that can be imported into any
LMS platform. Interactivity means that the
instructor has the opportunity to use gaming
features to reinforce learning at the knowledge
and comprehension levels. This foundational
learning is essential to higher order learning.
Well-constructed gaming can facilitate learning
assessments and be engaging and fun! Instructors
can close the feedback loop by creating a quiz
associated with the gaming activities. Learners
can be instructed to take a quiz once they have
mastered the essential concepts presented in the
SCORM.
SCORM modules are created with third party
software whose features make interactive learning
virtually “painless” for the instructor.
Authoring SCORM modules can begin by importing
existing electronic files (Microsoft PowerPoint®
or Word® documents) into the software. Once
imported, files are converted to a Flash®
format and compressed into a zipped (.zip) file.
The instructor simply uploads the .zip file as a
SCORM module into the LMS.
Instructors have a choice of many SCORM gaming
software solutions. All include tutorials that
shorten the learning curve for users. For
example, Hot Potatoes® (http://hotpot.uvic.ca/)
is a free download for use by educators. Users
can create the following interactive learning
activities: matching, jumbled sentences, crossword
puzzles, fill-in-the-blank, multiple-choice/short
answer quizzes, or a combination of any.
Commercial third party solutions are also
available – all with 30-day free trials. With
just a click of a mouse, Studymate Author by
Respondus® provides a means of creating
numerous learning activities from a glossary or
from existing quizzes. TechSmith Camtasia Studio®
(http://www.techsmith.com/camtasia.asp)
and Adobe Captivate® (http://tryit.adobe.com/us/captivate/)
allow users to create Flash video learning
resources with or without audio. Both allow for
import of slides from Microsoft PowerPoint®
and have a means to create associated quizzes. If
the learning assessment includes a quiz that is
saved in SCORM formatting, learners’ grades show
in the LMS gradebook. Table 3 shows a sampling of
software solutions for the design and development
of learning activities. For a comprehensive list
of tools for creating SCORM content, go to the
Moodle LMS web site at
http://docs.moodle.org/en/Tools_for_creating_SCORM_content.
Table
3.
Software Solutions for Learning Activities
|
Software Solution |
Cost |
Fact Cards |
Flash Cards |
Pick-a-Letter |
Matching |
Crossword Puzzles |
Quizzes |
Glossary |
Flash Video /Audio |
|
Hot Potatoes® |
Free* |
|
|
x |
x |
x |
x |
|
|
|
StudyMate Author® |
** |
x |
x |
x |
x |
x |
x |
x |
|
|
TechSmith Camtasia Studio® |
** |
|
|
|
|
|
x |
|
x |
|
Adobe Captivate® |
** |
|
|
|
|
|
x |
|
x |
* For educators
**30-Day Trial
Summative Assessments
Summative assessments are formal assessments
conducted at the end of lessons, projects, and/or
course to evaluate the learning achievement.
Summative assessments are graded and are reflected
in the final course grade. Examples of summative
assessments include papers, quizzes, tests, and
synthesis projects.
Grading rubrics.
When instructors wish to analyze learning
achievement demonstrated in papers or projects,
grading rubrics can be used to communicate
criteria to learners and facilitate the instructor
in providing fair and timely feedback to the
learner (Angelo & Cross, 1993; Huba & Freed, 1999;
Thaler, Kazemi, & Huscher, 2009; Walvoord &
Anderson, 1998). Rubrics reveal the expectations
of the instructor to learners. Not only do
rubrics provide a list of required elements in an
assessment, but also they specify the level of
performance needed to achieve learning objectives
(Lombardi, 2008). The scoring system associated
with the rubric closes the feedback loop between
the instructor and the learner.
LMSs include built-in tools for creating learning
assessment rubrics. In some LMSs, the term
“rubrics” is used and in others the term “grading
forms” is used. Regardless of the name of the tool
provided by the LMS, the function is to reveal
instructor expectations for a particular
assignment to learners. If the grading rubric is
not available in the LMS version, the instructor
can post the assignment and associated rubric as
an attachment in the assignment drop box,
discussion forum, or course e-mail.
Quizzes and Tests
Quizzes or tests in LMSs can consist of
multiple-choice, matching, and completion items.
The instructor can customize the design and the
deployment of the test as well as feedback
options. The instructor can show one item at a
time or the entire test. The test can be timed and
password protected. Grade availability can be
determined by the instructor as either immediately
after the test is finished or after the instructor
has completed a review of items and an item
analysis. Rationales can be provided for each
answer option of an item or for the item as whole.
Instructors will likely have students enrolled in
courses who have never taken an online test.
Practice tests can be developed with a similar
design of test items, deployment options, and
feedback as graded tests. This practice can
prepare learners for taking tests online so that
anxiety can be reduced.
General test security tips.
Even when tests are not proctored, instructors can
set deployment features to increase the security
of tests. The first security measure is to hide
the test until the release date and make the test
available to students who have a correct
password. Next, the instructor can limit the time
students see test items once the test is begun.
Typically, 1 minute per multiple-choice item is
sufficient for students who have studied for a
test. To reduce the possibility of students
getting answers from other students in the course,
instructors can develop item sets. These sets
have items with similar content but are asked in a
different manner or cover a slightly different
aspect of the concept. The instructor then makes
parallel forms of the same tests. Most LMSs have
a built in tool that can select an item from a
question set to make the parallel forms. In
addition to having parallel forms, instructors can
set the LMS testing tool to randomize the
presentation of questions in a test and randomize
the answer options of an item. When the
instructor uses parallel tests, randomization, and
timed items, students have more difficulty
consulting one another about test items.
Other methods to promote academic honesty with
non-proctored tests are to create the test as
open-book/notes. The instructor can make it
“legal” for students to use the Internet,
textbooks, and notes, but state what is
off-limits, such as discussing items with other
students. Instructors can use test items that
require higher-order thinking so that answers are
not found on the page of a textbook or in
students’ notes. These strategies can support
students’ synthesis of concepts, analysis of
problems, and development of solutions. If
discussion questions are used, instructors can
provide 15-30 minutes per item. Whatever the
design of a non-proctored test, a wise instructor
will assign low percentages of the course grade to
non-proctored quizzes or tests.
Reducing Opportunities for Cheating
Cheating is clearly a frequent behavior among
students. Studies report that 50-75% of students
self-report cheating (Burrus, McGoldrick, &
Schuhmann, 2007). Although it may be intuitive to
think that students who are distant from
instructors in online courses cheat more often
than students in face-to-face classes do, research
indicates the opposite. Online cheating is no more
prevalent than classroom cheating (Burrus et al.,
2007; Grijalva, Kerkvliet, & Nowell, 2006; Krsak,
2007). Never-the-less, cheating is a problem in
both settings. Recognizing why and how cheating
occurs is the first step to changes in course
policies to reduce cheating.
According to Grijalva et al. (2006), there are two
types of cheating, unintentional and intentional.
Unintentional cheating occurs in a moment of panic
when students realize that they have no clue as to
how to answer a question. In desperation, they
resort to unplanned, creative means of finding the
answer. In the traditional classroom, it often
means looking at a classmate’s test answers or
text messaging a friend. In a non-proctored
online environment, students can use notes,
textbooks, and ask friends for assistance.
The intentional cheater, on the other hand, makes
plans in advance. The intentional cheater may
arrange for a proxy to take the exam, plan with
another person to use text messaging during the
exam, sneak notes into the exam disguised as a
soda or water bottle, or sneak hidden notes into
the exam. Intentional cheating is planned
dishonesty.
Promoting honesty.
Because summative assessments are reflected in
course grades, the instructor should take cheating
issues into consideration when planning the
assessment activities. Students cheat when the
probability of being caught is low and when the
severity of punishment is low (Burrus et al.,
2007). Hard, Conway and Moran (2006) reported
that faculty members who are familiar with
university policies regarding student academic
misconduct and who use preventive techniques can
reduce the incidence of cheating. This prevention
starts with
an environment of honesty reflected in the
syllabus and course materials. Statements about
academic honesty, definitions of cheating, and
clearly stated consequences of cheating should be
provided to students. In addition, instructors
should reveal the capability of the LMS to monitor
student activities (audit trails) in the course.
Instructors can promote honesty with written
assignments by following several principles: (a)
make clear to students what plagiarism is, (b)
require students to complete the writing
assignment over a semester, and (c) require
documentation of originality (Fain & Bates, 2005).
Although instructors believe that students should
“know” what plagiarism is from writing assignments
in high school, this assumption is faulty.
Instructors need to be explicit about what
constitutes plagiarism. Instructors can discuss
copyright issues with published papers and
materials on the Internet. Students need to be
held accountable for learning a particular style
of citing references in text and constructing a
reference page. Instructors can provide links
inside online courses to excellent sites, such as
the Purdue Online Writing Lab (http://owl.english.purdue.edu/),
to assist students. Instructors can require
students to submit sections of the paper
throughout the semester for feedback. This
strategy provides students with deadlines along
the way (reducing panic cheating) and gives
instructors an opportunity to become familiar with
a student’s writing style. Finally, the instructor
can require all rough drafts and copies of cited
references be submitted with the final paper. The
final paper can be checked for originality by
using plagiarism detection tools such as Turnitin®
(www.turnitin.com)
and
PlagiarismDetect®
(www.PlagiarismDetect.com)
for written assignments.
Promoting honesty with high stakes tests.
When a test contributes more than 20% of the
course grade, instructors should consider giving
the test in a proctored environment. Students
enrolled in courses that use a combination of
face-to-face and online delivery can easily take a
test in the university’s computer center where the
instructor or teaching assistants can monitor
students. Rules such as not allowing students to
retain papers, backpacks, hats, cell phones, or
other electronic devices during the testing period
should be enforced.
Other strategies to reduce intentional cheating
include using assigned seating that is changed for
each test administration, administering parallel
forms of tests, and administrating tests at the
same date and time for multiple sections of the
same course (Strom & Strom, 2007). In addition, a
simple agree/disagree question at the beginning of
the test serves to remind students of the penalty
for dishonesty. For example, “I understand that
academic dishonesty results in penalties as
described in the syllabus and Undergraduate
Catalog. I will not engage in any academic
dishonesty during this examination”. Challenges to
test security exist in computer labs, and
instructors should strive to reduce dishonesty by
using computer monitor privacy screens, protecting
the test with a password until the exam begins,
and changing the password once students begin the
exam.
If available, instructors can set an Internet
surf-lock to remove the ability of students to
locate notes online. Instructors can talk with
the instructional technology department at their
universities to learn how to activate these
features. If no Internet lock exists, instructors
can suggest that the university purchase third
party software to lock out students from the
Internet. Some of these vendors include Vision
Classroom Management Software® - using
surf lock, Respondus LockDown Browser Powerlink®
(http://www.respondus.com/)
and Securexam Browser Powerlink® (http://www.softwaresecure.com/browser.htm).
Each of these products lock-out all applications
and the Internet, other than the testing function
in the LMS.
Conclusion
Learning assessments provide instructors with
concrete clues about learners’ achievement of
learning objectives. Though assessment techniques
have been used by some instructors in face-to-face
classes, online courses offer technologies that
make the design and reuse of assessment techniques
easier. When instructors embed formative
assessment techniques into online lessons, they
can obtain information to evaluate how well
students are learning concepts and make
adjustments to teaching plans as needed. Students
can receive immediate feedback on activities when
automated scoring with rationales is used by the
instructor. When designing assessments,
instructors need to state clearly their
expectations for honesty and the consequences of
breaking course and university policies. Learners,
instructors, programs, and the university benefit
from the use of well-designed learning
assessments.
