Introduction
This is a study of the use of individualized, graphical feedback to increase motivation in an online course. Student motivation in online courses deserves attention because learning depends on stimulating and sustaining students’ engagement through the design of effective interactions (Wang & Reeves, 2007). Feedback provides information about one’s competence and success in a specific task (Butler & Nisan, 1986). This information is critical for student motivation because it serves as a basis for self-evaluation and self-improvement. In addition, literature on effective online teaching emphasizes the importance of providing specific, objective, and individualized feedback in online learning environments (Mory, 2004; Tallent-Runnels et al., 2006). Online environments offer new opportunities for educators to systematically vary feedback derived from theories of motivation. Such research is important in view of the rapid growth in online learning at the K-12 (Watson et al., 2009) and higher education levels (Allen & Seaman, 2008).
Literature Review
The purpose of this study was to examine the effects of two different types of graphical feedback given to students who were enrolled in an online master’s level course. The two types of feedback being compared are self-referenced (how does my performance now compare to my own previous performance?) versus peer-referenced (how does my performance compare to my peers in the course?). The effects of the contrasting types of feedback on students’ self-reported motivation and their academic performance were studied.
Feedback in this study was presented in visually engaging graphical form in contrast to the more common feedback in verbal or numeric score forms. The development of these graphical representations draws on the work of Tufte (1986) who has emphasized the cognitive and emotional impact of the visual display of quantitative data. This study represents an attempt at using contrasting graphical displays to increase the motivational impact of feedback to individuals based on self-referenced and peer-referenced performance measures. A brief review of the motivational research literature provides the basis upon which these forms of feedback were created.
Achievement Goal Orientation Theory
The present study is based primarily on achievement goal theory. A goal orientation or achievement goal orientation can be defined as a purpose of academic engagement (Ames, 1992; Brophy, 2004; Kaplan & Maehr, 2006). Depending on the nature of the goals, students’ achievement goals can be labeled as mastery goals or performance goals. Mastery goal orientation can be defined as “an individual’s purpose of developing competence” whereas performance goal orientation can be defined as “an individual’s purpose of demonstrating competence” (Kaplan & Maehr, 2006). Students who are mastery goal oriented tend to engage in academic activities mainly because they want to improve their skills and develop knowledge, while students who are performance goal oriented tend to engage in the same type of academic activities because they are mainly concerned about demonstrating their superior skills and knowledge to their peers. In other words, some students are motivated by the desire to learn while others are more concerned about how their competence is viewed by others.
According to achievement goal theory, quality of engagement in schoolwork can vary depending on students’ goal orientations (Kaplan & Maehr, 2006). Barron and Harackiewicz (2003) examined the relations between individual goal orientations, classroom structures such as tasks, evaluation, recognition, and authority, and students’ grades and interest in the college classroom. They found that mastery goals (here referred to as self-referenced) have a significant main effect on students’ interest, while performance goals (here referred to as peer-referenced) have a significant main effect on their actual final grades. This suggests that students who are sensitive about their peers’ performance tend to receive better grades than those who are less conscious about other classmates’ performance.
Butler (1987) found that students who received individualized comments regarding their performance show higher mastery goals than those who receive ego-involving feedback like praise that are not directly tied to actual performance. Butler and Nisan (1986) also investigated the effects of three different feedback conditions on students’ intrinsic motivation and performance. The authors wanted to see if providing task-related comments that conveyed both positive and negative information were still more effective than providing feedback in the form of grades or no feedback at all. The results showed that the task-related comments group scored high on the given tasks, whereas the no feedback group scored low on them. Participants who received task-related comments expressed greater interest in the tasks than did those in the other two groups.
Ilies and Judge (2005) examined how different types of feedback influence subsequent goals predicted within individuals. One group was given self-referenced feedback referring to their own performance. The other group was given peer-referenced feedback relative to others’ performance. The results showed that when feedback was negative, both self-referenced and peer-referenced feedback did predict subsequent goals. However, when positive feedback was given, only the self-referenced feedback did. This study suggests that manipulating types of feedback can have an impact on students’ orientations to learning.
Online environments provide efficient tools such as email and grade books for providing feedback to help maintain students’ motivation (Johnson, 2007; Lebaron & Miller, 2005). Grant and Thornton (2007) conducted surveys on faculty who teach undergraduate courses online and proposed several best practices in course design and delivery. One of the principles they recommended was giving prompt feedback by setting up a time line to provide feedback on discussions and assignments. They noted that feedback plays an important role in helping students engage in learning in the online environment. Garrison and Cleveland-Innes (2005) also acknowledged that the quality of the feedback has a significant impact on developing strong learning communities that support the social and cognitive development of students.
Given the purpose of this study, to examine the effects of two contrasting forms of feedback presented to students in an online master’s level course, and based on the literature on achievement goal orientations (Ames, 1992; Brophy, 2004) and the impact of feedback (Butler, 1985; Butler & Nisan, 1986), the following hypotheses were generated:
- Students who receive self-referenced graphical feedback will report higher gains in mastery goal orientations than those who receive peer-referenced graphical feedback.
- Students who receive peer-referenced graphical feedback will report higher gains in performance goal orientations than those who receive self-referenced graphical feedback.
- Students who receive self-referenced graphical feedback will report higher gains in interest in the course than those who receive peer-referenced graphical feedback.
In addition, this study explored the following questions:
- Do students become more punctual in submitting their assignment after they receive feedback about punctuality?
- Do students write longer essays after they receive feedback about essay lengths?
Methods
Research Design
The study was designed as a cross-over experiment involving pre-, post-, and post-post-tests (Petersen, 1985; Ross, & Morrison, 2004). Participants were randomly assigned to Group PS (Peer-referenced first, self-referenced second) and Group SP (Self-referenced first, peer-referenced second). Both groups received the same amount of feedback throughout the course but the type of feedback varied.
Participants
The sample consists of graduate students enrolled in an online master's program during the summer semester of 2008 at a large Mid-western university. Twenty-nine students who were enrolled in the Capstone Seminars course voluntarily participated in this study.
Study Context
The study was conducted on a master’s level online portfolio course which was delivered on the ANGEL course management system. The course aimed to help in-service teachers build their own websites where they could showcase their work they completed in the program. During the first half of the course, students were asked to explore various options of Web design tools, choose one Web publisher, and create their own websites. Given that mastering these basic Web publishing skills early in the course was critical in creating a high quality portfolio, the course instructor emphasized that meeting the deadline for each assignment was important.
During the second half of the course, the focus of the class shifted from developing basic Web publishing skills to writing thoughtful essays. The other major goal of this course was to help students reflect on their learning experiences in the program. As part of their portfolio, they were asked to write four essays: one required that they reflect on goal statements, another on their future as a learner, a third addressed online tools and an individual student’s learning preferences and the final assignment was a synthesis of their experiences in the master’s program. Each of the four essays posed different questions that were designed to help students engage in more meaningful reflections on what they have learned during their two years in the master’s program. To ensure that each essay was written in a thoughtful manner, the course instructor provided a suggested length for each of the essay assignments.
Procedure
Pre-Test : During the first week of the semester, participants completed an online survey that included two measures: achievement goal orientations and interest (see Appendix). The measure of achievement goal orientations had five items for mastery goals and five items for performance goals. The measure of interest also had 10 items. All the items in the pretest survey were adapted from measures found reliable and valid in prior research (Barron & Harackiewicz, 2003; Harackiewicz et al., 2000).
Treatment 1 : During the first half of the course, group PS was provided with graphical feedback on how punctually students submitted their work for each week. The data on the submission time was gathered through the Reports Console in ANGEL and entered into an Excel spreadsheet. Using the punctuality data, a graph designed to prompt students to compare their performance with their peers was created. The yellow box indicated the student’s own performance (e.g., time of submission). The green dots indicated the assignments that were submitted before the deadline by other students, and the red dots indicated the late submissions (see Figure 1).
Figure 1. An example of peer-referenced feedback based on punctuality data
Group SP, on the other hand, was given graphical feedback that was designed to prompt students to look back on their own performance. Students could monitor their own performance (e.g., time of submission) over time (see Figure 2).
Figure 2. An example of self-referenced feedback based on punctuality data
Feedback about punctuality was given twice at the end of week 3 and week 4 via e-mail.
Post-Test: At the end of the first half of the semester, the participants completed the survey on achievement goal orientations and interest which was identical to the first survey. In addition, data on punctuality of assignments, which was an hourly difference between the submission time and the deadline, was collected.
Treatment 2: During the second half of the course, the group PS was provided with self-referenced feedback on the length of their assignments (see Figure 3). The data on the essay length was gathered through the Reports Console in ANGEL and entered into an Excel spreadsheet. Using the essay length data, a graph designed to prompt students to look back on their own performance was created.
Figure 3. An example of self-referenced feedback based on essay lengths
The group SP, on the other hand, was given peer-referenced feedback about the essay lengths that was designed to prompt students to compare their performance with their peers (see Figure 4).
The feedback about the essay lengths was given twice at the end of week 7 and week 8 via e-mail.
Post-Post-Test : At the end of the semester, the participants completed the survey on achievement goal orientations and interest which was identical to the first survey. In addition, data on the length of assignments, which was a difference score between the recommended essay length and the actual essay length that was submitted by a student, was collected.
Measures
Measures of Motivation: A self-report questionnaire, which was on a 5-point Likert scale, was used to assess the extent to which students pursue mastery goals (e.g., My goal in this class is to learn as much as I can about this topic), pursue performance goals (e.g., It is important for me to do well compared to others in this class) (Barron & Harackiewicz, 2003), and show interest in the course. (e.g., I enjoy this course very much) (Barron & Harackiewicz, 2003). For ease of interpretation, the means of each composite score was divided by 5, making it a zero to one scale.
Measures of Academic Performance: The punctuality of assignments, whether the student met the deadline of an assignment or not, was measured by the hourly difference between the deadline and the actual time the student submitted his or her assignment. In addition, the length of their essay was measured by the difference between the suggested length and the actual word count in each assignment.
Students’ achievement goal orientations (mastery and performance) and interest were measured on three separate occasions (pre-test, post-test, and post-post-test). In addition, data on punctuality and essay lengths were collected each week.
Figure 4. An example of peer-referenced feedback based on essay lengths
Results
Effects on Student Motivation
A 2 (group: PS vs. SP) x 3 (test occasion: pretest vs. posttest vs. post-posttest) repeated measures analysis of variance (RM ANOVA) was conducted to examine changes in students’ mastery goal orientations (MG), performance goal orientations (PG), and interest in the course (IN) after they received different types of feedback. Specifically, the analysis focused on the interaction effect between test occasion and group, which would tell whether changes in students’ motivation and academic performance depended on which type of feedback they received.
Table 1. Descriptive Statistics of Students’ Motivation
Group |
Dependent Variable |
Pre-test |
Treatment 1 |
Post-test |
Treatment 2 |
Post-post-test |
Mean |
SD |
Mean |
SD |
Mean |
SD |
PS |
MG |
.86 |
.07 |
Peer-Referenced Feedback |
.88 |
.10 |
Self-Referenced Feedback |
.88 |
.10 |
PG |
.66 |
.13 |
.67 |
.12 |
.70 |
.16 |
IN |
.86 |
.12 |
.88 |
.09 |
.89 |
.10 |
SP |
MG |
.89 |
.06 |
Self-Referenced Feedback |
.88 |
.07 |
Peer-Referenced Feedback |
.90 |
.07 |
PG |
.65 |
.11 |
.64 |
.10 |
.67 |
.11 |
IN |
.86 |
.09 |
.87 |
.11 |
.87 |
.12 |
RM ANOVA on mastery goals showed that there was no statistically significant interaction effect between the test occasion and group [Wilks’ Lambda =.97, F(2,19)=.32, n.s.]. The hypothesis that students who receive self-referenced feedback (Group SN between pre and post-test & Group PS between post and post-post-test) would report higher gains in mastery goals than those who receive peer-referenced feedback was not supported.
While the interaction effect between the test occasion and group on performance goals was not statistically significant [Wilks’ Lambda =.97, F(2,19)=.32, n.s.], the direction of changes in performance goals was consistent with the hypothesis that students who receive peer-referenced feedback (Group PS between pre and post-test & Group SP between post and post-post-test) would report higher gains in performance goals than those who receive self-referenced feedback. Therefore, a visual inspection of the direction of the changes in performance goals is included below (see Figure 5).
Figure 5. Performance goals score by test occasion and group.
Similar results were found in students’ interest in the course. Although the interaction between the test occasion and group was not statistically significant [Wilks’ Lambda =.99, F(2,19)=.11, n.s.], the direction of changes in interest was consistent with the hypothesis that students who receive self-referenced feedback (Group SP between pre and post-test & Group PS between post and post-post-test) would report higher gains in interest in the course than those who receive peer-referenced feedback (see Figure 6).
Effects on Student Performance
2 (group: PS vs. SP) X 3 (week: W2 vs. W3 vs. W4 for punctuality & W7 vs. W8 vs. W9 for essay length) RM ANOVAs on both the punctuality and essay length showed no main effect of feedback. In other words, the changes in students’ punctuality and essay length after receiving feedback were not statistically significant. However, it was observed that students who received self-referenced feedback during the second half of the course did write longer essays as the weeks progressed (see Figure 7).
Discussions and Conclusions
Although the results did not reach statistical significance, the directions of the changes in motivational measures were consistent with prior research suggesting that students’ performance goals are affected by peer-referenced feedback and that their interest in the course is influenced by self-referenced feedback (Senko & Harackiewicz, 2005). This is an encouraging finding because it indicates that the types of feedback instructors provide to students have an impact on their motivation, specifically goal orientations and interest.
Figure 6. Interest score by test occasion and group.
Table 2. Descriptive Statistics of Punctuality during the First Half of the Class
Group |
Type of Feedback |
Week 2 |
Week 3 |
Week 4 |
Mean |
SD |
Mean |
SD |
Mean |
SD |
PS |
Peer-Referenced |
44.50 |
45.43 |
28.14 |
39.82 |
45.43 |
98.80 |
SP |
Self-Referenced |
52.13 |
40.55 |
38.87 |
37.16 |
-9.00 |
97.27 |
Table 3. Descriptive Statistics of Essay Length during the Second Half of the Class
Group |
Type of Feedback |
Week 7 |
Week 8 |
Week 9 |
Mean |
SD |
Mean |
SD |
Mean |
SD |
PS |
Self-Referenced |
127.00 |
163.90 |
143.79 |
204.92 |
192.64 |
405.86 |
SP |
Peer-Referenced |
22.93 |
263.94 |
110.71 |
261.33 |
47.93 |
486.66 |
The present study represents an initial effort at applying motivational theory to the design of visual forms of feedback to learners in online environments. In contrast to much research on motivation in classrooms, where the feedback has typically been in the form of words spoken to a student or numbers such as test scores, the present study explored how visual displays might communicate in a more compelling way the students’ performance compared to their own prior performance or compared to their peers in the course. Online environments, including course manage systems such as ANGEL and Blackboard, offer possibilities for giving real-time, automatically generated feedback to students. What is needed now is a better understanding of how best to tailor individual feedback that informs and motivates learners.
Figure 7. Essay length score by group and week.
Several possible limitations should be acknowledged. First, although the cross-over design resolved the ethical issue of equal treatments of subjects, it affected the duration of each treatment as a result (e.g., three weeks of peer- and self-referenced feedback versus six weeks for each type of feedback). A true experimental design where the treatment does not switch would help us better understand the effects of peer-referenced and self-referenced feedback. Second, despite the fact that graphical feedback was customized to each student to an extent and contained detailed explanations on how to interpret the graph, some students might have difficulty making sense of the feedback. Future studies should include a manipulation check, which allows researchers to see whether the participants paid attention to the treatment as intended by researchers. Another important issue with this study was the small sample size. Given that the direction of the changes in performance goals and interest in the course was consistent with the hypotheses, it is possible that a replicated study with a larger sample might find significant effects of feedback on students’ motivation and academic performance.
Two major implications can be drawn from this study. From a theoretical perspective, this research can contribute to the existing body of research on students' achievement goals. The experimental nature of this study allows us to better investigate the effects of peer-referenced and self-referenced feedback on students' learning and motivation. This study also provides insights into student motivation in online courses. Most studies on student motivation were conducted in traditional classroom settings. This research was one attempt to apply traditional motivation theory to online learning environments. The results of this study also have strong ecological validity in the sense that it was conducted in an actual online course throughout a semester. Unlike other studies where feedback was given on a contrived non-academic task for a brief period of time, this research study examined the effects of actual feedback that was provided over several weeks.
From an instructional perspective, the current study has important practical implications as well. The study used a semi-automated feedback mechanism with data that was easily available in online courses like submission time of assignments and number of words in posts. The feedback strategies used in this study can be implemented by online educators and instructors in real online learning environments without too much expertise in technology. Given that providing individualized feedback in a timely manner is one of the best practices highly recommended for online instructors, developing and testing different instructional strategies is critical.
