Comparing Student Achievement in Online and Face-to-Face Class Formats

Cindy Ann Dell
Assistant Professor
College of Education
Montana State University-Billings
Billings, MT 59105 USA
cdell@msubillings.edu

Christy Low
Director of Distance Education
St. Ambrose University
Davenport, IA 52801
LowChristyE@sau.edu

Jeanine F. Wilker
Assistant Professor
Curriculum and Instruction
Western Illinois University- Quad Cities Campus
Moline, IL 61265 USA
JF-Wilker@wiu.edu

Introduction

Online teaching and learning is now commonly offered in teacher education programs, with students enrolled in either individual classes or entire programs online. A publication from Sloan C (2008) indicates that online enrollments are growing faster than the total higher education student population. Those statistics show that students taking at least one online class increased by 12 percent from fall 2006 to fall 2007 (3.9 million students), while the overall higher education enrollments increased only by 1.2 percent during the same one-year period. They also indicate that over twenty percent of all higher education students in the U.S. took at least one online class in the fall of 2007, nudging teacher education faculty toward teaching online.

Research regarding online pedagogical methods has helped teacher education faculty to improve their online teaching skills, addressing issues of assessment, use of multimedia, and increasing student interaction to improve learning. Even though there is increasing interest in online teaching and research into effective online teaching methods, there are also lingering concerns regarding the quality of student performance and learning for those in online classes as compared to those who attend traditional face-to-face classes (Parsons-Pollard, Diehl Lacks, & Hylton Grant, 2008). Faculty concerns may also include the fear of increased work load, concerns regarding the perceived lack of administrative support, the degree of technological knowledge and skill required of them, as well as the quality of student learning (Mills, Yanes, & Casebeer, 2009).

However, a meta-analysis conducted by Bernard et al. (2004) examined the reported results of over 200 qualifying studies comparing the differences among distance education (including online) and face-to-face classes. Results were mixed at best, and the researchers report that further study is needed to determine the effectiveness of instruction and quality of student work when taking classes online (or at a distance). Bernard et al. (2004) explain that the pedagogical methods and the medium by which the instruction is offered are separate constructs and should not be considered as one element of instruction. They explain that:

…the instructional method is the ‘active ingredient,’ not the medium—the medium is simply a neutral carrier of content and of method. In essence, he [Clark, 1983 and 1994 as cited in Bernard, et al.] argues that any medium, appropriately applied, can fulfill the conditions for quality instruction (p. 381).

Bernard et al. (2004) further explain that methods of instruction need to be explained and examined in any study, since the method can influence the degree of student learning, as in any course, regardless of the medium (online or face-to-face). They conclude that “methodology and pedagogy are more important than media in predicting achievement” (p. 399).

In another recent meta-analysis Means et al. (2009) report that of 51 studies comparing online and face-to-face classes, eleven were significantly positive, favoring online or blended instruction, and only two favored traditional face-to-face instruction, which was significant over what one would expect to find by chance. They state that “the overall finding of the meta-analysis is that classes with online learning (whether taught completely online or blended) on average produce stronger student learning outcomes than do classes with solely face-to-face instruction. The mean effect size for all 51 contrasts was +0.24, p < .001” (p. 18). They further state that this finding is more positive than previous results most of which concluded that distance learning “is as effective as classroom instruction but no better” (p. 18). They also stress that active learning is crucial to positive student outcomes, and that “evidence suggests that promoting self-reflection, self-regulation and self-monitoring leads to more positive online learning outcomes. Features such as prompts for reflection, self-explanation and self-monitoring strategies have shown promise for improving online learning outcomes” (p. 44).

Overall, researchers have concluded that there are no significant differences between online and face-to-face student achievement, or that some online methods may lead to stronger learning outcomes. (Bernard et al., 2004; Fortune, Shifflett, & Sibley, 2006; Herman & Banister, 2007; Koory, 2003; Means et al., 2009; Tallent-Runnels, Lan, Cooper, Ahern, Shaw, & Liu, 2006; Warren & Holloman 2005; Weber & Lennon, 2007 ). There are a few pedagogical variables that can have an influence including (1) the use of problem-based learning strategies , (2) the opportunity for students to engage in mediated communication with the instructor, (3) course and content information provided to students prior to class starting, (4) and the use of video provided to students by the instructor, to name a few. To that end, instructors of online classes should focus their effort on quality course design rather than the characteristics of media.

Although Bernard et al. (2004) report few differences among Distance Education (DE), including synchronous and asynchronous online instruction, they point out that the research methodology and design need to be improved in order to conclude that there are no significant differences. They found that over 60% of the studies examined had problems related to research design and/or reporting of the results as explained here:

The most persistent problem was the reporting of characteristics of the comparison condition (i.e., classroom instruction). Often, authors went to extraordinary lengths to describe the DE condition, only to say that it was being compared with a “classroom condition.” If we cannot discern what a DE condition is being compared with, it is very difficult to come to any conclusion as to what is meant by an effect size characterizing differences (p. 407).

Tallent-Runnels et al. (2006) and Means et al. (2009) also support this finding in their review of the literature on the topic and join the call for more research with better design, analysis, and reporting.

That being said, it is important to continue to study the issue of student achievement online, as compared to face-to-face classes, and provide results from well-designed projects. In addition, the reporting needs to include rich descriptions of pedagogical methods, assignments, and assessments, especially when comparing the two mediums. To that end, a study was designed to compare the quality of student work from both online and face-to-face platforms.

Methods

Since online instruction and research methodology are under scrutiny, especially in terms of student achievement, researchers should continue to focus on comparing face-to-face and online class student outcomes, as well as the development of best practices in online pedagogy. Results from this study include the findings of a study examining differences of student achievement in both graduate and undergraduate classes, from both platforms (online and face-to-face), and across different types of assignments, in accordance with recommendations from Bernard, et al. (2004) and Means et al. (2009).

The instructor of two separate classes, one graduate human development and learning class, and one undergraduate educational psychology class, had the unique opportunity to compare the quality of work submitted by students in equivalent sections of online and face-to-face formats. The graduate sections were taught the fall of 2005, while the undergraduate sections were taught the fall of 2007. In each case, the class content and instruction was consistent between sections, including readings, activities, assignments, and discussions. The assignments required students to apply their knowledge of theories and constructs of educational psychology and/or human development to analyze short stories, review journal articles and/or describe observations of the learning process (the zone of proximal development and the use of learning strategies), as well as answer an essay question on self-efficacy on a final examination.

Lists of each of the students in the two graduate sections and the three undergraduate sections of the undergraduate class, were arranged randomly, and assigned random numbers for identification purposes. From there the submitted assignments were identified by that student’s number so the platform format from which the assignment came was not known by anyone but the instructor prior to the analysis.

Subjects

In the two sections of the graduate level course there were a total of 32 students. In the face-to face section there were 18 students, 12 female, and 6 male. There were 14 students in the online class only one of whom was male. In the three sections of the undergraduate course, there were a total of 59 students, 13 from the online section four were male, nine female, and 23 from each of the face-to-face sections, 13 were male, 33 were female. The summary of the subjects’ basic demographics is in Table 1.

Table 1.Subjects by Section and Gender

*Includes two sections

Graduate Class Structure and Content

Bernard et al. (2004) suggest that researchers describe the instructional methods for both online and face-to-face classes in order to make meaningful comparisons. For both sections of the graduate course, the instructor maintained consistency in terms of the required content, the order of content and readings, opportunity for reflection, discussion, and feedback on submitted assignments. The four assignments used for comparison were a (1) a short story analysis, (2) a journal article analysis, (3) an observation of learning through the Zone of Proximal Development (ZPD), and (4) a final examination question on an explanation of self-efficacy.

Short story analysis . The online class was built upon one-week units while the face-to-face class met once a week for three hours. The format of both sections required students to read the required content, including text chapters, PowerPoint presentations, and additional readings as assigned. Each week a short story was assigned, for which students were required to reflect upon what they had learned to date (not just that week), and analyze the short story and the characters according to the psychological constructs from the class. The short story analysis used for comparison in this study was based upon Desiree’s Baby, by Kate Chopin (Chopin, 1893), and was used the second week of class.

For the face-to-face section, students brought their papers to class and after a short lecture and PowerPoint presentation on the week’s new content, groups convened to discuss their insights and analysis of the short story. The instructor circulated around the room, checking in on each group, but offered only clarification of content and limited feedback if students demonstrated misunderstandings. Occasionally the instructor would ask a question which would require additional critical thinking by the students. Overall, however, the groups operated independent of the instructor. After 30 – 45 minutes of discussion, each student wrote a second analysis in class, including any insights or ideas they had learned from their group members during discussions. Both papers were submitted to the instructor, who graded and provided detailed and corrective feedback, returning them the following week.

The online section had a similar format, whereby students read the required readings and short story, and previewed the same PowerPoint presentation as the face-to-face section. A specific area within the class called Office Hours was reserved for questions about content and course requirements, since the instructor was obviously unavailable on a live basis. A paper following the same guidelines as the face-to-face section was submitted online to the instructor, with a deadline of Sunday afternoon. On Monday morning online students then began discussing their insights and analysis in pre-determined groups. Asynchronous discussions took place from Monday morning until Wednesday evening. Each online group had an assigned facilitator, which rotated through the group members over the duration of the semester. Facilitators were told to begin the discussion, keep the group on topic, and ask thought provoking questions to encourage critical thinking. Students were required to be in discussions all three days in order to earn full credit. During this time the instructor was partially involved in the discussions, checking in on each online group, and like the face-to-face discussions, offered only clarification of content and limited feedback if students demonstrated misunderstandings. Occasionally the instructor would ask a question which would require additional critical thinking by the students. Overall, however, the groups operated independent of the instructor. Papers were also graded with extensive feedback, and returned to individual students during the period of the discussions. After discussions were complete, post-discussion analyses were also required of the students, and were due the day after the discussions ended. The major differences between the two sections were that face-to-face students waited a week prior to getting feedback from the instructor, and wrote their post-discussion paper during class, prior to feedback from the instructor.

Assignments and exam question . The other three student products used for comparison included: (1) an observation and analysis of a person learning a new skill or concept (with attention directed toward the stages of the Zone of Proximal Development); (2) a journal article related to the cognitive and social benefits of play for young children (Pellegrini & Bohn, 2005); and (3) an essay question from the final exam requiring a detailed explanation of the nature and development of self-efficacy. The requirements surrounding these three assignments did not include student discussions or other forms of interaction. Students had to apply their acquired knowledge to analyze their observation of the ZPD, the content of the Pellegrini and Bohn (2005) research, as well as the explanation of the self-efficacy question on the final. The final exam for the face-to-face section was scheduled during final exam week, and all students took the test during the pre-arranged time. Students in the online section were required to take the final exam with a pre-approved proctor. The items on the final exam were all essay questions.

Undergraduate Class Structure and Content

For the undergraduate class in educational psychology (taught fall 2007), three sections were used for the study, two of which were face-to-face; the third was online. The same instructor who taught the graduate class sections also taught the undergraduate sections. As with the graduate sections, the instructor maintained consistency in terms of content, the order of content and readings, opportunity for reflection, discussion, and feedback on submitted assignments. New concepts and readings were introduced in the same order, and assigned the same week in all three sections of the class. The two assignments used for comparison were: (1) analysis of an observation of learning and the ZPD, and (2) an observation of learning strategies. In addition, one final examination question on an explanation of self-efficacy was used for the comparisons.

Observation of learning and the ZPD. Learning the stages of the ZPD and the constructs provided by Vygotsky was the first lesson in the three sections of Educational Psychology. After reading from the text on the stages of the ZPD and the ways that instructors can scaffold learning, students were assigned to observe a learning and teaching episode. In their analysis, students were to describe the stages they observed, the types of scaffolding techniques used by the teacher or more capable peer, determine if the instruction was effective, and if the learner began to show signs of self-regulation.

The observation of learning strategies. The use of learning strategies was the last major assignment of the class, for all three sections. Students were required to run an experiment in learning with three individuals. Each subject was asked memorize a list of unrelated words in the order given ( duck, cat, iron, lake, sky, plate, spinach, and boat). Students were then to note and analyze the various learning strategies the subjects utilized to learn and remember the words, and return after a period of about 24 hours to check if the subjects still remembered the list of words.

Students in each of the three sections conducted the experiment and wrote a 2 – 3 page paper to describe the activities and analyze the results, highlighting the effectiveness of the strategies used by their subjects. Students in the face-to face sections returned to class to discuss their results and conclusions in small groups; those in the online section discussed the same in small asynchronous groups. Asynchronous discussions took place for one week after the assignment was due. During this time the instructor was partially involved in the discussions, checking in on each online group, and like the face-to-face discussions, offered only clarification of content and limited feedback if students demonstrated misunderstandings. Occasionally the instructor would ask a question which would require additional critical thinking by the students. Overall, the groups operated independent of the instructor. However, the discussions, both online and face-to-face did not influence what was included in the paper, since in all sections, students submitted their papers prior to discussions.

Exam question on self-efficacy. Each student was required to answer a question from the essay format final exam requiring a detailed explanation of the nature and development of self-efficacy, and why it is important to motivation. Since this was the final exam, students were not required to discuss or reflect further on the question. The final exams for the face-to-face sections were scheduled during final exam week, and all students took the test during the pre-arranged time. Students in the online section were required to take the final exam with a pre-approved proctor.

Analysis

Assignments from all sections were reviewed by two independent raters in a blind review. For each assignment the raters were provided copies of the requirements, scoring rubrics, and all of the work submitted by the students (The instructions and rubrics are included in the Appendix). The raters did not know if an assignment was from an online or face-to-face section. In addition, the raters were trained on the rubrics. For each assignment, the raters read the students’ papers, and rated them according to the rubric. To establish inter-rater reliability, the raters came to agreement on all ratings (1 being lowest to 5 being highest).

Since the ratings data generated for each assignment were ordinal, a Chi-Square for Independent Measures was used to determine if there was a relationship between the two variables—in this case the course format (online or face-to-face) and the quality of student work, rated from 1 - 5. The test uses frequencies for individuals classified into categories, in this case a 1 – 5 rating by the raters on specific assignments. The null hypothesis states that the populations have the same distributions or proportions within the distribution. The alternative hypothesis states that the proportions or distributions are different. The Chi-Square for Independent Measures was run for each of the data sets generated by the reviewers.

Results

For the undergraduate sections, the null hypothesis could not be rejected. There were no differences found between student work from the online section and the face-to-face sections. Therefore, it can be said with some confidence that there are no differences in the quality of work between the undergraduate groups (online vs. face-to-face students), or specifically, that the quality of work is not dependent upon the course format. Results of the Chi-Square analysis for the undergraduate assignments may be seen in Table 2.

Table 2. Undergraduate Class Comparison Results

The results for the graduate course were different, with ratings from two of the four assignments showing a significant difference between the groups quality of work. The two assignments for which the null was rejected were the analysis of the short story (p= .007) and the final exam explanation of self-efficacy (p=.037). In both cases, the online learners scored higher than those in the face-to-face section did. The analysis results for the Self-Efficacy Explanation are not as robust as those from the Short Story Analysis, and since the number of assignments rated was low, and other variables not controlled, the results should be interpreted with caution. Results of the Chi-Square analyses may be seen in Table 3.

Table 3. Graduate Class Comparison Results

*p<.05—Reject Null

Conclusions

The results of this study further confirm Bernard et al. (2004), Means et al. (2009) and others, that demonstrated that the platform or medium (online vs. face-to face) is not as important as the instructional strategies employed, since there were few differences between in the quality of work from the online and face-to-face sections. Since the comparisons made were based on consistent instruction, interaction, and even the instructor, the basis for comparisons was in the format, and not any instructional differences. It may be said then, that there were few differences in the quality of student work, regardless of the platform in which the class was taken. The differences that did emerge were in the graduate class assignments, and the results of one were so close the alpha level (.05), it is difficult to make any generalizations or absolute conclusions regarding that set of assignments. It is interesting to note, however, that the short story assignment , which showed a high level of significance (p=.007) was one of the first assignments of the class, given for the second week of the semester, while the self-efficacy explanation, also showing a significant difference (p=.037) was given on the final exam at the end of the semester.

Because of the results of this study, and others like it, online instructors should focus on providing high quality instruction for online learners. Interaction among the learners and with the instructor is important in face-to-face and online formats. Active learning, application of knowledge; effective interaction; facilitation of self-regulation and self-efficacy; and high expectations are all important methods of instruction. Wu and Hiltz (2004) report that the number of online classes a student takes has no relationship to the ability to perform well in online classes, and that the instructor is a larger factor to online learner success than past experience. They note that effective instructors gave more guidance for discussions, and provided structured and focused topics for discussions.

Kanuka, Rourke, & Laflamme (2007) discuss the importance of encouraging higher level thinking skills in online assignments. They found that students showed higher levels of cognitive performance when students engaged in well structured WebQuests and online debate activities. The criteria they utilized to judge high quality interaction activities required that activities are “well structured….provide clearly defined roles and responsibilities for the students… [and] provoke the students to explicitly confront others’ opinions” (p. 269).

Vonderwell and Zachariah (2005) conducted a case study to examine motivators for participation in online learning. In their study, they observed that students who were assigned specific roles in online discussions, such as facilitating discussions, checking facts, or gathering resources, participated more frequently in discussions than those who had not been assigned a specific role. They also noted that students who were assigned to facilitate commented that they learned more when they facilitated discussions than those for which they were not assigned that role.

Norton and Hathaway (2008) explain that providing self-regulating activities are essential for effective online teaching. They maintain that the design of the learning environment should include tools to help students with time management, pacing their work load, deadlines that facilitate the completion of their assignments, and appropriate learning strategies. These design features scaffold online learners’ self-regulation and lead to their sense of self-efficacy for online learning. Norton and Hathaway (2008) also give guidance for online instructors, explaining that they should be skilled in online instruction, understand the learning process, and be able to build learning communities within the class. It is important for the instructor to facilitate higher level thinking skills, reflection, and promote problem solving through interactive, problem-based activities.