Introduction

An online course is defined here to be a course in which the instructional and course materials are delivered via internet. Often online courses are password protected and access is provided only to enrolled students. Materials can include papers, assigned readings, recorded lectures, notes, exams and quizzes. Course management systems, such as Blackboard (n.d.) or Modular Object-Oriented Dynamic Learning Environment (MOODLE) (2004), are often used to organize such courses, allowing the instructor to control when students view material as well as what material they view.

Engineering Cultures is an elective course offered at Virginia Polytechnic Institute (VT) and the Colorado School of Mines (CSM). Two versions of Engineering Cultures have been developed and tested in the college classroom: online and in class. The course management system that is used to deliver the online version of the course is Blackboard. The primary difference between the two versions of the course is the mode of instructional delivery. This article describes the results of a pre and post assessment which compares the effectiveness of the Engineering Cultures curriculum under the two modes of delivery.

The work reported here was partially supported by the National Science Foundation (DUE-0230992). The opinions, results and interpretations are that of the authors and do not necessarily reflect that of the NSF. Early results of this work were presented at the annual meeting of the American Society for Engineering Education (Parkhurst, Moskal, Downey, Lucena, et al., 2006) and an extensive multi-semester report is available through the thesis work of the first author (Parkhurst, 2007).

Literature Survey

There are known benefits to online instruction. Many online courses are designed such that students can complete the course during a self-selected period, assuming they have internet access at that time. Conflicts with work and other components of students’ lives are minimized, allowing students to work on their own schedule and at their own pace. If a student does not have a solid hour to attend a weekly lecture, he or she may assemble small increments of time toward course completion (Bourne, Harris & Mayadas, 2005; Hannay & Newvine, 2006; Rivera & Rowland, 2008). Some studies (Bourne et al., 2005; Dutton, Dutton & Perry, 2001; Haag & Palais, 2002) have found that online students perform better than their classroom based counterparts on standardized assessments.

There are also known disadvantages to online learning. Since there is no set schedule for online courses, some students lack the motivation or time to complete the required materials, resulting in high levels of student attrition (Stanford-Bowers, 2008; Tyler-Smith, 2006). Online students may also experience difficulties accessing online programs (Collins & Berge, 1996) or may miss the personal element that is common to the classroom (Richarson & Swan, 2003; Rivera & Rowland, 2008). These negative components of online instruction can result in lower student satisfaction ratings with respect to online instruction when compared to classroom based courses ( Johnson , Aragon , Najmuddin & Palma-Rivas, 2000).

Much of what is known about online instruction is limited to how students feel about online learning or what they or their instructors report about their learning experiences (e.g., Cuthrell & Lyon, 2007; Grant & Thornton, 2007; Hannay & Newvine, 2006; Vesely, Bloom & Sherlock, 2007). Research has also addressed the factors that influence students’ persistency in such courses (Stanford-Bowers, 2008). Less is known with respect to the impact of online instruction on learning when compared to classroom based instruction. Additionally, researchers have speculated that the effectiveness of online instruction may differ based on the content of the given course (Block, Felix, Udermann, Reineke & Murray, 2008), supporting the importance of evaluating online courses in multiple contexts.

A course titled Engineering Cultures was designed and first implemented by Downey and Lucena at VP and CSM, respectively ( Downey , Lucena, Moskal, Parkhurst, Bigley, Hays et al., 2006). Engineering Cultures is designed to teach future engineers how the culture of engineering differs across various countries. As countries become more intertwined, engineers are more likely to collaborate across national boarders. To maximize the benefit of collaboration, engineers need to be able to understand and communicate with each other (Downey et al., 2006). The increase in international collaboration supports the importance of offering courses such as Engineering Cultures. Furthermore, according to the criteria set forth by the Accreditation Board for Engineering and Technology (ABET) (ABET, 2007), engineers need to be aware of global issues. A detailed description of the Engineering Cultures curriculum can be found in Downey et al. (2006).

Engineering Cultures continues to be taught at VT and CSM. The section at CSM is taught by a member of the faculty; while the sections at VT are taught by several teaching assistants under the direction of a faculty member. VT offers the course in both online and in class versions. The purpose of this article is to discuss the measured effectiveness, in terms of student learning, of the online version of the course in comparison to the classroom version of the course.

Research Questions

This investigation presents the result of an examination of whether students learned more, as measured by a multiple choice assessment, in the online version or the in class version of the Engineering Cultures course during the academic year of 2004-2005. The research questions are as follows:

Methods

The following sections provide an overview of the methodology used in this investigation. This includes a description of the subjects, course, instruments, and analysis techniques. All appropriate human subject procedures were followed.

Subjects

The subjects in this study were students enrolled in either the online version or in class version of Engineering Cultures during the 2004-2005 academic year. All online versions were taught at VT while in class versions were taught at VT and CSM. At VT, Engineering Cultures is a sophomore level course; at CSM, this course is offered at the junior level.

Course

The Engineering Cultures course curriculum was developed based on an extensive interview process which was completed by Downey and Lucena concerning the perceptions of engineers that are commonly held in six countries: U.S., France, Britain, Germany, Russia (pre, during and post Soviet Union) and Japan. Since there is no textbook for the course, published papers are used extensively. Although all sections of the course, online and in class, covered the same material, instructors had the option of defining the order in which topics were addressed. Within a given instructor’s courses, regardless of whether the course was taught online or in class, the ordering of the topics was consistent.

The online version of Engineering Cultures includes videos of a professor lecturing and electronic copies of all reading and writing assignments. There is also a discussion board where students interact with each other and hold online discussions concerning the course material. The online students were further required to attend a scheduled one-hour, weekly online discussion concerning the course. With the exception of this scheduled one hour each week, the remaining components of the course could be completed at the students’ discretion. The in class sections of Engineering Cultures differ only in that students attended a lecture and participated in classroom discussions.

Instruments

For the purpose of measuring changes in students’ knowledge from beginning to end of the course, a pre and post multiple choice content assessment was developed. This twenty-five question multiple choice assessment was administered online the first and last week of the course. Both the pre and post content assessments shared identical questions and have been designed to measure student understanding of the course content. This instrument was developed following published guidelines for multiple choice test construction (Kohoe, 1995; Frary, 1995) and the expert opinion of the curriculum developers. These efforts provide support for the validity of the interpretations that are made from the data that results from the use of this instrument. At the conclusion of the course, students also completed a self-report survey. The question that was of interest in this investigation was, “ I believe I learned more in the online version of the course than I would have learned in a classroom version” and was only administered to the students that completed the online version of the course. In asking this question, the assumption was made that the students had experienced classroom based courses throughout their education allowing them to adequately imagine how this course might be delivered in a classroom setting. This question used a Likert item format with four options, Strongly Agree, Agree, Disagree, and Strongly Disagree.

Limitations

The limitations of this study result from the use of a quasi-experimental design (Gay, 2005). Since the students selected to enroll in either the online or the classroom version of the course, random assignment was not possible. This is recognized as a common and often unavoidable limitation in research that investigates online and in class versions of courses (Olds, Moskal & Miller, 2005).

Analysis

To statistically compare students’ performances in the online and classroom versions of the course, a two-sample t-test was used to examine pretest, posttest and difference scores. Difference scores are defined to be the result of a posttest score minus a pretest score. The survey data is reported descriptively as the percentages of students that selected a given answer.

Results

This section presents the results of the two-sample t-tests and of the students’ responses to the survey question. The quantitative data was analyzed by the researchers using the statistical package MINITAB.

Pre- and Post-tests

At the beginning of each semester, the content exam was administered to all participating students. This instrument was administered again in all participating classrooms at the conclusion of the semester. Table 1 summarizes the average pretest, posttest and difference score in both the in class and online versions of the course.

The reader will notice that the students in the online version of the course performed, on average, lower than the students in the classroom version of the course on the pre-test. On the post-test, the online students on average performed better than the in class students. This suggests greater gains in the online version of the course than the in class version of the course, and this is further supported by the significant difference found in difference scores with a p-value of 0.001.

One factor that may impact these results is that different instructors taught the two versions of the course. In order to account for the impact of this variable, the next analysis compares online to in class performances when both version of the course were taught by the same instructor.

Instructors

Two instructors who were graduate student teaching assistants under the direction of the same faculty member taught both an online and in class version of the course at VT. They will be referred to here as Instructor 1 and Instructor 2. As was previously mentioned, within a given instructors’ courses regardless of whether these courses were taught online or in class, the ordering of the modules was consistent. This eliminates the module order as a variable of concern within instructor. Also, since both instructors were located at VT, the impact of institution was eliminated as an external variable in the comparison.

Table 1. Average performances on Pre-test, Post-test and Difference Scores

The results within Instructor 1’s courses are displayed in Table 2. The online students began the course with lower pretest scores but ended the course with comparable post-test scores. The difference scores were not found to be significantly different ( p=0.24). In other words, for this instructor, the gains in both the online and in class versions of the course were comparable.

A similar trend is witnessed for Instructor 2, as is displayed in Table 3. Online students began the class with lower pre-test scores and ended the course with comparable post-test scores. For this instructor, the difference scores were found to be significantly different ( p=.004), suggesting greater gains in the online version of the course. Considering these results for Instructor 1 and 2 together, the overall significant finding appears to be strongly impacted by the results of Instructor 2.

Survey Data

As previously mentioned, online students were asked to complete a self-report survey at the end of the course. The question of interest in this investigation was, “ I believe I learned more in the online version of the course than I would have learned in a classroom version.” Only online students were asked to respond to this question and a summary of the results are provided in Table 4.

As this table indicates, responses tended to be split between “Disagree” and “Agree”. It is surprising that a large number of students disagreed with the statement when the results of the content assessment indicate that online students had greater gains in learning than did in class students.

Table 2. Instructor 1, Average Pre-test, Post-test and Difference Score

Table 3. Instructor 2, Average Pre-test, Post-test and Difference Score

Table 4. Results of “ I believe I learned more in the online version of the course than I would have learned in a classroom version”

Conclusions

A surprising finding of this investigation is that online students had greater gains in knowledge as measured by the multiple choice assessment when compared to in class students. This was true across as well as within instructors’ courses. However, the result was not statistically significant for Instructor 1. There are several potential conclusions that can be drawn from these observations. It is possible that online instruction is more effective than in class instruction for the given course content. This, however, seems unlikely given that the materials provided in both courses were identical and the primary difference was whether or not these materials could be discussed verbally or electronically.

Another possibility is that there are student and teacher factors that were not measured which influenced these results. For example, students that are more motivated and that are confident in their own abilities may be more likely to select to complete online courses. These same students may be more likely than the typical student to acquire greater learning gains regardless of the course structure. This interpretation of the data can be supported by the finding concerning online students’ responses to the survey question. Although the online students displayed greater learning gains than the classroom students on the content assessment, many of the online students did not agree with the statement, “ I believe I learned more in the online version of the course than I would have learned in a classroom version.” If these students are correct in their own evaluation of their learning, then online learning is not necessarily preferable to classroom learning; these students may have performed even better in a classroom setting. Since random assignment was not possible in this investigation, it is difficult to determine whether these students would have learned more in a classroom version of the course.

This study also indicates that differences exist between the teachers who were implementing the online courses. Rivera and Rowland (2008) have indicated that two factors that support powerful online learning are: social interactions and social engagement. It is possible that the differences that were witnessed in student learning gains between Instructor 1 and Instructor 2 were a reflection of the individual instructor’s ability to electronically support the needed level of engagement among students.

Another factor that may be impacting these results is the use of a multiple choice assessment. Multiple choice assessments are most effective when measuring factual knowledge. It is possible that online learning is a useful mode of delivery when the desired outcome is factual knowledge rather than conceptual understanding. In an extension of this study, this possibility was explored and the results were published (see Parkhurst, Moskal, Downey , Lucena et al., 2008). This follow-up investigation found that in class students displayed greater advances in conceptual knowledge while online students displayed greater advances in factual knowledge.

The current study does back the conclusion that both the online and the in class versions of Engineering Cultures support student learning as reflected through the multiple choice exam. In both versions of the course, students displayed learning gains. Since online courses have the promise of reaching more students than is possible for in class versions, a next step in this research is examining in detail the student and teacher factors that influence success in the online environment.

Online learning is still new to the educational landscape. Few studies have investigated and empirically demonstrated the benefits of online learning. Based on the findings of this investigation, online learning does appear to be an effective mode of delivery for the Engineering Cultures curriculum when the desired learning outcomes is the acquisition of factual knowledge.

References

Accreditation Board for Engineering and Technology (2007). Criteria for Accrediting Engineering Programs, Retrieved June 1, 2008 , from the Accreditation Board for Engineering and Technology site: :http://www.abet.org/Linked%20Documents-UPDATE/Criteria%20and%20PP/E001%2008-09%20EAC%20Criteria%2012-04-07.pdf

Blackboard. Retrieved June 1, 2008 , from the Blackboard site: http://www.blackboard.com/us/index.bbb

Block, A., Felix, M. Udermann, B., Reineke, D. & Murray, S. (2008). “ Achievement and Satisfaction in an Online versus a Traditional Health and Wellness Course.” Journal of Online Learning and Teaching, 4(1). Retrieved June 1, 2008 , from the MERLOT site: https://jolt.merlot.org/vol4no1/block0308.htm

Bourne, J., Harris, D. & Mayadas, F. (2005). “Online Engineering Education: Learning Anywhere, Anytime.” Journal of Engineering Education, 94(1), 131-146.

Collins, M. & Berge, Z. (1996). “Facilitating Interaction in Computer Mediated Online Courses.” Retrieved June 1, 2008 , from the Emoderators site, FSU/AECT Distance Education Conference, Tallahassee , FL : http://www.emoderators.com/moderators/flcc.html.

Cuthrell, K. & Lyon (2007). “Instructional Strategies: What Do Online Students Prefer?” Journal of Online Learning and Teaching, 3(4). Retrieved June 2, 2008 , from the MERLOT site: https://jolt.merlot.org/vol3no4/cuthrell.htm

Downey , G.L., Lucena, J., Moskal, B., Parkhurst, R., Bigley, T., Hays, C., Jesiek, B., Kelly, L., Miller, J., & Ruff, S. (2006). “The globally competent engineer: Working effectively with people who define problems differently.” Journal of Engineering Education, 95 (2), 107-122.

Dutton, J., Dutton, M. & Perry, J. (2001). “Do Online Students Perform as Well as Lecture Students.” Journal of Engineering Education, 90 (1), 131-136.

Frary, R.B. (1995). “More Multiple Choice Item Writing Do’s and Don’ts.” Practical Assessment, Research and Evaluation,4 (11). Retrieved June 1, 2008 , from the Practical Assessment, Research and Evaluation site: http://pareonline.net/getvn.asp?v=4&n=11

Gay, L.R. (2005). Educational Research: Competencies for Analysis and Application (8 th Ed.). New York : Macmillan Publishing Company.

Grant, M.R. & Thornton, H.R. (2007). “Best Practices in Undergraduate Adult-Centered Online Learning: Mechanisms for Course Design and Delivery.” Journal of Online Learning and Teaching, 3(4). Retrieved June 1, 2008 , from the MERLOT site: https://jolt.merlot.org/vol3no4/grant.htm

Haag, S. & Palais, J. (2002). “Engineering Online: Assessing Innovative Education.” Journal of Engineering Education, 91 (3), 285-289.

Hannay, M. & Newvine, T. (2006). “Perceptions of Distance Learning: A Comparison of Online and Traditional Learning.” Journal of Online Learning and Teaching, 2(1). Retrieved June 1, 2008 , from the MERLOT site: https://jolt.merlot.org/Vol2_No1.htm

Johnson, S. Aragon , S., Najmuddin, S. & Palma-Rivas, N. (2000). “Comparative Analysis of Learner Satisfaction and Learning Outcomes in Online and Face-to-Face Environment.” Journal of Interactive Learning Research, 1 (1), 29-49.

Kehoe, J. (1995). “Writing Multiple-Choice Test Items.” Practical Assessment, Research and Evaluation, 4 (9). Retrieved June 1, 2008 , from the Practical Assessment, Research and Evaluation site: http://pareonline.net/getvn.asp?v=4&n=9

Modular Object-Oriented Dynamic Learning Environment (MOODLE). Retrieved June 1, 2008 , from the Moodle site : http://moodle.com/

Olds, B., Moskal, B. & Miller, R. (2005). "Assessment in engineering education: Evolution, approaches and future collaborations.” Journal of Engineering Education, 94 (1), 13-25.

Parkhurst, R., Moskal, B., Downey , G., Lucena, J., Bigley, T. & Elber, S. (2008). “Engineering cultures: Comparing student learning in on-line and classroom based implementations.” International Journal of Engineering Education, 24 (5), 955-964 .

Parkhurst, R. (2007). “Engineering cultures: A comparative analysis of online and in-class implementations.” Master’s Thesis, Colorado School of Mines.

Parkhurst, R., Moskal, B., Downey , G., Lucena, J., Bigley, T. & Ruff, S. (2006). “A comparative analysis of on-line and in-class versions of engineering cultures.” Paper in the proceedings of the annual meeting of the American Society for Engineering Education, Chicago , Illinois .

Richardson, J. & Swan, K. (2003). “Examining Social Presence in Online Courses in Relation to Students’ Perceived Learning and Satisfaction.” Journal of Asynchronous Learning Networks, 7 (1), 68-88.

Rivera, B. & Rowland, G. (2008). “Powerful E-Learning: A Preliminary Study of Learner Experiences.” Journal of Online Learning and Teaching, 4(1). Retrieved June 1, 2008 , from the MERLOT site: https://jolt.merlot.org/vol4no1/rowland0308.htm

Stanford-Bowers, D. “Persistence in Online Classes: A Study of Perceptions among Community College Stakeholders.” Journal of Online Learning and Teaching, 4(1). Retrieved June 1, 2008 , from the MERLOT site: https://jolt.merlot.org/vol4no1/rowland0308.htm

Tyler-Smith, K. “Early Attrition amoung First Time eLearners: A Review of Factors that Contribute to Drop-out, Withdrawal and Non-completion Rates of Adult Learners undertaking eLearning Programmes”. Journal of Online Learning and Teaching, 2(2). Retrieved June 2, 2008 , from the MERLOT site: https://jolt.merlot.org/Vol2_No2.htm

Vesely, P., Bloom, L., & Sherlock, J. (2007). “Key Elements of Building Community: Comparing Faculty and Student Perceptions.” Journal of Online Learning and Teaching, 4(1). Retrieved June 2, 2008 , from the MERLOT site: https://jolt.merlot.org/vol3no3/vesely.htm