The Purpose of the Study 

Online learning, whether a semester long credit class or a short non-credit professional development activity is a critical teaching and learning activity in the 21^st century.  As the number of students involved in online learning increases, the expertise of those willing to lead such learning must increase as well.  In 1970, there were a nominal number of computers connected to the internet.  With little time elapsing between the Internet explosion and the present, the interim period available for intensive study has been quite brief.  The unparalleled growth in online learning as a whole has not allowed for adequate, thorough, and methodical research to be completed covering issues like quality, capacity, and retention.  As the popularity of online learning continues to increase, so, too, does the need for continued research concerning such issues, in particular quality.  However, since the birth of online learning within community colleges throughout the state of Illinois and because of the brief amount of time that online learning has been available, only nominal research regarding any aspect of quality as it relates to online learning had been collected or analyzed.   

The study began by focusing on the identification of benchmarks considered important by students, faculty, and staff when considering quality as it relates to online learning.  The study used (with permission) the 24 quality benchmarks identified by the Institute for Higher Education Policy as a springboard.   In addition, the investigation of quality was directed by the following questions: 

1. What do students believe are important benchmarks of quality in online learning?
2. What does faculty believe are important benchmarks of quality in online learning?
3. What do staff members believe are important benchmarks of quality in online learning?

The Participants 

Of the 49 public community colleges in Illinois, 17 were selected for participation in this study.  Community colleges were selected based on their size, geographic location, enrollment, online learning enrollment, and willingness to participate.  Two distinct groups of community colleges were identified.  Group I consisted of 12 community colleges considered to be representative of all Illinois community colleges.  Group II consisted of nine community colleges, based on self-nomination, as having special programs for the development and offering of online learning.  There were four community colleges that were in both Groups I and II.   All participants in Group I completed an online survey.  All participants in Group II completed an online survey and were interviewed by telephone or participated in an on-campus focus group. 

All faculty members who were teaching online courses during the fall of 2003 and all staff members who were directly dedicated to supporting online courses at that time were asked to participate.  Finally, the student sample requested from each participating community college to complete an online survey was based on an institution’s total online enrollment in fall 2003.  Table 1 displays the percentage of students asked to participate in the study as determined by the institution’s total online enrollment in fall 2003: 

 After incomplete and duplicate entries were eliminated, 1024 respondents provided valid responses to the survey distributed as follows:  a) 42 staff members, b) 177 faculty members, and c) 805 students. 

Respective to Group II, researchers contacted three faculty members and one staff member at each college location and conducted a 45 minute phone interview.  In addition, each college in Group II set up a focus group consisting of fall 2003 online students that met with researchers for approximately 90 minutes.  

The Instruments 

The Online Surveys 

As discussed earlier, the surveys developed for this study were based on the Institute for Higher Education Policy’s 24 Quality Benchmarks (IHEP, 2000) with a separate survey designed with appropriate wording for each group of participants (faculty, staff, and students).  All surveys, regardless of respective respondents, were housed and completed via the ILCCO website and all response data were maintained, collected, and managed by ILCCO’s webmaster.   All survey information was submitted anonymously in order to protect the confidentiality of the participants. To prevent duplicate submissions, each person asked to complete the survey received a clearance code that could only be used once to gain access to the survey instrument.  

The first section of each online survey asked respondents for demographic information which assisted researchers in developing an accurate picture of faculty, staff, and students involved in online learning.  The second section of the survey asked respondents questions regarding quality and online learning.  Using a four point Likert Scale (Strongly Agree = 4, Agree = 3, Disagree = 2, and Strongly Disagree = 1), respondents were asked to rank the importance of IHEP’s quality benchmarks.  In addition, using the same four point Likert Scale (Strongly Agree = 4, Agree = 3, Disagree = 2, and Strongly Disagree = 1), respondents were provided with an opportunity to indicate whether or not the quality benchmark was occurring at their respective institution.  The researchers selected one Illinois Community College who was not among the Group I or Group II participants to pilot test the online surveys.  Once comments were received, the researchers made appropriate changes accordingly. 

The Interviews 

In order to complete the faculty and staff interviews, the researchers developed a list of key interview questions. One set was developed for faculty and one set was developed for staff, with both sets being very similar allowing for differences when discussing areas relevant to job activities.   The key interview questions were used when speaking with interviewees via telephone.  The average length of time for the interviews was approximately 1 hour.  The list of nineteen questions included a few demographic questions in order to allow the researchers to describe the setting / framework of those interviewed.  

The Focus Groups  

The researchers developed a separate set of focus group questions that were used to guide the student focus groups.  The focus group questions were similar to the interview questions asked via telephone of the faculty and staff; however, the focus group questions were tailored to student activities.  The only demographic information collected was the number of students who attended the focus group session. 

All faculty, staff, and students who either participated in telephone interviews or participated in focus groups were also asked to complete the online survey.  In addition to the online surveys and to the interviews and focus groups, an environmental scan of online learning within three state systems (Colorado, New York, and Texas) was completed to help provide a frame of reference for data collected within Illinois. 

Procedures 

Online Survey Data Collection 

To begin the data collection process, the surveys were placed on an ILCCO website.  The CAOs were contacted and asked to direct faculty, staff, and students from their respective institution to the website.  The researchers sent instructions to the CAOs at the participating Illinois Community Colleges which provided details regarding the entire process and included a deadline for survey completion.   The ILCCO webmaster developed a data collection system that funneled response data received from the respondents through the website to a spreadsheet that could be manipulated during the data analysis process.  Completion of the online survey was on a volunteer basis, and all information received from the surveys was held confidential.   

Interview and Focus Group Data Collection 

To begin the interview and focus group process, the CAOs were contacted and asked to submit names and contact information for at least three faculty members teaching online courses during the spring of 2005 and at least one staff member who was providing direct support for online learning during the same semester to the interview team.  The CAOs were also asked to assemble a group of students who were enrolled in online courses during spring 2005 and who were willing to participate in a focus group session.  The researchers sent the list of interview and focus group questions to the CAOs for their review prior to the activities.   

Data Analysis 

Creswell (2003) described data analysis as an ongoing process with continual reflection, involving data that has been obtained by the researcher from the participants that had been tailored to the specific strategy used in the study.  Merriam (2001) defined it as “the process of making sense out of the data” (p. 178).  Best and Kahn (1998) discussed changes in data analysis respective to the advent of the use of the computer and acknowledged that the computer as “one of the most versatile and ingenious developments of the technological age” has significantly altered the way in which researchers analyze data.  It has simplified complicated research designs and the data analysis process (p. 428). 

The data analysis for the survey portion of this study included the reporting of the number of valid responses, or useable responses returned from the respondents, as well as the number of missing responses or the number of invalid responses.  The data analysis also included the frequency of each type of response, the percentage of the total responses, the percentage of the valid responses, and the cumulative percentages of the valid responses.  All relevant quantitative data collected were analyzed using means, standard deviations, medians, and modes.  All data were collected and placed in an Excel workbook, using separate spreadsheets to assist when necessary.   

Using the statistical calculation formula features within the Excel workbooks, all statistical calculations were calculated for all questions that had been assigned Likert scales.  The calculations were completed on an aggregate basis as well as an individual basis when necessary. 

Finally, the qualitative data provided by the respondents throughout the interviews and the focus groups were analyzed by transcribing and placing all information into searchable Excel tables and seeking repetition of words, phrases, themes, and concepts.  An attempt was made to identify comment characteristics that had been repeated not only by an individual respondent, but by multiple respondents in multiple interviews and focus groups.  The more words, phrases, themes, or concepts had been repeated, the greater importance they were presumed to play. 

Findings 

The Survey Respondents 

Students.   According to the demographic data collected via the surveys, 45.6% of the respondents were between 18 to 25 years of age.  That percentage mirrors the percentage of total students age 25 or younger (45.7%) enrolled during fiscal year 2004 within the Illinois Community College System (Illinois Community College Board, 2004).  The majority of the respondents were full-time students (57%) with a very even distribution of first year students (35%), second year students (30.7%) and students having completed at least four college semesters (33.4%).  Seventy-five percent of the respondents were employed.  Sixty percent of the employed respondents were employed full-time.  

In addition, 39.7% of the respondents were new to online learning, having enrolled in an online course for the first time.  Most respondents (53.6%) were enrolled in only one online course during spring 2005, with 27% enrolled in two online courses and 18.2% enrolled in 3-4 online courses.   An overwhelming 84.8% of respondents stated that they were using their home computers to complete their courses.  The remainder of the respondents identified using workplace computers (7.1%), school computers (5.1%), or computers located at other locations (3%).  The majority of the survey respondents stated that their technology expertise was at an advanced level (58%), with 31% stating that they were at an intermediate level and 10.2% at a novice level. 

Faculty.    According to the demographic data collected from the faculty respondents to the survey, overall they were a very experienced group.  Over 70 % of the faculty respondents had more than 10 years teaching experience (including K-12, community college, and university experience) with 15.4% having 6 to 10 years experience and 12.3% having 2-5 years experience.  Over 54% of the respondents had taught for more than 10 years in a community college setting, and 45.2% had taught more than 10 years at their current institution.  Not only did the respondents have strong experience levels, the majority (57%), albeit narrow, was full-time faculty.   Nearly 37% of the respondents taught between 12-24 credit hours face-to-face during the combined fall 2004 and spring 2005 academic semesters, with 24.8% having taught between 1-11 credit hours.  Furthermore, the respondents had a high level of experience teaching online with 57.6% having taught courses online between two to five years and 25% between six to ten years.  More than 60% had taught online for their current institution between two to five years, and 61% had taken an online course as a student.  The overwhelming majority of the faculty respondents rated their tech; The survey respondents categorized as online support staff provided similar demographic information as the faculty respondents.  Three fourths of the staff respondents had been in their current position for five years or less, with 17.5% in their position less than one year, 12.5% between 6-10 years and 12.5% more than ten years.  Nearly 58% had been employed at their current institution in some capacity for at least six years and 65% had spent at least six years employed in a community college setting.  Nearly 60% of the staff respondents indicated that no less than 40% of their workload was devoted to online learning.  Approximately one fourth of the staff respondents had online teaching experience and slightly more than 50% had been a student in an online course.  Three fourths of the staff respondents rated their technology expertise as advanced and one fourth rated it as intermediate.  

Participants Interviewed.  As all individuals who participated in the interview / focus groups were asked and expected to complete the online survey, the demographic information corresponded to that collected through the survey.  Some additional information was gathered that provides some additional details about the participants interviewed.  One faculty participant interviewed had taught over 100 online sections, and one was teaching online for the first time.  The faculty participants taught a wide variety of course including composition, history, humanities, mathematics, speech, psychology, science, word processing, automated office technology, management, accounting, criminal justice, nursing, and computer technology.   The staff supporting online learning did so in a variety of ways including instructional design, technical training, course management support, faculty website development, internal online committee participation, and institutional representation on statewide online learning initiatives.  Nearly 100% of the staff participants interviewed reported to the academic side of the college.   

The Quality Benchmarks 

Throughout the survey, all three groups of respondents were asked to rate the importance of each quality benchmark using a four point Likert Scale (Strongly Agree = 4, Agree = 3, Disagree = 2, and Strongly Disagree = 1).  Table 2 displays the benchmarks with the highest means based on all respondents (see Table 2).    Table 3 displays the benchmarks with the lowest means based on all respondents (see Table 3).   

Table 2. Quality Benchmarks Rated Most Important Based on Highest Means – All Respondents

Table 3. Quality Benchmarks Rated Least Important Based on Lowest Means – All Respondents

The data collected were also separated according to the distinct respondent groups of faculty, staff, and student.  Upon completing appropriate statistical analysis, none of the mean scores of the individual faculty, staff, and student groups were found to be different at a statistical significant level of .05.  In addition, the faculty and staff were grouped according to several other distinctions including: 1) faculty who had taken classes online and those who had not, 2) faculty who had taught fewer than six sections online and those who had taught six or more sections online, 3) those who identified their computer skills as novice, intermediate, and expert, and 4) staff who had taught online and those who had not.  Students were divided into various groups as well including: 1) those who were 25 years or younger and those who were 26 and older, 2) those who were employed full-time and those who were either employed part-time or not employed, 3) those who had taken two or fewer online courses, those who had taken three or four online courses, and those who had taken five or more, and 4) those who identified their computer skills as novice, intermediate, and expert.  For each of these groups, multivariate tests were completed in regards to the means of their responses and the quality benchmarks.  As a result of this analysis, no statistically significant differences were found between the means of any of the groups.  The lack of statistically significant differences reinforced the consistency among the groups as far as the ranking of the quality benchmarks. 

Following their rating of the importance of each quality benchmark, the respondents were asked to rate the extent to which the quality benchmarks were occurring at their respective institution. Again, the respondents used a four point Likert Scale (Strongly Agree = 4, Agree = 3, Disagree = 2, and Strongly Disagree = 1) to complete the survey.   Table 4 shows the means of the responses received regarding the extent to which they believed that the quality benchmark occurred at the respondents’ respective institutions.   

Again, upon completing appropriate statistical analysis, none of the mean scores of the individual faculty, staff, and student groups were found to be different at a statistical significant level of .05; however, it is notable that the faculty means were discernibly higher than those of students in regards to 1) student interaction with faculty and students, 2) the timeliness of faculty feedback, 3) the agreement on expectations for assignments, and 4) students’ access to technology assistance.  In addition, the faculty means were notably higher than those of the staff on:  1) the constructive quality of faculty feedback, 2) the timeliness of faculty feedback, 3) instruction in proper methods of research, 4) online information and training given on library resources, and 5) the assessment of teaching and learning. Finally, the means of the staff responses were higher than students on: 1) students’ access to virtual library, 2) students’ access to technical assistance, 3) timely responses from student services, and 4) the system of feedback about online courses. 

Table 4. Extent to which Quality Benchmark Occurs at Institution of Respondent based on Mean

 Benchmark

The Interviews 

At the conclusion of the interview and focus group activities, faculty, staff, and students consistently identified four components necessary for quality online learning programs including: 1) a strong administrative support team that collaborates with its faculty throughout the online learning process, 2) a system for continuous improvement of online teaching and learning, 3) reliable technology and reliable technical assistance, and 4) strong faculty, staff, and student online teaching and learning preparation / orientation programs. Throughout this study, the faculty, staff, and students were in nearly universal agreement that there was no significant difference between online teaching and learning.  In addition, the majority of the faculty interviewed stated that the outcomes between the face to face and online version of the same course were virtually identical.  Through the focus groups and interviews, the research team was able to extrapolate a list of components for maintaining and/or improving the quality of online courses (see Table 5). 

Table 5. List of components for maintaining / improving quality

Conclusions 

Throughout the entire study, one conclusion that can be gleaned is that online learning is in a more mature phase.  The range of data collected during this study in particular throughout the interviews and focus group activities did not focus on the original forefront issues within online learning including those such as its academic credibility, the calculating of load relative to the online modality, the calculation of faculty compensation, and ownership issues.  Online learning has moved beyond the initial issues perhaps and hopefully because those issues have been resolved. 

Moreover, the results of this particular study demonstrate a remarkable cohesiveness among all respondents (faculty, staff, and students) in terms of their agreement on the quality benchmarks they found to be both most and least important.   Their unanimity provides credence to the idea that faculty, staff, and students all have common expectations in terms of quality and online learning even though their interaction, use, or activity within this modality is markedly different.  Because of this unanimity, it can also be concluded that there is a level of confidence concerning what constitutes or defines online quality among its daily users. 

Of the top ten quality benchmarks considered to be most important, six were directly related to technology, and of the nine quality benchmarks considered to be least important, only one was directly related to technology.   Similarly in terms of the extent to which the quality benchmarks were occurring at the respondent’s respective institution, four of the top seven strongest and two of the top seven weakest quality benchmarks occurring at institutions involved technology or technical assistance; however, the remaining three strongest and five weakest quality benchmarks did not.  Overall, the most important and those strongest occurring quality benchmarks at local institutions focused on technology.  The least important and those weakest occurring quality benchmarks did not.  In addition, six of the seven strongest occurring quality benchmarks at local institutions identified by the respondents appeared within the top ten benchmarks of greatest importance with the majority focusing on technology.  

Within the online learning modality, the technology used as the teaching and learning framework no doubt is paramount to its existence which may explain why benchmarks related to technology were considered most important and why they were occurring strongly at respondents’ respective institutions.  

Of greater interest is why quality benchmarks commonly associated with quality teaching and learning irrespective of learning modality were ranked of lesser importance and occurring to a lesser degree.  These findings seem to indicate that the technological segment of online learning overshadows the non-technological segment.  Issues such as a strong focus on learning outcomes, evaluation, assessment, educational effectiveness, and critical feedback should somehow be of greater value to teaching and learning than having technical assistance during course development.    If and when the teaching and learning move to the forefront and the technology becomes simply the offering mechanism, the measuring of the quality of online learning will be much easier to complete.        

References

Blumenstyk, G. (1998).  Leading community colleges go nationwide with new distance-learning network.  Chronicle of Higher Education, 44(44), A16-A17.

Burton, L. & Goldsmith, D. (2002).  Students’ experiences in online courses:  A study using asynchronous online focus groups. Connectifont-family:Arial">De A'Moreli, R. (1995). Exploring the Internet: Spectrum Universal.

Illinois Board of Higher Education. (2004). Data Book. Retrieved August 15, 2006, from the Illinois Board of Higher Education Website:  http://www.ibhe.state.il.us/Data%20Bank/DataBook/default.asp.

Illinois Community College Board. (2004).  Data and Characteristics.  Retrieved February 26, 2006, from the Illinois Community College Board Website: http://www.iccb.state.il.us/pdf/reports/databook2004.pdf

Illinois Virtual Campus. (2005).  Distance Education Enrollments at Illinois Colleges and Universities Summer/ 2005 Academic Year 2004-2005. Retrieved August 15, 2006, from the Illinois Virtual Campus Website:  http://www.ivc.illinois.edu/pubs/enrollment/Summer_05.html

Milliron, M.D. (2000).  The dotcommies are coming [Video Recording].  San Antonio: San Antonio College.

Milliron, M.D., & Miles, C. L. (2000).  Taking a big picture look @ technology, learning, and the community college.  Mission Viejo: League of Innovation in the Community College.

Moschovitis, C. J. P. (1999). History of the Internet : a chronology, 1843 to the present. Santa Barbara, Calif.: ABC-CLIO.

Ohio Learning Network.  (2005).  Bounded exuberance: e-Learning in Ohio.  Retrieved August 15, 2006 from, the Ohio Learning Network Website:  http://www.oln.org/about_oln/pdf/Bounded-web.pdf.

Phipps, R. & Merisotis, J.  (2000).  Quality on the line:  Benchmarks for success in internet-based distance education. Washington, D.C.:  The Institute for Higher Education Policy.

Seagren, A., & Watwood, B. (1996, February). The virtual classroom:  Great expectations.  Delivering graduate education by computer: A success story. Paper presented at the Annual International Conference of the National Community College Chair Academy.

Selingo, J. (1998).  Small, private colleges brace for competition from distance learning.  Chronicle of Higher Education, 44(34), A33-A35.

Sikora, A., & Carroll, C. D. (2002). A profile of participation in distance education: 1999-2000 (No. NCES 2003-154). Washington DC: Government Printing Office.

U.S. Department of Education. (2003). Distance education at degree-granting postsecondary education institutions 2000-2001 (No. NCES 2003-017). Washington, D.C.: National Center for Educational Statistics. 

Manuscript received 2 Nov 2006; revision received 5 Mar 2007.

 

This work is licensed under a

Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License