The Purpose of the Study 

Online learning, whether a semester long credit class or a short non-credit professional development activity is a critical teaching and learning activity in the 21^st century.  As the number of students involved in online learning increases, the expertise of those willing to lead such learning must increase as well.  In 1970, there were a nominal number of computers connected to the internet.  With little time elapsing between the Internet explosion and the present, the interim period available for intensive study has been quite brief.  The unparalleled growth in online learning as a whole has not allowed for adequate, thorough, and methodical research to be completed covering issues like quality, capacity, and retention.  As the popularity of online learning continues to increase, so, too, does the need for continued research concerning such issues, in particular quality.  However, since the birth of online learning within community colleges throughout the state of Illinois and because of the brief amount of time that online learning has been available, only nominal research regarding any aspect of quality as it relates to online learning had been collected or analyzed.   

The study began by focusing on the identification of benchmarks considered important by students, faculty, and staff when considering quality as it relates to online learning.  The study used (with permission) the 24 quality benchmarks identified by the Institute for Higher Education Policy as a springboard.   In addition, the investigation of quality was directed by the following questions: 

1. What do students believe are important benchmarks of quality in online learning?
2. What does faculty believe are important benchmarks of quality in online learning?
3. What do staff members believe are important benchmarks of quality in online learning?

The Participants 

Of the 49 public community colleges in Illinois, 17 were selected for participation in this study.  Community colleges were selected based on their size, geographic location, enrollment, online learning enrollment, and willingness to participate.  Two distinct groups of community colleges were identified.  Group I consisted of 12 community colleges considered to be representative of all Illinois community colleges.  Group II consisted of nine community colleges, based on self-nomination, as having special programs for the development and offering of online learning.  There were four community colleges that were in both Groups I and II.   All participants in Group I completed an online survey.  All participants in Group II completed an online survey and were interviewed by telephone or participated in an on-campus focus group. 

All faculty members who were teaching online courses during the fall of 2003 and all staff members who were directly dedicated to supporting online courses at that time were asked to participate.  Finally, the student sample requested from each participating community college to complete an online survey was based on an institution’s total online enrollment in fall 2003.  Table 1 displays the percentage of students asked to participate in the study as determined by the institution’s total online enrollment in fall 2003: 

 After incomplete and duplicate entries were eliminated, 1024 respondents provided valid responses to the survey distributed as follows:  a) 42 staff members, b) 177 faculty members, and c) 805 students. 

Respective to Group II, researchers contacted three faculty members and one staff member at each college location and conducted a 45 minute phone interview.  In addition, each college in Group II set up a focus group consisting of fall 2003 online students that met with researchers for approximately 90 minutes.  

The Instruments 

The Online Surveys 

As discussed earlier, the surveys developed for this study were based on the Institute for Higher Education Policy’s 24 Quality Benchmarks (IHEP, 2000) with a separate survey designed with appropriate wording for each group of participants (faculty, staff, and students).  All surveys, regardless of respective respondents, were housed and completed via the ILCCO website and all response data were maintained, collected, and managed by ILCCO’s webmaster.   All survey information was submitted anonymously in order to protect the confidentiality of the participants. To prevent duplicate submissions, each person asked to complete the survey received a clearance code that could only be used once to gain access to the survey instrument.  

The first section of each online survey asked respondents for demographic information which assisted researchers in developing an accurate picture of faculty, staff, and students involved in online learning.  The second section of the survey asked respondents questions regarding quality and online learning.  Using a four point Likert Scale (Strongly Agree = 4, Agree = 3, Disagree = 2, and Strongly Disagree = 1), respondents were asked to rank the importance of IHEP’s quality benchmarks.  In addition, using the same four point Likert Scale (Strongly Agree = 4, Agree = 3, Disagree = 2, and Strongly Disagree = 1), respondents were provided with an opportunity to indicate whether or not the quality benchmark was occurring at their respective institution.  The researchers selected one Illinois Community College who was not among the Group I or Group II participants to pilot test the online surveys.  Once comments were received, the researchers made appropriate changes accordingly. 

The Interviews 

In order to complete the faculty and staff interviews, the researchers developed a list of key interview questions. One set was developed for faculty and one set was developed for staff, with both sets being very similar allowing for differences when discussing areas relevant to job activities.   The key interview questions were used when speaking with interviewees via telephone.  The average length of time for the interviews was approximately 1 hour.  The list of nineteen questions included a few demographic questions in order to allow the researchers to describe the setting / framework of those interviewed.  

The Focus Groups  

The researchers developed a separate set of focus group questions that were used to guide the student focus groups.  The focus group questions were similar to the interview questions asked via telephone of the faculty and staff; however, the focus group questions were tailored to student activities.  The only demographic information collected was the number of students who attended the focus group session. 

All faculty, staff, and students who either participated in telephone interviews or participated in focus groups were also asked to complete the online survey.  In addition to the online surveys and to the interviews and focus groups, an environmental scan of online learning within three state systems (Colorado, New York, and Texas) was completed to help provide a frame of reference for data collected within Illinois. 

Procedures 

Online Survey Data Collection 

To begin the data collection process, the surveys were placed on an ILCCO website.  The CAOs were contacted and asked to direct faculty, staff, and students from their respective institution to the website.  The researchers sent instructions to the CAOs at the participating Illinois Community Colleges which provided details regarding the entire process and included a deadline for survey completion.   The ILCCO webmaster developed a data collection system that funneled response data received from the respondents through the website to a spreadsheet that could be manipulated during the data analysis process.  Completion of the online survey was on a volunteer basis, and all information received from the surveys was held confidential.   

Interview and Focus Group Data Collection 

To begin the interview and focus group process, the CAOs were contacted and asked to submit names and contact information for at least three faculty members teaching online courses during the spring of 2005 and at least one staff member who was providing direct support for online learning during the same semester to the interview team.  The CAOs were also asked to assemble a group of students who were enrolled in online courses during spring 2005 and who were willing to participate in a focus group session.  The researchers sent the list of interview and focus group questions to the CAOs for their review prior to the activities.   

Data Analysis 

Creswell (2003) described data analysis as an ongoing process with continual reflection, involving data that has been obtained by the researcher from the participants that had been tailored to the specific strategy used in the study.  Merriam (2001) defined it as “the process of making sense out of the data” (p. 178).  Best and Kahn (1998) discussed changes in data analysis respective to the advent of the use of the computer and acknowledged that the computer as “one of the most versatile and ingenious developments of the technological age” has significantly altered the way in which researchers analyze data.  It has simplified complicated research designs and the data analysis process (p. 428). 

The data analysis for the survey portion of this study included the reporting of the number of valid responses, or useable responses returned from the respondents, as well as the number of missing responses or the number of invalid responses.  The data analysis also included the frequency of each type of response, the percentage of the total responses, the percentage of the valid responses, and the cumulative percentages of the valid responses.  All relevant quantitative data collected were analyzed using means, standard deviations, medians, and modes.  All data were collected and placed in an Excel workbook, using separate spreadsheets to assist when necessary.   

Using the statistical calculation formula features within the Excel workbooks, all statistical calculations were calculated for all questions that had been assigned Likert scales.  The calculations were completed on an aggregate basis as well as an individual basis when necessary. 

Finally, the qualitative data provided by the respondents throughout the interviews and the focus groups were analyzed by transcribing and placing all information into searchable Excel tables and seeking repetition of words, phrases, themes, and concepts.  An attempt was made to identify comment characteristics that had been repeated not only by an individual respondent, but by multiple respondents in multiple interviews and focus groups.  The more words, phrases, themes, or concepts had been repeated, the greater importance they were presumed to play. 

Findings 

The Survey Respondents 

Students.   According to the demographic data collected via the surveys, 45.6% of the respondents were between 18 to 25 years of age.  That percentage mirrors the percentage of total students age 25 or younger (45.7%) enrolled during fiscal year 2004 within the Illinois Community College System (Illinois Community College Board, 2004).  The majority of the respondents were full-time students (57%) with a very even distribution of first year students (35%), second year students (30.7%) and students having completed at least four college semesters (33.4%).  Seventy-five percent of the respondents were employed.  Sixty percent of the employed respondents were employed full-time.  

In addition, 39.7% of the respondents were new to online learning, having enrolled in an online course for the first time.  Most respondents (53.6%) were enrolled in only one online course during spring 2005, with 27% enrolled in two online courses and 18.2% enrolled in 3-4 online courses.   An overwhelming 84.8% of respondents stated that they were using their home computers to complete their courses.  The remainder of the respondents identified using workplace computers (7.1%), school computers (5.1%), or computers located at other locations (3%).  The majority of the survey respondents stated that their technology expertise was at an advanced level (58%), with 31% stating that they were at an intermediate level and 10.2% at a novice level. 

Faculty.    According to the demographic data collected from the faculty respondents to the survey, overall they were a very experienced group.  Over 70 % of the faculty respondents had more than 10 years teaching experience (including K-12, community college, and university experience) with 15.4% having 6 to 10 years experience and 12.3% having 2-5 years experience.  Over 54% of the respondents had taught for more than 10 years in a community college setting, and 45.2% had taught more than 10 years at their current institution.  Not only did the respondents have strong experience levels, the majority (57%), albeit narrow, was full-time faculty.   Nearly 37% of the respondents taught between 12-24 credit hours face-to-face during the combined fall 2004 and spring 2005 academic semesters, with 24.8% having taught between 1-11 credit hours.  Furthermore, the respondents had a high level of experience teaching online with 57.6% having taught courses online between two to five years and 25% between six to ten years.  More than 60% had taught online for their current institution between two to five years, and 61% had taken an online course as a student.  The overwhelming majority of the faculty respondents rated their tech; The survey respondents categorized as online support staff provided similar demographic information as the faculty respondents.  Three fourths of the staff respondents had been in their current position for five years or less, with 17.5% in their position less than one year, 12.5% between 6-10 years and 12.5% more than ten years.  Nearly 58% had been employed at their current institution in some capacity for at least six years and 65% had spent at least six years employed in a community college setting.  Nearly 60% of the staff respondents indicated that no less than 40% of their workload was devoted to online learning.  Approximately one fourth of the staff respondents had online teaching experience and slightly more than 50% had been a student in an online course.  Three fourths of the staff respondents rated their technology expertise as advanced and one fourth rated it as intermediate.  

Participants Interviewed.  As all individuals who participated in the interview / focus groups were asked and expected to complete the online survey, the demographic information corresponded to that collected through the survey.  Some additional information was gathered that provides some additional details about the participants interviewed.  One faculty participant interviewed had taught over 100 online sections, and one was teaching online for the first time.  The faculty participants taught a wide variety of course including composition, history, humanities, mathematics, speech, psychology, science, word processing, automated office technology, management, accounting, criminal justice, nursing, and computer technology.   The staff supporting online learning did so in a variety of ways including instructional design, technical training, course management support, faculty website development, internal online committee participation, and institutional representation on statewide online learning initiatives.  Nearly 100% of the staff participants interviewed reported to the academic side of the college.   

The Quality Benchmarks 

Throughout the survey, all three groups of respondents were asked to rate the importance of each quality benchmark using a four point Likert Scale (Strongly Agree = 4, Agree = 3, Disagree = 2, and Strongly Disagree = 1).  Table 2 displays the benchmarks with the highest means based on all respondents (see Table 2).    Table 3 displays the benchmarks with the lowest means based on all respondents (see Table 3).   

Table 2. Quality Benchmarks Rated Most Important Based on Highest Means – All Respondents

Table 3. Quality Benchmarks Rated Least Important Based on Lowest Means – All Respondents

The data collected were also separated according to the distinct respondent groups of faculty, staff, and student.  Upon completing appropriate statistical analysis, none of the mean scores of the individual faculty, staff, and student groups were found to be different at a statistical significant level of .05.  In addition, the faculty and staff were grouped according to several other distinctions including: 1) faculty who had taken classes online and those who had not, 2) faculty who had taught fewer than six sections online and those who had taught six or more sections online, 3) those who identified their computer skills as novice, intermediate, and expert, and 4) staff who had taught online and those who had not.  Students were divided into various groups as well including: 1) those who were 25 years or younger and those who were 26 and older, 2) those who were employed full-time and those who were either employed part-time or not employed, 3) those who had taken two or fewer online courses, those who had taken three or four online courses, and those who had taken five or more, and 4) those who identified their computer skills as novice, intermediate, and expert.  For each of these groups, multivariate tests were completed in regards to the means of their responses and the quality benchmarks.  As a result of this analysis, no statistically significant differences were found between the means of any of the groups.  The lack of statistically significant differences reinforced the consistency among the groups as far as the ranking of the quality benchmarks. 

Following their rating of the importance of each quality benchmark, the respondents were asked to rate the extent to which the quality benchmarks were occurring at their respective institution. Again, the respondents used a four point Likert Scale (Strongly Agree = 4, Agree = 3, Disagree = 2, and Strongly Disagree = 1) to complete the survey.   Table 4 shows the means of the responses received regarding the extent to which they believed that the quality benchmark occurred at the respondents’ respective institutions.   

Again, upon completing appropriate statistical analysis, none of the mean scores of the individual faculty, staff, and student groups were found to be different at a statistical significant level of .05; however, it is notable that the faculty means were discernibly higher than those of students in regards to 1) student interaction with faculty and students, 2) the timeliness of faculty feedback, 3) the agreement on expectations for assignments, and 4) students’ access to technology assistance.  In addition, the faculty means were notably higher than those of the staff on:  1) the constructive quality of faculty feedback, 2) the timeliness of faculty feedback, 3) instruction in proper methods of research, 4) online information and training given on library resources, and 5) the assessment of teaching and learning. Finally, the means of the staff responses were higher than students on: 1) students’ access to virtual library, 2) students’ access to technical assistance, 3) timely responses from student services, and 4) the system of feedback about online courses. 

Table 4. Extent to which Quality Benchmark Occurs at Institution of Respondent based on Mean

 Benchmark