Adoption of the Multimedia Educational Resource for 
Learning and Online Teaching (MERLOT) Among Higher Education Faculty: 
Evidence from the State University of New York Learning Network

Peter Shea
Dept. of Educational Theory and Practice and
College of Computing and Information
ED 114 - University at Albany
State University of New York
Albany, NY, USA
pshea@uamail.albany.edu

Stacey McCall
Dept. of Educational Theory and Practice
University at Albany
State University of New York
Albany, NY 12222


Asil Ozdogru
Dept. of Educational Psychology
University at Albany
State University of New York
Albany, NY 12222

   

Abstract

 

This paper examines higher education faculty adoption of the Multimedia Educational Resource for Learning and Online Teaching (MERLOT) through a case study of 710 online faculty teaching at thirty-three institutions in the State University of New York.  This research is framed in the literature of technology adoption and diffusion of innovation theory.  These conceptual approaches focus on the stages that individuals traverse in the process of adopting technologies and other innovations.  Results presented here indicate high levels of awareness of MERLOT, but lower levels of use among the targeted population. Data analysis reveals heterogeneity in adopter profiles, indicating that the most committed online faculty were significantly more likely to adopt MERLOT. Results also suggest that the stage-approach common to technology adoption models is appropriate in understanding some aspects of the data (the design of professional development); however, a more powerful organizing principle may be contextual relevance of the innovation, which precedes and predetermine levels of concern and stages of adoption.  Suggestions for faculty development and further research are included.

 

 

 Introduction  

The Multimedia Educational Resource for Learning and Online Teaching (MERLOT) is an international project that seeks to facilitate the evaluation and use of online teaching and learning materials.  The project, begun by the California State University System in 1995, has sought to make high-quality, discipline-specific online learning materials freely and easily available to faculty and their students. Through the participation of a large number of institutions – 23 system partners representing more than 8 million students, the MERLOT project utilizes a peer-review system that allows educators to locate, evaluate and use high-quality online teaching and learning materials across fourteen discipline areas.  

A broad scale approach taken by the project leaders seeks to transform the use of technology in higher education and thereby transform higher education itself. Hanley (2003) estimates that the MERLOT project requires an adoption rate of approximately 15% of faculty in U.S. colleges to facilitate this transformative effect.  Until recently there have been relatively few large-scale assessments of the project from the perspective of faculty adopters.  The following study examines background information and evidence that begins to address this gap with survey results from more than 700 online faculty members from thirty-three different institutions within the State University of New York system (SUNY).  

Review of Literature  

From a theoretical perspective, faculty acceptance and use of technology in higher education can be viewed through the lens of an adoption of innovation framework. A number of theoretical models on diffusion of innovation exist (e.g. Dooley, 1999; Fung, 1992; Hall, Wallace, & Dossett, 1973; Havelock , 1973; Hamelink, 1984; Knuetel, 1995; Prochaska, DiClemete, and Norcross, 1992; Rogers , 1963, 2003) all of which are relevant to this discussion.  A summary of these frameworks suggests the focus that researchers have placed on the “stages” that potential adopters of a technology must traverse before acceptance, adoption, or sustained use.

 

(Grunwald, 2004)  

Of these models, the most commonly cited is Rogers ’ five stages of the innovation process (2003). This model is a useful lens through which to analyze the adoption of MERLOT within SUNY.  Rogers defines the innovation-decision model as “the process through which an individual (or other decision-making unit) passes from gaining initial knowledge of an innovation, to forming an attitude toward the innovation, to making a decision to adopt or reject, to implementation of the new idea, and to the confirmation of this decision” (2003, p. 168). The stages of the innovation-adoption process include: knowledge, persuasion, decision, implementation, and confirmation (Figure 1). 

 

 Figure 1.   Rogers (2003) Diffusion of Innovation Mode

Within the knowledge stage, individual characteristics of the decision-maker bear on whether the process will continue to the next stages – these include socio-economic and personality variables and communication behaviors ( Rogers , 2003). Individual characteristics of the decision maker support or undermine the decision to be persuaded in the next stage.  This component of the model can enable and constrain administrative decisions about whom to consider for support or inclusion in online teaching initiatives.

In the persuasion stage, the individual considers the relative advantages, compatibility, “observability”, “triability”, and complexity of an innovation.  Relative advantage refers to the degree to which the adopter perceives the innovation to represent an improvement in either efficiency or effectiveness in comparison to existing methods.  The adoption of an innovation is, to a certain extent, contingent upon the existence and success of faculty development and training efforts. In these efforts, it is essential that potential adopters are made aware of the relative advantages of the innovation under consideration.  The model highlights the importance of the status and standing of peers and “near-peers” in the communication network through which the innovation is diffused, a facet that we will return to later.  

Observability refers to the ease with which the technology can be seen, imagined or described to the potential adopter ( Rogers , 2003).  Through a program’s faculty development process, new instructors should be provided with access to views and examples of the technology and pedagogy of technology-based teaching and learning.   

Triability refers to the capacity to experiment with the new technology before adoption. The greater the opportunity to test the new technology, the more likely it will be adopted ( Rogers , 2003). Compatibility refers to the degree to which an innovation is consistent with the existing values, past experiences, and needs of potential adopters.  New ideas incompatible with existing values and norms are unlikely to be adopted or adopted rapidly.  

The fifth characteristic in Rogers ’ model is complexity – the degree to which the innovation is perceived to be difficult to understand or apply ( Rogers , 2003).  Managing complexity is among the greatest challenges to the diffusion of innovation.  

The online program studied here (the SUNY Learning Network) provides both technical and human resource supports to assist faculty to deal with each of the stages in the Roger’s model and will be discussed in greater detail below.     

Research regarding faculty adoption of technology has focused primarily on the process through which the individual passes and not necessarily on the process associated with an organization.  An organization is defined as a “stable system of individuals who work together to innovation instead of the decision stage that characterizes individual adoptions.  /st1:place> , 2003).  The stages of the innovation process in organizations (i.e., agenda-setting, matching, redefining-restructuring, clarifying and routinizing) are represented in two encompassing activities: initiation and implementation (Figure 2).

 

 

 

 

 

 

 

 

 Figure 2. Stages of the innovation process in organizations ( Rogers , 2003)  

The initiation  phase includes agenda-setting and matching stages and is defined as “the information gathering, conceptualizing, and planning for the adoption of innovation” (Rogers, 2003, p. 420).  Agenda-setting is the initial process of defining an organizational problem and its perceived need for an innovation ( Rogers , 2003).  Matching is defined by Rogers as the stage at which an organization fits its particular agenda to a specific innovation. The critical decision involved in the matching stage is the “adoption” of an innovation by the organization, but the most important diffusion stages for an organization are followed in the implementation phase of the diffusion process.   

During the implementation phase, the diffusion stages of redefining-restructuring, clarifying, and routinizing ultimately determine sustainability of the innovation within an organization.  These stages cover all events, decisions, and actions made during the integration of an innovation ( Rogers , 2003).  The redefining-restructuring stage involves the process of altering the innovation to accommodate the needs of the organization or adapting the organization to be appropriate for the innovation.  This aspect of accommodation is extended in the clarifying stage by putting the innovation to more widespread use throughout the organization; in turn, this allows more members of the system to become familiar with the innovation and is believed to lead to Rogers ’ last stage of diffusion in organizations.  Routinization refers to the stage where the innovation becomes “incorporated into regular activities of the organization and loses its separate identity” (Rogers, 2003, p. 434).  The main questions to be addressed in this study involve investigating the diffusion of MERLOT within SUNY in light of the Rogers ’ framework.  

Technical and Human Resource Support to Targeted Adopters  

This section describes the faculty development process in which the MERLOT training is embedded and is based on the knowledge of the lead author who served as MERLOT Project Director for the State University of New York from 2001-2004. The State University of New York (SUNY) has been an institutional participant in MERLOT since 2001. In Rogers’ agenda setting and matching stages, several factors influenced the decision to join the project.   One motivator was the rapid growth of online teaching and learning that SUNY was witnessing at that time.  Enrollments in online courses offered through the SUNY Learning Network (SLN) were increasing dramatically with the online student population more than doubling in some academic years.  There was a growing recognition of the need for a large, shared and sustainable discipline-specific repository of quality instructional materials that faculty could use as a resource in the development of online courses and for the enhancement of traditional courses.  The MERLOT project matched this description well.  It should be noted, however, that participation in the MERLOT project was only one element in the strategy to employ learning object repositories.  Although beyond the scope of this paper, other components included an ongoing SUNY-wide contract with Academic Systems Inc. for their online library and the development of an annotated collection of tools useful for online teaching known as “Cool Tools,” edited by a member of the SLN instructional design group.             

In the redefining-restructuring stage, steps were taken to begin to adapt the organization to be appropriate for adoption of MERLOT. Through the office of SUNY Learning Environments, faculty from SUNY institutions were recruited and supported to participate on MERLOT Editorial Boards.  Initially, in alignment with the structure of the MERLOT program, eight SUNY professors were identified and agreed to participate.   The discipline areas that were identified as having strategic value to the SUNY system were Teacher Education, Mathematics, Physics, World Languages, Information Technology, Psychology, Music and Biology.   

Initial participants on the MERLOT editorial boards came from the four University Centers in SUNY ( Albany , Binghamton , Buffalo and Stony Brook) as well as from the Crane School of Music at Potsdam , Herkimer County Community College , the College at Brockport and the College at Oneonta. With guidance and training from the MERLOT organization, the editorial boards were charged with devising appropriate processes, procedures, and criteria by which online teaching and learning materials in the MERLOT collection could be reviewed and with carrying out the peer-reviews of the materials.  They were also asked to “grow” the collection and to enlist the assistance of professional societies in encouraging use of materials and in supporting the ongoing sustainability of the project.  It should be noted that SUNY was just one of eighteen systems participating and that each of the systems committed to initially support a team of eight editorial board members – thus more than 140 faculty initially participated on the MERLOT project editorial boards.  

As a second component of SUNY’s plan for participation in the MERLOT project, an aspect of Rogers ’ clarifying stage was to build awareness and use of the materials in the collection.  To this end, several strategies were developed.  First, key instructional support staff from campuses within the SUNY System were invited to attend MERLOT workshops designed to support trainers in integrating MERLOT into instruction. Fourteen staff members from campuses located in a variety of regions across the state were identified and supported to attend workshops in three consecutive years.  The other eighteen institutions created similar plans.  

Another element of the clarifying stage was to leverage SUNY-wide conferences as a mechanism for building awareness and offering training about MERLOT.  Members of the SUNY-MERLOT Editorial Boards were requested to present at the SUNY-wide Conference on Instructional Technologies (CIT) as a component of their responsibilities as supported participants in the program.  Staff affiliated with the SUNY Teaching, Learning, and Technology Program have offered hands-on workshops at CIT from 2001-2005.  Members of the MERLOT administrative and technical staff were invited to CIT and the SUNY-wide Teaching, Learning, and Technology Meeting and made presentations at these conferences from 2002-2005.  Again, the other eighteen MERLOT partners supported similar activities.  

A third aspect of the clarifying stage was to provide training to faculty developing online courses through the SUNY Learning Network.  The SUNY system is comprised of sixty-four institutions that include community, technology, and university colleges, university centers and graduate research universities that currently enroll more than 413,000 students, studying in 6,688 programs (SUNY, 2005).  The SUNY Learning Network provides a comprehensive program of technical support, infrastructure and training for online faculty from across the SUNY system. Since its creation in 1994, SLN has become one of the nation’s largest distance learning programs, providing educational opportunities through more than one hundred complete online degree programs offered by forty SUNY colleges.  This e-learning enterprise comprises more than 4,000 course offerings with complete online enrollments in excess of 100,000 annually.  Given the match between the scope and goals of SLN and those of MERLOT, and the ongoing success of the SLN faculty development program, faculty teaching within the program were targeted for training.  As a component of all faculty training participants were provided with an overview of MERLOT and learning activities within the sessions guided them to register as a member of MERLOT and to search for an online resource in the collection that they might be able to integrate into their online course.  

While an assessment of each training was conducted, a broader and more inclusive assessment of the program from the perspective of SUNY faculty has not been conducted; this study begins to fill that gap.  This paper examines issues of awareness, use, and dimensions of satisfaction with MERLOT among the online faculty in SLN.  The goal is to begin to understand online faculty adoption of the broad spectrum of technologies that reside within the MERLOT collection; to better gauge how and whether the materials benefit teaching and learning from the faculty perspective; to collect information on uses of alternatives to MERLOT; and to understand why some faculty may choose not to adopt online teaching and learning materials. Specific questions addressed along the way include:  

1) What is the current usage of MERLOT among a large segment of SUNY online faculty?

2) What are the main reasons given by SUNY faculty for and against adoption of MERLOT?

3) What factors influence the extent of the system’s diffusion of MERLOT so far?

4) How might the current diffusion plan be strengthened?

5) What do these results suggest for theory development and future research?

 
Method

 Materials

 To assess levels of use and satisfaction with MERLOT broadly across SUNY, data from three sources were collected and analyzed.  The data included log files of SUNY users of the MERLOT website; numerical and narrative reports gathered from online faculty surveys and a ten-part questionnaire soliciting information from past and current members of the MERLOT Editorial Board from the State University of New York.    

Design and Procedure

Log files from the MERLOT website were analyzed to gain a sense of global levels of use.  The MERLOT administrative staff provided a report of activity on the MERLOT website by members of the SUNY system.  The report was generated by MERLOT using a range of Internet-protocol (IP) addresses provided by SUNY that corresponded with its institutions.  

Survey data included in this report contain both numeric and narrative responses from SUNY’s online faculty teaching via SLN-CourseSpace programs in the Fall 2004 and Spring 2005 semesters.  The data were collected from participants using an online survey that sought to assess respondents’ attitudes to a broad range of issues in the online teaching environment and MERLOT was only one of the areas.  The data reflect the levels of use and attitudes of faculty for whom online teaching and learning materials are highly relevant, i.e., those developing Web-enhanced and online courses. The narrative survey data were analyzed using the analytic procedure of constant comparison.  In this process, two coders reviewed the data independently and then conceptually labeled the comments to reflect categories that emerged.  After completing initial review of the data, the coders compared categories to determine developing patterns in the responses; this led to the creation of a mutually agreed upon list.  Both coders then re-categorized the data using the agreed upon list to establish a baseline for reliability. Agreement was used to calculate levels of inter-rater reliability.  

The SLN program administration elicited the participation of faculty in completing the survey via email communications which were sent three times during the final five weeks of each semester.  710 faculty from 33 institutions responded to the survey.  This represents 60% of the total number of faculty teaching through the SLN program for those semesters. Given this rate of response, some care needs to be taken with regards to generalizing these findings beyond the scope of the current study.  

Finally, data from SUNY MERLOT Editorial Board were solicited during the summer of 2005.  A brief questionnaire asking for Board members’ perspectives was developed and distributed via email. Several follow-up email communications encouraging the editors to complete the questionnaires were also disseminated.  Summaries of editor responses are included in the results section.   

Results

Data on SUNY faculty’s usage of MERLOT were collected for the period between January 2003 and May 2005. The data were gathered from the MERLOT log files and analyzed by a tool called NetTracker. The data were filtered by the Internet protocol (IP) addresses; thus, SUNY’s computers were tracked using IP addresses.  This method is efficient but has considerable limitations in light of the present evaluative needs. For example, counting SUNY’s usage of MERLOT by means of registered IP collect the usage rates.  

Using NetTracker data, the total number of identified visitors from SUNY fluctuated with an overall increase of 18.46 percent from 1,392 in January-June 2003 period to 1,649 in January-May 2005 period. Figure 3 presents the usage data for 6-month periods.

 

            Figure 3.  Total number of MERLOT visitors for six-month periods between
                             2003 and 2005.

 The average number of visits per month for each 6-month periods showed an overall 33.8 percent increase from 245 to 328.  The total number of unique visitors (i.e., unique users no matter how many times they visited) also fluctuated with an overall increase of 9.67 percent from 765 to 839 unique visitors. Figure 4 shows the trend line in average visits per month and number of unique visitors for six-month periods from January-June 2003 to January-May 2005.  

 

Figure 4.Average number of SUNY visits per month and number of total unique visitors for six-month periods between 2003 and 2005.  

While the percentage of visitors who returned more than once to MERLOT was approximately 30 percent, the average length of visits at MERLOT showed a reducing pattern.  As presented in Figure 5, average length of visits showed an overall 32 percent reduction from 510 seconds to 347 seconds. 

 

Figure 5. Average duration of visits at MERLOT in seconds six-month periods between 2003 and 2005.  

Number of total registered users and personal collections showed a rapid increase (approximately 112%) between January 2003 and May 2005. Figures 6 and 7 demonstrate this trend.  Numbers in this report describes the users who registered in MERLOT with an identifiable partner campus email address, or self identified as belonging to a partner school or system. So, again it does not report on users from the system or campus who may use commercial email systems who did not identify their university affiliation.

Figure 6. Number of registered MERLOT users from SUNY for six-month periods between 2003 and 2005.  

Figure 7. Number of total personal collections and number of registered users with personal collections for six-month periods between 2003 and 2005.  

 

SUNY faculty activities within MERLOT also increased from January 2003 to May 2005.  Noted activities include: material and comment contribution, peer reviewing and material authoring. The percentage increase between these two periods in the total number of activities was 32% for comment contribution, 72% for material contribution, 80% for material authoring, and 105% for peer reviewing. Change for all of the activities of registered users are presented in Figure 8, 9 and 10.

 

Figure 8. Activities of SUNY faculty within MERLOT for six-month periods between 2003 and 2005: Comment contributions.

 

 

Figure 9. Activities of SUNY faculty within MERLOT for six-month periods between 2003 and 2005: Material contributions.

 

Figure 10. Activities of SUNY faculty within MERLOT for six-month periods between 2003 and 2005: Material authoring.

 

Figure 11. Activities of SUNY faculty within MERLOT for six-month periods between 2003 and 2005: Peer reviews.

   

Survey Data Analysis

 A total of 710 online SLN-CourseSpace faculty responded to items inquiring about their use of and satisfaction with MERLOT in two surveys at the ends of Fall 2004 and Spring 2005 semesters ( Fall 2004 n= 518, Spring 2005 n = 192). Results showed that although a majority of the faculty had heard about MERLOT, a smaller number of them had used it. Table 1 displays the answers of the online faculty to the questions about their familiarity with MERLOT, usage of MERLOT in their courses, and perceived usefulness of MERLOT

 

Table 1. Frequencies and percentages of faculty heard about MERLOT, adopted learning materials from MERLOT, and found it useful - 2004-05 academic years at SUNY.  

Question	Answer	Fall 2004  (n = 518)	Spring 2005  (n = 192)	2004-05 Year  (N = 710)
Heard of MERLOT	Yes	400	155	555
	No	103	33	136
Choose not to answer	15	4	19
Adopted MERLOT materials	Yes	93	29	122
	No	304	137	441
	Choose not to answer	60	23	83
	Blank	61	3	64
Found MERLOT useful	Yes	85	25	110
	No	101	142	243
	Choose not to answer	152	25	177
	Blank	180		180

Analysis of the “usefulness” data pointed to a connection with actual implementation of MERLOT materials. Faculty who adopted materials in MERLOT found it more useful than faculty who had heard of, but not tried MERLOT.   

In Fall 2004, while 81.72 % of the faculty (76 out of 93) who had adopted materials in MERLOT found it useful, only 2.96% of the faculty (9 out of 304)  who did not try it found it useful. Table 2 presents frequencies of faculty responses for their usage and perceived usefulness of MERLOT for Fall 2004 semester.  When only responses of the faculty who answered “Yes” or “No” to the usage and usefulness questions considered, it is seen that 86.36 percent of faculty (76 out of 88) who used MERLOT found amily:Arial">