MERLOT Journal of Online Learning and Teaching

Vol. 2, No. 3, September 2006


                                     

Adoption of the Multimedia Educational Resource for 
Learning and Online Teaching (MERLOT) Among Higher Education Faculty: 
Evidence from the
State University of New York Learning Network


Peter Shea
Dept. of Educational Theory and Practice and
College of Computing and Information
ED 114 - University at Albany
State University of New York
Albany, NY, USA
pshea@uamail.albany.edu

Stacey McCall
Dept. of Educational Theory and Practice
University at Albany
State University of New York
Albany, NY 12222


Asil Ozdogru
Dept. of Educational Psychology
University at Albany
State University of New York
Albany, NY 12222

   

Abstract

 

This paper examines higher education faculty adoption of the Multimedia Educational Resource for Learning and Online Teaching (MERLOT) through a case study of 710 online faculty teaching at thirty-three institutions in the State University of New York.  This research is framed in the literature of technology adoption and diffusion of innovation theory.  These conceptual approaches focus on the stages that individuals traverse in the process of adopting technologies and other innovations.  Results presented here indicate high levels of awareness of MERLOT, but lower levels of use among the targeted population. Data analysis reveals heterogeneity in adopter profiles, indicating that the most committed online faculty were significantly more likely to adopt MERLOT. Results also suggest that the stage-approach common to technology adoption models is appropriate in understanding some aspects of the data (the design of professional development); however, a more powerful organizing principle may be contextual relevance of the innovation, which precedes and predetermine levels of concern and stages of adoption.  Suggestions for faculty development and further research are included.

 

 

 Introduction  

The Multimedia Educational Resource for Learning and Online Teaching (MERLOT) is an international project that seeks to facilitate the evaluation and use of online teaching and learning materials.  The project, begun by the California State University System in 1995, has sought to make high-quality, discipline-specific online learning materials freely and easily available to faculty and their students. Through the participation of a large number of institutions – 23 system partners representing more than 8 million students, the MERLOT project utilizes a peer-review system that allows educators to locate, evaluate and use high-quality online teaching and learning materials across fourteen discipline areas.  

A broad scale approach taken by the project leaders seeks to transform the use of technology in higher education and thereby transform higher education itself. Hanley (2003) estimates that the MERLOT project requires an adoption rate of approximately 15% of faculty in U.S. colleges to facilitate this transformative effect.  Until recently there have been relatively few large-scale assessments of the project from the perspective of faculty adopters.  The following study examines background information and evidence that begins to address this gap with survey results from more than 700 online faculty members from thirty-three different institutions within the State University of New York system (SUNY).  

Review of Literature  

From a theoretical perspective, faculty acceptance and use of technology in higher education can be viewed through the lens of an adoption of innovation framework. A number of theoretical models on diffusion of innovation exist (e.g. Dooley, 1999; Fung, 1992; Hall, Wallace, & Dossett, 1973; Havelock , 1973; Hamelink, 1984; Knuetel, 1995; Prochaska, DiClemete, and Norcross, 1992; Rogers , 1963, 2003) all of which are relevant to this discussion.  A summary of these frameworks suggests the focus that researchers have placed on the “stages” that potential adopters of a technology must traverse before acceptance, adoption, or sustained use.

 


(Grunwald, 2004)  

Of these models, the most commonly cited is Rogers ’ five stages of the innovation process (2003). This model is a useful lens through which to analyze the adoption of MERLOT within SUNY.  Rogers defines the innovation-decision model as “the process through which an individual (or other decision-making unit) passes from gaining initial knowledge of an innovation, to forming an attitude toward the innovation, to making a decision to adopt or reject, to implementation of the new idea, and to the confirmation of this decision” (2003, p. 168). The stages of the innovation-adoption process include: knowledge, persuasion, decision, implementation, and confirmation (Figure 1). 

 

 Figure 1.   Rogers (2003) Diffusion of Innovation Mode


Within the knowledge stage, individual characteristics of the decision-maker bear on whether the process will continue to the next stages – these include socio-economic and personality variables and communication behaviors ( Rogers , 2003). Individual characteristics of the decision maker support or undermine the decision to be persuaded in the next stage.  This component of the model can enable and constrain administrative decisions about whom to consider for support or inclusion in online teaching initiatives.

In the persuasion stage, the individual considers the relative advantages, compatibility, “observability”, “triability”, and complexity of an innovation.  Relative advantage refers to the degree to which the adopter perceives the innovation to represent an improvement in either efficiency or effectiveness in comparison to existing methods.  The adoption of an innovation is, to a certain extent, contingent upon the existence and success of faculty development and training efforts. In these efforts, it is essential that potential adopters are made aware of the relative advantages of the innovation under consideration.  The model highlights the importance of the status and standing of peers and “near-peers” in the communication network through which the innovation is diffused, a facet that we will return to later.
 

Observability refers to the ease with which the technology can be seen, imagined or described to the potential adopter ( Rogers , 2003).  Through a program’s faculty development process, new instructors should be provided with access to views and examples of the technology and pedagogy of technology-based teaching and learning.   

Triability refers to the capacity to experiment with the new technology before adoption. The greater the opportunity to test the new technology, the more likely it will be adopted ( Rogers , 2003). Compatibility refers to the degree to which an innovation is consistent with the existing values, past experiences, and needs of potential adopters.  New ideas incompatible with existing values and norms are unlikely to be adopted or adopted rapidly.  

The fifth characteristic in Rogers ’ model is complexity – the degree to which the innovation is perceived to be difficult to understand or apply ( Rogers , 2003).  Managing complexity is among the greatest challenges to the diffusion of innovation.  

The online program studied here (the SUNY Learning Network) provides both technical and human resource supports to assist faculty to deal with each of the stages in the Roger’s model and will be discussed in greater detail below.     

Research regarding faculty adoption of technology has focused primarily on the process through which the individual passes and not necessarily on the process associated with an organization.  An organization is defined as a “stable system of individuals who work together to innovation instead of the decision stage that characterizes individual adoptions.  /st1:place> , 2003).  The stages of the innovation process in organizations (i.e., agenda-setting, matching, redefining-restructuring, clarifying and routinizing) are represented in two encompassing activities: initiation and implementation (Figure 2).

 

 

 

 

 

 

 

 

 Figure 2. Stages of the innovation process in organizations ( Rogers , 2003)  

The initiation  phase includes agenda-setting and matching stages and is defined as “the information gathering, conceptualizing, and planning for the adoption of innovation” (Rogers, 2003, p. 420).  Agenda-setting is the initial process of defining an organizational problem and its perceived need for an innovation ( Rogers , 2003).  Matching is defined by Rogers as the stage at which an organization fits its particular agenda to a specific innovation. The critical decision involved in the matching stage is the “adoption” of an innovation by the organization, but the most important diffusion stages for an organization are followed in the implementation phase of the diffusion process.   

During the implementation phase, the diffusion stages of redefining-restructuring, clarifying, and routinizing ultimately determine sustainability of the innovation within an organization.  These stages cover all events, decisions, and actions made during the integration of an innovation ( Rogers , 2003).  The redefining-restructuring stage involves the process of altering the innovation to accommodate the needs of the organization or adapting the organization to be appropriate for the innovation.  This aspect of accommodation is extended in the clarifying stage by putting the innovation to more widespread use throughout the organization; in turn, this allows more members of the system to become familiar with the innovation and is believed to lead to Rogers ’ last stage of diffusion in organizations.  Routinization refers to the stage where the innovation becomes “incorporated into regular activities of the organization and loses its separate identity” (Rogers, 2003, p. 434).  The main questions to be addressed in this study involve investigating the diffusion of MERLOT within SUNY in light of the Rogers framework.  

Technical and Human Resource Support to Targeted Adopters  

This section describes the faculty development process in which the MERLOT training is embedded and is based on the knowledge of the lead author who served as MERLOT Project Director for the State University of New York from 2001-2004. The State University of New York (SUNY) has been an institutional participant in MERLOT since 2001. In Rogers’ agenda setting and matching stages, several factors influenced the decision to join the project.   One motivator was the rapid growth of online teaching and learning that SUNY was witnessing at that time.  Enrollments in online courses offered through the SUNY Learning Network (SLN) were increasing dramatically with the online student population more than doubling in some academic years.  There was a growing recognition of the need for a large, shared and sustainable discipline-specific repository of quality instructional materials that faculty could use as a resource in the development of online courses and for the enhancement of traditional courses.  The MERLOT project matched this description well.  It should be noted, however, that participation in the MERLOT project was only one element in the strategy to employ learning object repositories.  Although beyond the scope of this paper, other components included an ongoing SUNY-wide contract with Academic Systems Inc. for their online library and the development of an annotated collection of tools useful for online teaching known as “Cool Tools,” edited by a member of the SLN instructional design group.             

In the redefining-restructuring stage, steps were taken to begin to adapt the organization to be appropriate for adoption of MERLOT. Through the office of SUNY Learning Environments, faculty from SUNY institutions were recruited and supported to participate on MERLOT Editorial Boards.  Initially, in alignment with the structure of the MERLOT program, eight SUNY professors were identified and agreed to participate.   The discipline areas that were identified as having strategic value to the SUNY system were Teacher Education, Mathematics, Physics, World Languages, Information Technology, Psychology, Music and Biology.   

Initial participants on the MERLOT editorial boards came from the four University Centers in SUNY ( Albany , Binghamton , Buffalo and Stony Brook) as well as from the Crane School of Music at Potsdam , Herkimer County Community College , the College at Brockport and the College at Oneonta. With guidance and training from the MERLOT organization, the editorial boards were charged with devising appropriate processes, procedures, and criteria by which online teaching and learning materials in the MERLOT collection could be reviewed and with carrying out the peer-reviews of the materials.  They were also asked to “grow” the collection and to enlist the assistance of professional societies in encouraging use of materials and in supporting the ongoing sustainability of the project.  It should be noted that SUNY was just one of eighteen systems participating and that each of the systems committed to initially support a team of eight editorial board members – thus more than 140 faculty initially participated on the MERLOT project editorial boards.  

As a second component of SUNY’s plan for participation in the MERLOT project, an aspect of Rogers clarifying stage was to build awareness and use of the materials in the collection.  To this end, several strategies were developed.  First, key instructional support staff from campuses within the SUNY System were invited to attend MERLOT workshops designed to support trainers in integrating MERLOT into instruction. Fourteen staff members from campuses located in a variety of regions across the state were identified and supported to attend workshops in three consecutive years.  The other eighteen institutions created similar plans.  

Another element of the clarifying stage was to leverage SUNY-wide conferences as a mechanism for building awareness and offering training about MERLOT.  Members of the SUNY-MERLOT Editorial Boards were requested to present at the SUNY-wide Conference on Instructional Technologies (CIT) as a component of their responsibilities as supported participants in the program.  Staff affiliated with the SUNY Teaching, Learning, and Technology Program have offered hands-on workshops at CIT from 2001-2005.  Members of the MERLOT administrative and technical staff were invited to CIT and the SUNY-wide Teaching, Learning, and Technology Meeting and made presentations at these conferences from 2002-2005.  Again, the other eighteen MERLOT partners supported similar activities.  

A third aspect of the clarifying stage was to provide training to faculty developing online courses through the SUNY Learning Network.  The SUNY system is comprised of sixty-four institutions that include community, technology, and university colleges, university centers and graduate research universities that currently enroll more than 413,000 students, studying in 6,688 programs (SUNY, 2005).  The SUNY Learning Network provides a comprehensive program of technical support, infrastructure and training for online faculty from across the SUNY system. Since its creation in 1994, SLN has become one of the nation’s largest distance learning programs, providing educational opportunities through more than one hundred complete online degree programs offered by forty SUNY colleges.  This e-learning enterprise comprises more than 4,000 course offerings with complete online enrollments in excess of 100,000 annually.  Given the match between the scope and goals of SLN and those of MERLOT, and the ongoing success of the SLN faculty development program, faculty teaching within the program were targeted for training.  As a component of all faculty training participants were provided with an overview of MERLOT and learning activities within the sessions guided them to register as a member of MERLOT and to search for an online resource in the collection that they might be able to integrate into their online course.  

While an assessment of each training was conducted, a broader and more inclusive assessment of the program from the perspective of SUNY faculty has not been conducted; this study begins to fill that gap.  This paper examines issues of awareness, use, and dimensions of satisfaction with MERLOT among the online faculty in SLN.  The goal is to begin to understand online faculty adoption of the broad spectrum of technologies that reside within the MERLOT collection; to better gauge how and whether the materials benefit teaching and learning from the faculty perspective; to collect information on uses of alternatives to MERLOT; and to understand why some faculty may choose not to adopt online teaching and learning materials. Specific questions addressed along the way include:  

1) What is the current usage of MERLOT among a large segment of SUNY online faculty?

2) What are the main reasons given by SUNY faculty for and against adoption of MERLOT?

3) What factors influence the extent of the system’s diffusion of MERLOT so far?

4) How might the current diffusion plan be strengthened?

5) What do these results suggest for theory development and future research?

 
Method

 Materials

 To assess levels of use and satisfaction with MERLOT broadly across SUNY, data from three sources were collected and analyzed.  The data included log files of SUNY users of the MERLOT website; numerical and narrative reports gathered from online faculty surveys and a ten-part questionnaire soliciting information from past and current members of the MERLOT Editorial Board from the State University of New York.    

Design and Procedure

Log files from the MERLOT website were analyzed to gain a sense of global levels of use.  The MERLOT administrative staff provided a report of activity on the MERLOT website by members of the SUNY system.  The report was generated by MERLOT using a range of Internet-protocol (IP) addresses provided by SUNY that corresponded with its institutions.  

Survey data included in this report contain both numeric and narrative responses from SUNY’s online faculty teaching via SLN-CourseSpace programs in the Fall 2004 and Spring 2005 semesters.  The data were collected from participants using an online survey that sought to assess respondents’ attitudes to a broad range of issues in the online teaching environment and MERLOT was only one of the areas.  The data reflect the levels of use and attitudes of faculty for whom online teaching and learning materials are highly relevant, i.e., those developing Web-enhanced and online courses. The narrative survey data were analyzed using the analytic procedure of constant comparison.  In this process, two coders reviewed the data independently and then conceptually labeled the comments to reflect categories that emerged.  After completing initial review of the data, the coders compared categories to determine developing patterns in the responses; this led to the creation of a mutually agreed upon list.  Both coders then re-categorized the data using the agreed upon list to establish a baseline for reliability. Agreement was used to calculate levels of inter-rater reliability.  

The SLN program administration elicited the participation of faculty in completing the survey via email communications which were sent three times during the final five weeks of each semester.  710 faculty from 33 institutions responded to the survey.  This represents 60% of the total number of faculty teaching through the SLN program for those semesters. Given this rate of response, some care needs to be taken with regards to generalizing these findings beyond the scope of the current study.  

Finally, data from SUNY MERLOT Editorial Board were solicited during the summer of 2005.  A brief questionnaire asking for Board members’ perspectives was developed and distributed via email. Several follow-up email communications encouraging the editors to complete the questionnaires were also disseminated.  Summaries of editor responses are included in the results section.   

Results

Data on SUNY faculty’s usage of MERLOT were collected for the period between January 2003 and May 2005. The data were gathered from the MERLOT log files and analyzed by a tool called NetTracker. The data were filtered by the Internet protocol (IP) addresses; thus, SUNY’s computers were tracked using IP addresses.  This method is efficient but has considerable limitations in light of the present evaluative needs. For example, counting SUNY’s usage of MERLOT by means of registered IP collect the usage rates.  

Using NetTracker data, the total number of identified visitors from SUNY fluctuated with an overall increase of 18.46 percent from 1,392 in January-June 2003 period to 1,649 in January-May 2005 period. Figure 3 presents the usage data for 6-month periods.

 

            Figure 3.  Total number of MERLOT visitors for six-month periods between
                             2003 and 2005.

 The average number of visits per month for each 6-month periods showed an overall 33.8 percent increase from 245 to 328.  The total number of unique visitors (i.e., unique users no matter how many times they visited) also fluctuated with an overall increase of 9.67 percent from 765 to 839 unique visitors. Figure 4 shows the trend line in average visits per month and number of unique visitors for six-month periods from January-June 2003 to January-May 2005.  

 

Figure 4.Average number of SUNY visits per month and number of total unique visitors for six-month periods between 2003 and 2005.  

While the percentage of visitors who returned more than once to MERLOT was approximately 30 percent, the average length of visits at MERLOT showed a reducing pattern.  As presented in Figure 5, average length of visits showed an overall 32 percent reduction from 510 seconds to 347 seconds. 

 

Figure 5. Average duration of visits at MERLOT in seconds six-month periods between 2003 and 2005.  

Number of total registered users and personal collections showed a rapid increase (approximately 112%) between January 2003 and May 2005. Figures 6 and 7 demonstrate this trend.  Numbers in this report describes the users who registered in MERLOT with an identifiable partner campus email address, or self identified as belonging to a partner school or system. So, again it does not report on users from the system or campus who may use commercial email systems who did not identify their university affiliation.

Figure 6. Number of registered MERLOT users from SUNY for six-month periods between 2003 and 2005.  

Figure 7. Number of total personal collections and number of registered users with personal collections for six-month periods between 2003 and 2005.  

 

SUNY faculty activities within MERLOT also increased from January 2003 to May 2005.  Noted activities include: material and comment contribution, peer reviewing and material authoring. The percentage increase between these two periods in the total number of activities was 32% for comment contribution, 72% for material contribution, 80% for material authoring, and 105% for peer reviewing. Change for all of the activities of registered users are presented in Figure 8, 9 and 10.

 

Figure 8. Activities of SUNY faculty within MERLOT for six-month periods between 2003 and 2005: Comment contributions.

 

 

Figure 9. Activities of SUNY faculty within MERLOT for six-month periods between 2003 and 2005: Material contributions.

 

Figure 10. Activities of SUNY faculty within MERLOT for six-month periods between 2003 and 2005: Material authoring.

 

Figure 11. Activities of SUNY faculty within MERLOT for six-month periods between 2003 and 2005: Peer reviews.

   

Survey Data Analysis

 A total of 710 online SLN-CourseSpace faculty responded to items inquiring about their use of and satisfaction with MERLOT in two surveys at the ends of Fall 2004 and Spring 2005 semesters ( Fall 2004 n= 518, Spring 2005 n = 192). Results showed that although a majority of the faculty had heard about MERLOT, a smaller number of them had used it. Table 1 displays the answers of the online faculty to the questions about their familiarity with MERLOT, usage of MERLOT in their courses, and perceived usefulness of MERLOT

 

Table 1. Frequencies and percentages of faculty heard about MERLOT, adopted learning materials from MERLOT, and found it useful - 2004-05 academic years at SUNY.  

Question  Answer  Fall 2004 
(n = 518) 
Spring 2005 
(n = 192) 
2004-05 Year 
(N = 710) 
Heard of MERLOT  Yes  400 155 555
No  103 33 136
Choose not to answer  15 4 19
Adopted MERLOT materials  Yes  93 29 122
No  304 137 441
Choose not to answer  60 23 83
Blank  61 3 64
Found MERLOT useful  Yes  85 25 110
No  101 142 243
Choose not to answer  152 25 177
Blank  180    180

Analysis of the “usefulness” data pointed to a connection with actual implementation of MERLOT materials. Faculty who adopted materials in MERLOT found it more useful than faculty who had heard of, but not tried MERLOT.   

In Fall 2004, while 81.72 % of the faculty (76 out of 93) who had adopted materials in MERLOT found it useful, only 2.96% of the faculty (9 out of 304)  who did not try it found it useful. Table 2 presents frequencies of faculty responses for their usage and perceived usefulness of MERLOT for Fall 2004 semester.  When only responses of the faculty who answered “Yes” or “No” to the usage and usefulness questions considered, it is seen that 86.36 percent of faculty (76 out of 88) who used MERLOT found amily:Arial">  

 

MERLOT useful

Total

Yes

No

Choose not to answer

Blank

Adopted MERLOT Materials

Yes

76

12

5

-

93

No

9

88

102

105

304

Choose not to answer

-

-

44

16

60

Blank

-

1

1

59

61

 

Total

85

101

152

180

518

 

Figure 12.  Percentage of SUNY faculty who found MERLOT useful or not according to their usage of MERLOT in Fall 2004.  

Additional insight regarding faculty adoption of MERLOT can be gained by examining respondents’ ratings of the usefulness of this resource by their level of online teaching experience.  One can see that faculty who were more experienced with online teaching were more likely find MERLOT to be a helpful resource. The data from Fall 2004 indicated that faculty who had taught more than one online course were more likely to report that MERLOT was useful in their teaching. Among the faculty who had taught more than once, 54% found MERLOT useful compared to only 27% of faculty who were teaching online for the first time.  

 Table 3. Frequency of faculty who had Found MERLOT Useful by Online Teaching by Experience

Online Teaching Experience  I found MERLOT useful for my Teaching  % Yes  Total 
  Yes  No       
         
First time  14 37 27% 51
          
More than one time  70 60 54% 130
         
Total  84 97  “Overall I found MERLOT to be very useful in my teaching” on a 5-point Likert scale where scale of (1) “Strongly Agree,” (2) “Agree,” (3) “Neutral,” (4) “Disagree,” and (5) “Strongly Disagree.” Results showed that 29 faculty who adopted MERLOT materials found it more useful (M = 2.1, SD = .94) than 90  Since fifty eight of the non-user faculty chose not to answer to usefulness question, only ninety faculty out of 148 were included in the analysis.  

 

Figure 13.  Numbers of SLN faculty who had adopted MERLOT and their perceptions of usefulness in Spring 2005.  

Another approach to understanding utilization of MERLOT by the SLN online faculty is to examine the numbers of faculty who had reported using multimedia of any sort in their online courses.  Faculty who reported using multimedia of any sort were significantly more likely to report having used MERLOT in their courses. For example in the Spring 2005 survey, among the faculty who reported using multimedia of any sort, 27% of these faculty also reported using MERLOT in their courses.  Of those who reported that they had not used multimedia in their course, only 4% reported using MERLOT.  

 

  

  

Used MERLOT in this online course  % used MERLOT in online course  Total 
Yes 

No 

     
Used Multimedia of any Sort in this Course         
Yes  27 89 27% 116
          
No  2 47 4% 49
         
Total  29 136    165
Table 4. Faculty who had used any multimedia in their course and use of MERLOT.


Responses for the question about faculty use of other resources similar to MERLOT in the Spring 2005 questionnaire turned out a low level of use of other resources. When faculty were asked “Have you ever used another resource similar to MERLOT to locate peer-reviewed, discipline specific, online teaching materials?” 165 (out of 189) provided an answer for this question. While 15.2 percent had used another similar resource, 84.8 percent had not.
 

Finally, the use of the statistical analytic technique called Chi Square Automatic Interaction Detection (CHAID) allows for a multivariate approach to determine what factors are most important with regard to the adoption of MERLOT.  Using the items from the Fall 2004 survey data, all items can be combined to determine which indicate significantly different subgroups within the entire data set.  Using this method reveals that the highest level break occurs on the item asking whether the online instructor would like to teach additional online courses in the future.  The second highest level break occurs within that sub-group in response to the item asking how much the instructor felt the students in the course learned compared to similar courses they had taught in the classroom.  The results suggest that the most committed online faculty (those most likely to want to teach online again) were significantly more likely to adopt MERLOT.  Additionally, within that group, those that felt students learned more online than in the classroom were also significantly more likely to use MERLOT than others who were less sure about the learning effectiveness of the online format.

 


Figure 14.
Decision tree (CHAID) indicating faculty adoption of MERLOT


Analysis of Narrative Data

In addition to the quantitative analysis, narrative data focusing on the SLN faculty’s use of MERLOT were collected and interpreted by two independent raters. The data provided insight into the faculty’s reasons for adopting or not adopting materials in MERLOT.  One survey question was used to gather narrative data in Fall 2004, while additional questions were added to the survey in Spring 2005 to provide further insight into faculty use of MERLOT.  

The first narrative item asked respondents who indicated that they had not used MERLOT to explain why they had not done so. Of a total of 518 completed faculty surveys in Fall 2004, 221 included narrative responses.  In the Spring of 2005, a total of 136 such responses was included within the 192 surveys collected.  

Two raters initially screened all of the responses and developed independent coding schemes based on the data.  The raters then compared the coding schemes and collapsed these into one mutually agreed upon coding list.  The list consisted of ten separate codes, which were then used to recode the data.  Following the second coding process, the results were compared to calculate the percentage of agreement between the two raters.  An inter-rater reliability score of .95 was measured between two coders for Fall 2004 and .93 for Spring 2005.

 


Figure 15. Faculty reasons for not using MERLOT in Fall 2004.


The faculty’s top three reasons for not using MERLOT in Fall 2004 were that they ‘simply chose not to use it (26%),’ ‘time constraints (24%),’ and the ‘lack of relevant content available in the subject area taught by the faculty’s (18%). Figure 15 provides the percentages of faculty reasons for not using materials in the MERLOT collection.
 

The results for Spring 2005 differed from the 2004 results in that the faculty noted ‘time constraints (13%)’ as their major reason for not using MERLOT. Also topping the Spring 2005 list was ‘that the faculty already had a lot of other resources,’ so they expressed no need for additional resources (11%), and ‘faculty did not know about MERLOT’ (9%) or ‘simply chose not to use MERLOT’ (9%).  Figure 16 provides the percentages of the faculty’s reasons for not using MERLOT in the Spring 2005 data.


Figure 16.  Faculty reasons for not using MERLOT in Spring 2005.

 

Additional narrative data regarding SLN faculty use of MERLOT were drawn from the Spring 2005 survey.  One item gave the faculty the opportunity to provide any additional comments they had regarding MERLOT.  Approximately 8.8% of the faculty offered additional comments. Items were coded for dominant themes and inter-rater reliability score of .93 was measured between two coders.  Figure 17 provides the percentages of frequency for the additional comments noted by the faculty. 

 

Figure 17. Additional comments noted by the faculty. 

 

SUNY Members of the MERLOT Editorial Board Responses  

In addition to the SLN-CourseSpace faculty surveys, the SUNY administration also collected feedback in the form of narrative responses from its six members of the MERLOT Editorial Board in the summer of 2005.  The questionnaire was given to Editorial Board members to obtain each member’s perceptions regarding MERLOT’s strengths, advantages, benefits, weaknesses, deficiencies, areas in need of improvement, and opinion regarding SUNY’s continued participation in the MERLOT project.   

The significance of data collected from Editorial Board narratives is due in part, to the members’ unique positions within the MERLOT and the SUNY system.  As board members, SUNY’s editors are change agents in SUNY’s efforts to promote and train faculty how to use MERLOT; while also being ‘adopters’ and users of the innovation themselves. All board members have promoted MERLOT via participation in MERLOT training workshops provided to online faculty and have made presentations aimed at promoting MERLOT in professional conferences nationally or at the annual SUNY Conference for Information Technology (CIT).  The board members’ unique positions therefore allow for both evaluative responses and recommendations for improvements to SUNY’s diffusion plan.   

In the six questionnaires collected, similarities and differences are visible between responses provided by board members and responses collected from the online SUNY faculty via the SLN-CourseSpace surveys.  For example similarities were found when considering usage and views of relative advantage.  Five of the six (83%) editorial board members indicated that MERLOT offers advantages that aid them in their own teaching, research and development.  Several board members’ statements serve as persuasive testimonials that speak directly to the relative advantages of MERLOT: “in my work with graduate students in language and technology, MERLOT is an invaluable resource through which to study instructional design, the peer review process, and the formation of academic community;” and “it offers a single access point to a wide variety of online learning resources from a global consortium: through the Federated Searches I can access materials world-wide.”   These results are similar to responses collected from SLN-Courhe board members’ comments of relative advantage (i.e., usefulness) are similar to patterns found in the online faculty comments, it should be noted that one potential determinant of use and relative advantage is the user’s views of the compatibility of MERLOT’s resources with his/her needs and or interests.  The compatibility of resources is obvious in editors’ comments, “in physics, right now, MERLOT materials serve the purpose of being attendant/supporting materials. These materials often provide simulations, exercises, visualizations. The collections are extensive and straddle all levels, middle school to advanced graduate level;” and “within my discipline area—music education—I find that MERLOT offers access to interactive and multimedia content that has never been available in tetyle="text-align:justify;mso-layout-grid-align:none; text-autospace:none">Perhaps more telling than the SUNY members’ views as adopters of MERLOT; are their “perspectives on the weaknesses, deficiencies, or areas in need of improvement regarding MERLOT”. One editorial board member recognized the need to increase SUNY’s participation rates in using MERLOT. In this response, the member’s comment appears to view MERLOT as a vehicle to be used in creating an individual SUNY learning resource repository that could eventually be incorporated in MERLOT.  The member stated: “there is an urgent need to lower the barriers to participation for SUNY faculty, professional staff, and students by providing incentives and improving our own technology to make it easier to integrate with MERLOT.”  He also suggested, “improvements in overall access might be to establish a SUNY wide e-portfolio solution, a digital image/assets library, a learning object repository or better yet, a broader and all-encompassing learning resource repository…  

When focusing on other editors’ recommendations for improvements of MERLOT, two suggestions standout.  These suggestions are “a more logical topical organization that partitions (in a mathematical/logical sense) a collection should be developed,” and “MERLOT should add a section to its website on the design of effective learning objects.”  In the end, all but one editor found MERLOT useful and recommended SUNY’s continued participation. 

Discussion  

MERLOT has always been one facet in an overall strategy to provide a scaleable and sustainable resource in support of the deployment of online and distributed learning in SUNY.  While no single resource can meet all the needs of the technology-mediated teaching and learning environment of the size and scope of the State University of New York – these results indicate a generally high level of awareness of MERLOT, a high level of use relative to other, similar resources, and a trend line indicating growth or improvement in nearly every indicator of use and satisfaction with this resource.    This was particularly evident in the log files that seemed to indicate some success with regards to the faculty development and training efforts. These logs suggest growth that parallels the activities carried out in the “clarifying stage” to increase awareness of MERLOT among potential adopters.  

The results indicating that faculty were more likely to find MERLOT useful for their online teaching were those who were more experienced suggests that the complexity of online teaching itself represents a barrier to adoption of additional innovations when viewed through the Rogers’ model.  It is common wisdom to advise first time online instructors to “keep it simple”.  The learning curve and the time required to develop the first instantiation of a complete online course may represent such a significant challenge that utilizing MERLOT, and therefore finding it useful at this stage, is unlikely.  

The multivariate CHAID analysis indicating that the most committed online faculty were also the most likely to adopt MERLOT is also suggestive. It appears that this group has the most experience and a greater understanding of the utility of this resource for their online teaching.  The second level CHAID result indicating that those who felt that their online students learn more than their classroom students also suggests that the issues of commitment and relevance are quite important.    It may be more practical to target the diffusion of this innovation to more experienced faculty who already exhibit greater understanding of and faith in technology-mediated instruction. In Roger’s model this is an issue of compatibility of the innovation with the values and norms of the potential adopters, in this case experienced and committed online faculty.  

The comments from the spring and the fall semester can also be interpreted as they reflect stages in Rogers ’ model.  The categories of narrative responses suggest potential adopters in various phases of the adoption process.  Many faculty appear unaware of the innovation and in need of more information to begin the process – reflecting issues of “observability.”  A second group struggles with understanding “relative advantage” with comments that suggested that they had materials already and did not see the need or benefit of additional resources.  Other narrative responses suggest that “triability” is an issue – that respondents understood the potential but had not been able to utilize the resource yet. Another group reflects respondents that had “tried it and liked it” indicating those that had traversed the “implementation” and “confirmation” stages of Rogers ’ model.   Generally speaking, additional effort is required to match the stages of potential adopters with appropriate supports that address specific needs.  The professional development provided should be designed to address each of the particular stages at which faculty reside. This suggests the need for beginning, intermediate and advanced sessions to help faculty move forward in the adoption and use process.  Successful users of MERLOT would assist these efforts, as these users are likely to have a beneficial peer or “near peer” relationship with other potential adopters, in line with aspects of the Roger’s model predicting successful innovation diffusion.  

Limitations of Current Research

This study has several limitations.  The first is the difficulty of obtaining accurate and reliable data on the overall use of MERLOT by SUNY faculty.  While the use of IP addresses allows some tracking, it is an incomplete measure.  

The instrument that was used in this study was a small component of a larger survey of online faculty concerns, and thus is limited in the quantity and quality of data it can provide. There is a need for additional questions  for a closer look at key barriers to adoption, ‘time constraints’ for example are actually considerations of priority – understanding why MERLOT occupies a low priority for these respondents requires additional follow-up items.  Responses to surveys indicating “simply chose not to” which were common, require other methodologies (interviews for example) to elicit more meaningful responses. We need additional research to understand why SUNY users of the MERLOT site, while increasing in number, spend less time at the site.  Does the reduced time reflect more efficient usage or reduced interest? Additionally, more work also needs to be done to understand the adopter profiles – those who had “tried it and liked it” to understand why.   Future research should conduct a more formal review of the system-wide diffusion plan, and its implementation.   Additional data and insight acquired from such further study may facilitate the diffusion of MERLOT, moving this innovation closer to the routinization stage in Roger’s model and helping to achieve the original goals of the SUNY System with regard to their participation in the MERLOT project, and those of the MERLOT organization itself.  

The research conducted here suggests that previous work emphasizing the importance of “stages” of adoption and use of new technologies is justified to some extent. Faculty development needs to be tailored to the stage of concern that potential adopters are likely to exhibit.   However, some of the results also suggest that the stage focus may be insufficient to describe and explain diffusion of innovations such as MERLOT.  Results indicate a recurring theme that technology will be accepted and faculty concerns addressed to the extent that the innovation is perceived to be an efficient and effective solution to a real-world problem embedded in a specific context. Comments suggesting lack of relevant content in the MERLOT collection hint at the “problem-based” focus that future research might take.  The higher level of use of MERLOT by more experienced online instructors found here is also evidence supporting this contextual-application approach in technology adoption.  Future investigators might use the following question as a foundation for research in technology adoption,  

“What is the current, important, individual and/or institutional problem to which innovation X is an efficient, effective, and institutionally supported solution?”  

Gaining a better understanding of both the individual and institutional goals and priorities in which technology choices are embedded will better prepare us to more fully describe and explain diffusion of innovations such as MERLOT. 


References

Dooley, K.E. (1999). Towards a holistic model for the diffusion of educational technologies: An integrative review of educational innovation studies. Educational Technology & Society, 2(4), 35–45. Retrieved August 25, 2005, from http://ifets.ieee.org/periodical/vol_4_99/kim_dooley.pdf.  

Fung, A. C. (1992) Management of educational innovations: The "Six-A" process model. Paper presented at the Regional Conference of the Commonwealth Council for Educational Administration, Hong Kong .

Grunwald, H. (2004). Institutional contexts and faculty traits that predict use of the Web by college faculty teaching in traditional classrooms.  Doctoral Dissertation. University of Michigan . 

Hall, G. E., Wallace, R. C., & Dossett, W. A. (1973). A developmental conceptualization of the adoption process within educational institutions, Austin , TX : Research and Development Center for Teacher Education, The University of Texas.

Hanley, G. (2003). Remarks at the 2004 MERLOT International Conference Opening Session. Vancouver , Canada .

Hamelink, J. (1984). Integration of technology. Performance and Instruction, 23(2), 26-27.

Havelock , M. C. (1973) The Change Agent's Guide to Innovation in Education. Englewood Cliffs, NJ: Educational Technology Publications.

Jacobsen, M.D. (1998).Adoption patterns of faculty who integrate computer technology for teaching and learning in higher education.  Proceedings of the ED-Media and ED-Telecom 98: World Conference on Educational Multimedia and Hypermedia & World Conference on Educational Telecommunications, Freiburg , Germany , June 20-25.

Knuetel, P. (1995). Factors affecting faculty adoption of instructional technology. Unpublished doctoral dissertation, University of Michigan , Ann Arbor .

Multimedia Education Resource for Learning and Online Teaching (MERLOT). (1997). Retrieved October 10th, 2005 from  MERLOT Tasting Room Access: http://taste.merlot.org/initiatives/fac_dev.htm.

Prochaska, J. O., DiClemente, C.C. & Norcross, J.C. (1992). In search of how people change: Applications to addictive behaviors. American Psychologist, 47(9), 1102-1114.

Rogers, E. (1963). Diffusion of Innovations. New York : Free Press

Rogers, E. (2003). Diffusion of Innovations. (5th ed.) . New York : Free Press.

Sane Solutions, LLC., (1996). NetTracker. [Computer Software]. North Kingstown , RI .  Retrieved, August 25, 2005, from NetTracker Web Analytics Solutions Access: http://www.sane.com.

SUNY Learning Network (SLN). (2006). About SLN. Retrieved October 3, 2005, from SUNY Learning Network Access: http://sln.suny.edu/sln_aboutsln.htm.

The State University of New York (SUNY). (1994, June). Short history of SUNY. Retrieved October 3, 2005, from The State University of New York: SUNY  Access: http://www.suny.edu/Student/university_suny_history.cfm.

Zaltman, G., Duncan, R., & Holbek, J. (1973). Innovations and organizations. New York : John Wiley and Sons.

 


Received 31 May 2006; revised manuscript received 5 Sep 2006

Creative Commons License


This work is licensed under a
Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License.

 
Copyright © 2005 MERLOT. All Rights Reserved.
Portions Copyright by MERLOT Community Members. Used with Permission.
Questions? Email: jolteditor@merlot.org
Last Modified : 2005/04/14