Ref
 

MERLOT Journal of Online Learning and Teaching
Vol. 6, No. 2, June 2010

Share on Facebook


A Model for Developing Multimedia Learning Projects

Barbara A. Frey
University of Pittsburgh
Pittsburgh, PA USA
bafrey@pitt.edu

 Jann Marie Sutton
University of Delaware
Newark, DE USA
jsutton@UDel.Edu

 

Abstract

The purpose of this position paper is to present novice faculty and designers with a succinct multimedia development guide based on a review of the literature and a Delphi technique with expert educators, designers, and programmers. After two rounds of review and feedback from a panel of experts, the following steps were identified to comprise a multimedia development model: (1) define the instructional goals, objectives, and audience, (2) review and investigate existing options, (3) determine format, budget, and timeline, (4) determine the content, activities, and assessment strategies, (5) develop evaluation strategies, criteria, and instruments to determine the effectiveness of the project, (6) develop the flowchart, site map, and/or storyboard, (7) develop a prototype, (8) perform a formative evaluation, (9) complete the design, and (10) perform a summative evaluation of product and process. These steps are demonstrated through the development of a specific multimedia project, an Architect’s Scale to measure distances.

Keywords: Cognitive load; Instructional design; Online learning; Learning objects; Computer-based projects


Introduction

Multimedia learning projects are "those that integrate media objects such as text, graphics, video, animation, and sound to represent and convey information [which have] the potential to connect key learning objectives in a prescribed curriculum to real world contexts, integrate diverse curriculum areas, support student decision-making, and foster authentic collaboration" (Crichton & Kopp, 2006, Para. 4). Multimedia learning projects are integrated into a course when current teaching methods do not effectively achieve the learning objectives. Perhaps the laboratory setting for teaching the concepts is too dangerous, the materials are costly, the time frame is unrealistic, or the opportunities for practice are limited (Alessi & Trollip, 2001). Multimedia learning allows for a level of interactivity that can enhance both the online text and traditional lecture-based classroom environments, and is growing at all levels of education.

Is multimedia still relevant in today’s Web 3.0 world?  With Web 3.0 defined as the next generation of Web innovations which combine the social networking capabilities of Web 2.0 with semantic Web data applications (Hendler, 2009), is there still a place in the online educational environment for multimedia learning? As the ability to quickly and efficiently communicate using blogs, wikis, Twitter, You Tube, Google Wave, Flickr, Jing, DimDim, and other tools, does it really matter what multimedia can do in this whirlwind of technological advances?  Despite the potential immediacy of the Web 3.0 world, engaging multimedia content continues to have educational value for practice activities, learner-content interaction, and time on task for the learner. 

Furthermore, instructor-controlled multimedia learning objects provide a level of stability that may be lacking with third-party Web products. Tools and files hosted outside of the institution may not always be accessible. When a site is down or content is lost, the responsibility to retrieve this information lies outside of the institution. From the authors’ experience, technology support is also an issue because many institutions do not have the staff to support the increasingly complex list of third -party Web applications.

The advanced computer skills of many educators and the user-friendly technology tools available at affordable prices make it possible for a growing number of users to envision engaging interactive learning content or even create computer-based multimedia instructional materials. Based on a review of literature, a concise step-by-step model for developing these multimedia programs is not readily available to the academic community. The purpose of this manuscript is to present a succinct multimedia development model based on a review of the literature and experts' experiences as educators, designers, and programmers.

Background: The Need

Within the last decade, we have seen the continued growth and advancement of Internet resources in every academic discipline. Publisher Websites that enhance student learning with resources and multimedia activities are common in many courses. Multimedia programs can provide authentic, safe, cost-effective practice exercises that many faculty members seek for their learners. Content development tools such as Adobe Flash®, Sun’s Java, Microsoft PowerPoint®, video, audio, Captivate®, Articulate®, Softchalk® and many more applications that were once the exclusive domain of computer programmers, are now readily available for many computer-savvy users. Faculty members who struggle with communicating a learning concept often have creative ideas to solve the learning issue and are now empowered to either individually or as part of a team develop multimedia learning content. Small grants are available at many institutions to provide funding for the development of innovative learning projects. Unfortunately, there is not a concise multimedia development model that is readily available to guide educators through the complex development process. Even within business and industry, Barry and Lang (2001) found no uniform approach to multimedia development. Several texts (for example, Multimedia for Learning by Alessi & Trollip [2001] and Multimedia Learning by Mayer [2001]) explain many phases of multimedia development; however, a concise model or set of guidelines that encompasses important aspects of instructional project planning is not available. Moreover, a development model might help faculty evaluate existing multimedia projects they are considering for integration into their courses. For example, does the existing learning object support their instructional goals and objectives? And is it presented in a manner that does not overwhelm the students’ cognitive load?

Literature Survey

For purposes of this position paper, a literature review was conducted to examine existing models for designing and developing multimedia projects. While numerous articles addressed various aspects of multimedia development in higher education, a succinct set of guidelines that depicted the development process for multimedia learning objects is not readily available.

Multimedia Development Models

Designing instruction can be a monumental task, and a model would provide a systematic structure for the process. In one development approach, Stoney and McMahon (1998) adapted Gould’s (1995) model of multimedia development and identified the following four basic phases: (1) information design including the planning of the content and an analysis of the audience, (2) interface design which connects the learner with the content in the most functional and intuitive way possible, (3) navigation to connect the pages of content in a logical structure, and (4) interaction design which determines how the program works and how the learner uses the program. Alessi and Trollip (2001) also identified a four-phase model including (1) presenting information, (2) guiding the learner, (3) practicing, and (4) assessing learning.

Instructional Design Models

Closely related to the multimedia development process are the instructional design (ID) models that offer guidance to practitioners. One of the seminal ID models, developed by Dick, Carey, and Carey (1978; 2007), is a ten-step iterative, interactive process referenced in Table 1. Morrison, Ross, and Kemp (2007) developed a nine-step ID process, described as a "comprehensive instructional design plan" (p. 7). Their model is a learner-centered continuous cycle . From the dozens of ID models, Seels and Glagow (1998) identified five common components and developed the popular ADDIE model with the following five logical steps: (1) analyze, (2) design, (3) develop, (4) implement, and (5) evaluate.

Within the context of ID, Greer (1992) introduced a ten-step ID Project Management Model that includes the following steps: (1) determine project scope, (2) organize the project, (3) gather information, (4) develop the blueprint, (5) create draft materials, (6) test draft materials, (7) produce master materials, (8) reproduce, (9) distribute, and (10) evaluate. ID models help developers focus on the learning content and establish a vision that breaks the materials into manageable chunks of instructional content. In general, ID models focus on the design and development of learning content, and not on larger administrative or management issues such as budgets and staffing. 

Table 1. Instructional Design Models of Dick, Carey & Carey and Morrison, Ross & Kemp

Dick, Carey, and Carey (1978; 2007)

Morrison, Ross, and Kemp (2007)

1. Identify instructional goal(s).

2. Conduct instructional analysis.

3. Analyze learners and contexts.

4. Write performance objectives.

5. Develop assessment instruments.

6. Develop instructional strategy.

7. Develop and select instructional materials.

8. Design and conduct formative evaluation of instruction.

9. Revise instruction.

10. Design and conduct summative evaluation.

1. Identify instructional problems, and specify goals for designing an instructional program.

2. Examine learner characteristics that should receive attention during planning.

3. Identify subject content, and analyze task components related to stated goals and purposes.

4. State instructional objectives for the learner.

5. Sequence content within each instructional unit for logical learning.

6. Design instructional strategies so that each learner can master the objectives.

7. Plan the instructional message and delivery.

8. Develop evaluation instruments to assess objectives.

9. Select resources to support instruction and learning activities.

Software Development Models

Software development models can be informative to the multimedia development process, but lack an educational perspective. Prior to a society with computers in almost every home, Royce (1970) proposed a highly structured sequential development process that became known as the “waterfall” model of (1) requirements/analysis, (2) design, (3) coding, (4) testing, and (5) maintenance. Highsmith (2000) later developed the more flexible adaptive software development model that applies a very different iterative, continuous adaptation cycle consisting of (1) speculate, (2) collaborate, and (3) learning. Two components that are clear and consistent in the phases of software development and that may be transferred to multimedia development are (1) the need for careful analysis of the end user and (2) continuous testing of the product.

Multimedia Learning

Among the articles that reference multimedia production, Clark and Mayer's (2003) have significantly contributed to the understanding of how the cognitive processes of interpreting visual and auditory information can affect learning. Their guidelines (see Table 2) are integral to the development of multimedia learning objects with regard to how to incorporate images, audio, video, written and spoken narration. With this background knowledge, practitioners can critically analyze the components of multimedia in order to prevent an overload of cognitive stimulation which can impede learning (Clark & Mayer, 2003).

In addition to the learning considerations, there is some evidence that multimedia instruction may be more motivating, interesting, and accessible at anytime and anyplace than conventional learning methods  (Zheng, 2009; Astleither & Hufnagl, 2003). Furthermore, Bernard and others (2004) reported learners who were actively engaged in their coursework were motivated to achieve at higher levels than learners who were not engaged. To increase this engagement, Lui, Toprac, and Yuen (2009) identified the following five factors as contributing to intrinsic motivation: (1) problem solving, (2) having fun or playing, (3) information processing, (4) self control or voluntary acting, and (5) socializing.

Table 2: Six Media Principles from E-Learning and the Science of Instruction (Clark and Mayer, 2003).

PRINCIPLES

DESCRIPTIONS

Multimedia

People learn more deeply from words and graphics than from words alone.

Contiguity

When corresponding printed words and graphics are placed close to one another on the screen or when spoken words or graphics are presented at the same time.

Coherence

People learn more deeply from multimedia lessons when distracting stories, graphics, and sounds are eliminated.

Modality

People learn more deeply from multimedia lessons when graphics are explained by audio narration rather than onscreen text.

Redundancy

People learn more deeply from multimedia lessons when graphics are explained by audio narration alone rather than audio narration and onscreen text.

Personalization

People learn more deeply when the speaker uses conversational style rather than formal style.

Methods

The authors drafted a 12-step guide for multimedia development (Frey & Lightcap, 2008) based on a literature review and their experiences as instructional designers and educators. To further develop and refine the 12-step guide, a modified Delphi technique was used to obtain group feedback regarding the model or process (Skulmoski, Hartman & Krahn, 2007). The 12-step guidelines were distributed to ten multimedia development experts who were highly experienced practitioners working in higher education. Most of the experts had ten years or more experience in project management, multimedia development, Web or graphic design, and/or computer programming. The experts reviewed the steps with descriptions, and then provided feedback on each step (Babbie, 1995). The team of experts provided suggestions on how to clarify, consolidate, and improve the development process. The authors revised the guidelines based on the experts' comments, and re-distributed the document for final feedback. The proposed multimedia model encompasses the development of the entire project from start to finish, which is broader than previous instructional design models. As a result of this Delphi process, the final guidelines consisted of the following 10-step Multimedia Development Model:

Step 1 Define the instructional goals, objectives, and audience.

Step 2: Review and investigate existing options.

Step 3: Determine format, budget, and timeline.

Step 4: Determine the content, activities, and assessment strategies.

Step 5: Develop evaluation strategies, criteria, and instruments to determine the effectiveness of project.

Step 6: Develop the flowchart, site map, and/or storyboard.

Step 7: Develop a prototype.

Step 8: Perform a formative evaluation.

Step 9: Complete the design.

Step 10: Perform a summative evaluation of product and process.


Proposed Multimedia Development Model

The proposed ten steps provide a foundation for faculty to incorporate multimedia education into their teaching resources. Detailed considerations within each step are provided to help educators accomplish the complex development process. 

Step 1: Define the instructional goals, objectives, and audience.

An essential element of any multimedia project is to begin with clear pedagogical goals and objectives. Depending on the scope of the project, the goals and objectives should not be expansive, but rather specific with clear measurable outcomes. However, this well known step of any instructional project can sometimes be overlooked in the excitement of developing an innovative and technologically dynamic project. It is critical that this step be reviewed during every decision and every phase of the project.

Identifying the goals will help the faculty member define what the students should learn and understand, while the objectives provide specific and measurable outcomes (Gagne, Wager, Golas & Keller, 2005). Additionally, knowing and understanding the needs of the audience is crucial to having an effective learning experience. Ensuring that all three of these components--goals, objectives and audience analysis-- are addressed will confirm that the final project meets the intended outcome.

The following questions will help to facilitate this step:

  • What should students be able to do or to master when they complete the activity?

  • Are there prerequisites to this skill set?

  • Should prerequisites be noted or included in this activity?

  • Who is the target audience for this project?

  • Are there physical or cognitive limitations of the student audience that need to be considered?

  • What is the typical connectivity and level of computer experience of this group?

Step 2: Review and investigate existing options.

Before spending a great deal of money or investing time in creating a multimedia project, it is critically important to complete a thorough review of existing applications. A learning program may be discovered that meets most of the required educational objectives. A multimedia learning object may also exist that could serve as a template for the newly proposed project. For example, the Cameron Balloon Factory (http://courses.chalkface.com/cameron/welcome.htm) business simulation served as a helpful guide for an instructor creating a virtual health care facility. It helped her to visualize the various departments and to integrate case studies with images and video. Researching the many resources that are currently available is a way to gain insight into the proposed project, providing guiding examples as well as what to avoid. See the Appendix for an adapted list of popular learning objects repositories.

It is important to note that if an existing learning object is found that might achieve the desired learning outcomes, a thorough review of the project should take place. Many of the steps of this model can be used as a guide to ensure that the newly discovered project is congruent with the instructor's or institution's needs. In particular a summative evaluation as described in Step 10 of this model can aid in confirming its validity. Dick, Carey & Carey (2001) describe that summative evaluations can be used to evaluate a single instructional project or several.

Consider the following questions during Step 2:

  • Has the same project or a similar one already been created?

  • Has the team checked MERLOT and other learning object repositories?

  • Are there existing resources available from the textbook publisher?

Step 3: Determine format, budget, and timeline.  

Experienced multimedia production experts often state that projects can be produced in a manner that is “fast, cheap, or good” (Brown, Green, & Zatz, 2001, p. 28), but it is rare to have all three qualities in one project. It takes time to develop and test programs, and shortcuts often hinder the overall quality.

Multimedia learning objects may be simulations, games, puzzles, problem-based learning activities, tutorials, presentations, animations, case studies, or assessments. The format of a project will depend upon the learning goals and resources. Developers must consider issues such as student equipment and computer skills, production budget and staff availability, and bandwidth and server requirements. The projects may be Web-based, created for mobile applications, or distributed on a CD-ROM or DVD.

Defining a budget or the estimated costs for a project may be referred to as an art form because it blends knowledge, experience, and creativity. Hourly estimates for the production team, escalating , competing high priority projects, changes in staff and team members, and advancements in the tools and resources are challenging to determine. In education, most budgets costs for development focus on the direct costs or costs such as labor, equipment, and materials needed for the project. Because of the many considerations required for a comprehensive budget and timeline, it should be a team effort with members who actually complete the tasks providing the cost estimates. It helps to review previous projects to determine performance and costs. 

The following questions will contribute to a justifiable budget that is realistic:

  • Does the project fit with the budget and time frame?

  • Who is needed on the production team? How much time will each team member spend on specific areas of the application?

  • What are the areas of risk or where is the budget likely to face unforeseen issues/barriers?

  • What applications, equipment, technology, or server space is needed to create the project?

  • Will the project be a face-to-face activity, Web page, DVD, video, CD-ROM, or another digital format?

  • If project costs become higher than the budget allowances, how can the team reconcile project and budget costs? 

Step 4: Determine the content, activities, and assessment strategies.

Although educators have seen a transformation in the methods and technologies used to deliver instruction, how people learn remains constant. Effective multimedia projects demonstrate alignment or a clear connection between the content, activities, and assessments. Gagne’s (1985) Nine Events of Instruction model is frequently used to plan an instructional multimedia intervention including: (1) gain attention, (2) inform learners of objectives, (3) stimulate recall of prior learning, (4) present the content, (5) provide guidance, (6) elicit performance (practice), (7) provide feedback, (8) assess performance, and (9) enhance retention and transfer. Multimedia projects likely begin with a foundation of information presented in text, audio, video, or image format and chunked into manageable components to support the learning objectives.

In addition to the specific content presented in a multimedia project, the design of learning material requires attention to cognitive functioning and overload. Clark and Mayer (2003) note that one of the challenges of multimedia learning projects "is to build lessons in ways that are compatible with human learning processes. To be effective, instructional methods must support these processes. That is, they must foster the psychological events necessary for learning" (p.30).

Mayer’s six principles from the cognitive theory of multimedia learning (see Table 2) can be applied to most multimedia projects thus guiding the development of effective learning. For example, the contiguity principle advises designers to keep scrolling to a minimum, place the words near their corresponding images, and keep instructions near activities. The coherence principle eliminates extraneous images, sounds, and text that do not contribute to the learning. The critical reason to pay close attention to these principles relates to the important distinction between “information delivery theory” and “cognitive theory of multimedia learning.” Mayer (2001) proposes that the cognitive theory of multimedia learning addresses the competing nature of the dual channels (auditory and visual) for processing information in the construction of knowledge in contrast to basic information delivery theory which focuses on the methods of delivery (e.g. audio, video, text). In other words, designers want to use the most effective medium for delivery of information and also to balance the demands placed on the learner’s auditory and visual channels of communication processing.

With awareness of these cognitive limitations, multimedia developers may incorporate interaction in the form of making decisions, analyzing problems, constructing knowledge, or practicing skills or concepts. Designing interaction is challenging because the developer is limited to keyboard and/or mouse manipulation--clicking, dragging, dropping, or typing. Along with the student-to-computer interaction, constructive feedback is also an integral component of the instructional design process. Ultimately, the feedback should enhance learning and lead to mastery of the concepts in future practice and learning attempts (Alessi & Trollip, 2001).

The following questions contribute to the effective design of content, activities, and assessment:

  • How can the team align activities and assessments with the learning objectives?

  • Will the developers provide multiple examples of major concepts?

  • How can the developers use Gagne’s (1985) events to assist students with performance and retention?

  • How do Mayer’s principles apply to the development of the multimedia module?

  • Does the project include an introduction, guidance, practice, feedback, and assessment?

Step 5: Develop evaluation strategies, criteria, and instruments to determine the effectiveness of the project.

"Evaluation examines the effectiveness of the instruction by considering how well the outcomes, assessments, and activities are aligned within the instruction and whether they are appropriate for the needs and characteristics of the learners" (Cennamo & Kalk, 2005, p. 124). When the assessments are drafted and the formation of the module is taking shape, the instructor must consider how to measure the overall effectiveness of the project. While the assessments will measure the student's understanding of the specific content, how can it be determined if the actual teaching strategy is the stimulus for the learning effect?

A well designed evaluation can confirm the value and validity of the tool. Does it achieve the intended objectives and how will its effectiveness be confirmed? Creating the broader assessment criteria in the early stages of the project can inform the development process.

Pretesting, embedded testing, and post-testing are strategies to determine the effectiveness of any learning experience. By assessing the same skills with different questions at three different points in the learning process, an educator can determine the effectiveness of the teaching model. The pre-test determines current understanding and stimulates prior knowledge, and the embedded test can help the learner practice and transfer knowledge to confirm that learning is taking place. Additionally, checklists, journals, and rubrics can also be of great value in determining learning (Cennamo & Kalk, 2005).

Throughout the development of the project, assessment strategies should be analyzed. If the project is not meeting the set objectives, it should be reviewed for its viability. Consider the following aspects of planning evaluation strategies:

  • Will the project include pre and post testing?

  • How will the team determine the effectiveness of the multimedia module?

  • What outcomes need to be measured to confirm the validity of this module?

  • If the project was not effective, were there enough opportunities for learning transfer to occur?

  • Do the developers need to reconsider the content, activities, and assessment strategies (Step 4)? 

Step 6: Develop the flowchart, site map, and/or storyboard. 

Flowcharts, site maps, and storyboards are visual methods which present a clear overview of the content. They are a screen-by-screen demonstration of the program which promote communication and clarity among members of the development team. The flowchart or storyboard confirm each slide's purpose and relationship to the learning process. More specifically, the flowchart helps designers to plan the audio, text, graphics and/or user interaction on each screen (Cennamo & Kalk, 2005).

Flowcharts can bring the learning project to life by demonstrating navigation within the program. Multimedia programs are not limited to a linear presentation of content; therefore, the branching and content pathways can become extremely complex. These visual tools provide team members with information about the development process and identify decision-making points. Storyboards, flow charts, and site maps can be created using a variety of methods ranging from software tools specifically designed for each of these purposes to yellow sticky notes organized on a bulletin board.

Review the questions below regarding the organization of a storyboard:

  • What will the project look like?

  • What learning events need to happen and when?

  • How can the developers organize the progression of content to achieve the learning objectives?

  • What is the basic layout of the project and does it require branching?

  • Is the navigational structure clear and concise for the learner?

Step 7: Develop a prototype. 

A prototype is a draft or working model of the project or some portion of the project that helps the team to apply and test ideas before too much time and money are invested into the development process. At the very lowest and simplest level, the prototype could be a storyboard of interactive pages and at the most advanced level it might be a full-functioning Website. During the prototype phase, developers use performance testing to consider both the effectiveness of the content in achieving the learning goals and the usability of the technology.  

As the project progresses, the interface or screen design (e.g., images, graphics, fonts, color, and layout) should be visually attractive and uncluttered. Team members test the look and performance with different browsers, display settings, and bandwidths. In a consistent and intuitive navigation structure, menus, and maps allow students to know where they are in the program. As part of the formative evaluation plan, this is an opportunity to pilot the program with real learners. It is possible that there will be several iterations of the prototype in a major project. The prototype serves as a model and the last chance to make changes (Cennamo & Kalk, 2005).

The following questions are considered in the prototype phase:

  • Is the application easy to use and is the navigation intuitive?

  • Is the screen interface visually attractive and clear?  Are colors appropriate?

  • Are the font sizes and labels legible?

  • Does the project require the branding of the school?

  • Does the prototype need to be chunked into smaller segments?

  • Does the prototype fully meet the team student needs?

  • What must be changed to make it work or perform more effectively?

Step 8: Perform a formative evaluation.

The purpose of a formative evaluation is to collect data and information from the target audience as the project is developed. "Formative decisions are those in which the results of the evaluation provide guidance on how to improve instructional materials or the procedures by which they are designed and developed" (Gagne et al., 2005, p. 38). The formative process should include the feedback of several types of stakeholders from the student to the content expert and client (Cennamo & Kalk, 2005).

Gagne et al. (2005) propose five types of evaluation that can be employed during both the formative and summative phases of the project: (1) materials evaluation, (2) process evaluation, (3) learner reactions, (4) learner achievement, and (5) instructional consequences. Survey tools can be created based on these categories, which can then be used during the collection of user data. According to Dick, Carey and Carey (2001), there are three phases of obtaining information: (1) a clinical phase (one-on-one), (2) a group phase (small groups), and (3) a field study (up to 30 participants).

Obtaining feedback is a critical step in developing a project that will not only achieve its learning objectives but also be intuitive to the user. It is important to determine how the student perceives the instructional aids and how the navigational prompts are interpreted. This step can be time-consuming and difficult to organize; however, the benefit to the learner is immense and should not be overlooked.

The following questions may be addressed during the formative evaluation process:

  • Will an external reviewer check the facts in the project?

  • Will an editor review the content for grammar, clarity, tone, and reading level?

  • Will students perform a beta test to provide valuable formative feedback?

  • Has the prototype been tested (and retested) with students?

Step 9: Complete the design.

When team members are satisfied with the working prototype, the project is ready to advance to its final form as a fully functioning learning application. This involves completing the video, audio, graphics, photographs, animations, assessments, activities, and any other instructional materials. Completing the product may mean contracting with professional actors, narrators, or videographers to obtain the skills needed for the project. Sometimes, there are additional support resources such as a student user's guide or instructor's guide.

Once all of the components are produced, they must be assembled. Version control for multiple project drafts and time management issues are challenges during this completion phase. Care must be taken to name, re-name, and back-up each version of the project (Alessi &   Trollip, 2001). Time management becomes an issue as final details are often tedious and time consuming. 

In Step 9, team members consider the following questions:

  • What is needed to complete the project?

  • Were the design and development objectives accomplished?

  • How can the design team keep the project within its original scope?

  • Does the team have a standard naming convention for labeling multiple versions of the evolving project?


Step 10: Perform a summative evaluation of product and process.

"Summative evaluation leads to decisions about the worth and value of a product or activity" (Gagne et al., 2005, p. 38). While formative evaluations occur during the creation of the project, summative evaluations occur after the early issues and problems of the project are resolved (Cennamo & Kalk, 2005). This step confirms that the objectives have been achieved. Did the project resolve the learning dilemma that prompted its development (Dick, Carey & Carey, 2001)? 

Summative or final evaluation is divided into two phases: an expert review and a field study (Dick, Carey & Carey, 2001). It is recommended, when possible, to have a third party conduct the summative evaluation (Cennamo & Kalk, 2003). What seems obvious to the developers may not be apparent to a new reviewer who can provide an objective perspective on the project. In the second phase, the proposed field trial provides feedback directly from the targeted learning audience. One of the key concepts of an expert review and field study is to confirm that the needs of the instructor or organization are being achieved.

If the formative evaluation process is used effectively, the anticipated effect is that the summative evaluation will obtain positive results. Additionally, establishing the project's validity is important to the integrity of the developers' work as well as to the integrity of the educational process.

When preparing for the summative process, review these questions:

  • Did this multimedia project solve the teaching and learning problem?

  • How will the team assess the effectiveness of the project and to what degree?

  • Will the developers assess content, achievement of learning goals, satisfaction of the learner, functionality, and design?

Application of Multimedia Development Model: Architect’s Scale Project

The Architect’s Scale (See Figure 1) developed at the University of Delaware demonstrates an effective project piloted with the proposed multimedia development model. Dr. Bruck teaches undergraduate students how to accurately use an architect’s scale for her face-to-face course, Introduction to Landscape Design.

Figure 1: Image of the Architect's Scale demonstrating how to use the scale to measure
the length of a line.


Becky Kinney, University of Delaware technical designer, and Professor Bruck collaborated to create the tool for students to learn the fundamentals of the scale with practice exercises to reinforce student learning.

Figure 2: Image of Architect’s Scale exercise showing assessment and feedback
incorporated into the learning process.

 


 Figure 3: Image of Architect’s Scale depicting a measurement exercise.

Table 3 provides an overview of how the multimedia development model was applied to the scale. The complete timeframe for the project was about three months.

Architect's Scale Interactive Learning Module 

Table 3: Summary of the proposed ten-step model applied to the Architect’s Scale.

Multimedia Development Model Steps

Architect’s Scale Project Development

Step 1  Define the instructional goals, objectives, and audience.

Professor Bruck found that although she spent a great deal of time in class teaching students how to use the Architect's Scale, many students continued to have difficulty understanding how to effectively use this tool. Students expressed frustration and would then stop asking for clarification, presumably because they were embarrassed among their peers.  

Her objective was to find or create an online interactive tool to reinforce the Architect's Scale foundations she taught in class. Equally important was her objective to provide learning opportunities for her students to practice these concepts at their own pace without peer classroom pressures and time constraints.

Step 2: Review and investigate existing options.

Professor Bruck and her colleagues reviewed Web resources and determined that an online exercise did not exist that met her objectives. 

Step 3: Determine format, budget, and timeline.

Professor Bruck applied for a technology grant to secure time from an internal resource at The University of Delaware called the PRESENT.

The project team consisted of an instructional designer, a technical designer, and the faculty member with a proposed timeline of 3 months. The decision to utilize a Flash-based format was easily agreed upon due to the functionality that the faculty member envisioned.

Step 4: Determine the content, activities, and assessment strategies.

Professor Bruck required that the multimedia application demonstrate several learning exercises for the students to observe how the measuring functions of the Architect's Scale are applied. She also wanted to provide measuring simulations using the scale (See Figure 1) and practice exercises to allow the students to reinforce their learning. It was not required that this project record student progress for grading purposes, but rather was used as a self-paced, learner controlled practice module. Immediate feedback on performance was provided to students. See Figure 2. A glossary was also added for easy student access.

Step 5: Develop evaluation strategies, criteria, and instruments to determine the effectiveness of the project.

The technical designer through a series of meetings with the faculty member interpreted the in-class exercises and learning content to translate them into a Flash-based multimedia module. Because Professor Bruck wanted this tool to be self-assessing to allow the students to control the pace of their learning in a private setting, student evaluations were conducted in-person to determine the tool’s effectiveness.

Step 6: Develop the flowchart, site map, and/or storyboard.

Through several meetings the technical designer and the faculty member discussed ideas and concepts of how to present the Architect’s Scale to students. Emails and rough drafts were exchanged until the initial concept was approved.

Step 7: Develop a prototype.

The technical designer created the prototype which was refined several times at this stage. Then the technical designer and the faculty member had to agree upon the functionality of a moveable scale that students could manipulate. The student needed to be able to drag the scale to various points to answer questions and then re-set the scale for the next question. See Figure 3.

Step 8: Perform a formative evaluation.

After the prototype had been refined, the Architect's Scale was again evaluated by the design team for ease of navigation, learning opportunities, and overall ease of use. It was confirmed that additional audio instructions were needed for clarification at certain points in the project. Various evaluation strategies were discussed and it was decided that the initial effectiveness of the tool would be measured by developing an in-class survey to obtain immediate feedback.

Step 9: Complete the design.

The design was completed after the formative evaluation results were applied. Because of the scope of the project, all multimedia production remained within The University of Delaware.

Step 10: Perform a summative evaluation of product and process.

Students’ tutorial scores were automatically collected through an email account. Adjustments were made based on the student scores. For example, the display for student correct/incorrect responses was modified to provide immediate feedback.

Lessons Learned

The Delphi technique using a panel of experts to review the multimedia development model proved to be an effective means to organize the planning and structured development process with a progression of concrete tasks. The expert panel agreed that a strong foundation must be established prior to creating any multimedia components. The ten-step model is not a straight forward progression, but a blended, iterative process. Some steps can be completed simultaneously and other steps will likely be revisited. The larger the project, the more important it is to have a structured model, clear expectations, and consistent communication among the team members.

As the project develops, the faculty member or client's vision for the project often changes. Midway through the project, the instructor or client might propose modifications, enhancements, or additions that ultimately improve the student's learning experience, but also change the scope of the project. The final version of the multimedia application is not always clear--thus, the popular term "project creep" which refers to a gradual change in scope. Having a firm multimedia development model in place may help to define the process for less experienced team members including the instructor.

With additional application and review, the authors believe it may be possible to combine some of the steps. For instance, if the formative and summative evaluation steps were combined, would the user better understand the relationship between these two concepts?

More research is needed regarding the process of developing multimedia projects. Perhaps there should be adapted development models based on the type of project (e.g., tutorial, simulation, and game) or based on the technology tools used to create the application. Additionally, new mobile technologies are growing in popularity and may demand modifications to the model. Accessibility for students with physical and cognitive limitations continues to be a critical issue for developers of online material. More research is needed in the area of accessibility design issues in multimedia applications. The tools to create accessible applications are not always readily available, cost effective, or easy to use. Designers need to be aware of developing alternate versions of accessible material early in the design process.

Conclusion

Multimedia learning projects in the classroom are not a new phenomenon, but advanced computer technology tools allow multimedia projects to be developed with enhanced interactivity, greater ease, and less cost than ever before. There is no uniform approach for faculty who are new to the design and development of multimedia projects, but there is a logical structured approach. The proposed ten-step model leads novice faculty and designers through the development process. This model does not suggest that development is a clean, linear process; on the contrary, it is an iterative, flexible, problem-solving process. Development is presented as a team effort which constantly strives to balance the capabilities of the technology with the needs of the learner and content. Effective multimedia projects evolve through a consistent planned effort based on a measurable learning outcome. Due to advances in cognitive science research, designers can develop learning projects that consider the complexities of cognition with various media. Technology is changing and students are changing, the methods for delivering education must change as well.

Acknowledgments

Expert Delphi Panel:

David Holzemer, E-Learning & Business Training Specialist, Carnegie Mellon University
Bill Johnston, Manager Rich Media Production, University of Pittsburgh
Denise King, Instructional Technologist, University of Pittsburgh
Becky Kinney, Multimedia Developer, University of Delaware
Ruth Newberry, Director of Educational Technology, Duquesne University
Nancy O’Laughlin, Instructional Designer, University of Delaware
Holly Schifflet, Associate Director of Online Programs, University of Pittsburgh
Karen Girton-Snyder, Graduate Assistant, University of Delaware
Timothy Snyder, Graduate Assistant, University of Delaware

Additional Support:

Jules Bruck, Assistant Professor, Department of Plant and Soil Sciences, University of Delaware
Millie Sass, Research Assistant, University of Pittsburgh

References

Alessi, S.M. & Trollip, S.P. (2001). Multimedia for learning: Methods and development. Boston, MA; Allyn and Bacon.

Astleither, H. & Hufnagl, M. (2003). An aptitude-treatment-interaction-approach on motivation and student's self-regulated multimedia-based learning. Interactive Educational Multimedia, 13, pp. 11-23 Retrieved February 13, 2010 from http://greav.ub.edu/iem/index.php?journal=iem&page=article&op=view&path[]=90&path[]=113

Babbie, E. (1995). The Practice of Social Research (7th Ed), New York: Wadsworth Publishing Co.

Barry, C. & Lang, M. (2001) A survey of multimedia and web development techniques and methodology usage. IEEE Multimedia. 8(3), 52-60.

Brown, A., Green, T. & Zatz, D. (2001). Guidelines for multimedia production projects, Educause Quarterly, 2 (4), 26-29.  Retrieved January 29, 2010 from http://net.educause.edu/ir/library/pdf/eqm0142.pdf

Cennamo, K. & Kalk, D. (2005). Real world instructional design. Belmont, CA: Wadsworth/Thompson Learning.

Clark, R. C., & Mayer, R. E. (2003). E-learning and the science of instruction. San Francisco, CA: Jossey-Bass.

Crichton, S. & Kopp, G. (2006). Multimedia technologies, multiple intelligences, and teacher professional development in an international education project. Innovative Online, 2 (3), Retrieved December 13, 2009 from http://www.innovateonline.info/pdf/vol2_issue3/Multimedia_Technologies,_Multiple_Intelligences,_and_Teacher_Professional_Development_in_an_International_Education_Project.pdf

Dick, W. Carey, L. & Carey, J. (2001). The systematic design of instruction. New York: Addison-Wesley Educational Publishers Inc.

Frey, B.A. & Lightcap, J.M. (2008, November). Multimedia projects: Build it right and they will come. Sloan-C International Conference on Online Learning, Orlando, FL.

Gagne, R. (1985). The conditions of learning (4th ed.). New York: Holt, Rinehart & Winston .

Gagne, R., Wager, W., Golas, K., & Keller, J. (2005). Principles of instructional design.  Belmont, CA: Wadsworth/Thompson Learning.

Greer, M. (1992). ID project management: Tools and techniques for instructional designers and developers. Amherst, MA: HRD Press.

Gould, E.J. (1995). Empowering the audience: The interface as a communication medium. Interactivity, 1 (4), 86-88.

Hendler, J. (2009). Web 3.0 Emerging, Computer, 42 (1), 111-113.

Highsmith, J.A. (2000). Adaptive software development: A collaborative approach to managing complex systems. New York: Dorset House Publishing.

Moreno, R. (2006). Learning in high-tech and multimedia environments. Current Directions in Psychological Science, 15 (2), 63–67.

Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press.

Liu, M., Toprac, P., & Yuen, T. (2009). What factors make a multimedia learning environment engaging: A case study. In R. Zheng, (Ed.) Cognitive Effects of Multimedia Learning (pp. 173-192). Hershey, PA: Idea Group Inc.

Morrison, G. R., Ross, S. M., Kemp, J. E., & Kalman, H. K. (2007).  Designing effective instruction. (5 th Ed). Hoboken, NJ: John Wiley & Sons, Inc.

Royce, W. (1970), Managing the development of large software systems, Proceedings of IEEE WESCON. Retrieved February 12, 2010 from http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/waterfall.pdf

Skulmoski, G.J., Hartman, F.T. & Krahn, J. (2007). The Delphi method for graduate study. Journal of Information Technology Education, 6. Retrieved April 22, 2010 from http://informingscience.org/jite/documents/Vol6/JITEv6p001-021Skulmoski212.pdf

Stoney, S. & McMahon, M. (1998). An alternative model of multimedia development: Small projects within an academic environment. ED-MEDIA/ED-TELECOM 98 World Conference on Educational Telecommunications Proceedings, Freiburg, Germany.   ED 428727

Zheng, R.Z., Cognitive effects of multimedia learning. Hershey, PA: IGI Global.


 

Appendix. Learning Object Repositories

NAME

URL

SUBJECT(S)

MATERIALS

NOTES

1. BioDiTRL Biological Digital Teaching Resource Library

http://bio-ditrl.sunsite.ualberta.ca/

Biology

9,000+

Digital media: images, animations, video clips, text

Requires free registration for full access

2. DLESE: Digital Library for Earth System Education

 

 

http://www.dlese.org/dds/histogram.do?group=subject

Earth Science

13,000+

Lesson plans, maps, images, models, data sets, activities, visualizations, assessment curriculum, online courses

K-12 and Higher Education

 

3. Exploratorium Digital Library

 

 

http://www.exploratorium.edu/educate/dl.html

http://www.exploratorium.edu/imaging_station/gallery.php?Section=Introduction

Science

Two databases: (1) Digital library and museum materials and (2) Microscope images

Teaching resources including images, activities, PDFs, video, and audio files

All levels

Microscope images are searchable

4. FLORE: French Learning Object Repository for Education

http://flore.uvic.ca/welcome.php

French

1000+

Lessons in culture, grammar, history, writing, vocabulary, games, and other areas

Advanced search function

5. Fathom

 

http://www.fathom.com/

 

Archive of global educational resources on a variety of topics

100+

Lectures, articles, interviews, exhibits and free seminars from libraries, museums, universities, and film institute

Search by institution or topic

Project discontinued in 2003, but archive available

6. General Physics Java Applets

http://www.surendranath.org/

Physics

60+

Applets

Easy to use menus

Last updated 2006

7. HEAL: Health Education Assets Library

 

http://www.healcentral.org/index.jsp

 

 

Health Sciences

24,000+

Medical education images, tutorials, videos, PowerPoint presentations, and quizzes

Detailed meta data Search by subject or collection

Peer reviewed

8. Illumina: National Science Digital Library

http://www.ilumina-dlib.org/

National Science Foundation

University of North Carolina Wilmington

Undergraduate Math, Biology, Computer Science, Chemistry & Physics

Applets, videos, images, lessons

Search by subject or collection

 

9. INTUTE UK

 

http://www.intute.ac.uk/

Wide variety of topics

120,000+

Lessons, images, Applets, articles by British specialists

Search by keyword, subject, or resource type

10. MERLOT*

 

https://www.merlot.org/Home.po

https://www.merlot.org/merlot/index.htm

 

Wide variety of topics

21,000+

Simulations, tutorials, case studies, assessments, presentations, websites, Applets, and sample assignments

Search or browse

Peer reviewed

Registration required to submit materials

 

11. NEEDS Digital Library

 

http://www.needs.org/needs/

Engineering

Science & Math

Lessons, projects, simulations, demos, games, case studies

Easy-to-use search fields

Peer reviewed

12. Physics Education & TechnologyPhETU. of Colorado

http://phet.colorado.edu/web-pages/index.html

Physics, Chemistry, Biology, Earth Science and Math

 

Simulations using Java and Flash

Lessons and notes

 

13. SMETE

http://www.smete.org/smete/

Science, Math, Engineering, Technology Education

6000+

Wikis, courses, applets

Search or browse

All levels

 

14. Statistics Online Computational Resource (SORC)

http://www.socr.ucla.edu/SOCR.html

 

Probability and Statistics Education

Applets, simulations, resources, e-book

 

Browse by topic

 

15. Tufts Open Courseware

http://ocw.tufts.edu/

 

Life Sciences

45+

Online courses

 

Complete course with syllabus, content, quizzes

16. MIT OpenCourseWare

Reflects topics from graduate and undergraduate courses at MIT

1900+

Online courses

Complete courses with syllabi, lectures, problem sets, and exams

17. University Channel Princeton University

http://uc.princeton.edu/main/

 

International and Public Affairs

 

Video and audio recordings of lectures, seminars, panels and interviews from academic institutions

Menu font size indicates collection size

18. Digital Research Library

D-Scribe

http://www.library.pitt.edu/dscribe/az.html

Variety of topics

1000+ digital resources

Images from historic books and other requests from faculty

Growing collection of special projects

 19. Connexions

 

 http://cnx.org/

 

Variety of topics

15,000+

Educational material in reusable modules

 Primarily text-based modules that can be incorporated into courses

All levels

20. Carnegie Mellon University Open Learning Initiative

http://oli.web.cmu.edu/openlearning/

 

Chemistry, Economics, Physics, Engineering, and French

Tutoring , virtual laboratories, and simulations, with assessments

Self-contained online courses

21. Wisc-Online

 

 http://www.wisc-online.com/

Wide variety of topics

2000+

 Multimedia learning objects

Developed by faculty for online courses

 


Manuscript received 14 Feb 2010; revision received 26 Apr 2010.




This work is published under a Creative Commons Attribution-Non-Commercial-Share-Alike License

For details please go to: http://creativecommons.org/licenses/by-nc-sa/3.0/us/


   
Copyright © 2005-2010  MERLOT. All Rights Reserved.
Portions Copyright by MERLOT Community Members. Used with Permission.
ISSN: 1558-9528
Questions? Email: jolteditor@merlot.org
Last Modified : 2010/06/15