Using a Video Game as an Advance Organizer: Effects on Development of Procedural and Conceptual Knowledge, Cognitive Load, and Casual Adoption

Jennifer J. Vogel-Walcutt
Senior Scientist
Cognitive Performance Group, Inc.
Orlando, FL 32828 USA
jennifer@cognitiveperformancegroup.com

Katie Del Guidice
Research Associate
Design Interactive, Inc.
Oviedo, FL 32765 USA
katie@designinteractive.net

Logan Fiorella
Doctoral Student
Department of Psychological and Brain Sciences
University of California, Santa Barbara
Santa Barbara, CA 93106 USA
logan.fiorella@psych.ucsb.edu

Denise Nicholson
General Manager
MESH Solutions, LLC – A DSCI Company
Orlando, FL 32826 USA
dnicholson@dsci.com

Abstract This paper reports on a study aimed at investigating whether a video game can act as an advance organizer for teaching a military call for fire task in order to improve learning efficiency. Participants were 23 males and 45 females, randomly assigned to one of four conditions. Conditions were developed by a 2 x 2 matrix using video games with and without advance organizers to enhance decision-making skills. Participants completed two increasing levels of training that were followed by posttests in several areas. Results indicate that commercial, off-the-shelf video games do not appear to increase learning effectiveness. However, individuals who utilized an educationally relevant video game prior to learning reported more interest in continuing to learn. Unfortunately, they also reported investing higher cognitive load to acquire equal knowledge compared to the traditional outline advance organizer. These data support recent findings that suggest that ill-structured game-based learning environments can impede learning outcomes due to the extraneous cognitive load imposed by gaming elements. Keywords: advance organizer, video game, knowledge acquisition, cognitive load, casual adoption

Introduction and Literature Review

Military training is being revolutionized by research in simulation technologies. Realistic virtual environments and computer-based simulation training systems allow soldiers and recruits to train in conditions similar to those encountered on the battlefield and can help soldiers to maintain a high level of performance on crucial tasks outside of classroom training time (Macedonia, 2002, 2005). Despite advancements in training technology, the overhead required to train individuals on how to use and integrate military simulation platforms into instructional curricula remains high, often increasing the time needed to train a soldier on any one task (Ally, 2008). A potential solution to this issue is to capture non-instructional time to prepare students for their time in a simulator, thereby reducing the total instructional time needed for training. Therefore, the use of game-based learning may increase a trainee's desire, or motivation, to learn prior to formal learning in the classroom, and because the game provides a simulated context, the individual's ability to develop complex schema in long-term memory (LTM) may be enhanced. Consequently, learning efficiency, or the ability to understand more using less formal learning time, may be improved. In this paper, "training" refers to the acquisition of information that can be mastered, often involving action-oriented tasks, while "learning" refers to the acquisition of conceptual knowledge, or an understanding of the material.

Decision Making

Military tactics have been evolving to accommodate the ever-changing battlefield requirements. Specifically, distributed decision making, rather than procedural execution on command, has become a common tactic in military combat (Conway, 2008). Consequently, methodologies for training must also shift, despite the lack of additional training time availability. Thus, innovative interventions that capitalize on non-instructional time and that support the development of higher order cognitive skills, such as decision making, are of particular importance. Due to the differences between procedural and conceptual knowledge, the same training methodologies cannot be used interchangeably to teach these different types of knowledge. Rather, when teaching a deeper understanding of a situation or concept, it is necessary to improve the structure and organization of long-term memory (LTM) schema. Stated another way, it is not effective to simply memorize a series of steps to execute, as would be effective for procedural knowledge. Rather, one must be able to situate information within a contextual environment, understand the nuances of the material, and be able to effectively retrieve and apply that information in a real-world setting (Craig, Graesser, Sullins, & Gholson, 2004). A video game can provide the requisite environmental cues and context necessary to ground incoming information, but how these platforms are best utilized to improve the efficiency of training is not yet clear.

One learning strategy previously shown to reduce training time is the use of advance organizers (AOs) (Luiten, Ames, & Ackerson, 1980). AOs, as defined by Ausubel (1960), are "the advance introduction of relevant subsuming concepts" (p. 267). They aid in instruction by: (1) helping learners to attend to the most important material; and (2) allowing learners to better organize material conceptually in LTM by providing the learner with an initial schema, or representation, of the material to be learned (Ausubel, 1960; Neisser, 1967). If this initial schema is successfully transferred into the learner's LTM, it is expected to reduce working memory (WM) load during training by excluding extraneous cognitive load and focusing attention (Chalmers, 2003; Sweller, 1988).

Advance Organizers

The theory behind AOs proposes that information is stored hierarchically within LTM, with more concrete elements of information stored under more abstract and inclusive elements of information. Thus, giving learners an AO prior to training provides them with an initial high-level abstract framework, allowing them to better organize more concrete information into this framework as training occurs (Ausubel, 1960). This allows more material to be assimilated with LTM during training sessions.

AOs can be graphically or visually implemented and occur most often in the form of a narrative or outline (Mayer, 1979b; Moore & Readence, 1984; Schwartz, Ellsworth, Graham, & Knight, 1998). Research on the effectiveness of AOs has produced mixed results. A meta-analysis of 135 studies involving AOs indicated they have been shown to increase learning across a number of subject areas, grade levels, and presentation formats (Luiten et al., 1980). However, the effectiveness of an AO in any given training environment also often depends on factors within the individual learners, the structure of the material to be learned, and the form of AO used. Specifically, AOs have been shown to be more beneficial for lower ability learners, are better for preparing students to learn information related to complex, tasks that are highly dependent on declarative knowledge, and, for certain domains, are more effective when produced in a graphical format rather than a written format (Cannon-Bowers, Rhodenizer, Salas, & Bowers, 1998; Hatch & Dwyer, 1999; Mayer, 1997).

Most of the prior research on AO effectiveness deals with the use of an AO in an educational setting such as a classroom (Luiten et al., 1980). For example, AOs in this setting have led to improved learning within scientific domains, which are characterized by high declarative knowledge demands (Domin, 2008; Preiss & Gayle, 2006). Training contexts also often have high demands, including an awareness of the proper procedure and order of tasks to be completed. In these settings, AOs may serve to focus attention on important concepts to be learned during training through the organization of a large database of complex rules, procedures, and tasks into a meaningful structure (Cannon-Bowers et al., 1998). Unfortunately, there is much less available research on the use of AOs in these settings compared to classroom research.

Video Games as Advance Organizers

One promising delivery vehicle for an AO is within the structure of a video game. Research on video games as learning tools has demonstrated the potential of games to enhance learning, including the development of decision-making skills and the ability to teach complex and abstract concepts (Dondlinger, 2007). One reason postulated for the enhancement of learning with video games versus more traditional methods is that playing a video game is a form of experience-based, or active, learning (Ruben, 1999). In a simulation or video game, participants must actively engage with material and/or practice the desired behaviors in order to reach a goal (Garris, Ahlers, & Driskell, 2002). However, researchers point out that there is still much to be learned regarding when, with whom, and under what conditions games can be used to increase learning (Becker, 2005; Van Eck, 2006).

AOs have been demonstrated to enhance learning in prior research; thus, an interactive multimedia AO delivered within the context of a video game may also be effective for increasing learning (Chun & Plass, 1996; Mayer, 2002). However, recent research investigating the use of narrative games versus a matched slideshow presentation showed that learners performed worse in all areas assessed (e.g., posttest scores, retention, and learning time) (Adams, Mayer, MacNamara, Koenig, & Wainess, 2012), suggesting that too much erroneous or distracting information may be present in some game-based educational systems. It is also possible that the information provided in game-based form is too abstract for the learner to effectively organize and store within LTM and later retrieve and apply it to new situations.

Most multimedia AOs used in prior research contain, at a minimum, audio and video components explaining the to-be-learned material. Others have allowed for interactivity by presenting buttons on the screen that allow learners to move to the next segment of material (Mayer, 2002). Video games combine these multimedia elements in a unique format. Games contain audio, video, and textual elements and require users to be highly interactive. Learners must actively integrate the information provided by the game in order to successfully complete presented tasks (Mayer, 1979a).

Video games meet several of the requirements for AOs as outlined by Mayer (1979a). A video game as an AO provides a means for generating logical relationships among elements of to-be-learned information. Learners playing through an AO video game are able to view how the elements of to-be-learned information interact to create the scene through which they play. Video game AOs are frequently abstract in nature and do not contain any of the specific content from to-be-learned information. Instead, all information is presented in an abstract form that is used to complete the game, but this information is usually not specifically presented to learners.

Using a video game as an AO may be advantageous for several reasons. First, the rise of the Millennial generation has coincided with increased interest in using games for learning (Macedonia, 2005). Some of these generational qualities include the desire for multiple streams of information, preference for inductive reasoning, and the desire for quick and frequent interactions with material (Jones, 2003). Because material delivered in a game-based format is likely to appeal to Millennials, it is hypothesized that an AO presented in the context of a game will have higher rates of casual adoption (i.e., engaging with the material outside of the learning environment).

Second, using a video game as an AO may act to increase the self-efficacy of learners, if it is able to convey sufficiently detailed schema. Learners may become more confident in the learning situation by having domain-specific schema in LTM (Zimmerman, 2001). Self-efficacy is proposed to make learning easier by increasing self-confidence in the ability to gain mastery over the training material. This self-confidence changes learners' perception of the incoming material, focusing their attention and allowing them to more effectively allocate WM resources toward consolidating the information (Linnenbrink & Pintrich, 2003).

Third, it is likely that the use of a video game as an AO will lead to increased procedural and conceptual knowledge of tasks to be trained. In this study, procedural knowledge is defined as learning a series of steps to complete a task while conceptual knowledge is defined as understanding underlying information. Video games are an inherently abstract format, where players both consume and produce information present in the game and actively create their own meaningful structures (Dondlinger, 2007). Thus, the learner must interact with the concepts within a gaming environment versus simply reading or listening to them. Past research has suggested that learning effects are often very strong when learners actively construct knowledge to be learned (Ally, 2008). In addition, learners' schemas are likely to be better refined because of their experiences in the game, facilitating later retrieval (Gosen & Washbush, 2004; Sweller, van Merriënboer, & Paas, 1998). Finally, it is hypothesized that because material is presented in an abstract format via the video game, learners will develop more flexible schemas that will allow them to better apply acquired knowledge to novel situations.

Other Considerations

It is important to also consider some of the ways in which video games may not serve as effective AOs. One potential downfall is an increase in extraneous cognitive load during AO exposure. Cognitive load theory (Sweller, 1988) is a psychological theory based on the idea that WM has a finite capacity that can be optimized using appropriate instructional design. Based on this theory, extraneous load refers to the strain placed on WM capacity that does not aid in the acquisition of knowledge. It is expected that this will largely be a function of the usability requirements in the game that serve as the AO (Davis & Wiedenbeck, 2001). Thus, to the extent that the game is easy to use, there will be fewer adverse effects on cognitive load (Chandler & Sweller, 1996; Sadowski & Stanney, 2002).

Another area of concern regarding extraneous cognitive load is the additional information provided by the game. An AO presented in outline form provides only the information necessary to organize the upcoming material. By contrast, a game incorporates several other components (e.g., supplementary text, graphics, and sounds), and if learners' attention is directed towards this extraneous information, learning may be impaired (Paas, Tuovinen, Tabbers, & Van Gerven, 2003). Further, a learner may develop misconceptions from interacting with the game. For example, the learner may form erroneous schema through their interactions with the game that ultimately may act as a hindrance to learning.

AOs have been shown to be effective at increasing learning in highly nonlinear environments, such as a video game, but to date no research has explored whether video games themselves can act as delivery vehicles for an AO prior to training (McManus, 2000). Previous research has examined the effectiveness of either AOs or the use of games for training, but not the simultaneous combination of the two (Luiten et al., 1980). It is hypothesized that using a video game as a vehicle for AO delivery may be an especially promising technique for schema construction due to its highly interactive nature (Brant, Hooper, & Sugrue, 1991). Also, using a video game as an AO is likely to increase students' readiness for training in a computer-based simulator since the mode of instruction is similar. Because of the potentially high-payoffs (e.g., generational appeal, accessing non-instructional time, more refined and flexible schemas, and increased self-efficacy and learning efficiency) as well as the potential pit-falls (e.g., cognitive overload, information loss, misuse, or distraction by the game), further exploration is necessary to determine whether video games might serve as effective AOs, as well as how and when they might be effective.

The Current Study

The aim of the present study was to explore the use of a video game AO to train concepts from the military's call for fire (CFF) task. CFF is a procedure that allows a team on the ground to call in close air, artillery, or mortar support to attack enemy targets. One member of the fire support team (FiST), the forward observer, radios the components of a CFF to the supporting armed teams. Forward observers must make decisions about the method and type of fire to use depending on the type of enemy they are engaging and must be knowledgeable about both the capabilities of the enemy and their fire teams (U.S. Department of the Army, 1991).

The study was an investigation of: (1) using an AO to support training the CFF task; and more specifically (2) using a video game-based AO to support training the CFF task. Several impacts on training were measured, including development of procedural knowledge, development of conceptual knowledge, cognitive load, self-efficacy, decision-making efficiency, casual adoption, and integrated knowledge.

The hypothesized effects of the advance organizer were as follows:

• Hypothesis 1. The AO groups will exhibit greater procedural knowledge, as measured by the procedural knowledge test, than non-AO groups following training.
  
• Hypothesis 2. The AO groups will exhibit greater conceptual knowledge, as measured by concept maps, after training than the non-AO groups.
• Hypothesis 3. The AO groups will exhibit lower cognitive load, as measured by the Cognitive Load Questionnaire, than non-AO groups during training.
• Hypothesis 4. The AO groups will exhibit greater self-efficacy, as measured by the self-efficacy questionnaire, than the non-AO groups before and during training.
• Hypothesis 5. The AO groups will exhibit greater decision-making capabilities as exhibited by enhanced performance in the simulation and efficiency of decision making as exhibited by enhanced performance with lower cognitive load while performing within the simulation.

There were also two hypothesized effects of the video game:

• Hypothesis 6. The game groups (AO and non-AO) will have higher rates of casual adoption than the outline groups (AO and non-AO).
• Hypothesis 7. The AO game group will exhibit greater integrated knowledge, as measured by the integrated knowledge questionnaire, and will do so while utilizing lower cognitive load, than the AO outline and non-AO (game and outline) groups.

Method

Participants and Design

There were 68 participants, 23 male and 45 female, in the study. The mean age of participants was 19.34 years (SD = 2.27). Four participants (two males and two females) were excluded from the analysis because of previous participation in experiments using the CFF task, leaving 64 participants in the final analysis. Participants were sampled from a university using an online recruitment system available through the university's psychology department and received course credit for participation. Participants were randomly assigned to one of the four conditions given in Table 1. A power analysis was completed based on the four groups and power of .95 (1-β error probability) recommending a total sample size of 68 to detect an effect size of .52.

Table 1. Experimental conditions used in the study

Paper-Based Materials

• Demographics questionnaire. This 13-item questionnaire addressed personal identifiers such as age, race, gender, military experience, and degree of comfort with and frequency of use of computers (e.g., "Have you ever served in the military or ROTC?").
  
  
• Game biography questionnaire. A 14-item video game biography questionnaire was used to obtain detailed information about each participant's gaming background, including how long he/she had been playing video games and his/her level of adoption of gaming technology (e.g., "At what age did you begin playing video games?") (Adams & Ip, 2002).
  
  
• Prior knowledge questionnaire. Participants answered four free-entry questions regarding their prior knowledge of or experience with the FiST, forward observers, CFF tasks, and military simulators. This questionnaire was developed by our team. An example question is "What do you know about fire support teams?" Individuals who were able to correctly answer any of the questions were removed from the study. Four individuals were excluded for excessive prior knowledge.
  
  
• Cognitive load questionnaire (CLQ). This one-item questionnaire (described in Paas et al., 2003) asks, "In solving or studying the preceding problem I invested: ...". Participants then rated their perceived cognitive load during a task or set of tasks ranging from "1. very, very low mental effort" to "9. very, very high mental effort." Other studies, including the study by Paas (1992), have produced reliability (Cronbach's alpha) scores as high as .9. Scores ranged from 1 to 9.
  
  
• Self-efficacy questionnaire. Self-efficacy was measured using a 15-item questionnaire based on the constructs of self-efficacy described by Linnenbrink and Pintrich (2003). The questionnaire contains questions such as "I can understand the differences in ammunition types and can choose the correct one for each target type," "I am confident in my ability to use the GPS [global positioning system]," and "I am proficient at keeping my team safe from friendly fire." Participants rated themselves on a scale from 0 to 100. Ratings were summed to obtain a composite score and total scores ranged from 23 to 292.
  
  
• Simulator sickness questionnaire. The Simulator Sickness Questionnaire was administered following exposure to the video game and the deployable virtual training environment (DVTE) simulator (Kennedy, Lane, Berbaum, & Lilienthal, 1993). The 29-point Pre-Exposure Symptom Checklist was used to screen individuals. Participants rated the severity of current symptoms on a 4-point Likert scale (1 = None, 2 = Slight, 3 = Moderate, 4 = Severe). Example symptoms included "headache" and "eye strain." Scores were summed within three subscales: Nausea (N), Oculomotor (O), and Disorientation (D). Items within each subscale were then multiplied by a weight specific to each subscale (N = 9.54, O = 7.58, D = 13.92) and then summed and scaled by a factor of 3.74. Total scores ranged from 0 to 56.1.
  
  
• Fatigue questionnaire. Fatigue was measured using the 13-point Pearson–Byars Fatigue Feeling Checklist (Pearson, 1957; Pearson & Byars, 1956). Participants were asked to rate feelings on a 3-point Likert-type scale. Examples included "Very Lively" and "Slightly Tired." Total scores were summed and ranged from 1 to 34. These scales were used only as screeners to ensure that the data collected was valid. They were not used as moderator variables in the analysis.
  
  
• Casual adoption questionnaire. Self-ratings of likelihood of casual adoption were assessed using a 12-item Casual Adoption Questionnaire developed for this study. Separate Casual Adoption Questionnaires were developed for participants who received the outline and those who played the game. An example statement from the outline group was "I enjoyed studying the outline" and for the game group, "If given the opportunity, I would play this game in my free time." Participants rated how well these statements described them on a 7-point Likert-type scale, with 1 representing "Strongly Disagree" and 7 representing "Strongly Agree." Scores ranged from 6 to 39.
  
  
• Procedural knowledge test. Participants answered lab-developed multiple-choice questions regarding how to execute a CFF task and how to use the DVTE simulator. Questions included the order of steps that should be followed to complete a certain task and the pieces of equipment that should be used to accomplish a certain step in the procedure. For example, "In this simulation, what is the order in which a forward observer should communicate with the artillery team in order to call for fire?" One point was assigned for each correct response. Scores were summed and ranged from 17 to 32. Test-retest reliability showed a 90% agreement between pilot participants and construct validity evaluations by experts showed an average rating of 4.0 (out of a possible 4 points). Higher scores indicated good ratings.
  
  
• Integrated knowledge test. Integrated knowledge refers to information that is retrieved from LTM and applied in a real-world or simulated setting. The term 'integrated' refers to a type of knowledge, not the mode of delivery. Participants answered nine lab-developed free-entry questions that required inferences about and deeper knowledge of the FiST. The questions presented situations a member of the FiST might face that were not mentioned in the training presentations, requiring participants to apply their conceptual knowledge to novel situations in order to properly answer the questions. An example question is "Why do you think it's important for the forward observer to tell his supporting unit whether to use a high or low arching trajectory?" Independent raters used a scoring rubric and awarded up to three points per question. Scores ranged from 2 to 10. Test-retest reliability showed a 60% agreement between pilot participants and construct validity evaluations by experts showed an average rating of 3.33 (out of a possible 4 points). Higher scores indicate good ratings.
  
  
• Outlines. The outline used in the OAO condition (see Table 1) outlined the CFF task; however, the outline used in the OWAO condition outlined an unrelated military-based task, how to camouflage one's body and equipment. The CFF task outline was developed from an Army field manual on procedures for observed fire (U.S. Department of the Army, 1991). The camouflage task outline was developed from information publicly available on the Web (QuinStreet, 2005). The OWAO outline was not intended to be used as a comparison task to the outline or the game conditions. It was intended to act as a non-educational placeholder so that the OWAO group was required to complete a similar activity to the other groups. As such, the information provided in the outline was military relevant, but not instructionally relevant to the proceeding CFF task.

Computer-Based Materials

• DVTE. The DVTE simulator was designed to help active-duty Marines and trainees learn and practice the elements of the CFF task, among other training experiences. Its usefulness in training the CFF task has been supported in previous research (Bailey & Armstrong, 2002). During the simulation, trainees act as the forward observer and must call in artillery fire onto targets in a simulated battlefield. Trainees have a variety of simulated field tools available, including a compass, map, radio, and laser rangefinder. The radio call is simulated through the use of on-screen commands issued by the trainee via a point-and-click interface. Forward observers must radio in at least six pieces of information when issuing a CFF. These are the observer identification, the warning order (adjust fire or fire for effect), the target location, the target description, the method of engagement, and the method of fire and control (U.S. Department of the Army, 1991). The six major components of the CFF order are issued in three separate radio calls. In the first call, the forward observer gives his/her identification and warning order. In the second call, the observer gives the target location. Finally, in the third call, the observer gives the description of the target, the method of engagement, and the method of fire and control (U.S. Department of the Army, 1991).
  
  
• ARMA video game. Both the VGAO and VGWAO conditions met the criteria outlined by Vogel et al. (2006) for simulation video games. These criteria include having goals or conditions for completion, interactivity in the scenario, and feedback on performance during or following the scenario. The video game ARMA: Combat Operations (Bohemia Interactive, 2007) was used as the video game AO in the study. The game was chosen for its realistic combat game play scenarios, which closely match the style and tone of the DVTE simulation trainer used in the study (Bailey & Armstrong, 2002).
  
  A CFF task scenario is available as an add-on for the game and can be downloaded from the Internet. Participants who completed the VGAO condition played through the ARMA CFF task. In the scenario, players were first given a short mission briefing describing their objective. Players were required to conduct a CFF exercise on two targets (a tank and an enemy officer camp), while avoiding one non-target (a field hospital). Players had to use a map to determine the coordinates of each target and select a target by clicking on the map. They then called orders to their FiST team members such as the type of artillery and the number of rounds to be used. Following each call, the player received visual (explosions on screen) and auditory (artillery noises and explosions) feedback confirming that the target was hit or missed. Players then moved on to the next target until both targets had been destroyed and the mission objective was completed. A screenshot of the ARMA CFF task used in the VGAO condition is given in Figure 1.
  

Figure 1. Screen shot of game play in ARMA for the CFF task (VGAO)

• Participants who completed the VGWAO condition played an unrelated scenario in the ARMA game on a practice firing range. During this scenario, participants used the keyboard and computer mouse to fire on various targets on a simulated practice range. They were required to hit at least 15 of the 30 targets presented during the scenario. A screenshot of the VGWAO condition is given in Figure 2. Again, similar to the OWAO group, this game intervention was not intended to be used as a direct comparison task to the outline or the game conditions. It was intended to act as a non-educational placeholder so that the VGWAO group was required to complete a similar activity to the other groups. As such, the activity completed was militarily relevant, but not instructionally relevant to the CFF task.

Figure 2. Screen shot of game play in ARMA for the shooting range task (VGWAO)

Procedure

• Pre-training. Participants first signed an informed consent form. They then completed the demographics questionnaire, video gaming biography questionnaire, and prior knowledge questionnaire. Next, during the pre-training portion of the study, participants were randomly assigned to one of the four study conditions presented in Table 1. In the VGAO condition, participants played through the portion of the ARMA video game that included a CFF task for 10 minutes. In the VGWAO condition, participants played through the portion of the ARMA video game that included a shooting range task, but that did not include any components of the CFF task for 10 minutes. In the OAO condition, participants viewed a paper-based outline of the CFF task for 5 minutes. Finally, in the OWAO condition, participants viewed a paper-based outline of how to camouflage themselves and their equipment that did not include any components of the CFF task for 5 minutes. Participants in the gaming conditions were provided extra time due to the inherent time required to set up and execute the computer components (e.g., learn how to use the commands in the game), which were not an issue in the outline AOs. Following the pre-training task, participants in all groups completed the simulator sickness questionnaire, the cognitive load questionnaire, the casual adoption questionnaire, the procedural knowledge questionnaire, and the self-efficacy questionnaire. This portion of the experiment took about 25 minutes.
  
  
• Training. Next, participants in all groups viewed a training video indicating the role the participant would play in the simulator exercise to follow (the forward observer) and instructions as to how to perform the radio calls needed to complete a CFF using the simulator menu options. Following the training video, participants once again completed the cognitive load questionnaire, the procedural knowledge questionnaire, and the self-efficacy questionnaire. Participants also completed the fatigue questionnaire to ensure that they were not experiencing undue fatigue.
  
  
• Simulation-based assessment. Next, participants in all groups completed a practice exercise in the DVTE simulator where they destroyed two tanks using the procedure they learned in the training video. Participants then completed an assessment exercise measuring decision making in a simulator that contained four enemy tanks and four friendly tanks. Participants were instructed to execute as many CFF missions as were needed to destroy all the enemy tanks, and to destroy the enemies in a specific order based upon the threat levels they learned during the training video (e.g., enemies closer to the forward observer are destroyed before enemies farther away). The exercise was complete when participants indicated to the experimenter that they believed they had destroyed all the enemy tanks. Participants then once again completed the simulator sickness questionnaire, the cognitive load questionnaire, and the procedural knowledge questionnaire.
  
  Decision-making scores were based on performance in the simulator and calculated to indicate overall decision-making skills across the full assessment. Targets were rank ordered a priori to indicate the best neutralization sequence. This ranking was based on the differential levels of threat of each target (as described in the training video). Decisions were awarded increasingly higher penalty points according to this ranking, with the best decisions (e.g., destroying the right enemy target at the right time) acquiring no points and the worst decisions (e.g., destroying a friendly target) receiving as many as 8 penalty points. These individual scores were then averaged across the number of decisions made. Decision-making scores were the primary dependent variables in this study because the goal was to determine if the use of video games as AOs would affect individuals' abilities to efficiently organize their knowledge so that they could better retrieve and apply that information in a novel situation. In this study, the application of that knowledge was measured by assessing the quality of decisions made in the simulator. This portion of the experiment took about 40 minutes.
  
  
• Conceptual knowledge assessment. Participants in all groups next completed a concept map of the CFF task using computer-based concept mapping software that is part of the Intelligent Training Suite developed at the Team Performance Laboratory at the University of Central Florida (2001). Concepts to be mapped were developed from an expert's concept map of the CFF task. These concepts were presented to participants on a computer screen. Participants indicated relationships between the concepts using arrows drawn with the computer mouse. Following the concept-mapping task, participants completed the integrated knowledge test. Finally, participants once again completed the fatigue questionnaire to ensure that the study procedure had not caused them undue fatigue. This portion of the experiment took about 25 minutes.

Results

Descriptive Data

• Screening for vertigo and fatigue effects. Participants' simulator sickness scores on the SSQ were compared to average total severity scores obtained in other simulators, with a range of 0 to 108.6, as documented by Kennedy et al. (1993). Average scores ranged from 5.38 (less than the 60th percentile – the lowest percentile defined in Kennedy et al. to 8.71 (less than the 70th percentile). While 8.71 appears to be quite high in percentile, it is significantly lower than the maximum possible score of 108.6. Using the Kruskal–Wallis test, group differences on simulator sickness scores were compared prior to playing the game/reading the outline, after the same experience, before using the simulator, and again after the simulator experience. No significant differences between SSQ administrations were found, suggesting that simulator sickness effects did not unduly impact the findings. The means and standard deviations for each group's SSQ scores are presented in Table 2. A Kruskal–Wallis test was also used to compare fatigue scores across the experimental groups. No significant differences between groups were noted, suggesting that levels of participant fatigue did not influence the results of the study. The means and standard deviations for each group's fatigue scores are given in Table 2. Median scores are presented in Table 3.

Table 2. Means and standard deviations of screened variables

Table 3. Medians by variable

• Video game experience. A video game dedication score was calculated for each participant using results from the Video Game Biography. Among the participants, 44% were classified as ultra casual or non-gamers, 22% were classified as casual gamers, 25% were classified as transitional or moderate gamers, and only 9% were classified as hardcore gamers. There were no differences between the four AO groups on video game comfort scores, F(1, 54) = 0.682, p = .412, or between the AO and non-AO game groups, F(1, 54) = 0.323, p = .572. Therefore, video gaming skills or familiarity were not expected to influence the results of the study. Median scores are presented in Table 3.

Data Screening

Missing data was deleted listwise from analyses. All experimental variables were assessed to determine if they met the criteria for analysis of variance (ANOVA). Specifically, Levene's test was used to determine if each of the variables met the assumption for homogeneity of variance. Two variables did not meet the assumption for homogeneity of variance using this test (casual adoption, p = .048; decision making, p = .001). Each variable was also tested for skewness and kurtosis to determine if the variable distributions met the assumption of normality. Two variables' distributions were kurtotic (concept map, -2.26; decision making, -2.34), and two variables' distributions were significantly skewed (efficiency of decision making, -10.67; efficiency of integrated knowledge application, 2.97), and therefore did not meet this assumption. Consequently, for these five variables, planned comparisons were developed based on the hypotheses, and Kruskal–Wallis non-parametric tests were conducted. For all other variables, an ANOVA was conducted. Means and standard deviations for all experimental variables are provided in Table 4. Median scores are presented in Table 3.

Table 4. Means and standard deviations by hypothesis

Data by Hypotheses

A 2 (AO) x 2 (game) ANOVA was conducted to examine Hypotheses 1, 3, 4, and 7. Kruskal–Wallis tests were used to examine all other hypotheses for variables that did not meet the assumptions of ANOVA.

• Hypothesis 1. The independent variable was the advance organizer and the dependent variable was procedural knowledge, as measured by the procedural knowledge questionnaire. The hypothesis was not supported as there were no significant differences between the experimental groups on tests of procedural knowledge following the simulator exercise, F(1, 54) = 3.34, p = .073. However, it is noted that this result approached significance suggesting that the AO group, regardless of presentation type, learned less procedural knowledge than those not receiving an AO.
  
  
• Hypothesis 2. The independent variable was the advance organizer and the dependent variable was conceptual knowledge, as measured by concept map performance. The hypothesis was not supported as there were no significant differences between the experimental groups on concept map scores, (by condition χ²(3, 60) = 1.02, p = .796; by game group χ²(1, N = 62) = 0.535, p = .465; by AO χ²(1, N = 62) = 0.884, p = .347).
  
  
• Hypothesis 3. The independent variable was the advance organizer and the dependent variable was cognitive load, as measured by the CLQ. Significant interaction effects between AO and game groups were found in reported cognitive load levels during the simulator assessment, F(1,54) = 4.78, p = .033, partial η² = .082, suggesting that those receiving the OAO invested the least cognitive load to complete the simulator exercise than those in all other groups.
  
  
• Hypothesis 4. The independent variable was the advance organizer and the dependent variable was self-efficacy, as measured by the Self-Efficacy Questionnaire. The hypothesis was not supported, as there was not a significant difference between the AO and non-AO groups on self-efficacy scores, F(1, 54) = 0.807, p = .373.
  
  
• Hypothesis 5. The independent variable was the advance organizer and the dependent variable was decision-making efficiency, as measured by the decision-making score within the simulator and the CLQ. Decision-making efficiency was calculated using the following formula (Paas & van Merriënboer, 1993):

  

  where

  P = Performance (decision-making score); and
  E = Effort (cognitive load rating).

  There was not a significant difference between the AO and non-AO groups on decision-making scores, χ²(1, N = 63) = 1.93, p = .586.

• Hypothesis 6. The independent variable was the game and the dependent variable was casual adoption as measured by the casual adoption questionnaire. Those in the game group reported a higher rate of casual adoption than those in the outline group, χ²(1, N = 63) = 11.612, p < .001. Significant interaction effects between AO and game groups were found in reported casual adoption rates, χ²(1, N = 63) = 17.54, p = .001. These data suggest that those receiving the game without the AO component were more likely to continue playing and learning than those receiving the game with the AO.
  
  
• Hypothesis 7. The independent variable was the VGAO and the dependent variable was integrated knowledge efficiency, as measured by the IKQ and the CLQ. Integrated knowledge efficiency was again calculated using the formula noted above. No significant differences were found between the AO outline and non-AO groups.

Discussion and Conclusion

In the present study, the combination of the characteristics of a combat game and its educational content failed to improve learning, but appeared to improve interest. More specifically, willingness to continue learning during non-instructional time, as measured by the casual adoption questionnaire, was higher in those using the video game before learning. These findings suggest that individuals who are provided educational material embedded within an environment that has motivational components may be more willing to continue self-study, disguised as self-play, outside the classroom. Recent researchers investigating the use of problem-based learning in the classroom to achieve similar goals have found similar results (Rotgans & Schmidt, 2012), lending further support to the use of motivational educational activities for use as cognitive readiness interventions.

Regarding cognitive load, however, a higher level of effort was reported to attain the same level of knowledge acquisition when completing the video game prior to learning compared to receiving a traditional outline AO. It was expected that the presence of an AO would act to reduce WM load and allow participants to better focus their attention during the training exercises because they were given a conceptual framework to help structure and file incoming information. While this effect was found in the paper-based outline AOs, it was not replicated in the game-based AO, suggesting that either the game-based AO was too abstract to act as an effective AO or the CFF task was too procedural in nature to effectively assess changes in conceptual understanding. These data support the recent findings by Adams et al. (2012) suggesting that video games that fail to follow the principles of cognitive load theory may provide too much extraneous cognitive load and distract learners.

Still, these data suggest that through "play," arousal and attention may increase prior to learning and improve individuals' willingness to continue learning. Typically, as arousal increases during learning, so too does knowledge acquisition because the individuals' attention is better focused, leading to improved cognitive efficiency (Esmaeili, Karimi, Tabatabaie, & Moradi, 2012). However, if the motivational multimedia material is distracting, rather than enhancing, motivation to play may increase without supporting the implicit educational goals (Mayer, 2005). Thus, it is important to balance the amount of motivational material that positively affects attention and knowledge acquisition while not overloading the learner with too much extraneous material that will lead to task shedding, or loss of requisite information.

Those participants using a video game, and particularly, those using the video game without an AO reported the highest rate of casual adoption in terms of enjoyment and interest. Playing through the shooting range task was likely more intuitive and interactive than the game-based AO. Thus, it appears that interactivity and enjoyment may be strong predictors of subsequent adoption. It may be crucial to maintain an element of "play" in any game to be used as an AO, or for any game used for training or learning purposes.

Limitations

Several limitations are noted in this study. First, sample size may have affected the data's ability to achieve significance. Specifically, regarding the first hypothesis, the data approached, but failed to attain, statistical significance. A larger sample size likely would have clarified this finding. Additionally, a university sample was used for this study, but the focus was military training content. While this is a convenient sample, results may have differed if a military sample was used. For example, there is a possibility that casual adoption, motivation, and self-efficacy would have been affected. Finally, while using an off-the-shelf, ready-made game is pragmatic, the lack of customization to the training goals may have reduced its impact.

Recommendations and Future Research

The potential for games to act as effective AOs for training should be explored in future research, with a particular focus on which specific elements of games allow them to act as effective AOs (e.g., ease of use, enjoyment, collaborative effort). Additionally, the impact of games on learning efficiency and the application of conceptual knowledge represents an important area to expand upon in future research. For example, recent research on game-based learning has indicated that utilizing metacognitive prompting to direct learners' attention to the underlying principles of the game can enhance knowledge transfer (Fiorella & Mayer, 2012). Therefore, using metacognitive prompting to aid in schema construction may improve the efficacy of video game AOs.

References

Adams, E., & Ip, B. (2002, June 5). From casual to core: A statistical mechanism for studying gamer dedication. Gamasutra. Retrieved from http://www.gamasutra.com/view/feature/2988/from_casual_to_core_a_statistical_.php

Adams, D. M., Mayer, R. E., MacNamara, A., Koenig, A., & Wainess, R. (2012). Narrative games for learning: Testing the discovery and narrative hypotheses. Journal of Educational Psychology, 104(1), 235-249. doi:10.1037/a0025595

Ally, M. (2008). Foundations of educational theory for online learning. In T. Anderson (Ed.), The theory and practice of online learning (2nd ed., pp. 15-44). Edmonton, Canada: Athabasca University Press. Retrieved from http://www.aupress.ca/books/120146/ebook/01_Anderson_2008-Theory_and_Practice_of_Online_Learning.pdf

Ausubel, D. P. (1960). The use of advance organizers in the learning and retention of meaningful verbal material. Journal of Educational Psychology, 51(5), 267-272. doi:10.1037/h0046669

Bailey, M. P., & Armstrong, R. (2002). The deployable virtual training environment. In Proceedings of the 2002 Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) [CD-ROM]. Arlington, VA: National Training and Simulation Association.

Becker, K. (2005). How are games educational? Learning theories embodied in games. In Changing views – worlds in play: Proceedings of the DiGRA 2005 Conference. Tampere, Finland: Digital Games Research Association. Retrieved from http://www.digra.org/dl/db/06278.23299.pdf

Bohemia Interactive. (2007). ARMA: Combat operations [Computer software]. Los Angeles, CA: Atari.

Brant, G., Hooper, E., & Sugrue, B. (1991). Which comes first: The simulation or the lecture? Journal of Educational Computing Research, 7(4), 469-481. doi:10.2190/PWDP-45L8-LHL5-2VX7

Cannon-Bowers, J. A., Rhodenizer, L., Salas, E., & Bowers, C. A. (1998). A framework for understanding pre-practice conditions and their impact on learning. Personnel Psychology, 51(2), 291-320. doi:10.1111/j.1744-6570.1998.tb00727.x

Chalmers, P. A. (2003). The role of cognitive theory in human-computer interface. Computers in Human Behavior, 19(5), 593-607. doi:10.1016/S0747-5632(02)00086-9

Chandler, P. A., & Sweller, J. (1996). Cognitive load while learning to use a computer program. Applied Cognitive Psychology, 10(2), 151-170. doi:10.1002/(SICI)1099-0720(199604)10:2<151::AID-ACP380>3.0.CO;2-U

Conway, J. T. (2008). A concept for enhanced company operations. Marine Corps Gazette, 92(12), 56-61.

Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29(3), 241-250. doi:10.1080/1358165042000283101

Chun, D. M., & Plass, J. L. (1996). Facilitating reading comprehension with multimedia. System, 24(4), 503-519. doi:10.1016/S0346-251X(96)00038-3

Davis, S., & Wiedenbeck, S. (2001). The mediating effects of intrinsic motivation, ease of use and usefulness perceptions on performance in first-time and subsequent computer users. Interacting with Computers, 13(5), 549-580. doi:10.1016/S0953-5438(01)00034-0

Domin, D. S. (2008). Using an advance organizer to facilitate change in students' conceptualisation of the role of creativity in science. Chemistry Education Research and Practice, 9(4), 291-300. doi:10.1039/B818463C

Dondlinger, M. J. (2007). Educational video game design: A review of the literature. Journal of Applied Educational Technology, 4(1), 21-31. Retrieved from http://www.eduquery.com/jaet/JAET4-1_Dondlinger.pdf

Esmaeili, M. T., Karimi, M., Tabatabaie, K. R., & Moradi, A. (2012). The effect of post-learning positive arousal on memory consolidation. Procedia – Social and Behavioral Sciences, 32, 104-107. doi:10.1016/j.sbspro.2012.01.018

Fiorella, L., & Mayer, R. E. (2012). Paper-based aids for learning with a computer-based game. Journal of Educational Psychology, 104(4), 1074-1082. doi:10.1037/a0028088

Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 33(4), 441-467. doi:10.1177/1046878102238607

Gosen, J., & Washbush, J. (2004). A review of scholarship on assessing experiential learning effectiveness. Simulation & Gaming, 35(2), 270-293. doi:10.1177/1046878104263544

Hatch, C. A., & Dwyer, F. M. (1999). Effect of varied advance organizer strategies in complementing visualized prose instruction. International Journal of Instructional Media, 26(3), 311-328.

Jones, S. (2003). Let the games begin: Gaming technology and entertainment among college students. Washington, DC: Pew Internet & American Life Project. Retrieved from http://www.pewinternet.org/~/media//Files/Reports/2003/PIP_College_Gaming_Reporta.pdf.pdf

Kennedy, R. S., Lane, N. E., Berbaum, K. S., & Lilienthal, M. G. (1993). Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. The International Journal of Aviation Psychology, 3(3), 203-220. doi:10.1207/s15327108ijap0303_3

Linnenbrink, E. A., & Pintrich, P. R. (2003). The role of self-efficacy beliefs in student engagement and learning in the classroom. Reading & Writing Quarterly, 19(2), 119-137. doi:10.1080/10573560308223

Luiten, J., Ames, W., & Ackerson, G. (1980). A meta-analysis of the effects of advance organizers on learning and retention. American Educational Research Journal, 17(2), 211-218. doi:10.3102/00028312017002211

Macedonia, M. (2002). Games soldiers play. IEEE Spectrum, 39(3), 32-37. doi:10.1109/6.988702

Macedonia, M. (2005). Games, simulation, and the military education dilemma. In M. Devlin (Eds.), The Internet and the university: Forum 2004 (pp. 157-167). Boulder, CO: EDUCAUSE. Retrieved from https://net.educause.edu/ir/library/pdf/ffpiu018.pdf

Mayer, R. E. (1979a). Can advance organizers influence meaningful learning? Review of Educational Research, 49(2), 371-383. doi:10.3102/00346543049002371

Mayer, R. E. (1979b). Twenty years of research on advance organizers: Assimilation theory is still the best predictor of results. Instructional Science, 8(2), 133-167. doi:10.1007/BF00117008

Mayer, R. E. (1997). Multimedia learning: Are we asking the right questions? Educational Psychologist, 32(1), 1-19. doi:10.1207/s15326985ep3201_1

Mayer, R. E. (2002). Cognitive theory and the design of multimedia instruction: An example of the two-way street between cognition and instruction. New Directions for Teaching and Learning, 89, 55-71. doi:10.1002/tl.47

Mayer, R. E. (2005). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 31-48). New York, NY: Cambridge University Press. doi:10.1017/CBO9780511816819.004

McManus, T. F. (2000). Individualizing instruction in a web-based hypermedia learning environment: Nonlinearity, advance organizers, and self-regulated learners. Journal of Interactive Learning Research, 11(3), 219-251. Retrieved from EdITLib Digital Library. (8486)

Moore, D. W., & Readence, J. E. (1984). A quantitative and qualitative review of graphic organizer research. Journal of Educational Research, 78(1), 11-17. Available from JSTOR database. (27540086)

Neisser, U. (1967). Cognitive psychology. New York, NY: Appleton-Century-Crofts.

Paas, F. G. W. C. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84(4), 429-434. doi:10.1037/0022-0663.84.4.429

Paas, F. G. W. C., Tuovinen, J. E., Tabbers, H., & Van Gerven, P. W. M. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist, 38(1), 63-71. doi:10.1207/S15326985EP3801_8

Paas, F. G. W. C., & van Merriënboer, J. J. G. (1993). The efficiency of instructional conditions: An approach to combine mental effort and performance measures. Human Factors: The Journal of the Human Factors and Ergonomics Society, 35(4), 737-743. doi:10.1177/001872089303500412

Pearson, R. G. (1957). Scale analysis of a fatigue checklist. Journal of Applied Psychology, 41(3), 186-191. doi:10.1037/h0045095

Pearson, R. G., & Byars, G. E., Jr. (1956). The development and validation of a checklist for measuring subjective fatigue. Randolph Air Force Base, TX: U.S. Air Force School of Aviation Medicine. Retrieved from http://www.dtic.mil/dtic/tr/fulltext/u2/128756.pdf

Preiss, R. W., & Gayle, B. M. (2006). A meta-analysis of the educational benefits of employing advanced organizers. In B. M. Gayle, R. W. Preiss, N. Burrell, & M. Allen (Eds.), Classroom communication and instructional processes: Advances through meta-analysis (pp. 329-344). Mahwah, NJ: Erlbaum.

QuinStreet. (2005). 052-191-1361 (SL1) – Camouflage yourself and your individual equipment. Retrieved from http://www.armystudyguide.com/content/SMCT_CTT_Tasks/Skill_Level_1/0521911361-sl1-camouflage.shtml

Rotgans, J. I., & Schmidt, H. G. (2012). Problem-based learning and student motivation: The role of interest in learning and achievement. In G. O'Grady, E. H. J. Yew, K. P. L. Goh, & H. G. Schmidt (Eds.), One-day, one-problem: An approach to problem-based learning (pp. 85-101). Singapore: Springer. doi:10.1007/978-981-4021-75-3_5

Ruben, B. D. (1999). Simulations, games, and experience-based learning: The quest for a new paradigm for teaching and learning. Simulation & Gaming, 30(4), 498-505. doi:10.1177/104687819903000409

Sadowski, W. J., & Stanney, K. M. (2002). Presence in virtual environments. In K. M. Stanney (Ed.), Handbook of virtual environments: Design, implementation, and applications (pp. 791-806). Mahwah, NJ: Erlbaum.

Schwartz, N. H., Ellsworth, L. S., Graham, L., & Knight, B. (1998). Accessing prior knowledge to remember text: A comparison of advance organizers and maps. Contemporary Educational Psychology, 23(1), 65-89. doi:10.1006/ceps.1997.0958

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257-285. doi:10.1207/s15516709cog1202_4

Sweller, J., van Merriënboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251-296. doi:10.1023/A:1022193728205

University of Central Florida. (2001). Intelligent training suite [Computer software]. Orlando, FL: Author.

U.S. Department of the Army. (1991). Tactics, techniques, and procedures for observed fire (Field Manual 6-30). Washington, DC: Author. Retrieved from http://community.marines.mil/news/publications/Documents/FM%206-30.pdf

Van Eck, R. (2006). Digital game-based learning: It's not just the digital natives who are restless. EDUCAUSE Review, 41(2), 16-30. Retrieved from http://www.educause.edu/ero/article/digital-game-based-learning-its-not-just-digital-natives-who-are-restless

Vogel, J. J., Vogel, D. S., Cannon-Bowers, J., Bowers, C. A., Muse, K., & Wright, M. (2006). Computer gaming and interactive simulations for learning: A meta-analysis. Journal of Educational Computing Research, 34(3), 229-243. doi:10.2190/FLHV-K4WA-WPVQ-H0YM

Zimmerman, B. J. (2001). Theories of self-regulated learning and academic achievement: An overview and analysis. In B. J. Zimmerman & D. H. Schunk (Eds.), Self-regulated learning and academic achievement: Theoretical perspectives (2nd ed., pp. 1-37). Mahwah, NJ: Erlbaum.