Introduction

Building a Mental Health Research Career

The path to career advancement and positions of advanced leadership in fields of research is often marked by roadblocks. For early career researchers, obtaining independent funding (i.e., transitioning from junior faculty to independent research scientist) is one of the first challenges that must be surmounted (Chang, Hughes, & Chamberlain, 2008; Kupfer, Hyman, Schatzberg, Pincus, & Reynolds, 2002; Pion & Cordray, 2008; Reynolds, Pilkonis, Kupfer, Dunn, & Pincus, 2007). Similarly, moving from first independent grant award to second—a transition that impacts both the likelihood of future funding as well as faculty promotion—represents a second major challenge faced by many mid-career researchers (Chang, Hughes, & Chamberlain, 2008; Kupfer et al., 2002; NSF, 2004). In addition to achieving these metrics of career success, researchers must stay up-to-date with shifts in the field, such as an increased focus on inter-disciplinary and multi-site research (NRC, 2005a; BWF-HHMI, 2004; Zerhouni, 2005). Unfortunately, despite the years of training completed prior to beginning a research career, many researchers lack the necessary skills to negotiate these potential barriers to career growth.

Key research career transition points are particularly difficult for early- and mid-career researchers who are women and/or persons of color because of unique barriers that these groups experience (Donovan et al., 2005; NAC, 2001; NSF, 2004). In the past two decades, a growing body of research has identified a number of common barriers to success reported by researchers and other academic professionals. In addition to the struggles faced by all early- to mid-career researchers, researchers who are women and/or persons of color face additional challenges, such as a lack of diverse role models, historical exclusion from leadership status as a minority, and an insufficient supply of diverse mentors (Turner, Gonzalez, & Wood, 2008). Without adequate support and specialized training resources, researchers who are women and/or persons of color are at elevated risk for leaving the research career path (Donovan et al., 2005; NIMH, 2008; NRC, 2005a; NAC, 2001).

Benefits of Training

Advanced training and mentorship are instrumental to retaining individuals on the research career path (Jeste et al., 2007; NIMH, 2008; O’Hara et al., 2010). For example, availability of post-doctoral training and external support for research predicts the successful transition from early career scientist to independent investigator (Kupfer et al., 2009; Reynolds et al., 2007). Further, contact with mentors–for practical advice regarding funding (e.g., understanding different grant mechanisms), administering projects (e.g., managing budgets), and managing career growth (e.g., negotiating salary)–is essential throughout one’s career (Kupfer et al., 2009; Reynolds et al., 2007). Researchers who receive training and mentorship in these areas will be at an advantage when navigating key research career transition points. Furthermore, for researchers who are women or persons of color, additional training that addresses specific challenges they experience (e.g., access to diverse role models, addressing institutional stigma) may promote overall career advancement (Shavers, Fagan, & Lawrence, 2005; Yager, Waitzkin, Parker, & Duran, 2007).

Online and In-person Training

Despite this identified need for training, which has been recognized at the federal level, currently available training resources have some limitations that impact access (Balon et al., 2011). Advanced professional training is typically conducted through in-person conferences or research training institutes, but these opportunities are often perceived as limited in both number and accessibility (e.g., geographic, financial, and time barriers) (Jeste et al., 2007; Li & Atkins, 2005; Watson & Rutledge, 2005). These same constraints limit the availability of mentors from diverse backgrounds who can participate in the training. An alternative training model that addresses some of these obstacles is the development of online, Web-based training programs that target the same key training areas essential to career advancement while maximizing the likelihood that more researchers will have the opportunity to participate (Ernst, 2008; Stumpf, McCrimon, & Davis, 2005).

Online education is becoming more widely used for training in a variety of professional fields and the research literature regarding this training is growing (Githens, Crawford, & Sauer, 2010). Overall, available studies of online education indicate a “no significant difference” phenomenon (Russell, 1999). Specifically, multiple studies have shown that learning outcomes (e.g., knowledge gains) are generally comparable for those enrolled entirely online versus those who participate in face-to-face courses (Chumley-Jones, Dobbie, & Alford, 2002; Ferguson & DeFelice, 2010; Lapsley et al., 2008; Russell, 1999; Siebert, Siebert, & Spaulding-Givens, 2006; Feldhaus & Fox, 2004). In fact, several reviews of online and in-person training for medical professionals (e.g., physicians, dentists, medical students) revealed equivalent knowledge and skill gains for online and in-person delivery modes (see Chumley-Jones et al., 2002; Wutoh, Boren, & Balas, 2004; and Cook, Levinson, Garside, Dupras, Erwin, & Montori, 2008). However, this research has focused on knowledge gains for academically-focused learning (e.g., disorder etiology, treatment methods, historical and theoretical groundings), with no prior studies comparing online and in-person training that targets professional career growth and advancement. The primary goal of this study was to empirically test the effectiveness of three different training modes (i.e., in-person and/or online) for promoting the acquisition of knowledge essential for career advancement of social, behavioral, and mental health researchers who are women or persons of color.

Another limitation of the current online education literature is a lack of longitudinal studies evaluating training delivery modes beyond a simple pre-post comparison. In fact, Cook and colleagues (2008) identified 63 studies between 1997 and 2007 that compared knowledge gains for health professionals using online versus non-Internet training. Of these, only one study included a follow-up beyond post-training assessment (i.e., 6 months post-training) (Bell, 2000). While the research literature consistently supports online education as better than nothing and equivalent to in-person training for knowledge acquisition, little is known about whether these gains are maintained over time or whether retention of information varies by training delivery mode. To address these questions, knowledge gains for each group of trainees in this study was gathered over a nine month period following completion of their training (in-person or online).

Access to specialized training, mentoring, and networking opportunities significantly impact research career advancement (NIMH, 2008; Kupfer et al., 2009; Reynolds et al., 2007). This paper describes development of the Leadership Training Institute, a training program designed to provide specialized resources in each of these areas to help diverse researchers achieve a successful, long-term career in social, behavioral, and mental health research. The following sections describe the process by which the LTI training curriculum was developed, the live LTI training event was transformed into a parallel Web-based training program, and a randomized longitudinal research evaluation comparing training delivery modes was conducted.

Development of the Leadership Training Institute

Curriculum Development

In recognition of the scope of training needs for research career advancement, the National Institutes of Health (NIH) instituted a call for research proposals to address the critical training needs of researchers, in particular those early career researchers who are women and/or persons of color. This call was largely in response to a growing literature suggesting an area of high need for this group of researchers (NAC, 2001). The Leadership Training Institute (hereafter LTI) was developed and tested through funding from two federal contracts awarded in response to this call (see Acknowledgements). Development included a two-year iterative process beginning with a review of the literature, preliminary program development, a series of focus group meetings with experts in relevant fields (e.g., psychology, social work, psychiatry), and full program development based on accumulated input and feedback (see DeRosier & McMillen, 2010 for full details). Through all iterations of development, modifications were continuously implemented to ensure training topics addressed the fundamental barriers to success for researchers who are women or persons of color. The final LTI curriculum included a series of 19 courses organized into five training modules (see Table 1) designed to help researchers establish career goals, address challenges and opportunities specific to diverse researchers, navigate funding opportunities, build support networks, and expand their team management and leadership skills.

In-person Training

The LTI was conducted as a four-day intensive workshop led by 16 faculty members (i.e., women and/or persons of color who were selected based on significant contributions to their scientific field) and attended by early- to mid-career social, behavioral, and mental health researchers who were women and/or persons of color. During the live LTI workshop, training was provided through multiple presentation formats, including lectures, symposia, and experiential exercises. In addition, there was set aside time for trainees to network with faculty and other trainees as well as meet individually with faculty for one-on-one consultations.

Online Training

During the course of the live LTI workshop, all faculty presentations were captured via videotape for digitization, transcription, and editing. Presentations were then translated into online e-courses using 3-C ISD’s e-training platform (DeRosier, Kameny, & Hehman, 2007, 2010). The e-courses incorporated synchronized presentation of video, audio, slides, and written transcripts; trainees were able to select the presentation format that best matched their learning style (e.g., video plus slides, audio only), an online feature that has been found to significantly increase engagement in e-learning as well as overall training effectiveness (Chen, 2001; Hobbs, 2002; Villaverde, Godoy, & Amandi, 2006, Walters & Egert, 2000). The e-training platform also accommodated assistive technologies (e.g., screen readers) for enhanced accessibility (US DHHS, 2003). The LTI e-courses were self-paced with the ability to stop, restart, and pause during presentation with information indexed by slide so trainees could easily navigate to any point in the e-course (Hobbs, 2002). A variety of interactive features were interwoven throughout each e-course to increase engagement and learning (Chen, 2001; Sun, Tsai, Finger, Chen, & Yeh, 2008 ), including periodic brief quizzes and experiential exercises to simulate the in-person experience (e.g., branching role plays, Q&A audience participation simulations). It was also possible for trainees to download course materials (MP3 for audio, printable PDFs for slides and transcripts) to extend learning opportunities beyond the Internet.

Table 1. Leadership Training Institute Modules

Training Module	Topics Covered
Career Path Planning	Setting goals and career plans
	Knowing your career options
	Being a leader when in the minority
	Balancing personal and professional priorities
Navigating the Funding World	Role of funding in career advancement
	Overview of funding options and sources
	Grant review process (with mock reviews)
	Conducting general vs. specialized projects
Navigating the Workplace	Challenges and opportunities for women
	Challenges and opportunities for persons of color
	Career navigation (e.g., getting hired, negotiating contracts)
	Navigating and managing institutional policies
Networking for Success	Networking opportunities for different purposes
	Peer coaching and mentorship at different career stages
	Establishing and managing mentor-mentee relationships
Building Management Skills	Developing and maintaining successful teams
	Working with research staff
	Working across research teams
	Personnel and career development

In addition to training materials, the LTI website (www.lti.4researchers.org) incorporated a number of tools and resources to address logistical barriers to career advancement that are frequently cited by diverse researchers, such as limited access to networking opportunities due to geographic distance, few research collaborators at one’s institution, and insufficient guidance and support from advanced-level researchers (El-Guebaly & Atkinson, 1996; Johnson et al., 2010; Turner et al., 2008). Table 2 describes the various Web-based features offered to trainees through the LTI website, including: Ask an Expert, Discussion Boards, LTI Wiki, and Quick Tips.

Table 2. LTI Website Online Training Features

Feature	What is it?	Who uses it?	Other features?
Ask an Expert	Text-based discussions between faculty and trainees	Trainees view faculty areas of expertise and send questions to the relevant expert	Responses delivered to trainee and stored in searchable archive for all trainees
Discussion Boards	Online forums for sharing ideas that may then be threaded by discussion and organized by topic area	Trainees or faculty create a post by e-mailing the discussion board	RSS reader is used to subscribe to discussion board and receive alerts of new postings
Wiki	Web-based platform that allows for active participation, collaboration, and sharing of knowledge	Trainees and faculty access the wiki Web pages, create new Wiki pages, or edit content	Index and list of links are available for browsing; FAQ page provides support to users
Quick Tips	Short (1-4 minute) video clips featuring faculty speaking about research interests, personal experiences, and practical advice	Trainees select and access by topic, contributor, most recently posted, and most frequently viewed	Links to content that might be of interest (i.e., related Quick Tips, other information from the same faculty member)

Method
Participants

Participants were recruited for this study via faculty referral and advertisements on applicable listservs (e.g., Society for Research in Child Development member list). Interested individuals submitted an application, which was reviewed by project staff to determine eligibility. Eligibility requirements included being: (1) a female and/or a person of color; (2) a researcher in a social, behavioral, or mental health field; and (3) an early- or mid- career professional (e.g., advanced post-doctoral position, assistant or associate professor). Over 160 researchers applied to participate in the study with 122 meeting all eligibility criteria. From these applicants, 38 trainees were selected to attain the following sample goals: (1) geographic representation across the United States, (2) significant diversity across trainees (ethnicity, race, gender, scientific interests), and (3) high level of commitment to a research career. One selected individual withdrew prior to study initiation, resulting in a total sample of 37 trainee participants. Table 3 displays demographic characteristics of the trainee sample. Participants were 79% female and represented diverse racial/ethnic groups. The sample was evenly divided across early-career and mid-career level trainees and represented a broad array of research fields.

Table 3. Trainee Participant Characteristics

Characteristic	Percent
Gender
Female	79%
Male	21%
Race
Caucasian	33%
Black/African American	43%
American Indian/Alaska Native	7%
Asian	17%
Ethnicity
Hispanic/Latino	24%
Non-Hispanic/Latino	76%
Career Level
Early Career	54%
Mid Career	46%
Field of Research
Social Work	30%
Psychology	26%
Psychiatry	12%
Education	11%
Public Health/Community Medicine	8%
Sociology	8%
Nursing	5%

Procedures

The primary objective of this study was to compare live and online training delivery modes for knowledge gains specific to research career development as well as trainee satisfaction. A second objective was to examine the usability and value of the LTI website. For this project, participants were randomly assigned to one of three conditions: Live Training Only (LTO; n = 11), Live and Web Combination Training (LWC; n = 12), and Web Training Only (WTO; n = 14). LTO trainees attended the live LTI workshop, but had no access to the LTI website during the study period. LWC trainees attended the live LTI (with the LTO group) and subsequently had access to the LTI website, including online courses and web-based resources and tools. Finally, WTO trainees did not attend the live LTI, but instead had full access to the online training courses and web-based resources and tools via the project website.

In addition, following completion of all training modules (in-person or online), trainees in all conditions participated in coaching sessions with an assigned faculty member. Once every three months, faculty-trainee pairs met (in person or by phone) for approximately one hour, during which the faculty member provided the trainee with guidance, feedback, and suggestions regarding career development topics, such as how to contact a funder, negotiate a workplace contract, prepare a tenure case, or manage a work-related contract.

As Figure 1 displays, data were collected at multiple time points. During the initial data collection period, demographic information was collected as well as pre-training knowledge for content covered in the five training modules. Knowledge was again assessed immediately following completion of the training modules (in-person or online) as well as at three-, six-, and nine-months post-training. While data were collected on a parallel schedule for all conditions, data collection with LTO and LWC trainees was initiated before the WTO group. Several months were needed to convert the live LTI courses into e-courses, such that the LTI website was launched 3-months following the live LTI workshop. LWC trainees received access to the LTI website approximately two weeks prior to the 3-month data collection period.

 

  
Figure 1. Data Collection Time Points by Condition

Measures

Knowledge evaluation. Trainees completed a knowledge questionnaire at each of the five data collection time points (pre-training and immediately post-training as well as 3-, 6-, and 9-month post training follow-ups). This questionnaire included 58 questions designed to assess knowledge and understanding of content covered within the LTI training modules. For example, a question for the NIH funding process was “How many study sections are there in an integrated review group?” For building better management skills, an example question was “Which of the following is an example of building organizational support?” Response formats included multiple-choice, short-answer, and fill-in-the-blank. The full set of items was administered at each time point, but items and response choices (for multiple-choice items) were varied to alter presentation across time points. At each time point, the number of correct responses across items was summed and the percent of questions answered correctly was calculated.

LTI training evaluation. Following completion of all five training modules (in-person or online), trainees in each study condition completed a questionnaire rating their satisfaction with the content covered within the modules using a 5 point Likert scale (1=Not satisfied to 5=Highly satisfied). At the final time point, trainees rated their satisfaction with the post-training coaching sessions (on same scale) as well as the relevance and applicability of the LTI training program for different types of trainees (1=Strongly disagree to 4=Strongly agree). Trainees also compared the LTI to other career development programs (1=Much worse to 5=Much better).

LTI website evaluation. Participants who had access to the LTI website (WTO and LWC groups) completed an additional questionnaire at the final time point. Using a 5 point Likert scale (1=Not at all to 5=Extremely), trainees rated the quality and usability of the overall LTI website as well as individual LTI Web resources. LWC trainees also rated (on same scale) the degree to which access to the LTI website enhanced their overall training experience.

Results

Attrition Analyses

Over the course of the entire project, a total of 12 trainee participants (32%) dropped out of the study. Chi-square analyses revealed no selective attrition as a function of study condition, race, gender, or career level. Reasons cited for dropping out included limited time, loss of job or tenure opportunities, and changing of institutional affiliation during the course of the study.

Knowledge Analyses

Descriptive analyses were conducted to examine trainees’ mean percent correct scores on the knowledge questionnaire at each time point by study condition. Table 4 displays these results. Then, a one-way Analysis of Variance (ANOVA) with study condition as the between-subjects factor was conducted to test whether these mean percent correct scores varied significantly across study conditions for any time point. There were no significant differences across groups prior to LTI training, immediately following LTI training, or at 3-month follow-up. At these time points, trainees demonstrated the same level of knowledge regardless of study condition. However, significant differences across groups emerged at the 6-month follow-up (F (2,19)=3.72, p<.05) and became more pronounced at the 9-month follow-up (F (2,22)=9.05, p<.01). Post-hoc mean comparison tests revealed that, at 6-months following LTI training, the LWC group demonstrated significantly higher knowledge than the WTO or LTO groups, which were not significantly different from one another. At 9-months, the LWC group was still significantly higher than the other two groups, but the LTO group’s knowledge level had dropped, such that this group was now significantly lower than WTO.

 Table 4. Average Percent Correct for Knowledge Questionnaire at Each Time Point by Study Condition.

	Data Collection Time Point
Study Condition	Pre-training	Post-training	3 month follow-up	6 month follow-up	9 month follow-up
Live Training Only (LTO)	59%	68%	67%	68% B	61% C
Web Training Only (WTO)	61%	69%	68%	69% B	69% B
Live Web Combo (LWC)	66%	70%	73%	80% A	79% A
Note. At 6- and 9-month follow-up, means varied significantly by study condition. Groups with the different letters were significantly different from one another at that time point.

In order to investigate trends in knowledge over time, repeated measures Multivariate Analyses of Variance (MANOVA) were conducted with time point as the within-subjects repeated factor and study condition as the between-subjects factor. The first MANOVA examined changes in knowledge from pre- to post-training to test whether LTI training resulted in significant knowledge acquisition. Findings revealed a significant time effect (F (1,32) = 23.88, p<.0001) that did not vary significantly across study conditions. Therefore, trainees in all three study conditions showed significant knowledge gains from before training to immediately after participating in the LTI training (a 7% gain on average from pre- to post-training).

Next, a repeated measures MANOVA was conducted to examine retention of knowledge over the course of the study, i.e., changes in knowledge gains from post-training to 9-month follow-up. Results indicated retention of knowledge over time varied significantly by study condition (F (2,22) = 4.75, p< .01). As Figure 2 shows, scores for LTO participants were relatively stable over time, but dropped at 9-months to a level essentially equivalent to the group’s pre-training level. Scores for WTO participants showed a highly stable pattern over time with no change in retention level from post-training to 9-month follow-up. Scores for LWC participants showed an increase from 3- to 6-months that continued to be present at 9-month follow-up.

Figure 2. Patterns of Knowledge Scores over Time by Study Condition

LTI Training Evaluation

To examine whether satisfaction with training differed across delivery modes, satisfaction scores were aggregated across the LTO and LWC study conditions (i.e., because both groups participated in the live LTI training workshop). ANOVAs revealed no significant difference in post-training satisfaction ratings for trainees who participated in the live LTI training workshop versus those who participated in LTI online e-trainings. Each set of trainees reported equivalently high levels of satisfaction with the content covered within the LTI training modules regardless of delivery mode [live training average satisfaction M = 4.24 (SD = .49) versus e-training average satisfaction M = 4.19 (SD = .32)]. In addition, all trainees reported high satisfaction with the post-training coaching experiences with LTI faculty [live training average satisfaction M = 4.61 (SD = .50) versus e-training average satisfaction M = 4.64 (SD = .49)]. However, when asked whether any LTI faculty member was currently serving as a mentor for the trainee, chi-square analyses revealed a significant difference across study conditions (X 2 (2)=6.81, p<.05). While 14% of WTO and 42% of LWC trainees reported having an on-going mentoring relationship with an LTI faculty member, 0% of LTO trainees reported such a relationship.

At the end of the study (9-month follow-up), trainees rated the degree to which the overall LTI training program was applicable and relevant for different types of trainees. A one-way ANOVA was conducted to test whether ratings differed by study condition. As Table 5 displays, trainees in all conditions reported the LTI was highly relevant and applicable for all researcher sub-groups as well as for researchers more generally. In fact, 97% of all trainees recommended the LTI for female researchers and 100% recommended the LTI for researchers who are persons of color. While 100% of all trainees recommended the LTI for more junior researchers, significant disagreement across groups was present regarding applicability for more advanced researchers (F (2,21)=4.80, p<.05). Only 64% of WTO trainees reported the LTI training program was applicable to more advanced researchers versus 90% of the other two groups.

Table 5. Overall LTI Evaluations by Study Condition

	Mean (std)
Area Assessed	LWC	LTO	WTO
Recommend for any researcher	3.91 (.30)	3.91 (.30)	3.64 (.67)
Recommend for women	3.67 (.49)	3.86 (.23)	3.41 (.63)
Recommend for persons of color	3.79 (.40)	3.86 (.23)	3.86 (.45)
Relevant for more junior career	3.92 (.29)	3.91 (.20)	3.55 (.57)
Relevant for more advanced career	3.58 A (.52)	3.36 A (.71)	2.50 B (.55)
Note. Ratings were made on a 4-point scale from 1=Strongly disagree to 4=Strongly agree. Columns with the different letters were significantly different from one another for that item.

When asked whether the LTI was comparable to other career development programs the trainees had attended, 100% of LWC, 89% of LTO, and 57% of WTO trainees reported the LTI was “better” or “much better.” The remainder of each group reported the LTI was similar in quality and content to other career development programs. In addition, 88% of LWC trainees reported that having access to the LTI website enhanced their overall training experience.

LTI Website Evaluation

For the subset of participants who had access to the LTI website (LWC and WTO), analyses were conducted to examine ratings of the online tools and features associated with the LTI training program. Results indicated the e-courses were seen as the most important feature of the website (M = 4.25, SD = 1.1), followed by the Ask-an-Expert feature (M = 3.70, SD = 1.14) and the discussion boards (M = 3.35, SD = .99). Trainees reported the online tools and features of the LTI website were very easy to use (M = 3.76, SD = .92) and well organized (M = 4.06, SD = .99).

Discussion

Researchers face a variety of challenges launching their career and many of the skills needed for success are not well taught—if at all—during formal academic training (Reynolds et al., 2007; NRC, 2005a; BWF-HHMI, 2004). While training typically focuses on research skills, little attention is given to the career navigation skills (such as planning, negotiation, and management) needed to successfully establish and maintain an independent research career. Researchers who are women or persons of color face additional obstacles, such as lack of similar role models and stereotyping in the workplace, which further increase the risk that they will leave the research career path (NAC, 2001; NSF, 2004). Over the past several decades, there has been an alarming decline in the researcher workforce in the social, behavioral, and mental health fields—both generally and for researchers of diversity more specifically (Sung, Crowley, Genel, et al., 2003; NRC, 2005b). Without adequate numbers of researchers, our nation’s ability to move these sciences forward to improve public health is threatened. In response to a federal call to action, the goal of the current project was to develop and test the Leadership Training Institute (LTI) to support the career growth of social, behavioral, and mental health researchers who are women and/or persons of color.

Through a two-year iterative process of testing and revision with relevant stakeholders, the LTI curriculum was finalized with training content specifically designed to help diverse researchers: 1) establish career goals that align with their personal and professional priorities, 2) navigate funding issues, 3) build support networks for research and career advancement, and 4) expand research team management and leadership skills. However, traditional in-person training programs can only reach a small number of researchers and logistical barriers (e.g., travel, time) often limit accessibility. Therefore, in an effort to increase access to and use of the LTI, both an in-person and a parallel online training program were developed with the primary objective of examining the relative impact of each delivery mode—alone and in combination.

This project represents the first attempt to conduct a rigorous, longitudinal evaluation of in-person versus online training delivery modes for research career development. Overall, findings from this study underscore the potential value of using the Internet to increase opportunities for diverse researchers to participate in specialized training and networking to support career advancement. Consistent with past studies examining online education for professionals (e.g., Chumley-Jones et al., 2002), findings from this project supported the “no significant difference” phenomenon across delivery modes for knowledge acquisition as well as trainee satisfaction. In other words, trainees who participated in the LTI training modules solely via the website demonstrated equivalent knowledge gains from pre- to post-training as did trainees who participated in the live, in-person LTI workshop. Further, trainees were equally highly satisfied with their training experience regardless of delivery mode.

Little is known in the research literature about whether these equivalent pre-post knowledge gains are maintained over time (Cook, 2009). Unlike most prior studies, however, this project extended its follow-up period to 9-months post-training (online or in-person) in order to examine retention of knowledge gains over time for each delivery mode. Results revealed that knowledge levels across the three study conditions began to diverge at six months and continued to reveal more pronounced differences across groups at nine months following completion of the LTI training modules. Specifically, at nine month follow-up, trainees who only participated in the live LTI workshop (LTO group) showed a decline in knowledge gains to essentially the same level as pre-training. In contrast, the group of trainees who only had access to the LTI via the website (WTO group) maintained their post-training knowledge gains throughout the 9-month period. These findings suggest that on-going access to online training resources and tools may play a role in maintaining knowledge gains over the longer term and that, of the two delivery modes, online training appears to be equally or more effective than in-person training alone.

In contrast to the LTO and WTO groups, results for the combined delivery mode—trainees who both participated in the live LTI workshop and had on-going access to the LTI website (LWC group)—showed not just maintenance of knowledge gains, but also enhancement over time. It is important to note that LWC trainees did not have access to the LTI website until 3-months following the live event and it was at the 6-month follow-up when the group’s increase in knowledge level occurred. Thus, the combination of in-person and online training appeared to be most effective for knowledge gains. Interestingly, the likelihood a trainee would have an on-going mentoring relationship with an LTI faculty member paralleled the pattern found for knowledge. None of the LTO trainees had such a relationship at 9-month follow-up, whereas 14% of WTO did and LWC trainees were most likely (43%) to have an LTI faculty mentor.

It may be that attending the in-person event primed trainees so that they were motivated to take full advantage of the online resources. And, by providing a way to continue involvement after the live training event (via e-training materials and online networking tools), access to the LTI website significantly increased the trainees’ engagement as well as the overall effectiveness of the training program. LWC trainees clearly saw their training experience as exceptional, with the majority (88%) reporting the LTI was “much better” than other career development programs. Future research is needed to tease apart the degree to which these positive effects are due to online access to training resources versus networking tools for on-going communication with faculty and other trainees.

Overall, findings from this study provide considerable support for the LTI training program as a valuable and needed resource for diverse researchers. The training content was seen as generally applicable to any researcher, but also particularly useful for helping researchers who are women or persons of color more effectively address the career challenges they may face. The training program was also seen as most relevant for researchers early in their career as they transition to independent research scientist and tenured faculty. The in-person LTI training experience was significantly enhanced through subsequent access to the suite of online LTI tools and resources. Further, being able to access training modules, expert advice, and other career development tools via the LTI website was, in and of itself, a valuable and effective tool for enhancing outcomes for diverse researchers.

Prior research has repeatedly shown specialized training and networking opportunities to be essential ingredients for successful career growth and advancement (Nathan, 1998; Pincus et al., 1993; Reynolds et al., 2007). However, the high costs of travel, scheduling conflicts, and limited availability of in-person career development programs means that few researchers are actually able to benefit from such opportunities. By employing web-based technologies to reduce logistical barriers to participation, career development programs can be broadly disseminated so the number of researchers who participate can be maximized. Access to online training is a cost-effective way to provide researchers with the key knowledge and resources (e.g., mentoring, peer support) that positively influence career success. In effect, retaining diverse researchers on the research career path will translate into better science and better public health outcomes (Sung et al., 2003; NRC, 2005b)

Limitations

Several limitations to the current study should be noted. Due to funding parameters, the study included a small sample size, which limited the power to conduct sub-group statistical analyses. While review of sub-group means indicated similar findings across sub-groups, it was not possible to test whether findings differed by gender, race, or career level. Future research utilizing a larger sample size is needed to both replicate the findings across training delivery modes and to investigate whether these findings vary for key demographic sub-groups.

As with many longitudinal studies, it was difficult to retain these busy professionals throughout the entire data collection process (i.e., 9 months past their completion of the training modules). Therefore, attrition was significant, which further impacted the ability to conduct more detailed statistical analyses. Though no selective attrition by study condition or demographics was found, anecdotally, it was observed that individuals with more experience or interest in web-based technologies were more likely to complete their online surveys on time (for all groups) as well as more readily engage with the online features (for WTO and LWC groups). It is possible that additional training or help functions may have eased the procedures for some researchers who were less comfortable with technology.

Generalizability of the research findings for the current study may also be limited. Although the LTI was well designed for its target population, some of the training content more narrowly focused on barriers to career success experienced by researchers who are w omen and/or persons of color, and the participant sample in this research reflected this more narrow set of trainees. Further empirical testing is needed to assess the LTI’s effectiveness for researchers more generally. At the same time, it should be noted that many topics addressed in the LTI reflect concerns shared by many researchers, particularly early career researchers (e.g., finding good mentoring, needing further training in grant writing and professional leadership skills), and could be applicable across a wide range of research fields (e.g. intervention trials, prevention health science, social and behavioral health across the lifetime, services economics). Though the LTI participants for this study reported the training program was highly applicable to researchers in general, future research should test this assumption.

Future Directions

Providing career development opportunities via online, e-learning platforms provides a cost-effective, time-efficient, and effective means to broadly disseminate these valuable resources to as many researchers as possible. Given the high levels of satisfaction reported by trainees and the positive benefits of the web-based components, the technology and framework of the online LTI training program may be a useful reference for future efforts to create online trainings and tools for career development of professionals more generally. For example, it is likely that any novice researcher (not necessarily just women and people of color) may benefit from the framework developed for the current project. However, in the literature on web-based professional training, little is known as to whether trainee satisfaction and positive changes in knowledge translate into substantive changes in practice or career progress (Cook, 2009). An important area for future research is to investigate the degree to which online career development training programs, such as the LTI, result in demonstrable growth and advancement in key career-related outcomes. Further exploration of the impact of training on career outcomes would also benefit from inclusion of a no-training condition control group in order to examine the impact of time (i.e., natural career development) on these outcomes,

Acknowledgements

This research was supported through funding from the National Institute of Mental Health (Contracts: HHSN271200774104C and HHSN278200553104C). The authors are immensely thankful to the LTI faculty members who devoted such time and energy to developing and implementing the LTI training curriculum as well as to all the trainees who participated in this research.

References

Balon, R., Guerra, E., Meador-Woodruff, J. H., Oquendo, M. A., Salloum, I. M., Casiano, D. E., & Nemeroff, C. B. (2011). Innovative approach to research training: Research colloquium for junior investigators. American Journal of Psychiatry, 35, 11-14.

Bell, D. S., Fonarow, G. C., Hays, R.D., & Mangione, C. M. (2000). Self-study from web-based and printed guideline materials: A randomized, controlled trial among resident physicians. Annals of Internal Medicine, 132, 938-46.

Burroughs Wellcome Fund & Howard Hughes Medical Institute. (2004). Making the right moves: A practical guide to scientific management for postdocs and new faculty. Research Triangle Park, NC. Retrieved from http://www.hhmi.org/labmanagement

Chang, S., Hughes, D. C., & Chamberlain, R. (2008). Works-in-progress: Guiding junior scientists through career development applications. Journal of Cancer Education, 23(3), 142-148.

Chen, M. (2001). Factors affecting the adoption and diffusion of XML and Web services standards for E-business systems. International Journal of Human-Computer Studies, 58, 259-279.

Chumley-Jones, H., Dobbie, A., & Alford, C. (2002). Web-based learning: Sound educational method or hype? A review of the evaluation literature. Academic Medicine, 77, S86-S93.

Cook, D. A. (2009). The failure of e-learning research to inform educational practice, and what we can do about it,Medical Teacher, 31(2), 158-162.

Cook, D., Levinson, A., Garside, S., Dupras, D., Erwin, P., & Montori, V. (2008). Internet-based learning in the health professions: A meta-analysis. JAMA: Journal of the American Medical Association, 300(10), 1181-1196.

DeRosier, M. E., Kameny, R., & Hehman, C. (2007). Effective dissemination of web-based training materials for mental health researchers. Final Report for the National Institute of Mental Health. Bethesda, MD: NIMH.

DeRosier, M. E., Kameny, R., & Hehman, C. (2010). Web-based training and networking tool to promote broader dissemination of mental health research training and educational programs . Final report for the National Institute of Mental Health. Bethesda, MD: NIMH.

DeRosier, M. E., & McMillen, J. (2010). Increasing diversity in advanced research and executive leadership in mental health: The leadership training institute . Final report for the National Institute of Mental Health . Bethesda, MD: NIMH.

Donovan, C., Hodgson, B., Scanlon, E., & Whitelegg, E. (2005). Women in higher education: Issues and challenges for part-time scientists. Women's Studies International Forum, 28(2-3), 247-258.

El-Guebaly, N., & Atkinson, M. (1996 ). Research training and productivity among faculty: The Canadian Association of Professors of Psychiatry and the Canadian Psychiatric Association survey. Canadian Journal of Psychiatry, 41(3), 144-149.

Ernst, J. V. (2008). A comparison of traditional and hybrid online instructional presentation in communication technology. Journal of Technology Education, 19(2), 40-49.

Feldhaus, C., & Fox, P. (2004). Effectiveness of an ethics course delivered in traditional and non-traditional formats. Science & Engineering Ethics, 10(2), 389-400.

Ferguson, J. M., & DeFelice, A. E. (2010). Length of online course and student satisfaction, perceived learning, and academic performance. International Review of Research in Open and Distance Learning, 11(2), 73-84.

Githens, R.P., Crawford, F.L., & Sauer, T.M. (2010). Online occupation education in community colleges: Prevalence and contextual factors. National Research Center for Career and Technical Education. Louisville, KY: University of Louisville.

Hobbs, D. (2002). A constructivist approach to Web course design: A review of the literature. International Journal on E-Learning, 1, 60-65.

Jeste, D. V., Halpain, M. C., Trinidad, G. I., Reichstadt, J. L., & Lebowitz, B. D. (2007). USCD’s short term research training programs for trainees at different levels of career development. Academic Psychiatry, 31, 160-167.

Johnson, M. O., Subak, L. L., Brown, J. S., Lee, K. A., & Feldman, M .D. (2010). An innovative program to train health sciences researchers to be effective clinical and translational research mentors. Academic Medicine, 85, 484-489.

Kupfer, D.J., Hyman, S.E., Schatzberg, A.F., Pincus, H.A., & Reynolds, C.F. (2002). Recruiting and retaining future generations of physicians scientists in mental health. Archives of General Psychiatry, 59(7), 657-660.

Kupfer, D. J., Schatzberg, A. F., Grochocinski, V. ., Dunn, L. O., Kelley, C. A., & O'Hara, R. M. (2009). The Career Development Institute for Psychiatry: An innovative, longitudinal program for physician-scientists. Academic Psychiatry, 33(4), 313-318.

Lapsley, R., Kulik, B., Moody, R., & Arbaugh, J. B. (2008). Is identical really identical? An investigation of equivalency theory and online learning. Journal of Educators Online, 5(1), 1-19.

Li, Q., & Akins, M. (2005). Sixteen myths about online teaching and learning in higher education: Don't believe everything you hear. TechTrends: Linking Research & Practice to Improve Learning, 49(4), 51-60.

Nathan, D. G. (1998). Clinical research: Perceptions, reality, and proposed solutions. National Institutes of Health Director's Panel on Clinical Research. Journal of the American Medical Association, 280 (16), 1427-1431.

National Advisory Council (2001). Report of the National Advisory Mental Health Council’s Workgroup on Racial/Ethnic Diversity in Research Training and Health Disparities Research. Retrieved from fhttp://www.nimh.nih.gov/council/diversity.pdf

National Institute of Mental Health (2008). National Advisory Mental Health Council Workgroup on Research Training Report:Investing in the future. Retrieved from www.nimh.nih.gov/about/advisory-boards-and-groups/namhc/reports/nimh-diversity.pdf www.nimh.nih.gov/about/advisory-boards-and-groups/namhc/reports/nimh-diversity.pdf

N ational Research Council (2005a). Bridges to independence: Fostering the independence of new investigators in biomedical research. Washington, DC: National Academic Press.

National Research Council (2005b). Addressing the nation’s changing needs for biomedical and behavioral scientists. Washington, DC: National Academies Press.

National Science Foundation (2004). Women, Minorities, and Persons with Disabilities in Science and Engineering. Division of Science Resource Statistics. Retrieved from http://www.nsf.gov/statistics/wmpd

O'Hara, R., Cassidy-Eagle, E. L., Beaudreau, S. A., Eyler, L. T., Gray, H. L., Giese-Davis, J., Hubbard, J., &Yesavage, J. A. (2010). Increasing the ranks of academic researchers in mental health: A multisite approach to postdoctoral fellowship training. Academic Medicine, 85, 41-47.

Pincus, H. A., Dial, T. H., & Haviland, M. G. (1993). Research activities of full-time faculty in academic departments of psychiatry. Archives of General Psychiatry, 50, 657-664.

Pion, G. M., & Cordray, D. S. (2008). The Burroughs Wellcome Career Award in the Biomedical Sciences: Challenges to and prospects for estimating the causal effects of career development programs. Evaluation & the Health Professions, 31(4), 335-369.

Reynolds, C. F., Pilkonis, P.A., Kupfer, D.J., Dunn, L., & Pincus, H.A. (2007). Training future generations of mental health researchers: Devised strategies for tough times. Academic Psychiatry, 31(2), 152-159.

Russell, T. L. (1999). The No Significant Difference Phenomenon. Chapel Hill, NC: Office of Instructional Telecommunication, North Carolina State University Press.

Shavers, V. L, Fagan, P, & Lawrence, D, (2005). Barriers to racial/ethnic minority application and competition for NIH research funding. Journal of the National Medical Association, 97, 1063–1077.

Siebert, D., Siebert, C., & Spaulding-Givens, J. (2006). Teaching clinical social work skills primarily online: An evaluation. Journal of Social Work Education, 42(2), 325-336.

Stumpf, A. D., McCrimon, E., & Davis, J. (2005). Carpe diem: Overcome misconceptions in community college distance learning. Community College Journal of Research and Practice, 29, 337-367.

Sun, P., Tsai, R. J., Finger, G., Chen, Y., & Yeh, D. (2008). What drives a successful e-learning? An empirical investigation of the critical factors influencing learner satisfaction. Computers & Education, 50, 1183-1202.

Sung, N., Crowley, J., Genel, M., Salber, P., Sandy, L., Sherwood, L., & Rimoin, D. (2003). Central challenges facing the national clinical research enterprise. Journal of the American Medical Association, 289(10), 1278-1287.

Turner, C., González, J., & Wood, J. (2008). Faculty of color in academe: What 20 years of literature tells us. Journal of Diversity in Higher Education, 1(3), 139-168.

U.S. Department of Health and Human Services. (2003). Research-based web design & usability guidelines. Washington, DC: U.S. Department of Health and Human Services.

Villaverde, J. E., Godoy, D., & Amandi, A. (2006). Learning styles recognition in e-learning environments with feed-forward neural networks. Journal of Computer Assisted Learning, 22, 197-206.

Walters, D., & Egert, C. (2000). Can consideration of learning styles improve web-based learning? Proceedings of the AACE World Conference on the WWW and the Internet (WebNet) San Antonio, TX.

Watson, S. W., & Rutledge, V. C. (2005). Online course delivery and student satisfaction. Retrieved from: http://eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED490363

Wutoh, R., Boren, S., & Balas, E. (2004). eLearning: A review of internet-based continuing medical education. Journal of Continuing Education in the Health Professions, 24(1), 20-30.

Yager, J., Waitzkin, H., Parker, T., & Duran, B. (2007). Educating, training, and mentoring minority faculty and other trainees in mental health services research. Academic Psychiatry, 31, 146-151.

Zerhouni, E. (2005). Translational and clinical science – Time for a new vision. New England Journal of Medicine, 353 (15), 1621-1623.