Faculty Learns Curriculum and Teaching Capacities: Online Training Evaluation 

Luis M. Villar
Professor in the School of Education
Universidad de Sevilla
Seville, Spain
mvillar@us.es

Olga M. Alegre
Professor in the School of Education
Universidad de La Laguna
Canary Islands, Spain
oalegre@telefonica.net

     Abstract

This article addresses the choice of an appropriate procedure for the assessment of digital portfolios used in academic staff development at the two Canarian universities. The study includes a comprehensive definition of Curriculum and Teaching Capacities (CTC) in higher education and a formative assessment of a Professional Development of University Faculty (PDUF) programme as a model for faculty training. Using purposive sampling, twenty-nine university teachers were involved in an online course over an 11-week period. Criteria used for analysis were measures of instructors' attitudes and learning tests. Overall, faculty reported a change in their acquisition of CTCs, heading towards a more comprehensive and quality teaching. Examining the learning experiences of faculty has university significance for PDUF and CTC.

Keywords: Curriculum and teaching capacities; Digital portfolios; Online faculty development program; Formative assessment.

Introduction

In order to understand the complexity of European Convergence in higher education, Spanish universities have developed many short length faculty-training programmes. These programmes are coherent series of professional meetings and learning activities spread over periods of months, usually with an element of formal assessment. As expressed by Gibbs and Coffey (2004: 89):  

“Much training is explicitly oriented towards developing teachers’ teaching skills, especially their classroom practice.”

Staff development programmes have many varied goals, rationales and training processes. These programmes are concerned with improving instruction, particularly in their content and orientation. They provide models of instruction and sources of new emerging curriculum information for the faculty member to choose from and make use of, such as sample discipline guidelines, objectives, instructors’ skills, organizational methods, assessment, and books on teaching techniques and instructional strategies (Aleamoni, 1997). Besides, most continual professional improvement programmes are designed for instructors at the early stage of their university teaching careers although some also include more experienced academics, thus supporting stated institution’s “capacity to survive”. Practically speaking, this implies providing the time and institutional support necessary for ensuring an ongoing, collaborative staff development. In this sense, Camblin and Steger (2000: 4) advocate the dynamics of faculty and institutional vitality in the following terms:  

"Both the personal and professional well-being of faculty and the organizational structure of institutions have been affected by the changing nature of higher education."

The Case of the Professional Development of University Faculty (PDUF)

Essentially, the Professional Development of University Faculty (PDUF) is a voluntary programme involved in a continuous process of advancing the specific disciplinary expertise, pedagogical competencies and renewal designed to enhance personal awareness in individual faculty members, and which includes factors such as strong teaching beliefs and values that demand pedagogical excellence in the university profession (Caffarella and Zinn, 1999). Additionally, PDUF encompasses three types of components: (1) online faculty learning experiences in a reliable platform from which to operate digital portfolios, (2) formal teaching innovations connecting basic capacities and interdisciplinary knowledge, and (3) promotion of a greater cross-disciplinary communication among faculty for organizational development strategies. Universities need to tie their mission goals more explicitly to the teaching requisites of the faculty.

PDUF recognizes that faculty at various stages of their academic careers have different objectives regarding the advancement of academics, hence requiring diverse training strategies. Accordingly, the amount of time needed to complete the programme must be negotiated with the institutions. As maintained by Fitzgibbon and Jones (2004), the social dimensions of learning and contextual factors have to be emphasised. Therefore, before starting the course, a workshop was developed for a face-to-face induction into PDUF. Finally, faculty recognition was included through an official policy statement (i.e., learning certificate as an extrinsic reward), which gave internal motivation and encouragement to participate in the course.  

Web-based PDUF course: principles of instructional design

PDUF includes planning, organizing, structuring, tracking, reporting, communicating assessments, and many other principles, that take time and require orderliness on the part of the online programme advisers, which are critical issues in its design (Nijhuis and Collis, 2003). In this article, a staff training process is described which the authors have used successfully in other Spanish universities to guide the design and implementation of Web-based training courses that promote teaching knowledge construction. Currently, instructional design applied to Web-based learning environments is guided by the principles of an instructional systems design (Oliver and Herrington, 2003; Kandlbinder, 2003). The authors' experience has shown the following four-stage design process, which is customized to participant instructors' needs, to be an efficient strategy for conducting this study.  

·   The design of sequenced, structured and comprehensive lessons. The requirement of learning activities to engage and direct the participant instructors in the process of teaching knowledge acquisition. And also, the development of capacities in teaching and learning that are applied or transferred to practical classroom settings.

·   The design and condition of communication supports for the online participant instructors to scaffold the teaching-learning process. Furthermore, to provide meaningful forms of feedback, and to share ideas and problems with colleagues.

·   The design and arrangement of the learning resources needed by the participant instructors to successfully complete the set activities and to facilitate the guidance.

·   The design and specification of the PDUF to give the universities and institutions feedback on matters relating to participant instructors’ learning.  

Such processes are included in the PDUF programme design for its success, as some researchers have recommended (Motiwalla and Tello, 2000; MacKenzie and Staley, 2000; Grant, 2004; Smith and Bath , 2004).

Following are some key features of the PDUF online delivery system found on the Desarrollo Profesional Docente Universitario website.

·   Participant instructors use two texts books (Villar, 2004; Villar and Alegre, 2004).

·

Each CTC includes a four-step approach to reflection following a particular order: Goals, Uses, Teaching scenarios and Case study.

·   Participant instructors discuss two topics in asynchronous forums: European Convergence issues, and Student mental effort to cope with the new European credit system. These are organized and released on a fortnight basis, but remain accessible throughout the course. The last forum includes postings that pose reflective questions (Socratic questions).

·   Participant instructors access e-mail from the browser for one-on-one interactions with PDUF advisers or other participant instructors.

·   Participant instructors browse the material with URL links to related articles and institutions, notes and grades from any location, and at flexible time schedules.

·   Generally speaking, participant instructors download Microsoft Power Point presentations, key concept maps and study guides and resources on to their personal computer.

·   Participant instructors submit learning activity assignments online using Web forms interface, or via e-mail. These are authentic activities that have real-university relevance and which present complex teaching-learning tasks to be completed over a sustained period of time.

·   Assessment related activity tasks attract instructors’ attention in virtue of non-assessed information activities.

·   Participant instructors complete ten online exams using Web forms with the responses recorded in the appropriate database on the server. Each CTC exam is programmed (random selection) to be unique and provides instant feedback and results to the participant instructors. In other words, it provides authentic assessment, which is seamlessly integrated with the learning activity assignments to formatively assess their understanding of basic concepts, and possibly gain the sense of progress.

·   Participant instructors’ satisfaction with the PDUF online course. They assess the quality of materials and training process as a formative evaluation for course revision.

·   Participant instructors meet with the two PDUF researchers and colleagues during real-time in a chat room to discuss course progress and content.

Operational faculty competence

The authors develop a framework for CTCs in higher education (Villar, 2004) adjusted to university organizations with a student-centred education, as they are collaborative, and focus on the learning experiences and processes in the social context (Badley, 2000). Participant instructors are expected to have a deep understanding of their scientific field together with pedagogical and didactic capacities of a particular specialized aspect of their discipline. Therefore, CTCs are defined as an integrated set of knowledge, beliefs, abilities and attitudes that are basic for good performance in various university teaching settings. Common elements in our PDUF programme are to develop capacities in the design of curriculum and course material, as well as acquiring didactic and guidance capacities (Tigelaar, Dolmans, Wolfhagen and Van Der Vleuten, 2004). Three principles predominate in the PDUF in training towards a CTC approach: helping participant instructors understand that academics and students are different, thus designing curriculum and implementing classroom methods respectful of diversity and identity; that professoriate are dependent on one another in social relations and classroom interactions; and that the online course increases self-decision making and learning by assessment. Hence, Table 1 shows the seven modules and ten CTCs that are proposed and that are consistent with teaching problem solving research findings.

Table 1. Modules and Capacities Framework  

Module I. Personal Identity

CTC1. Knowledge of student motivation and ability to promote students’ positive attitudes.

CTC2. Awareness of students’ diversity in all its forms.

Module II. Social Relations

CTC3. Capacity to solve students’ problems.

Module III. Curriculum

CTC4. Capacity to develop metacognitive skills in the trainee.

Module IV. Methodology

CTC5. Capacity to provide effective and free curriculum time.

Module V. Decision Making

CTC6. Knowledge of area being supervised (learning tasks, research, assessment, etc.).

Module VI. Interaction

CTC7. Teaching and didactic skills for large groups.

CTC8. Knowledge of questioning skills.

Module VII. Evaluation

CTC9. Knowledge of formative and summative evaluation.

Networking and assessment

As in other faculty development programmes, online training programmes vary with regard to several characteristics: lesson presentation (e.g., text only, text with multimedia materials), interaction with exercises (e.g., questions, quizzes), and interpersonal interaction (e.g., electronic mail, synchronous chat, asynchronous discussion in forums) (Sargeant et al., 2004).  

The current study looks more closely at the thought-processes of academics engaged in interpersonal interaction and the way they treat teaching as a scholarly activity. Gaining a better understanding of how academics naturally read and make judgments about CTC may help in the practice of reviewing teaching. This research also compiles digital portfolios (Woodward and Nanlohy, 2004). Participant instructors thus give added depth and understanding to learning through the portfolio process, an approach to reasoning about the teaching-learning practices widely used for assessment and feedback purposes (Quinlan, 2002; Smith and Tillema, 2003). Consequently, participant instructors first acquire the necessary information, and communication and technology skills to operate in the aforementioned personal-made platform.  

Since Web training is based on asynchronous structures, prompt feedback is much more difficult than in face-to-face situations (Song, Hu, Olney and Graesser, 2004). Therefore, a commitment was made to provide quick and responsive feedback, which required that the course leaders were readily available on a daily basis. Following their exams at the end of each CTC, participant instructors were asked to provide online feedback on the CTC they studied. The instruments for evaluating CTCs are Web based, thus enabling participant instructors to monitor the feedback they provide (Tucker, Jones, Straker and Cole, 2003).  

Finally, this article offers the PDUF as a training model which supports faculty development and captures the themes of collaborative learning, discussing, reflecting, and consultation, thus responding to Pittas (2000: 108) assumption:  

“Perhaps a more important measure of a programme’s success is to be found in the climate it creates for faculty development.”  

Purpose and hypotheses of the study

A major purpose of this study was to assess the relative importance of the characteristics and academic factors of participants in relation to both the PDUF evaluation and CTC learning. The study was designed with the intention of evoking faculty reactions to several PDUF programme factors (e.g. lessons content and structure, delivery method, and time) which could be key in detracting or enhancing the likelihood of faculty appertaining to the universities of the Canary Islands to take part in the training process. Finally, the study also sought to determine faculty’s perceptions on how development processes might benefit their learning (Brown and Kiernan, 2001). Hence, the specific aim of the study was as follows:  

·         To assess if faculty master, in their involvement with the online PDUF programme, a series of ten CTCs.

The research questions which were used to investigate the evidence on CTCs were:

1.   What are the CTC needs of participants?

2. Are there significant differences among participants in their knowledge, attitude, and behaviour toward CTC learning?  

Two types of statistical analyses guided the study. The first hypothesis was tested by descriptive statistics. Hypothesis 2 was tested at the 0.05 level of significance using two-tailed tests. The hypotheses were as follows:  

Hypothesis No. 1. All participants will affirm the recognition of CTC needs.

Hypothesis No. 2. There are statistically significant differences among participants’ knowledge, attitudes, and behaviours toward CTCs due to demographic characteristics and academic attributes.

Method

Participants and Settings

The study was set in the two Canarian Universities: La Laguna (ULL) and Las Palmas de Gran Canaria (ULPGC). All participant instructors were volunteers and met the following selection criteria: (a) University campus, (b) scientific field, and (c) professional merits. Demographics and other factors (academic, professional and preparation) are given in Table 2. A mix of experiences, roles and technical ability were broad amongst the two university groups. Regarding the age of faculty members, there was a mix of life cycles with two groups of the professoriate between the ages of 35 and 44. All of the respondents were full-time faculty at the two public Canarian Universities. With regard to degree, most of participants had a Ph.D. degree. From the participating faculty, seventeen were tenured at the rank of university ‘titular’, and fourteen held a contract appointment. In terms of teaching experience, this sample was biased towards the senior faculty. Disciplines from the five scientific fields like arts an

Table 2. Distribution of faculty members by demographics and other factors

Faculty members from across the departments were invited to participate in a distance education course, specifically through a specially-designed course delivery system. The two researchers conducted the face-to-face workshop and the distance course. The focus of the workshop was to introduce faculty to the university course training and delivery system. Faculty members from across the colleges and departments attended the course with the aim to develop skills and future online class teaching. The face-to-face workshop was compacted into two 2-h sessions at each university in order to give the faculty an opportunity to integrate aspects of distance education, such as synchronous and asynchronous communications.

Measures  

Four basic types of data were collected:

1.             Attributes, what faculty are (demographic characteristics). This was done by means of an online questionnaire.

2.             Curriculum and teaching beliefs and needs, what faculty know to be true (an online three-point scale of ten declarative statements used as a teaching diagnostic tool).

3.             CTC opinions and attitudes, what faculty think might be true and say they want (ten online five-point Likert-type scale CTC sheets with an additional open-ended question. Each sheet consisted of ten declarative statements). Besides, the authors decided for the instructors’ learning activities to be assessed on the basis of coherence, comprehensiveness, and clarity of their reflective portfolio (Darling, 2001).

4.             Capacity learning, what faculty actually know (ten online capacity multiple-choice item tests. Each test was composed of ten declarative statements).

Independent variables were organized into three areas:

1.      Demographic variables (gender and age).

2.      Academic variables, or personal qualities of participants that were essential to mastering those aspects of academic work (degree, professional appointment, teaching experience range, University, scientific field, area of knowledge, department, undergraduate degree programme teaching, school, major subject matter teaching), and

3.      Career development variables, or faculty’s productive pedagogical knowledge (educational training, and European culture). (Missing data, particularly involving certain variables with many levels, created constraints for testing them; as a result variable analyses were limited).

Dependent variables. Three different measurements were used to judge faculty’s prior experience, and to rate the PDUF quality and CTC learning.  

1.      Prior experience. This variable was defined by two items referring specifically to educational knowledge. For each item, respondents were asked to indicate the extent to which the educational training was a personal characteristic on a 5-point scale.

2.      PDUF quality. Eleven opinion and attitude questionnaires on capacity, adapted from common themes in the University training literature were employed to capture potential attitude change among all participants (e.g. ‘This capacity is pertinent to my teaching’). A Cronbach’ alpha (α = .998 standardized) computed for this instrument indicated a high degree of internal consistency.

3.      Self-assessment. Ten teacher-made multiple response CTC tests were used for measuring learning attainment. All the more, taking a test was understood as an on-task learning activity (e.g. ‘A process of group dynamics can be constituted by the following phases’). Once again, Cronbach’ alpha (α = .98 standardized) for all tests showed a high degree of internal reliability. Responses required selecting from a range of four item possibilities, and tests were administered at the end of each CTC lesson. Face validity, stem clarity, correct keying answer, and spelling of distracters were some of the determinants to be considered of the quality of the capacity tests. Overall, these α scores indicated that respondents were very likely to answer consistently on items belonging to the same instrument or test.  

Frequencies and means were generated using the statistical package SPSS 12.0. Independent samples t-tests and analyses of variance were computed to compare means for the independent variables analysed. To determine the significance of differences in frequencies, the χ² test was used.  

Result findings  

Information regarding instructors’ CTC needs was obtained in order to better examine the relevance of the PDUF in relation to participants' learning of a CTC. The scale was 1-3, with values of “1 = Not So Necessary,” “2 = Moderately Necessary,” and “3 = Very Necessary”. Figure 1 displays the percentage of all CTCs considered as moderately and very necessary by the respondents to the survey.  

Hypothesis No. 1 was supported. As Figure 1 indicates, all participants felt they needed professional training in all ten CTCs of the current UTPD programme. A majority of respondents indicated they would need training in: firstly, knowledge of student motivation and ability to promote students’ positive attitudes, capacity to solve students’ problems, and knowledge of formative and summative evaluation (93.1 per cent); secondly, awareness of students’ diversity in all its forms, capacity to provide effective and free curriculum time, and capacity to conduct own self-assessment process (89.7 per cent); thirdly, knowledge of questioning skills (86.2 per cent); and fourthly, knowledge of area being supervised (learning tasks, research, assessment, etc.), and teaching and didactic skills for large groups at 79.3 per cent.

Figure 1. Perceived CTC needs expressed by the participant instructors  

Chi-square difference tests were used to compare whether two independent variables (participants’ demographics and academic variables, i.e., a nominal variable – university degree - and an interval variable – age cycle -) had significantly different distributions across the PDUF capacity needs. Data were cast into several contingency tables. For instance, participant instructors among the 35-39 age-range were in much need of acquiring knowledge for conducting an own self-assessment process, χ ² = (4, N = 29) = 9.97, p < .041. Regarding the relationship between participants without experience in European convergence, learning the large groups teaching and didactic skills capacity was considered of some need, χ ² = (2, N = 29) = 6, p < .050.  

In terms of the quality of the PDUF, means and standard deviations on the ten CTC rating scale items are shown in Table 3. On each item, a one-way within-subjects ANOVA, or a t- test was conducted.

Table 3. Means and standard deviations for the PDUF quality scale items

                                                                                    M                                  SD    

Relevance                                                                     1.21                              .84

Usefulness                                                                    1.36                              .92

Appropriateness                                                            1.59                              1.12

Adaptation                                                                     1.72                              1.12

Tips                                                                              1.65                              1.12

Structure                                                                       1.58                              1.02

Reading                                                                        t-align:justify">Impact                                                                          1.83                              1.19

Time-Consuming                                                            1.39                              .93

Hypothesis No. 2 was partially supported. (See Tables 3 and 4 for t-test and ANOVA result summaries).  

T-tests revealed significant differences between the two university groups on the European convergence programme preparation. With regard to genre, faculty at the two universities revealed a significant difference of quality CTC items (tips and pertinence). Regarding university degree, independent t tests revealed significant differences on quality CTC items (relevance, usefulness, appropriateness, adaptation, tips, structure, pertinence, impact, and time-consuming) and CTCs (Knowledge of student motivation and ability to promote students’ positive attitudes, Capacity to solve students’ problems, Capacity to develop metacognitive skills in the trainee, and Knowledge of area being supervised (learning tasks, research, assessment, etc.) between PhD. Instructors and Bachelor participants.

Table 4 Significant t-Test Results for Demographic and Academic Factor Comparisons

Contrast                                                Variable                                    t                         p

ULL vs. LPGC                           European convergence                           2.381                < .028

Male vs. Female                        Tips                                                      2.520                < .018

Pertinence                                             2.329                < .028

Doctor vs. Bachelor                   

Relevance                                              -3.246               < .003

Usefulness                                             -2.804               < .009

Appropriateness                                     -2.638               < .014

Adaptation                                             -3.136               < .004

Tips                                                       -3.984               < .000

Structure                                                -3.074               < .005

Pertinence                                              2.875               < .008

Impact                                                    -2.249               < .033

Time-consuming                                     -2.072               < .048

CTC1                                                     2.268               < .032

CTC3                                                     -2.631               < .014

CTC4                                                     -2,147               < .041

                                       CTC6                                                    -2,187               < .038  

ANOVA results indicated effects for the scientific field groups on the quality CTC items (relevance, usefulness, appropriateness, adaptation, tips, structure, reading, impact, and time-consuming) and CTCs (Awareness of students’ diversity in all its forms, Capacity to solve students’ problems, Capacity to develop metacognitive skills in the trainee, and Knowledge of formative and summative evaluation).  

Table 5. ANOVA for Academic Factors: Scientific Field  

Variable                                    df                                 F value                          P value

Relevance                                 4                                  6.448                                        < .001

Usefulness                                4                                  7.342                                        < .008

Appropriateness                         4                                  4.409                                        < .008

Adaptation                                 4                                  3.140                                        < .033

Tips                                          4                                  2.808                                        < .048

Structure                                   4                                  3.927                                        < .014

Pertinence                                 4                                  5.396                                        < .003

Reading                                    4                                  3.146                                        < .033

Impact                                      4                             bsp;        < .037

Time-Consuming                        4                                  3.900                                        < .014

CTC2                        &nbbsp;           2.750                                        < .052

CTC3                                        4                                  4.782                                        < .006

CTC4                                        4                                  4.868                                        < .005

CTC9                                        4                                  3.798                                        < .016

Assessing the learning activities. As Oliver and Herrington (2003: 114) pointed out:  

"Designing a learning environment by commencing with the design of learning activities creates a setting where the focus of the planning centres on formulating the forms of learning outcome being sought rather than considering what content will be covered."  

Learning activities reflected the way the curriculum and didactic knowledge will be used in real university environments. The activity(ies) gave meaning and structure to the study of the PDUF course. In this sense, participant instructors completed 2,176 learning activities (see Table 6). Also, a principle of the learning process was peer assistance and review through providing guidance and feedback to the participant instructors in their learning process. Online help was often needed. Thus, coaching and scaffolding of learning was done by the two PDUF leaders, who diagnosed the strengths and weaknesses of a given participant instructor and tailored support accordingly. Table 3 reported participant instructors’ changes in their interest in and willingness to respond to learning activities as the course progressed. The initial frequency of learning activities was, however, higher than final activity responses. Data demonstrated that time commitment to CTCs was not equally distributed. While the second CTC (awareness of students’ diversity in all its forms) took on a high frequency dedication, the last CTC (to conduct own self-assessment process) had a low or limited response frequency. At any rate, learning was fluent and participant instructors became aware of new possibilities regarding their teaching.  
 

   Table 6. Frequency of participant instructors’ CTC activities

     Note: A (Activity), Task (T), Practice (P) or Strategy (S).  

Moreover, data from Item 11 (open question) of the PDUF quality survey seemed to suggest that participant instructors’ concerns were focused on how university teachers could secure extra time to attend staff development courses. A brief outline of the kinds of remarks found under the most used CTCs illustrates this point.  

·