Introduction

The Open Source Portfolio (OSP) software for eportfolio learning and assessment has seen widespread adoption since its first release in 2003. It has grown into a fully nuanced open source eportfolio system, with capabilities that compare favorably to many commercial systems, and is now integrated into Sakai, an equally competitive collaborative learning environment (CLE). Several institutions that have piloted or implemented OSP to support learning and assessment have conducted research to determine its impact. This article surveys the current state of OSP development and use, shares the results of several of these research projects, and charts the implications of this work for the future of OSP and eportfolios in higher education. Research suggests that the use of OSP tools can have a significant impact on student learning and be used effectively as one component within programmatic and institutional assessment processes.

Overview of OSP

The Open Source Portfolio (OSP) is a comprehensive and flexible set of tools for composing, sharing, and assessing eportfolios within the Sakai Collaborative Learning Environment (CLE) (Open Source Portfolio, 2008). Its development and use is supported by a committed group of developers, educators, and learning support staff as part of the larger Sakai Project. OSP is a community source project (Wheeler, 2004). The code of the OSP tools is open source under the Educational Community License, and OSP’s development and maintenance is supported through the investments of several universities and vendors in the United States and Europe, with logistical and infrastructural support from the Sakai Foundation. At least 50 colleges and universities from around the world currently use the OSP tools, and the Sakai CLE, which includes OSP, has been implemented by a minimum of 200 institutions in at least 28 countries.

The development of the Open Source Portfolio began in 2003 with the formation of the Open Source Portfolio Initiative (OSPI), a collaboration of several institutions that were interested in developing eportfolio software. The initial open source version of OSP was based on code from a system in production at the University of Minnesota (Treuer & Jensen, 2003). OSPI also secured grant funding from the Mellon Foundation in 2004 to develop the next generation of OSP. This new version was developed by a functional team composed of eportfolio experts from seven institutions working with developers primarily from Indiana University and rSmart. Version 2.0 of OSP was released in June 2004. In 2005, the OSPI became part of the Sakai Project, and OSP tools were incorporated into the core of Sakai in the 2.4 release in 2007.

Over the last six years, numerous institutions have made significant contributions to the development of Sakai through creating functional requirements, contributing code, and offering community leadership. A partial list includes the Carnegie Foundation for the Advancement of Teaching, George Mason University, Indiana University, Portland State University, LaGuardia Community College, Marist College, Portfolios4U, RINET, rSmart, Serensoft, Syracuse University, the University of Amsterdam, the University of Michigan, the University of Minnesota, the University of Rhode Island, the University of Washington, and Virginia Tech University. The shared investments of these partners, combined with the Sakai Foundation’s strong commitment to ongoing support, warrants confidence in the sustainability and vitality of the OSP project.

The Open Source Portfolio Platform

OSP is a flexible and interconnected set of tools within the Sakai CLE. Together, the tools comprise a platform that supports a wide range of eportfolio teaching, learning, and assessment practices. Tools currently available in version 2.5 of OSP are listed in Table 1. (Additional tools, such as one that exports portfolios in IMS ePortfolio format, are available from OSP developers but are not yet part of the core distribution.) Out of the box, OSP tools, although flexible, require customization to meet the needs of the institution, instructors, and students who will use it. The system is designed with multiple levels of technical complexity to enable customization by users of varying expertise. Some tools, such as the Matrices and Wizards, are fully configurable via the interface so that an instructor can easily create his or her own matrix or wizard. Others tasks, such as creating a portfolio template for use in the Portfolios tool require XML coding. More extensive customization, such as integrating OSP tools with a student information system or modifying the way a particular tool operates, can require additional scripting or actual modification of the Java source code.

The OSP community offers ample support for institutions undertaking implementation and customization of OSP. Institutions can choose either to utilize in-house expertise or to purchase services and support from software companies such as rSmart and Serensoft. Launched in 2007, an OpenEd Practices library enables members of the OSP and Sakai communities to share the components of customized data structures (forms, matrices, wizards, portfolio templates) they have created. The library also contains supporting educational materials such as syllabi, handouts, and assessment rubrics (Opened Practices, 2008). Email lists and a documentation wiki also help community members share expertise on using OSP effectively (OSP Portfolios, 2008).

With careful design and customization, OSP can effectively support portfolio processes across a wide variety of contexts. OSP tools can also be used in conjunction with the larger set of tools for teaching, learning, and research available within the Sakai CLE. For example, the Goal Management tool in OSP is integrated with the Assignments tool in Sakai to enable links between non-portfolio-specific assignments and learning outcomes (Goal Management Tools, 2007).

Table 1: OSP Tools

Examples of three common types of portfolio illustrate how OSP tools interact. Portfolios for personal representation guide users in managing their virtual identities via online resumes, professional portfolios, and freeform portfolios. Portfolios for teaching and learning establish workflows that guide students in reflecting upon and sharing learning within and across disciplines. Portfolios for assessment and accreditation provide systematic reporting of results from courses and programs for institution-wide assessment. Typical scenarios for each type might take the following form:

Personal Representation: Students create an on-line resume using a Wizard to collect data in Forms for education, experience, and interests before accessing Portfolio Templates via the Portfolios tool to combine the Forms with uploaded files in chronological, skills-based, or teaching resumes. These can be shared with other users through email and the Web.

Teaching and Learning: General Education instructors use a Matrix to guide students in addressing learning outcomes. Students select and reflect upon evidence of learning, submit it for instructors to rate with the Evaluations tool, and share Matrix content using a Portfolio Template through the Portfolios tool.

Assessment and Accreditation: High school teachers create portfolio-worthy Assignments and link them with standards in the Goal Management tool. Students submit assignments for teachers to rate according to associated standards. Reports allow students to assess their work for inclusion in a graduation portfolio, allow teachers to manage student work across courses, and allow administrators to aggregate and display assessment data for each high school.

Uses of OSP

Across the many campuses that have implemented it, OSP is being used in diverse ways towards purposes that fall into each of the three categories: personal representation, teaching and learning, and assessment and accreditation.

Resumes are an important form of personal representation for many students. Numerous institutions, such as Rider University, include the ability to create and manage resumes in OSP as part of their portfolio offerings. Because typical resumes often fail to convey how accomplished people truly are, colleges and universities are using OSP to help their students, faculty, and community partners craft more powerful portfolio-based self-representations. Institutions such as the University of Michigan and Charles Sturt University in Australia guide students through the process of creating portfolios by using templates that offer some of the structure provided by the resume format without its limitations on content and scope. Others, such as Kapi’olani Community College and ITT Tech, invite faculty and employers to compose portfolios that represent their professional identities as well. Charles Sturt and LaGuardia Community College are also exploiting the freeform portfolio composition capabilities of OSP to offer additional flexibility and expressive range to students.

In addition to using OSP to support personal representation, many institutions are taking advantage of OSP tools to support teaching and learning. OSP is seeing widespread use in general education and honors programs at such institutions as Portland State University, Washington State University Vancouver, and UMass Boston. Learning communities in Douglass College at Rutgers University have access to a streamlined portfolio process that provides structure for documenting learning along with great opportunity for self-expression. At schools such as the University of Delaware, Virginia Tech, and Bentley College, OSP helps students document and synthesize their learning in their majors and professional fields. OSP is also being used to support co-curricular learning, such as leadership development at George Mason University and the University of Michigan, as well as for documentation of that learning through co-curricular transcripts at institutions such as Rider University, the University of Southern Maine, and Richard Stockton College.

A final, and for many institutions the central, function of OSP is to support assessment and accreditation. OSP has been used successfully to collect assessment data for re-accreditation of professional schools, such as the Syracuse University School of Education (see section 4 below). Many other institutions, such as Indiana University-Purdue University Indianapolis (IUPUI) and Weber State University are planning or piloting OSP to assess general education outcomes institution-wide. Secondary schools across Rhode Island, through the coordination of RINET, use the Rhode Island Electronic Portfolio System, a custom version of OSP, to track students’ progress achieving state educational standards throughout their high school years, culminating in a graduation portfolio that chronicles their learning.

Most institutions investigating the effectiveness of OSP have focused on the second and third categories of use, supporting teaching and learning and facilitating assessment and accreditation. Several institutions using OSP have conducted such research. The remainder of this article surveys the results.

Matrices and Matrix Thinking

Matrices was one of the first mature OSP tools. Because it has been available for over four years and has been central to many institutions’ work with OSP we have accumulated a significant body of data on its use in practice. Although Matrices is designed to manage the workflow of an assessment process, we have found that it also supports learning in a variety of contexts. Assessment processes supported by Matrices can be structured as learning experiences in and of themselves, a characteristic of effective assessment practices more generally (Maki, 2005). Research on OSP at IUPUI and Kapi’olani Community College, conducted through the Inter/National Coalition for Electronic Portfolio Research (I/NCEPR), shows that Matrices can have a notable effect on learning and engagement when used to promote "matrix thinking." Coined by Sharon Hamilton and Susan Kahn at IUPUI, matrix thinking is the process of reflecting on work originally created in another context in relationship to the intersection of multiple dimensions of a shared conceptual framework (Hamilton, 2006). While it can take multiple forms depending on the context, matrix thinking is characterized by integration of ideas across disciplines and learning experiences. Both IUPUI and Kapi’olani Community College use the Matrices tool to support matrix thinking, and both have produced data suggesting that that thinking has enhanced student learning.

Matrix Thinking in Capstone at IUPUI

Like many other institutions that have embraced electronic portfolios, IUPUI believes that the ability to critically self-reflect will be essential to its graduates’ success in professional and civic life. Research conducted in an English Capstone Seminar at IUPUI has found that Matrices can provide a conceptual framework that prompts powerful reflective thinking (Kahn & Hamilton, 2008). The seminar is intended as a culminating academic experience that provides students in their final semester an opportunity to apply their cumulative learning to an integrative project—development of a reflective senior portfolio, in this case—and to think about the future. The capstone course instructors designed a customized version of the OSP matrix, as illustrated in Figure 1.

Figure 1. IUPUI English Capstone Matrix

In this matrix, the rows represent learning outcomes—in this case, one of IUPUI’s six general education learning outcomes or “Principles of Undergraduate Learning” (PULs) and one of the English Department’s outcomes for the major. Student s select the PUL and English major outcome on which they wish to focus. The columns represent domains of personal development and intellectual maturation: lifelong learning (i.e., the development of independent learning capabilities); career (i.e., professional development); and “who I am as a citizen” (i.e., development of cultural competence, civic engagement, and appreciation of twenty-first century global complexities).

For each matrix cell, the assignment asks students to upload two or three pieces of past work that meet the two criteria for each cell. To complete the cell, they then reflect on that work in the context of the intersection between the two criteria. For example, if their most significant PUL is “critical thinking,” and their chosen career is “editor,” they articulate what aspects of critical thinking are most salient to a career in editing, and reflect on their chosen artifacts in relation to that conceptualization. That requirement seems to stimulate students to make connections across disciplines and learning experiences that are more insightful and profound than the instructors initially anticipated, leading them to label this kind of thinking “matrix thinking.” “Matrix thinking” results when students combine the elements of a matrix and use the resulting conceptual framework to think integratively about work initially created in a different context. For example, in a reflection on the critical thinking PUL and her career aspirations, one student wrote:

Islam.Y107.doc exemplifies my ability to be a critical thinker because I had to put forth significant effort to separate my emotions from the facts and research. I believe that objectivity and rationality are at the core of every serious student – this paper shows me that I can be a serious student. Every class that I have taken in political science, English, and philosophy has emphasized the importance of looking past the surface of things. Additionally, my education in the liberal arts has taught me that there is much more to things than what my emotions tell me there are. … My emotions are central only to my own experiences, and my critical thinking skills allow me to leap outside of my own experiences.

We see here an instance of a student bringing multiple disciplinary frameworks to bear on her understanding of the concept of “critical thinking” and identifying a common aspect of those disciplines: all emphasize “the importance of looking past the surface of things.”

This is but one example of the ability of a matrix to stimulate new integrative thinking and learning. In 2005, IUPUI’s research team analyzed numerous matrix reflections from both the capstone class and first-year learning communities. IUPUI researchers used a version of the Developmental Framework for Reflective Thinking, which was developed by a research team at Alverno College, another member of the I/NCEPR, through analysis of over one thousand eportfolio reflections (Rickards, et.al., 2008; Rickards & Guilbault, 2008). (Appendix 1 presents IUPUI’s adaptation of the Framework.) The framework classifies reflections as beginning, intermediate, or advanced in relation to students’ ability to self-assess, to use the assessment to improve their performance, and to situate their experiences in the context of lifelong learning. In adapting the framework for IUPUI, the researchers noted that those capstone students who completed reflections as directed tended to exhibit characteristics of advanced reflective thinking as defined by the Alverno researchers. 

Matrix Thinking at Kapi’olani

While Kapi'olani Community College does not use the matrix to track learning outcomes in the conventional sense, it engages students in reflection on their work in relationship to the intersection of multiple dimensions of a shared conceptual framework, which is by definition matrix thinking. In one of the programmatic matrices in Kapi’olani’s application of the Open Source Portfolio, the framework is cultural, using a voyaging metaphor to connect learning to six native Hawaiian values (Kapi’olani Community College, 2008). (See Figure 2.) The team of faculty researching the application through the I/NCEPR hypothesized that it would promote students’ abilities to learn successfully in college and stimulate a sense of empowerment that enables them to take responsibility for their own education. The central research question this study explored was whether this approach is, indeed, learning-centered. The students’ eportfolio works were placed, by the student, within cells of a matrix at the intersections of the six values and four stages of growth. This process gave students ownership of their works, experience, and learning, and validated their cultural intelligence. The result is a digital archive that provides evidence of what the student knows and a record of growth, both academic and personal.


Figure 2: Kapi’olani Hawaiian Values Matrix

In addition to uploading examples of work for each Matrix cell, students wrote a reflective analysis of their placement of works that explained the origins of the works, discussed how the selection and related experiences have challenged them to grow, and considered their plans for future learning. The reflective analysis requires a sense of self-awareness and displays evidence of an emotional connection to academic assignments that influence thinking and information processing. The research team designated first-year composition and second-year Hawaiian language courses for the study, and for first-year composition, designated a control class taught by the same instructor without the use of the e-portfolio system.

The results of two instruments, the Learning and Study Strategies Inventory (LASSI), and a survey based on the Community College Survey of Student Engagement (CCSSE) , along with analysis of the students’ reflections, suggest that the approach is highly learning-centered ( Kirkpatrick, Renner, Kanae, & Goya, 2008). The first instrument, LASSI, is a 10-scale, 80-item assessment of students’ learning strategies that provides immediate feedback on where they need to improve their knowledge, attitudes, beliefs and skills. The eportfolio students demonstrated significant improvement on eight of the ten LASSI scales, while control group students demonstrated improvement on five of ten of the scales; thus, three of the scales show improvement for only the eportfolio students: anxiety, time management, and test-taking strategies. In addition, the participating faculty reported that the values-driven eportfolio project was transforming such aspects of their teaching as their interactions with each other and their classroom strategies.

The second instrument, an eportfolio survey, administered at the end of the semester, used 12 questions based on the CCSSE for which the College has local and national benchmarks, and three instructor-generated questions. The CCSSE questions measure levels of student engagement in learning. In six out of the 12 CCSSE survey questions, the eportfolio results were significantly more positive than College and national benchmarks, while the other six questions showed no significant difference. The third measure examined students’ reflective analyses as they explained how they chose to place a given work in the Matrix. Analysis of these reflective comments yielded four general categories of personal growth: awareness of one’s strengths, recognition of areas where improvement is needed, increased understanding of the learning process, and recognition of learning beyond course content.

The findings from the LASSI and CCSSE surveys, combined with the analysis of students' reflective analyses and teacher feedback, present a picture of students who feel connected to their learning and empowered to extend what they have learned to other contexts. The I/NCEPR research team hypothesized that the act of developing the eportfolio would generate learner-centered behaviors such as motivation, orientation toward learning, and a deeper understanding of the relationship of the subject matter to Hawaiian values and the metaphor of the college experience as a journey. The LASSI and CCSSE results demonstrate that students in eportfolio courses developed a better ability to pursue learning in college and demonstrated an increased sense of empowerment. The central research question, whether this approach was, indeed, learning-centered, was answered in the affirmative.

The Hawaiian values in this approach are basically universal and easy to grasp, yet open to interpretation that demands student-instructor interaction and analytical thinking. Because the framework is based on an indigenous cultural metaphor rather than an imposed model of learning, the students informed their self-assessment and reflections with existing knowledge and cultural intelligence.

The Matrix, Assessment, and Institutional Research

A primary original purpose of the Matrices tool was to enable multi-dimensional assessment of learning outcomes on a programmatic and institutional level. While a number of institutions are moving towards the goal of undertaking such large-scale assessment, research to date indicates that the capabilities of an eportfolio technology are only some of many factors that influence the success of such initiatives; many institutions have a good deal of non-technical work to accomplish in order to use OSP effectively for this purpose. However, Matrices has proven an effective tool for gathering data and managing workflow and could prove a key component in larger systems of assessment and institutional research when campus cultures are ready to embrace them. Research on the use of OSP in systematic processes that examine learning suggests that they may provide not only the "hard data" needed for accreditation and public accountability but also richer accounts of student learning and development than are possible to obtain otherwise. Two case studies, which again make central use of the matrix tool, illustrate this.

Learning Through Assessment of General Education at Weber State

In 2006, Weber State University piloted the Open Source Portfolio (OSP) software as part of a larger initiative to assess its general education programs. As in the cases of IUPUI and Kapi’olani, the Matrices tool was central to the implementation. The goal of the project was to gauge whether students understood the university’s general learning outcomes, whether students felt that general education courses were helping them to develop the intellectual competencies associated with these outcomes, and whether OSP could help to answer the above questions. These questions had particular urgency because accreditors were examining Weber State’s general education assessment efforts.

Weber State used OSP to enable students to upload essays (or other artifacts) that they had produced in their courses. After submitting the artifacts, students were prompted to write short reflection pieces expressing how each artifact related to one of nine university-wide general education outcomes. The assessment involved ninety-nine students enrolled in four classes, who were introduced to OSP through class presentations and online tutorials. (See, for example, Fernandez, 2008b.) After the students had submitted their reflections, ten reviewers went through the reflections and attempted to gauge, through the use of scoring rubrics, whether the reflections provided evidence that students were achieving the intended program competencies.

The OSP pilot reported two general outcomes. First, the portfolio process facilitated formative assessment. As at IUPUI and Kapi’olani, OSP helped students think more deeply about the purposes and skills conferred through general education. Second, the pilot revealed that more work needed to be done to achieve effective summative assessment at Weber State. OSP could help Weber in its efforts to measure the quality of its general education comprehensively and programmatically but could not do so single-handedly.

A perennial concern among faculty at Weber State is whether modern information technology serves to vitalize or desiccate intellectual activity. Faculty continue to express these concerns about OSP. It is not clear to some whether the technology really promotes deeper and more considered reflection than a lighter weight technology such as, for example, a paper and pencil portfolio. Among the students who participated in the pilot, however, this concern was not so much in evidence: post-pilot surveys reveals that most felt OSP facilitated at least some forms of learning associated with the goals of general education (Fernandez, 2008a). Moreover, the student survey data doesn’t yield the type of anecdotes (or complaints) that would suggest that a lighter weight solution would have been superior (although, to be fair, the survey did not prompt students to make the contrast). And the quality of some of the submitted reflections suggests that the technology facilitated reflection. For instance, one student ruminated on how reading and writing about Alan Paton’s Cry The Beloved Country enhanced her appreciation of one type of learning that the Weber general education program attempts to instill:

By providing students with a space for reflection the technology demonstrated that at least some students were grappling with the core questions of general education and that OSP could play a positive role in catalyzing this reflection. Of course, the question remains as to whether OSP could catalyze this better than a paper and pencil portfolio and the pilot didn’t provide any definitive answers to this question. However, because an eportfolio is available anytime and anywhere a networked computer is available, the promise is that it will habituate students into doing reflections often, something that might be harder to catalyze with a paper and pencil portfolio that would need to be lugged around physically, and that students might be further motivated to reflect by being able to share their writing and receive feedback more easily and frequently.

While OSP catalyzed formative assessment, it was not a panacea for the methodological challenges that Weber State faces in summative assessment. The pilot found that OSP tools can facilitate data collection, but they cannot magically resolve the measurement problems that arise in deciding what information to gather and how to interpret it. To use OSP effectively for summative assessment at Weber State, sampling techniques need to be expanded and refined, curricular cultures need to be in place to encourage students to submit reflections over many semesters, scoring rubrics and reflection prompts need more refinement, and more time needs to be spent on calibration exercises.

But if the core challenges of summative assessment are related to methodology and the proper interpretation of evidence, it is still the case that assessment committees need systematic and reliable ways of collecting that evidence. The pilot found that in this capacity OSP holds much promise, since it is specifically designed to facilitate the collection of essays, reflections, and other digital learning artifacts that can be easily accessed for institutional assessment.

The pilot demonstrated one promising way in which this evidence can be collected and stored, and it clearly improved students’ opportunities to weigh their own progress in meeting general education goals. At Weber State many methodological, curricular and cultural challenges remain in deploying OSP on an institution-wide basis and steps are being taken to meet these challenges. Notwithstanding, the pilot demonstrated that OSP can help create an environment for systematic assessment of general education curriculum.

Leadership Identity Development at George Mason

At George Mason University, students developed portfolios representing themselves as leaders in a co-curricular setting, using evidence from both the classroom and their broader engagements. To truly understand how George Mason supports students’ development as leaders and increases understanding of leadership, Mason faculty and student affairs educators believe that they have to look beyond narrowly defined descriptions of knowledge or skills. While nearly everyone sees leadership as important, its definition is contested and often context-dependent. It means different things for different students, depending on their identities and aspirations, and these differences don’t necessarily align with the familiar distinctions between disciplines and between students and academic affairs around which many university programs are organized.

Rather than measuring progress towards pre-determined outcomes, as at Weber State, the leadership portfolios at George Mason are designed to provide a rich picture of what Yancey (1998) calls the "experienced curriculum,” shedding light on how students experience and make sense of the formal, “delivered” curriculum represented in syllabi, program descriptions, and statements of learning outcomes. The portfolios also situate that experience within the “lived curriculum” of learning throughout students’ lives during the years in which they are enrolled at George Mason. Students’ eportfolios capture their own conceptions of what leadership means and how the institution’s programs, curricular and co-curricular influence their evolving identities and capabilities, as evidenced by their artifacts and reflections.

As in the cases from the previous section, the Matrices tool was used to promote matrix thinking. In this case, however, use of a matrix was one stage of a process of developing a final portfolio for self-representation. During the first half of the semester, students used a hierarchical Wizard to reflect on and gather documents that evidenced their identity, relationships, and community engagements, three key dimensions of leadership according to the social change theory (Dugan, 2006). Students then used a Matrix to examine the reflective writings and artifacts they had collected within a theoretical framework that connects the three dimensions to four categories of leadership experiences, yet another example of matrix thinking. The results of this thinking then informed how they represented themselves as leaders in a final portfolio, which used a template also based on the three categories but offering students a great deal more flexibility. The contents of the final portfolios and the audiences to which they were addressed were left up to the students, based on their emerging understanding of their own leadership practice.

As at Weber State that composition process was a learning experience for students that also generated important data for the institution. Collaborative grounded theory analysis of the contents of the students’ portfolios yielded key insights into the dynamics of leadership at Mason. For example, faculty and staff had expected that students would use evidence from their coursework to demonstrate that they had leadership-related skills or knowledge and would perhaps go further to demonstrate how they had applied these capabilities and understandings to specific situations, for example by showing how improved public speaking skills helped them advocate for a policy change in a student organization. More often, however, students used such evidence to demonstrate character traits they associated with leadership.

For example, one student from Russia included essays from several courses documenting her development as a writer of English, showing her rapid progress towards accomplishment. Rather than demonstrating her skills at persuading others to agree to join in causes for which she advocated, she presented this evidence as proof of her integrity and determination. New to the United States during her first semester at Mason, she struggled but persisted:

The student drew on her academic work, situated within the larger context of her life, to demonstrate her understanding of how one acts as a leader and how that understanding shapes her work in the classroom, involvement in student organizations, and decisions about her future after graduation. Such rich representations of student learning and identity are powerful resources for guiding teaching and program design. Through systematic analysis of students’ portfolios, educators are beginning to see tensions among the delivered, experienced, and lived curricula that would not be visible without the kinds of eportfolios OSP has proven effective in helping students create. In relationship to leadership, educational experiences designed to help students acquire specific knowledge and skills are felt more powerfully as opportunities for developing and demonstrating character. Educators may be able to engage students more fully in these activities by capitalizing on this alternative framing. While coursework may focus on leadership as an evolving set of acquired capacities, a more trait-based theory of leadership may guide how students invest their time and energy as leaders throughout their lives. Programs designed to help broaden this understanding of leadership are likely to be more effective if they acknowledge where students begin.

Bridging Learning and Assessment with Goal Aware Tools

The George Mason research suggests that the Matrix, which was designed to be the primary site of assessment activity in OSP, may work best as part of a broader system of tools used in coordination both to promote learning and to yield information that helps an institution improve its performance. These findings are also corroborated by the Weber State study, which observed that the technology worked adequately but not to its full potential because the institution had yet to find ways of embedding it into day-to-day teaching and learning practices.

The school of education at Syracuse University has successfully used the Goal Management tools, which it developed, in conjunction with both OSP and more general-purpose Sakai tools to connect students’ portfolios to a larger assessment process. Using Syracuse’s “goal aware” version of the Sakai assignments tool, faculty could link students’ work submitted to them in any course to a standardized set of learning outcomes the school wished to track as part of the re-accreditation process. Because links to institutional goals were made through the existing process of submitting assignments, the structure of the Matrices and Wizards intended to guide students’ portfolio composition was thus not constrained by the outcomes that needed to be measured as it was at Weber State. Because students’ portfolios themselves no longer were the sole source of information about student learning, students were able to focus on creatively representing their identities as new professional teachers, articulating their emerging theories of teaching and learning, and demonstrating their ability to critically reflect, which research has shown is a key to professional success (Schön, 1983). The portfolios captured students learning to be teachers, not just learning the skills of teaching. The goal aware tools enabled the program to examine those skills without requiring that the portfolios themselves be structured around the standards. The portfolios’ rich representations of student learning were joined with other data provided by the goal aware tools in order to offer a comprehensive account of the school’s performance in preparing its graduates to teach. This account played a key role in the School of Education’s successful re-accreditation.

Several other institutions within the OSP community are now building on the goal aware tools to enable more comprehensive assessment linked to the use of portfolios to support learning. The University of Michigan is developing a suite of reporting and analysis tools that track assessment of student learning across multiple contexts. IUPUI is moving in a similar direction as it expands its work from course-specific to program-wide and institution-wide use. rSmart is working with RINET to assess student learning over multiple years using goal aware and portfolios tools in some twenty-five high schools across Rhode Island, demonstrating the scalability of this approach. Students complete portfolio-worthy assignments in each of their courses that their teachers have linked to one or more of some three thousand state standards and district expectations for learning. Teachers rate student work on assignments according to associated standards and expectations. Over the course of four years, students select assignments from their many courses to add to district-constructed graduation portfolios as a means of satisfying district expectations for graduation. Students and teachers use reports to monitor the status of assignments and portfolios and administrators use reports to prepare data on student learning for reporting to the Rhode Island State Department of Education.

As at Syracuse, evidence gathered through these distributed, embedded assessment processes can be integrated with student-owned portfolios that provide room for creativity and student ownership, both key principles in the tradition of portfolio pedagogy. Research conducted through the I/NCEPR at LaGuardia Community College, which is migrating to OSP, and at the University of Washington, which has contributed to OSP's functional requirements, suggests that providing students with control over the visual design and organization of their portfolios contributes to student learning (Eynon, 2008; Lane, 2008). LaGuardia and Michigan are working together to combine the flexibility of freeform portfolios with the ability to provide some structure to guide students and smooth the composition process. This work has yielded an improved WYSIWYG authoring and commenting interface, the Scaffolded Portfolio and Page Composer, for students and faculty that better encourages guided creativity and is ea sier to use (Botimer, 2008).

Future Directions

OSP institutions' investigations into the use of eportfolios for both enhancing learning and addressing the need for accountability points to a key challenge for the future: balancing the need for integrative learning with the need for assessment. Integrative learning is supported by offering students broad latitude in composing their portfolios, while assessment requires gathering data that can be compared across programs and institutions. It is difficult to serve both these purposes at once. As OSP continues to innovate, we will need to shape the technology to enable us to do both. The development of the goal aware tools is a first step towards such capabilities. Future development is likely to embrace Semantic Web technologies that will allow for more granular analysis of the contents of students’ portfolios and for more agile comparisons of data across programs and institutions.

As the OSP campuses' experiences with large-scale assessment suggest, success depends on not simply on improving the quality of the software, but on developing best practices and understanding of key outcomes that become part of the campus culture. Several institutions that are engaged with OSP—Kapi’olani, LaGuardia, Michigan, Portland State, and George Mason—are serving as leadership campuses in the Association of American Colleges and Universities’ Valid Assessment of Learning in Undergraduate Education (VALUE) project (Association of American Colleges and Universities, 2008b), which is developing meta-rubrics for assessing students’ progress, demonstrated through eportfolios, towards the liberal education outcomes identified by the Liberal Education and America’s Promise project (Association of American Colleges and Universities, 2008a). The VALUE project hopes to develop a shared process for evaluating portfolios that produces assessment data that can be compared across programs and institutions. Such comparability has been a key component of recent demands for increased accountability, most notably the report of the Spellings Commission (Commission on the Future of Higher Education, 2006).

As important as these challenges are, the potential of OSP ultimately does not stop there. For example, Syracuse’s experience using OSP suggests that portfolios can effectively capture professional identity, and George Mason’s research implies that portfolios may help make tacit knowledge explicit. OSP might be used to represent the identity, and to capture the knowledge, of experienced professionals, not just newly minted ones. Such portfolios would be powerful resources for those entering and progressing through professions. In this way, eportfolios may serve a key role in addressing the coming need for intergenerational knowledge transfer as the baby boomers move into retirement. The research and practice discussed in this article demonstrate that OSP is proving successful in supporting reflective and integrative learning and that it can serve as a key component in assessment processes targeted towards the improvement of teaching and learning. These successes are likely to pave the way towards addressing further challenges that lie just over the horizon. The vitality and diversity of the OSP community suggests that it will be ready to help chart the path of eportfolio technology and practice in the coming years.

References

Association of American Colleges and Universities. (2008a). Liberal education and America's promise. Association of American Colleges and Universities. Retrieved August 27, 2008, from http://www.aacu.org/leap/.

Association of American Colleges and Universities. (2008b). VALUE: Valid assessment of learning in undergraduate education. Association of American Colleges and Universities. Retrieved August 27, 2008, from http://www.aacu.org/value/.

Botimer, N. (2008). Michigan scaffolded portfolio authoring work. Sakai Project. Retrieved August 27, 2008, from http://confluence.sakaiproject.org/confluence/display/OSP/Michigan+Scaffolded+Portfolio+Authoring+Work

Commission on the Future of Higher Education. (2006). A test of leadership: Charting the future of U.S. higher education. Washington, DC: Department of Education.

Dugan, J. (2006). Explorations Using the Social Change Model: Leadership Development among College Men and Women. Journal of College Student Development 47(2), 217-225

Eynon, B. (2008). Making connections: The LaGuardia ePortfolio. In Cambridge, D., Cambridge, B., & Yancey, K. B. (Eds.), Electronic Portfolios 2.0. Sterling, VA: Stylus.

Fernandez, L. (2008a). Piloting OSP: Using online technology to assess Weber State University’s general education curriculum. Retrieved August 27, 2008, from http://osportfolio.org/practice/piloting-osp-using-online-technology-assess-weber-state-university%E2%80%99s-general-education-curr.

Fernandez, L. (2008b). Weber ePortfolio tutorial. Retrieved November 17, 2008, from http://ourspace.weber.edu/access/content/user/lfernandez/Videos/osp_flash_audio/osp_ve.html.

Goal Management Tools. (2007). Sakai Project. Retrieved August 27, 2008, from http://bugs.sakaiproject.org/confluence/display/GM/Goal+Management+Tools.

Kahn, S. & Hamilton, S. (2008). Demonstrating intellectual growth and development: The IUPUI ePort. In Cambridge, D., Cambridge, B., & Yancey, K. B. (Eds.), Electronic Portfolios 2.0. Sterling, VA: Stylus.

Hamilton, S. (2006). A Principle-based ePort goes public (and almost loses its principles!) In A. Jafari & C. Kaufman (Eds.), Handbook of ePortfolio Research, (pp. 434-446). Hershey, PA: IDEA Group.

Kapi’olani Community College. (2008). E|portfolio. Kapi’olani Community College, Retrieved August 27, 2008, from http://eportfolio.kcc.hawaii.edu

Kirkpatrick, J., Renner, T., Kanae, L., & Goya, K. (2008). A Values-Driven ePortfolio Journey: Nā Wa ̒ a. In Cambridge, D., Cambridge, B., & Yancey, K. B. (Eds.), Electronic Portfolios 2.0. Sterling, VA: Stylus.

Lane, C. (2008). Technology and change. In Cambridge, D., Cambridge, B., & Yancey, K. B. (Eds.), Electronic Portfolios 2.0. Sterling, VA: Stylus.

Opened practices: A community of practice for teaching and learning with open/community-source tools. (2008). Retrieved August 27, 2008, from http://openedpractices.org/

Open Source Portfolio. (2008) Open Source Portfolio. Retrieved August 27, 2008, from http://osportfolio.org

OSP Portfolios. (2008). Sakai Project. Retrieved August 27, 2008, from http://bugs.sakaiproject.org/confluence/display/OSP/Home

Rickards, W. H., Diez, M. E., Ehley, L., Guilbault, L. F., Loacker, G., Reisetter Hart, J. & Smith, P. C. (In press). Learning, reflection, and electronic portfolios: Stepping toward an assessment practice. Journal of General Education, 57(1).

Rickards, W. H. & Guilbault, L. (2008). Studying student reflection in an electronic portfolio environment: An inquiry in the context of practice. In Cambridge, D., Cambridge, B., & Yancey, K. B. (Eds.), Electronic Portfolios 2.0. Sterling, VA: Stylus.

Schön, D. (1983). The reflective practitioner: How professionals think in action. New York, NY: Basic Books.

Treuer, P., & Jenson, J. D. (2003). Electronic portfolios need standards to thrive. EDUCAUSE Quarterly, 26(2), 34-42.

Wheeler, B. (2004). Open Source 2007: How did this happen? EDUCAUSE Review, 39(4), 12-27.

Yancey, K. B. (1998). Reflection in the writing classroom. Logan, UT: Utah State University Press.

Appendix: Development in Reflective Thinking