Listening for the Squeaky Wheel: Designing Distance Writing Program Assessment


Virginia M. Tucker
Old Dominion University
vmtucker@odu.edu

Abstract

Distance writing programs still struggle with assessment strategies that can evaluate student writing as well as their ability to communicate about that writing with peers at a distance. This article uses Kim, Smith and Maeng’s 2008 distance education program assessment scheme to evaluate a single distance writing program at Old Dominion University. The program’s specific assessment needs include the ability to determine how well students are developing expert insider prose and working together as a virtual community. Kim, Smith and Maeng’s assessment scheme was applied to six courses within the writing program, revealing that programmatic assessment weaknesses included providing varied methods of embedded assessment and encouraging collaboration in writing. Findings further showed that few courses were using summative assessment in the form of exams and quizzes, and several lacked instruments of team or self assessment. The assessment scheme identified other assessment weaknesses across courses, including a lack of opportunities for these distance students to engage in peer discussion or evaluation and a need for greater assessment instrument variety. Applying this assessment method to courses within the single program revealed that electronic graduation portfolios best meet this program's unique interdisciplinary assessment needs.

Introduction

Assessment in distance education is a topic of relatively recent study, and to date very little has been said about assessment in distance writing programs specifically. Writing programs that are offered at a distance face particular challenges as writing can be very social and specialized, and students need effective interaction with professors and peers in order to develop their writing and editing skills. Assessing the quality of assignments and activities is key, but it can become too easy for distance writing program administrators (dWPAs) to overlook the necessity of effective program assessment. Lack of existing literature in the area of distance learning assessment exacerbates the issue for dWPAs, who may have little training in the area of distance education assessment and must learn on the job. This article examines current assessment methods within one distance writing program in order to pinpoint areas in need of improvement. In particular, I’m looking for an assessment method that will not only determine how well students are learning to write at a distance, but how effectively they develop “expert insider prose” within the disciplines (MacDonald, 1994, p.187) in addition to how successfully the program’s courses mimic the academic community experience felt by on-site students. This study will begin by examining the assessment scheme created by Kim, Smith, and Maeng (2008), originally designed to compare the assessment strategies of entire distance programs. Their assessment categories are not only purposeful for understanding assessment across programs, but can easily be used to compare individual courses within a single program in order to determine assessment weaknesses and needs. A sample of distance writing courses will be used in order to identify their assessment weaknesses and determine if an easily implemented method of programmatic assessment exists among the findings.

Research Problem and Need for Assessment Study

Many colleges and universities offer expert assistance for faculty to re-design their courses for distance learning environments. However, a distance program coordinator has the sole responsibility to administer, oversee, and evaluate assessment for that program. With an effective assessment strategy, a dWPA could provide the intellectual leadership that gives voice to a program (Leverenz, 2008) when meeting with instructional designers to discuss assessment strategies. Understanding the particular assessment needs of distance writing programs allows a dWPA to better lead a conversation about programmatic assessment strategies.

The IDS-Professional Writing degree program is just one of 50 distance degrees offered at Old Dominion University, and the only one of the undergraduate distance degrees that specializes in writing. There are several promised goals for this distance writing program; it “provides a foundation that includes interpersonal and organizational communication skills, promotes compositional skills, and provides a basic understanding and skill level in management, marketing and public relations” (ODU, 2011). The program emphasizes writing across the disciplines, so any programmatic assessment method would need to evaluate students’ ability to write within a specialization they choose for their upper division electives. Presently, the distance professional writing program depends on the learning assessments that take place in each individual course. While there are a variety of embedded-assignment assessment methods, including a senior electronic portfolio, results are not currently provided to program administrators aside from a final course grade, and this may not be sufficient as distance learning research suggests that the majority of faculty still require training to understand the needs of this unique population of students (Valentine, 2002).

Kim, Smith, and Maeng’s assessment scheme may be used to identify an efficient, sustainable method of programmatic assessment when used to analyze the current assessment practices of a selection of six professional writing courses. This allowed me to determine some specific curricular improvements that may better reflect the goals of the distance writing program. These improvements would be designed to enhance the sense of a virtual learning community for distance education students and strengthen their development of expert-insider prose. Most important would be changes to the portfolio project course, which could easily produce assessment portfolios. A number of programs already use portfolios as a method of “authentic assessment,” and electronic portfolios, specifically, have grown in popularity as they are hypertextual, contemporary, and multimedia (Kimball, 2003). Once the current assessment practices of the writing program are placed into Kim, Smith, and Maeng’s assessment categories, the results show that assessment variety and evaluation of interdisciplinarity in writing are target areas for improvement. E-portfolios coupled with an increase in interaction-based interdisciplinary assignments will create a stronger method of assessment for the distance writing program.

Review of Prior Research on Online Assessment and Distance Writing Programs

As institutions of higher education begin to offer more distance learning programs, they must also tackle the issue of assessing such programs. Traditional programmatic assessment models are not as effective for distance education, which often faces greater criticism concerning the quality of learning (Mateo, 2007). To counter this perception, distance programs require valid and reliable methods of assessing student learning outcomes. Historically, e-learning assessment has been largely quantitative often using written examinations, but this does not enable the kind of formative assessment strategies that are so important to learning (Mateo, 2007). Mateo argued that portfolios are our most important assessment tool, as they offer evidence of student competencies from a global perspective that elicits concrete discussion of the student's development while promoting reflection on the learning process. Studies on the use of electronic portfolios have found that they are effective tools for the assessment of problem-solving competencies, particularly in the form of text-based resources, but still lack the ability to assess team-based learning (Pereira et al., 2009). Studies across the disciplines have found that team-based learning in distance courses results in higher individual course grades and test readiness (Letassy et al., 2008). Portfolio assessment would then need to be combined with other forms of assessment in order to provide a well-rounded evaluation of student collaborative skills and meta-cognitive competencies.

Scholars in e-learning assessment have identified three principles for assessment in online distance education: that it be integral to instruction, continuous, and maximize feedback (Meyen et al., 2002; Kim et al., 2008). In essence, distance education assessment must measure performance and provide feedback. These principles suggest that assessment may be most effective in the form of embedded assignments relying on formative assessment. According to Meyen, Aust, and Isaacson (2002), distance learning technology allows for frequent and varied assessments that may be used to allow student progress to influence further assessments. Students then become a more integral player in their assignment assessments as they work toward performance improvement and away from periodic exams and graded activities toward more formative and self-assessment methods (p. 7). Meyen et al.(2002) recommended self-assessment, ungraded exams and quizzes, and informal student-to-student and student-to-teacher dialogue (7-8). Feedback on student performance then becomes imperative as students are able to recognize how well they are meeting the course objectives and what more needs to be done to reach a better understanding of those objectives. Studies have shown that such student-instructor communication is an essential factor in student motivation within a distance course (Kelsey & D’souza, 2004; Bradley, 2011).

According to Mateo (2007), the ability to construct real learning communities in cyberspace requires that we focus more on formative assessment that encourages identification and application of improvements while assisting in the collaborative construction of virtual community among students. Bradley (2011) argued that online learning should not be too assessment intensive, but there does need to be a plan for the weighting and assessment of student interaction in distance courses. In their study of the effect of social presence on distance courses, Mackey and Freyberg (2010) found that an increased sense of social presence through interaction with the instructor and peers can improve the learning experience and provide a learning environment more familiar to students. But their research is inconclusive about the effects this has on cognitive learning. Research on collaborative learning indicates that it improves self-directed learning, interaction with peers and faculty, and faculty feedback to students (Letassy et al., 2008). The continued interest in research on the subject of distance learning assessment indicates a persistent need to examine and address current strategies for continual improvement. As distance technologies and pedagogies are constantly being updated, so too should our assessment practices, requiring further research on proven assessment methods in distance education.

There exists a unique bond between distance learning and interdisciplinary writing in that both benefit most from active student engagement. Writing across the disciplines in many ways mirrors McLeod’s (2001) description of the writing across the curriculum movement’s progression “to a model of active student engagement with the material and with the genres of the discipline through writing, not just in English classes but in all classes across the university” (p. 150). Bruffee (1997) argued that student conversations should be designed to mirror the kind of writing and communication they will eventually be required to do. Hence, all courses across the disciplines should utilize assessment methods that allow students to actively participate within the academic discourse community of that discipline. Such active student engagement involves written assignments where substantive formative response is provided by instructors. For McLeod (2001), students must know how to both write and communicate within these disciplines, which again validates the importance of varied embedded assignment assessment methods. Students will simultaneously gain content knowledge of the discipline as they gain writing genre knowledge and learn to develop their “expert insider prose.” In McLeod’s (2001) words, “teaching the genres of the discourse community is therefore inseparable from teaching the disciplinary knowledge of the discipline” (p. 156). This indicates that writing assessment remains of the utmost importance to distance writing programs and courses, particularly those teaching writing across the disciplines. A writing portfolio may be a knee-jerk response to assessment in such courses, but research suggests a need to examine other possibilities for more well-rounded assessment.

Purpose and Research Questions

As with any writing program, the key factor in assessment is how well students are learning to write. In an interdisciplinary program that emphasizes writing across the curriculum, it is necessary to assess student writing across the disciplines in preparation for workplace writing. MacDonald (1994) called this “expert insider prose,” a concept that describes student engagement in discipline-specific discourse as well as their ability to feel like an “insider.” In other words, students can demonstrate the ability to communicate and write within the various discourse communities they become part of during their undergraduate career, then they have not only developed the skills necessary to meet the objectives of the IDS-Professional Writing program but they are also more fully prepared to merge into the workplace after graduation. Presently, however, there is no reliable method of program assessment beyond that provided in individual coursework. The learning assessments provided by individual faculty teaching the distance courses are highly valuable, but we must also examine how well students retain the rhetorical skills necessary to move across and between discourse communities and how their continued learning in forms of professional communication across the disciplines impacts their overall writing abilities.

The program should also support the development of a virtual academic community. It is too often believed that distance students prefer to work alone from the isolation of their computers, which is why they are enrolled in off-campus classes. This is not usually the case as a new generation of students has come to develop a dependency on virtual socialization and learning. In his examination of the iGeneration’s learning processes, Rosen (2010) described how this new generation of students learns differently: they are highly social and choose to interact in a variety of social landscapes. Their growing attachment to the World Wide Web increases their access to knowledge, which in itself  drives a new process of learning that is largely social (Unceta & Masa, 2011). The benefits of social, collaborative learning have long been known to educators. Studies of collaborative learning report that students find such activities helpful, stimulating, and fun (Gokhale, 1995). By encouraging students to practice knowledge-sharing, distance educators help students to internalize tacit knowledge more effectively, and provide effective sharing activities that encourage participation (Ardichvilli, et al., 2003).

The creation of a “collaborative culture” requires active participation, where students are members of the learning organization rather than clients. Enrolling in a distance degree program shouldn’t mean that students will feel isolated from their peers, instructor, and campus community; on the contrary, a distance program should do its best to be as good as if not better than comparable on-campus programs. Valentine (2002) noted that, too often, distance education is viewed as inferior to its traditional counterpart, and that distance faculty must plan effectively to create the feeling of a “true class.” In his study of cyberculture, Jones (1995) agreed that face-to-face communication is the ideal that we seek to mimic in virtual communication—it is the real to our virtual—and “we seek community [in] other places as it dissolves in the spaces we physically inhabit” (p. 16). The need to develop a learning community still exists in distance courses and programs, and such communities must be formed through collaborative learning process (Valentine, 2002). Accommodating this need will require changes to the way writing is traditionally taught, or “a redefinition of writing based on the new kinds of writing digital technology makes possible” (Leverenz, 2008, p. 43). Peer-to-peer collaboration during the writing process makes students feel as if a community exists, and the collaborative learning that takes place within these knowledge communities allows students to practice the discourse of the academic and professional worlds (Bruffee, 1997). Within these virtual learning communities, students should be encouraged to interact with one another through synchronous and asynchronous communication as well as group assignments.

The IDS-Professional Writing program has four upper division core courses required of all students. Although these may be the only courses students are required to share, the frequency of shared courses outside of these upper division core courses is still quite high. Students may have many choices, but those choices are still limited to what is offered and in what medium. Only the four core courses are offered each spring and fall, while the upper division methods, communications, and writing courses are offered interchangeably every two or three semesters. Some courses are offered very rarely because so few instructors and professors are interested in teaching distance courses. As a result, students entering into the program at the same time tend to follow similar paths of coursework until graduation. While it is certainly a goal to increase course offerings and, thus, flexibility and options for students, students should still develop a strong sense that a community exists even if they are not attending classes with the same people. The existence of a virtual community, then, must in many ways mirror the campus community: one may feel part of the group without necessarily knowing everyone in the group. Therefore, a second set of goals in researching the program’s assessment will examine how course syllabi reflect such attention to collaboration and how programmatic assessment methods can effectively evaluate virtual community.

To summarize, the research questions will focus on how well current assessment strategies within the distance writing program 1) evaluate students’ skills in interdisciplinary writing; 2) support the development of academic community through collaborative writing activities and the ability to evaluate those activities; 3) and provide a viable method of programmatic assessment, which currently does not exist.

Methods

Working within the scope of the above principles to design a usable distance program assessment method, Kim et. al. (2008) constructed several assessment categories for analyzing online program courses. They begin by identifying the percentage of formative and summative assessments in a program’s courses. Summative assessment, in the form of traditional tests, quizzes, and exams, are less efficient in the distance classroom. The lack of constructive feedback on student progress makes it difficult for students to see their learning task from a different perspective and difficult for teachers to identify the unique learning needs of each student (Kim et al., 2008). In contrast, formative assessment takes place during and throughout the learning process, providing necessary feedback to the learner and important communication between the teacher and student. Huot (2002) described the goals of such teacher-student communication in formative responses:  “We must help her set the rhetorical and linguistic targets that will best suit her purpose in writing and then we have to help her evaluate how well she has met such targets, using this evaluation to help her reach additional targets and set new ones” (p. 69). To motivate students and enhance their immersion into the discourse community, teachers must communicate with students using the language of the discipline. Proper, timely feedback to student writers becomes central to the learning process in the distance classroom. Studies that compared traditional classroom feedback to distance classroom feedback found the latter to be more effective and strategic than the former (Meyen et al., 2002; Kim et al., 2008). This suggests that assignments that allow response, either from peers or teachers, may be preferable to distance learning than summative methods such as quizzes and exams.

Kim, Smith, and Maeng (2008) cited the remarkable effects of distance collaborative learning as the idea behind their second category, which indentifies percentages of team versus individually assessed assignments. Collaborative assignments and peer reviews allow students to develop a stronger sense of academic community within their classroom as they communicate and learn with their peers. It also gives them practice in social strategies unique to virtual settings. Team assessment troubles many students, even in the traditional classroom, who see peer reviews as unhelpful and collaborative grading as unfair (Kim et al., 2008). As is often the case, teachers can avoid this argument by scoring group members individually, or allowing groups to submit anonymous peer assessments that could be presented in aggregate form (Kim et al., 2008). This could, however, discourage the feeling of a true learning community if responses are anonymous. For this category, all collaborative essays, presentations, and activities, including graded synchronous and asynchronous discussions, are considered team-based.

Seven assessment instruments were selected for categorization: essays, quizzes and exams, discussions, projects and simulations, reflections, portfolios, and peer evaluations (Kim et al., 2008). Whether individually or collaboratively created and assessed, each assignment type within a course should be able to be placed within one of these categories. Distance classrooms that use a lot of synchronous or asynchronous discussion through virtual chats or discussion boards would have these assignments placed in the discussion/chat category, for example. Peer reviews of essays or discussion posts would be considered peer evaluation. Interestingly, reflections are categorized separately from portfolios, though the two are often considered to go hand-in-hand. Courses at Old Dominion that require a portfolio typically require a reflective essay as part of that assignment. For that reason, courses requiring portfolios also required reflective essays, thus the percentages are equal. 

To assess a reasonable sample of IDS-Professional Writing courses, syllabi were collected from six of the program’s upper division distance courses: two English courses, two interdisciplinary studies courses, and two courses in other disciplines (communications and computer science). These courses represent three upper division core courses that all majors in the program must take: one professional writing option, one communication option, and one organizational methods option. Six courses are a representative sample, as they comprise 18 of the total 48 credit hours required in upper division courses within the IDS-Professional Writing major. The assessment scheme requires that one identify the types of assessment, whether it is team-based or individual, and what tools or instruments are used. Each syllabus was examined for references to course assessment. Of particular interest was the list or descriptions, if available, of course assignments and any policies related to collaboration and course grading. If it was not made clear in the assignment whether an assignment was team-based or individual, the course instructor was contacted for verification. 

Table 1 mimics the assessment categories provided in Kim, Smith, and Maeng’s (2008) comparison of three distance program assessment methods. Their original table provides percentage values of assessment types for a representative sample of courses with each of the three programs. Because the table examines only one distance program, it outlines data for individual courses. The table depicts which courses, and thereby which disciplines, are emphasizing certain assessment types and instruments. It is important to note that percentage values do not indicate grade weights, so while some courses may require several assignments involving peer evaluation, the grading of such assignments may not be substantial. Also, the syllabi may not reflect the specific rubrics used in a course, thus the table can only attest to the quantity of assessment types and methods used, rather than their perceived quality. Finally, the assessment scheme is designed to determine ratios of assessment methods and types currently in use, and is not necessarily designed to evaluate the quality of assessment taking place in these courses. Despite attempts to converse with faculty teaching these courses, the correspondence often did not result in more than distribution of the syllabus and assignments, further indicating the challenges faced by the program administrator when working across disciplines.

Syllabi represent the following courses: IDS 493 E-portfolio Project, COMM 495Special Topics in Communication: Positive Communication in Human Relating, CS 300 Computers in Society, ENGL 327W Advanced Composition, IDS 300W Introduction to Interdisciplinary Studies, and ENGL 334 Technical Writing. Table 1 features a breakdown of the assessment types, percent of assessments based on individual or team assignments, and assessment instruments. It replicates the table used in Kim, Smith, and Maeng’s (2008) comparison of three distance programs.

Table 1: Assessment ratios for six IDS-PW courses using Kim, Smith, and Maeng’s (2008) Assessment Categories


Courses

 

IDS 493

COMM 495

CS
300

ENGL 327

IDS 300

ENGL 334

Assessment Type

Formative

87%

75%

40%

90%

86%

89%

Summative

13%

25%

60%

10%

14%

11%

Total

100%

100%

100%

100%

100%

100%

Team vs. Individual Assessment

Individual

40%

100%

80%

25%

25%

28%

Team

60%

0%

20%

75%

75%

72%

Total

100%

100%

100%

100%

100%

100%

Assessment Instrument/Method

Paper/Essay

20%

25%

20%

15%

20%

11%

Exam/Quiz/ Problem Set

0%

25%

60%

0%

6%

11%

Discussion/Chat

33%

0%

0%

50%

33%

61%

Project/Simulation/ Case Study

6%

50%

20%

5%

7%

6%

Reflection

7%

0%

0%

5%

7%

0%

Portfolio

7%

0%

0%

5%

7%

0%

Peer evaluation

27%

0%

0%

20%

20%

11%

Total

100%

100%

100%

100%

100%

100%

Assessment Desults and Discussion

The assessment analysis reveals how courses within the various disciplines are engaging their students in the assessment process. Importantly, all courses are requiring essays and papers among their assessment methods, indicating that students are practicing writing in the disciplines. English courses, surprisingly, are less focused on essays as assessment instruments (at 15% and 11%, respectively), but appear to use discussion more heavily than other courses (at 50% and 61%). While essays are not the predominant assessment tool in any of the courses, all of them use at least two other instruments of assessment that were either more quantitative, such as quizzes and tests, or more collaborative, such as discussion. Portfolios and reflection were among the lowest ratios across the courses, and were being used in only half of the six courses examined. However, these assessment tools held the most substantial grade weight in courses where they were being used.

Each of the six courses utilize a variety of formative and summative assessment methods, often with an emphasis on the kind of formative assessment that assists students with ways to improve their writing. Formative assessment, considered the most important form in distance learning (Mateo, 2007), is well-represented in the program through the use of assessment tools that highlight faculty response to student work. English courses use formative assessment the most, at 90% and 89%, while computer science was the only course to use summative assessment (60%) more than formative (40%). All courses examined use some form of writing as an assessment instrument, strengthening the program’s ability to assess interdisciplinary writing and "expert insider prose." These, too, support the distance learning assessment principles, which value integral and continuous formative assessment.

There is an emphasis on the kind of team assessment valued by distance learning theorists, with such assessment taking the form of group discussions, peer reviews, and group projects and presentations. However, the assessment scheme reveals that collaboration isn't occurring in all courses. COMM 495 and CS 300 are relying almost exclusively on individual assessment, at 100% and 80% respectively. It is not surprising then that these courses also do not indicate a method for assessing discussion or peer evaluation among students. This is not to say that discussion does not take place in these courses, but the syllabi and assignments did not indicate a grade weight for this type of assignment, so any discussion went without assessment. Those courses with a higher ratio of discussion and peer evaluation, both English and IDS courses, also had higher ratios of team-based assessment. It should be noted, however, that syllabi indicated other tools were being used for team-based assessment, particularly projects and simulations.

The results suggest some areas of weakness as well. Kim, Smith and Maeng’s (2008) assessment principles highlight assessment that is integral, continuous, and provides frequent feedback. The sample of courses examined in the above table indicates that very few courses are still relying on summative assessment in the form of exams and quizzes. Courses that do, such as CS 300 (60%) and to a minor extent COMM 495 (25%), are also those using few instruments of team or self assessment. These courses may not be providing opportunities for students to engage in student-to-student discussion or peer evaluation. An increase in assignments that allow for student interaction with one another will provide opportunities for self and peer assessment and bring students into the assessment process (Meyen et al., 2002). These will also increase students’ disciplinary discourse practice and help them further develop their “expert insider prose.”

Another obvious area for improvement is in assessment instrument variety across the courses. While some courses, such as IDS 300, IDS 493, and ENGL 327, offer many varied methods of assessment for students, half of the courses examined focused heavily on just one area of assessment. Courses like CS 300 and COMM 495 primarily assess assignments that are individually created, which does not provide enough emphasis on the development of academic community through collaborative activities. The instrument findings were also interesting, indicating that a course like ENGL 334, technical writing, does not require more projects/simulations than other courses, while COMM 495 does not require any team assessment. It should be noted, however, that the percentages in the table represent the percent of assignments that make up each type of assessment. They do not represent the grade weight of each assessment instrument.

Among the courses with the most varied assessment instruments is the IDS 493 e-portfolio project course, a course that could easily become a program requirement for majors. Portfolios are commonly used to exhibit students’ development as writers over a period of time as well as their growing proficiencies across the disciplines. The electronic portfolio is easily transported and maintained, and meets the unique needs of the distance learning environment. Portfolios assist with the kind of self-assessment valued in distance learning theory. When students create their portfolios, they gain a “more representative and realistic concept of writing evaluation” and “acquire the types of assessment skills important and necessary for evaluating and responding to suggestions for revision” (Huot, 2002, p. 71). As students revise their written artifacts and discuss these revision decisions with their faculty, they develop important skills in critical self-assessment. This encourages what Huot (2002) referred to as instructive evaluation, which requires students be constantly involved in assessments of their work (p. 69). Students further articulate their revisions as they participate in peer evaluations, engaging them in the kind of dialogue appropriate for professionals within that field and brining them deeper into that community’s discourse strategies.

Students receive formative assessment throughout the portfolio development process provided by faculty readers from prior courses taken during their time within the program. Schön (1987) explained how the dialogue between the instructor and student will provide models to imitate, as well as questions, advice, and criticism to consider. This kind of formative assessment is key to learning because the student is in the midst of a task. In other words, “in the context of her attempts to design, both the coach’s telling and her listening have a heightened potential for efficacy” (p. 103). In the midst of portfolio creation, the student is better able to utilize instructor’s advice since it applies directly to the work rather than something more abstract. Success is still dependant on the student, who maintains the responsibility of applying formative comments received from portfolio reviewers.

E-portfolios are already widely used in distance program assessment to assess how students learn. Because they show concrete proof of abilities, encourage revision, and demonstrate learned skills better than standardized tests, portfolios are also a highly effective method of interdisciplinary assessment (Kimball, 2003). Making the e-portfolio a graduation requirement would enable all students in reflecting back on their undergraduate career and better equip them to discuss the merits of their unique interdisciplinary degree with future employers. In these ways, portfolios are a valid method of learning assessment. The e-portfolio course already requires students to compose a reflective essay, embedding self-assessment in the assignment. As students reflect upon their course artifacts, they may discuss how these artifacts support the various course objectives as well as their personal and professional objectives. Artifacts, which represent writings from across the disciplines, will attest to students’ ability to engage in “expert insider prose.” Finally, because portfolios directly measure student achievement in actual writing tasks across the disciplines, they represent an authentic assessment best suited for distance learning environments (Kim et al., 2008; Meyen et al., 2002).

Because the e-portfolio course is already part of the curriculum, it would require very few changes to the program to make this assignment a graduation requirement. Current assessment methods used in the course will remain the same (as seen in the first column of Table 1). Assessments will remain primarily formative with an emphasis on group interaction and assignments through peer reviews as well as synchronous and asynchronous discussions. The course itself would continue to maintain its focus on the process of portfolio development, reflective writing, and artifact collection and selection. Other programs using e-portfolios for assessment typically rely on the evaluations of multiple faculty readers, strengthening interrater reliability. The assessment process involves reviewing the portfolio’s organization, its artifacts, and the reflective essay, then filling out a simple rubric for each (see Appendix). The goal of the interrater assessment is to eventually include faculty across the disciplines who teach the various upper division and organizational methods courses. The rubric is adapted from the portfolio assessment rubric already in use in the IDS 493 E-portfolio Project course. Similar to the criteria checklist used by Saint Joseph College (WPA, 2005), this rubric requires readers to select check boxes indicating whether the criteria exceed, meet, or do not meet expectations. Criteria include general portfolio organization, which inspects the design, linking, and inclusion of all parts; reflections on learning, which examines depth of self-assessment; and overall effectiveness, which responds specifically to the program outcomes. Following that, readers have a place to record their formative comments to the student. This rubric ensures scoring methods are consistent across all e-portfolios regardless of the instructor.

Recommendations for Distance Learning Administrators

By applying an assessment scheme originally designed to compare assessment across distance programs, this study showed how this method might also be effectively used to compare assessment across courses within a single distance program, particularly when those courses cross disciplines and disciplinary assessment strategies. This particular distance program has little control over the assessment practices of the individual courses and faculty as they are situated within other departments, yet the findings improve the program administrator's understanding of how assessment is handled across courses and what areas need improvement to ensure program quality. Not all dWPAs may find themselves in the same situation regarding courses across the disciplines, but the assessment scheme developed by Kim, Smith, and Maeng can provide dWPAs with a visual, presentable look at gaps in course assessment strategies, launching talks with faculty about how the program's courses can improve assessment.

While several studies have found e-portfolios to represent the most effective method of assessment,  this study's findings indicate that courses using portfolios do not use them in isolation as they so not adequately assess how well the course or program develops a sense of community among students. For Bruffee (1997), collaboration and social engagement must be an integral part of the academic environment, and successful collaboration requires more than just “throwing students together with their peers” (p. 412). For these reasons, the portfolio alone cannot suffice as the only method of programmatic assessment. We must continually rely on the individual course assessments and student self-assessments to determine how well the program engages students with one another, encouraging them to practice communication within each discipline’s academic community and socialize with their peers. Collaborative assessment practices continue to motivate our new generation of students, develop their “expert insider prose,” and encourage social learning practices. Any distance program relying solely upon portfolios would need to re-examine how other assessment instruments are being used to evaluate meta-cognitive and collaborative skills.

Because portfolios are assessed individually, a distance program should accommodate embedded assessment of team-based learning through.  In their 2004 case study of student motivation in distance courses, Kelsey and D’souza (2004) found that while some learners felt their peers were essential to their success in a course, others viewed peers as a hindrance. A dWPA should not consider this fairly dated case study a sign that students prefer the romanticized isolation often associated with learning at a distance, rather as an indication that assignment variety is a necessity. Not all students see the importance of peer collaboration and evaluation, which is even the case in traditional classrooms. This should not convince us to avoid interactive assignments; rather, it should convince us that assessment variety will more effectively reveal how well students are meeting course and programmatic objectives.

Establishing a graduation requirement e-portfolio to evaluate program outcomes and ensuring the development of a virtual academic community through frequent and varied collaborative and individual assessment methods is just the beginning for this particular distance program. Distance writing program administrators must continuously scrutinize their program for areas of possible improvement. Though this study suggests an effective method that provides direction for programmatic assessment in one particular interdisciplinary writing program, graduation e-portfolios would still need to be implemented and tested to determine if they are truly the best fit. Future steps for the program include the evaluation of distance course faculty and their writing assignments using methods that mirror the program’s outcomes and expectations. Further, it would be beneficial to construct a truly effective method for assessing the program’s ability to construct a virtual academic community—or to indentify whether the program itself is even fully capable of constructing this sort of community entirely on its own. While many steps remain in the progress toward a distance writing program that matches or exceeds the capabilities of a traditional on-campus writing program, this small assessment provides an approach to a sustainable, reliable method of assessing a cross-disciplinary distance writing program.


References

Ardichvilli, A., Page, V., & Wentling, T. (2003). Motivation and barriers to participation in virtual knowledge-sharing communities of practice. Journal of Knowledge Management, 7 (1), 64-77.

Bradley, W. E. (2011). A conceptual Ffamework for the design and evaluation of online learning modules in professional training and academic education in business. The Business Review, Cambridge, 18 (1), 20-27.

Bruffee, K. A. (1997). Collaborative learning and the ‘Conversation of mankind’. In Victor Villanueva Jr. (Ed.) Cross-Talk in Comp Theory (pp. 393-414). Urbana, IL: NCTE.

Gokhale., A. A. (1995). Collaborative Learning Enhances Critical Thinking. Journal of Technology Education, 7(1).

Huot, B. (2002). (Re)Articulating writing assessment. Logan, UT: Utah State University Press.

Jones, S. G. (1995). CyberSociety. Thousand Oaks, CA: Sage.

Kelsey, K. D. & D’souza, A. (2004). Student motivation for learning at a distance: Does interaction matter? Online Journal of Distance Learning Administration. 7(2).

Kim, N., Smith, M.J. & Maeng, K. (2008).  Assessment in online distance education: A comparison of three online programs at a university. Online Journal of Distance Learning Administration. 11(1).

Kimball, M. A. (2003). The web portfolio guide. New York: Longman.

Letassy, N. A., Fugate, S. E., Medina, M. S., Stroup, J. S., Britton, M. L. (2008). Using team-based learning in an endocrine module taught across two campuses. American Journal of Pharmaceutical Education, 72 (5).

Leverenz, C. (2008). Remediating writing program administration. Writing Program Administration. 32(1), 37-56.

MacDonald, S. (1994). Professional academic writing in the humanities and social sciences. Carbondale, IL: Southern Illinois University Press.

Mackey, K. R. M., and Freyberg, D. L. (2010). The effect of social presence on affective and cognitive learning in an international engineering course taught via distance learning. Journal of Engineering Education, 99 (1), 23-34.

Mateo, J. (2007). Designing online learning assessment through alternative approaches: Facing the concerns. European Journal of Open, Distance and E-Learning, 2007/II.

McLeod, S. (2001). The pedagogy of writing across the curriculum. In G. Tate, A. Rupiper, & K. Schick (Eds.), A Guide to Composition Pedagogies. (pp. 149-64). New York: Oxford University Press.

Meyen, E. L., Aust, R.J. & Isaacson, R. (2002). Assessing and monitoring student progress in an e-learning personnel preparation environment. Teacher Education and Special Education. 25(2).

Old Dominion University (2011).  IDS-Professional Writing (BS). Retrieved from http://dl.odu.edu/programs/interdisciplinary-studies-professional-writing

Pereira, A., Oliveira, I., Tinoca, L., Amante, L., de Jesus Relvas, M., do Camo Teixeira Pinto, M., Moreira, D. (2009). Evaluating continuous assessment quality in competence-based education online: the case of the e-folio. European Journal of Open, Distance and E-Learning, 2009/II.

Rosen, L. D. (2010). Rewired. New York: Palgrave Macmillan.

Schön, D. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass Publishers.

Unceta, A. & Masa, M. (2011). The Ikastola from community to knowledge community: Learning and organizational change. In J. Echeverria, A. Alonso and P. J. Oiarzabal (Eds.), Knowledge Communities. (pp. 53-71).
Reno, NV: Center for Basque Studies.

Valentine, D. (2002). Distance learning: Promises, problems, and possibilities. Online Journal of Distance Learning Administration. 5(3).

WPA Assessment Gallery: Assessment Models.
(2005). Council of Writing Program Administrators. Retrieved from http://wpacouncil.org/assessment-model

Appendix: E-portfolio Assessment Rubric

Student Name:

Meets Expectations

  1. Organization

Exceeds

Yes

No

    1. Home page provides direction to artifacts and introduction to student and program

    1. Reflective Essay orients reader to content, purpose of portfolio

    1. Pages link logically and creatively constructed, with clear annotations

    1. Portfolio includes all required elements: reflective essay, 3 skill sets, 2-3 artifacts per skill set, & resume

  1. Reflections on Learning

    1. Links to appropriate artifacts of academic and professional writing that demonstrate “What was I supposed to learn/ accomplish?”

    1. Describes activities performed to complete artifact: “How did I accomplish it?”

    1. Summarizes the growth that occurred in knowledge, skills, and abilities as a result of the learning activities: “What did I learn? Where was I when I started? Where am I now? What would I do differently?”

  1. Overall Effect

    1. Does this portfolio demonstrate the student’s ability and efforts to learn and to reflect upon his or her learning?

    1. Are the portfolio artifacts well written and demonstrate expert-insider prose applicable to the disciplines in which the represent?

    1. Does the portfolio meet the expectations   for this program?

Reader Comments:


Online Journal of Distance Learning Administration, Volume XV, Number V, Winter 2012
University of West Georgia, Distance Education Center
Back to the Online Journal of Distance Learning Administration Contents