Assessment in Online Distance Education: A Comparison of Three Online Programs at a University



Nari Kim
Doctoral Candidate
Instructional Systems Technology
Indiana University in Bloomington
800 N. Union Street, APT #401
Bloomington, IN 47408
812-345-4605
E-mail: narkim@indiana.edu

Matthew J. Smith
Lieutenant Commander
US Coast Guard
USCGC Mobile Bay (WTGB 103)
1543 Pennsylvania Street
Sturgeon Bay, WI 54235
920-743-2646
E-mail: Matthew.j.smith@uscg.mil

Kyungeun Maeng
Junior Consultant
Training & Development Department
Korea Productivity Center
122-1, Juk-Sun Dong, Jong-no Gu, Seoul, South Korea
82-02-724-1134
E-mail:kemaeng@kpc.or.kr

 

Abstract

The purpose of this study was to investigate whether or not the principles of assessment in online education are reflected in the assessment activities used by the developers and administrators of actual online distance courses. Three online distance education programs provided at a large mid-west university were analyzed; the School of Continuing Studies – undergraduate distance program, the School of Business – distance MBA program, and the School of Education – distance graduate program. The results of the study showed that the assessment activities of online distance courses do not strictly follow the principles suggested in the literature.

Introduction

Despite the recent interest in online distance education in the higher education setting, there is scant literature concerning how to assess student performance in the online distance education environment.  Since assessment is an important lens through which education is viewed (Bransford, Brown, & Cocking, 2000), and a driver of student performance, the authors considered it an important component of any online distance education program that needed further study.  The authors examined the body of literature to define general principles for assessment of student performance in an online distance-education context, and investigated whether or not these principles are reflected in the assessment activities used by the developers and administrators of actual online distance courses.  Their research efforts were guided by two key questions:

In an attempt to answer these questions, the authors analyzed the assessment schemes of a selection of courses from three online distance education programs at a large mid-west university. By classifying their assessment schemes and quantifying the degree to which different methods were used, the authors were able to draw tentative conclusions about the state of assessment in a large mid-west university’s online courses. The authors hope that their work will serve as a pilot study to encourage further research in this area.

Definitions of Assessment

Before investigating which assessment strategies and methods are appropriate for online distance education, and the degree to which these methods are being applied at a university, the authors first established definitions of notions. Since evaluation, assessment, knowledge domains, online, and distance education are terms whose meanings can vary depending on one’s point of view, the definitions or frameworks the authors chose to describe them served as boundaries to make their analysis more manageable and their discussion more precise.

First, in this analysis the authors chose to look at online courses through the lens of the Kirkpatrick Four-Level Model. Specifically, the authors were interested in Level 2: Learning, which Kirkpatrick defined as “the extent to which participants change attitudes, improve knowledge, and/or increase skill as a result of attending the program” (Kirkpatrick, 1998). Even though the emphasis of Kirkpatrick’s model was the evaluation of training programs, the authors considered his framework to be the most useful way to narrow down the very broad idea of evaluation of online distance courses.

The authors chose to further narrow their scope to assessment in online distance education programs (the basis for Level 2 evaluation of the program) for two reasons. First, the authors considered an analysis of Level 3 (Behavior) and 4 (Results) assessment in online distance education programs to be too ambitious a goal considering their time and resource constraints, not to mention a general lack of assessment at these levels that became apparent to them in their initial review of the online programs. Second, the authors considered an analysis of assessment strategies and methods in the online distance education environment to be both the most directly useful aspect of the topic and the most relevant to instructional strategies for distance education as a whole. Third, Level 2 is more familiar with the stakeholders – instructors, students, administrators, etc.

Knowledge Domains of Assessment

Another important concept that impacted the authors’ investigation of this topic was that of knowledge domains. Seels and Glasgow (1990) discussed three commonly accepted knowledge domains as their psychological basis for instructional design – affective, cognitive, and psychomotor. In addition to these domains, Romiszowski (1981) identified a missing domain – social/interpersonal/interactive skills that may need to be developed, and, therefore, (logically) assessed depending on the outcomes of the course or program. The authors considered no investigation of online distance courses to be complete without looking at how these domains are addressed by their assessment schemes; therefore, the term “knowledge domains” means the four domains listed above.

Definition of Online Distance Education

Finally, determination of the scope of the authors’ analysis required a decision on the definition of “online distance education.” For “distance education,” the authors chose to use Keegan’s (1990, p.44) definition as their working model, which includes “quasi-permanent separation of teacher and learner throughout the length of the learning process”, “influence of an educational organization, use of technical media”, “provision of two-way communication”, and “quasi-permanent absence of the learning group throughout the learning process…so that people are usually taught as individuals and not in groups.”

The authors chose to define “online” as the use of the Internet (World Wide Web) as the ”technical media” used to provide the “two-way communication” required by Keegan’s definition. The authors decided that the degree to which a course enabled an instructor to use the internet to accomplish an organized body of learning outcomes for students working remotely (separated from the instructor as well as each other) was the degree to which it was an online, distance education course. Therefore, courses can be considered “online distance education courses” by this definition, despite the fact that they might have a small “face to face” or “residential” component.

Principles of Assessment in Online Distance Education

New technology has made frequent and varied assessments possible in the online distance education environment, compared to the traditional learning environment (Meyen, Aust, Bui, & Isaacson, 2002).  However, the authors should remember that the most important thing for assessment in the new online learning environment is to still focus on learners’ achievement in terms of instructional goals and objectives.  Therefore, even though technology can facilitate the process of assessment in effective and efficient ways, the authors must choose appropriate assessment opportunities only when assessments are essential during instruction.

Over the last few decades, many researchers have been convinced that assessment of learner achievement in online distance environments should be integral to instruction, be continuous, and maximize feedback (Meyen et al., 2002).  Based on these shared believes about online assessment, the authors will discuss several principles of assessment in the following paragraphs.

First of all, Pennsylvania State University (1998) developed a set of principles to guide assessment in online distance education.  These principles of assessment might be an initial guide for designing “big picture” evaluation of learner achievement.  Based on their assumption that assessment and measurement should serve valuable purposes for both instructors and students, the principles emphasized importance of integrating assessment with instruction as follows (p.7):

On the other hand, in a design plan for online assessment, Kibby (2003) explained that online learning and assessment should be considered not only in a student-centered approach but also in a teacher-center approach (e.g., management system).  She emphasized that web-based assessment might assist students in taking ownership of their learning because the assessment could provide integration of learning and assessment, and also immediate and effective feedback to students.  Thus, web-based assessment systems might have more potential than paper-based assessment systems in terms of access and flexibility for both students and teachers in effective and efficient management.  In order to develop web-based assessment, she suggested several key decisions to be made as follows (Kibby, 2003):

Features of Assessment in Online Distance Education

Based on several educational philosophies such as behaviorism and constructivism, there are various features of assessment in online distance education emphasized from different points of view. However, in this study, the authors did not discriminate between these different educational philosophies in order to search for assessment features in the online environment. Instead of dividing these educational approaches into opposite sides, the authors tried to figure out the most important features of assessment that could be used as appropriate assessment strategies in an online distance education environment. Therefore, based on the traditional assessment strategies suggested for the face-to-face instructional environment, in the sections below, the authors will discuss several assessment features crucial to success in web-based assessment.

Ongoing Assessment: Formative Assessment

According to the Concord Consortium (2002), the use of one “traditional high-stakes test" to measure learner achievement may be effective and efficient in a monitored classroom. However, online assessment should be a “continuous, ongoing process”. For instance, the Concord Consortium recommended that instructors should find evidence of achievement in individual participant’s daily contributions to their online learning group such as online discussion. Also, the instructors should try to find out “each student’s unique activity or approach to solve learning problems” through their posted ideas on the discussion board.

On the other hand, when the authors consider ongoing assessment as measuring the process of learning, this type of assessment can be called formative assessment. According to Bransford, Vye, and Bateman (2002), formative assessment serves students as well as instructors in many concrete ways. For example, “students can use feedback from formative assessments to help them know what they have not yet mastered and what they need to study on further” (p. 174). Through this formative assessment, students can have more opportunities to consider their learning task from a different perspective based on the instructor’s feedback. Also, with information from the formative assessment, instructors can change their instruction to be more effective and efficient and to target students who need further help (Bransford et al., 2002).

Therefore, ongoing assessment or formative assessment can be a very integral part of instruction in an online distance learning environment, which can track individual learning activities easily compared to a traditional classroom environment. However, in order to maximize these ongoing assessments’ advantages in web-based instruction, an online management system should be able to provide instructors with accumulated data of student learning activity and scores in effective and visual ways.

Feedback in Assessment

If assessment is to be integral to instruction as explained above, feedback must play a central role in the assessment process (Meyen et al., 2002). Compared to the traditional instruction environment, the online learning environment made this central role of feedback achievable in terms of time and access to information. In continuous assessment of the web-based environment, Kerka and Wonacott (2000) explained that the significance of instructional feedback could directly affect what students learn and how effectively they would do so. The especially easy use of electronic communications can support the central role of feedback in web-based assessment. Indeed, proper and immediate feedback can transform an assessment experience into an instructional experience for learners (Meyen et al., 2002). Collis, De Boar, and Slotman emphasized the importance of instructor supports for facilitating feedback in online learning environments. Also, they referred to “the practical implications of feedback in the context of time expenditures, clarity of expectations for students, and efficiency of managing the overall submission and feedback process (Meyen et al., 2002, p. 191).” As an example of feedback, Collis et al. presented “personal feedback by the instructor to an individual assignment, model-answer provided by the instructor, peer evaluation provided by the student(s), and automatic direct feedback provided by the computer (Meyen et al., 2002, p. 191).”

In discussion about effectiveness of feedback in online distance education, Meyen et al. (2002) confessed that, in a face-to face course (traditional learning environment), they could not deliver feedback strategically and provide the same level of feedback that they could in an online course situation, even though synchronous feedback was possible in a classroom. Such findings indicate that electronic feedback in an online distance course might be more effective than that of a traditional course.

Self Assessment

Self-assessment should be a major component of online distance education (Robles & Braathen, 2002). Some instructors might want to assess student learning only by themselves. However, Robles et al. believed that it would be very important for students to participate in assessment of their own learning because students could measure their own learning process and achievement. They also emphasized that students could have the ability to determine “if they have arrived at the required instructional objectives, and that if not, they could repeat the coursework “by themselves in order to attain their own goals (p.45). For example, online pre-tests could be considered for this self-assessment because students would be able to receive immediate feedback after taking their pre-tests in order to determine their existing knowledge level (Robles & Braathen, 2002). Through the pre-tests, students can know their current levels of knowledge before starting online courses, choose the proper levels of courses, and take the test again to measure their achievement after finishing the courses. These pre-tests can also allow students to feel more comfortable with the material itself or its instructional objectives.

Team Assessment and Peer Assessment

Because of the remarkable effects of collaborative learning in a classroom, “many online courses also aim to develop students’ ability to work as part of a team and include team assessment task such as presentation, projects, case studies, reports, debates and so on” (Freeman & McKenzie, 2002, p. 552). Gokhale (2003) explains that collaborative learning can be a good “instruction method in which students at various performance levels work together in small groups toward a common goal. The students are responsible for one another’s learning as well as their own. Thus, the success of one student helps others to be successful.” According to Freeman and McKenzie (2002), however, although many students feel the value of learning in teams and developing teamwork skills, they do not consider their team assessment to be “a fair assessment method if team members are equally rewarded for unequal contributions (p.552).” Thus, improving fairness of team assessment is essential to enhancing students’ learning from team tasks. Aggregate data in peer assessment can encourage the students to rate confidentially their own and their peers’ contributions to team tasks and team maintenance. Also, they believed that benefits of improving student learning from teamwork tasks, and saving time by automating the process of calculating self and peer adjustments of assessment grades can be especially attractive for large enrollments in university level courses.

Authentic Assessment

Grant (1990) insisted that assessment should be authentic when the authors would want to directly measure learner achievement on worthy intellectual tasks, instead of the type of indirect test items that traditional assessments rely on for their advantages as efficient and simplistic substitutes. In discussion of the features of authentic assessment, he explained that authentic assessment could provide students with the full range of tasks. These tasks could require students to reflect priorities and challenges presented in good instructional activities (e.g. collaborating with others on a debate) while conventional tests would be relatively limited to the paper-and-pencil or one-answer questions. Also, he suggested that authentic assessment could provide validity and reliability by standardizing appropriate criteria for scoring student products in contrast to traditional testing, which standardizes objective items and the one right answer for each item. However, beyond these technical considerations, he assumed that this new approach to assessment would be based on the premise that assessment should primarily support the needs of learners. Grant believed that the best assessment should teach students and teachers alike the kind of work that most matters.

In this way, electronic portfolios have been suggested as the best type of authentic assessment in an online distance learning environment (Meyen et al., 2002). For the above features described as authentic assessment, electronic portfolios could evolve as a management tool for both instructors and students with the emergence of online distance education. These electronic portfolios can monitor student processes and facilitate not only formative assessment but also summative assessment. Especially, through these electronic portfolios, formative assessment can serve to identify strengths and weakness of a student’s learning process with the proper feedback.

Methods of Assessment in Online Distance Education

Many articles suggest online assessment methodologies. According to Rovai (2000), however, general assessment principles are not different in online environment; only the manner in which the principles are applied is changed. In light of this, Rovai (2000) suggested some assessment methods for online courses. Among them, he emphasized proctored testing and online discussion. There are three kinds of proctored testing for distance courses: a delayed telephone conversation, online chat, or e-mail; proctored testing at decentralized locations and at centralized on-campus residencies. He presented that proctored testing promotes identity security and academic honesty, two difficult issues for distance education. Proctored testing is recommended for high-stakes, summative assessment.

Rovai (2000) also recommended online discussion as a good assessment method. The ability of online discussion to promote text-based communication can support the construction of knowledge. It would also promote reflection through asynchronous online interactions better than in traditional classroom settings. Instructors can use these online interactions for summative assessment as well as formative assessment. For authentic performance assessment, Rovai (2000) proposed projects and case studies that are unique and relevant to the individual learner, with the added benefit that they can help solve the identity security and academic honesty problems.

Robles and Braathen (2002) said that the assessment techniques used in traditional classroom settings could be modified to reflect the nature and pedagogy of distance settings. As they suggested several online assessment techniques, they argued that a variety of assessment tools could be used to determine whether the student had achieved the pre-established learning objects. The suggested assessment methods in the article are: self-test, assignments, electronic portfolio, online discussion, asynchronous threaded discussion group, one-minute paper, synchronous chatting, and e-mail content of questions.

Meyen and his colleagues (2002) said that e-learning assessment options are little different from those routinely employed in face-to-face instruction. They suggested several methods for online course assessment: literature review activity, collaborative projects, exams, student reports in real time, journal entries, and electronic portfolio. Those were implemented in an online course taught by Meyen in1997. He included a mid-term, final exam, a literature review exercise, a collaborative project, and approximately 30 activities. Among these methods, Meyen et al. (2002) emphasized the electronic portfolio. They stated that the electronic portfolio method can evaluate students’ achievement both formatively and summatively. They also believe that portfolio assessment provides a more accurate means of measuring academic and professional skills. “Through the use of technology, the electronic portfolio in hypermedia format can become a personal/professional information management system that contributes significantly to the pedagogy of e-learning in higher education in addition to professional development and as a tool for K-12 teachers.” (Meyen et al., 2002, p. 194)

The preference for electronic portfolio can be seen in Dewald, Scholz-Crane, Booth, and Levine’s article (2000). They argued that electronic portfolio assessment works well both for document and develop meta-cognitive skills. According to the article, as students work more and more electronically, electronic portfolios are becoming more common, especially in the distance learning environment. “At the end of a course, the portfolio serves as a representation of not only a student’s progress toward mastery of course content, but also of a student’s increasing awareness of his or her own skills. Finally, portfolios encourage students to develop meta-cognitive skills and allow the instructor to monitor the development of those skills (Dewald et al., 2000, p. 41).”

Methodology

To answer the research questions, the authors analyzed three different distance education programs provided at a large mid-west university; the School of Continuing Studies – undergraduate distance program, the School of Business – distance MBA program, and the School of Education – distance graduate program. The three programs are representative distance programs of a large mid-west university. Undergraduate courses of the School of Continuing Studies, graduate courses of the School of Education and MBA courses for professionals are representative of courses that are typically provided in higher education. This university is one of the large universities that can represent other higher education institutes in the mid-west that have similar conditions. A description of the characteristics of each program follows.

Descriptions of the programs
The School of Continuing Studies Undergraduate Program

The School of Continuing Studies established in 1975 is one of the largest distance education providers in the United States. The School of Continuing Studies used to offer correspondence distance education programs, but many of the correspondence courses have been converted to online courses along with the development of technology. The School offers a high school diploma, two undergraduate degrees in general studies (both available entirely online), one graduate degree in adult education (available online with one required on-campus weekend), more than 200 university courses and more than 100 high school courses and professional development and custom training opportunities. Among them, the authors investigated 16 online undergraduate courses.

The subject matters vary from accounting to art appreciation. All the courses of the School of Continuing Studies are for independent study, so there is only interaction between an instructor and a student, not among the students. In an online course, the students receive lessons, assignments, and grades on the World Wide Web. They communicate with the instructor via e-mail, and submit the assignments via Web browser. Some of the courses include interactive activities and virtual field trips that the students participate via the World Wide Web.

The School of Business Online MBA Program

The online MBA program at the School of Business is designed to allow professionals to keep working full-time and take care of family responsibilities while getting their MBA degree online. It is a two year program using a system of 12-week quarters. It requires a one-week “in-residence” course each year, but the rest of the courses are completed online (asynchronously with some synchronous components) using the following tools: discussion forums, online testing, audio/video streaming, and simulations/case-based learning. The courses are taught by tenured faculty at the School of Business. Graduates are granted a Master of Business Administration degree. The authors investigated 14 courses for which they could get information.

The School of Education Distance Graduate Programs

The School of Education Distance Education program offers students and educators fully accredited coursework delivered via the Internet and two-way interactive video. Topics of this School of Education Distance Education program range widely across the elementary and secondary curricula, including Instructional Technology, Language Education, and Educational Psychology courses. This program offers graduate-level credits to meet certification and recertification requirements in school districts across the country and around the world in order to be a part of a master's program, either at the mid-west university or another institution. This program also offers a few courses for undergraduates just beginning work towards certification. Especially, this distance program proves master’s degree programs for Instructional Technology and Language Education. Therefore, among the master’s degree programs, the authors selected 7 courses in Instructional Technology and 3 courses in Language Education to analyze their program assessment.

Instrumentation and Rationale

With the course information gathered, the authors categorized various kinds of assessment methods to investigate what kind of assessment methods is actually being used.  First, the authors divided them up as formative assessment and summative assessment.  Formative assessments refer to the methods that assess the learning process rather than learning outcome.  Rather, summative assessments are the assessment of learning results.  For example, the quizzes that are given at the end of the unit and evaluate their understanding of each are considered formative, while the mid-term or final exams for assessing their learning outcomes at the end of the semester or at the end of the bigger units are considered summative.  Interim parts of an ongoing project or paper are another example of formative assessment, while the final project outcome is a summative assessment method.

As discussed in the literature review, feedback is extremely important.  According to Bransford (2001), feedback is most valuable when students have the chance to use it to modify their thinking while they are on a unit or a project.  In terms of this, formative assessments that are given during the process of learning are important in a distance setting, too.  Through formative assessment, instructors can have an idea of how much the students have achieved their objectives, and can revise their instructions according to the results of the formative assessment.

Second, the authors categorized them as team assessment and individual assessment. Team assessment means that the grade is evenly given to the group of people worked together.  Individual assessment is that an individual gets his or her own grade for individual work.  The authors made this category because of the importance of collaborative work.  One of the disadvantages of the distance education setting is the difficulty of interaction or collaborative learning.  Through the interaction among students or collaborative learning process, they can learn from and give feedback to each other, as well as learn interpersonal skills.  Team assessment is also relevant to the authenticity of the assessment.  In business setting, most of the projects are completed as a team-base.  Therefore, even in a distance setting, collaborative learning should be done for the sake of authenticity, and the authors wanted to know how it is done in real courses. 

The authors next chose to categorize the assessment schemes of the selected courses by the various methods of assessment that were used:

Using these seven categories, the authors sought to determine how much these online courses consider authenticity, variety, and if they do not just evaluate students’ memory of simple fact and procedures, but higher level thinking and deep understanding or meta-cognition.

Table 1. Assessment Categories Used to Analyze Online Courses


Assessment Type

Formative Assessment

  1. Assessment of the learning experience progress (Pennsylvania State University, 1998)
  2. Continuous, ongoing assessment and feedback (Bransford et al., 2002; Concord Consortium, 2002; Meyen et al., 2002)
  3. Immediate and effective feedback during learning (Kerka & Wonacott, 2000; Kibby, 2003; Meyen et al., 2002)

Summative Assessment

  1. Measuring learning at the end of the process (Kibby, 2003, Meyen et al., 2002)
  2. Traditional tests (Concord Consortium, 2002)

Team vs. Individual Assessment

Individual Assessment

  1. Self assessment (Robles & Braathen, 2002)

Team Assessment

  1. Assessment in collaborative learning (Freeman & McKenzie, 2002; Gokhale, 2002)

Assessment Instrument / Method

Paper / Essay

  1. Short papers (Robles & Braathen, 2002)
  2. Student reports in real time (Meyen et al., 2002)

Exam / Quiz /
Problem Set

  1. Conventional tests such as paper-and-pencil or one-answer questions (Grant, 1990)
  2. Proctored testing (Rovai, 2000)
  3. Mid-term and final exams (Meyen et al., 2002)
  4. Self tests (Robles & Braathen, 2002)

Discussion / Chat

  1. Online discussion, online chat, and e-mail (Robles & Braathen, 2002; Rovai, 2000)

Project / Simulation / Case Study

  1. Authentic assessment (Grant, 1990; Kibby, 2003)
  2. Collaborative projects (Freeman & McKenzie, 2002; Meyen et al., 2002; Rovai, 2000)
  3. Case studies (Freeman & McKenzie, 2002; Rovai, 2000)

Reflection

  1. Meta-cognitive skills (Dewald et al., 2002; Grant, 1990)

Portfolio

  1. Electronic portfolio (Dewald et al., 2002; Meyen et al. 2002; Robles & Braathen, 2002)

Peer evaluations

  1. Peer contribution (Freeman & McKenzie, 2002)

Procedure

After investigating the online distance courses at the large mid-west university as a whole, the authors selected a convenient sample of courses from each of the three distance programs previously discussed. The authors got permission to access the course syllabi or websites from the course administrators or instructors of each program. For the analysis, they used only the information provided in the documents they were able to obtain. The authors decided upon our three types of assessment categories (formative/summative, team/individual, and assessment methods) based on the information in the documents and our literature review and used these categories to classify the methods used in the assessment schemes of our selected online distance courses. Then they analyzed the data quantitatively (percentages of assessment categories) and qualitatively (course descriptions) to determine the answers to the research questions.

Results

The data regarding the assessment schemes employed by online distance courses at a large mid-west university are summarized in Table 2. A closer look at the data in Table 2 sheds light on the question of whether or not the recommendations of the experts are reflected in the assessment schemes of online distance courses at a large mid-west university. In the following sections, the authors will discuss the extent to which each major principle or aspect of effective assessment in online distance courses is being applied in practice.

Table 2. Ratio of Assessment Categories of an Average Online Course in Three Distance Education Programs (See Appendix A: Course List)

 

SCS

SOE

SOB

Average

Number of Courses

16

10

14

13.33

Assessment Type

Formative Assessment

47.69%

78.00%

40.71%

55.47%

Summative Assessment

52.31%

22.00%

59.29%

44.53%

Total

100.00%

100.00%

100.00%

100.00%

Team vs. Individual Assessment

Individual Assessment

100.00%

78.80%

76.79%

85.2%

Team Assessment

0.00%

21.20%

23.21%

14.8%

Total

100.00%

100.00%

100.00%

100.00%

Assessment Instrument / Method

Paper / Essay

40.25%

54.20%

13.21%

35.89%

Exam / Quiz /
Problem Set

58.19%

4.00%

47.50%

36.56%

Discussion / Chat

0.00%

14.50%

10.36%

8.29%

Project / Simulation / Case Study

1.56%

10.25%

18.21%

10.00%

Reflection

0.00%

8.35%

0.00%

2.78%

Portfolio (collection of individual production)

0.00%

6.50%

0.00%

2.17%

Peer evaluations

 0.00%

2.20%

7.14%

3.11%

Total

100.00%

100.00%

100.00%

100.00%

SCS: School of Continuing Studies
SOE: School of Education
SOB: School of Business

Formative Assessment and Student Feedback

The data suggest that, to varying degrees, ongoing or formative assessment is a significant feature of each of the three programs. Overall, formative assessment of student performance accounts for 55% of the total assessment. Likewise, formative assessment is a feature of all of the School of Continuing Studies (SCS) and School of Education (SOE) courses, and half of the School of Business (SOB) courses the authors analyzed.

In the SCS courses, formative assessment typically takes the form of written assignments at the end of each lesson that includes multiple choice, short answer, and essay questions. The correct answers to these activities comprise the feedback that the students get. In the SOE courses, formative assessment actually comprises a much greater share (78%) of the total assessment than summative does. It is accomplished in the Instructional Technology (IT) department through the use of asynchronous discussion and ongoing documentation of group projects. The feedback given on these interim deliverables is applied to the next phase of the project in a formative manner. In the Language Education (LE) department, formative assessment takes the shape of simpler instruments such as short individual papers which are not part of a larger (group) project. As with the IT program, the feedback on these papers is used formatively to improve student performance. Finally, in the SOB courses, formative evaluation / student feedback is conducted through a mix of problem sets, short case analyses, interim project submittals, and discussion. Interestingly, there was a greater disparity in the use of formative vs. summative assessment in the SOB program; four courses relied entirely on summative measures, while three others relied entirely on formative measures for assessment of student performance in the course.

Authentic Assessment

As can be seen in Table 2, the vast majority of assessment in all three online programs remains relatively “traditional” in nature – papers, essays, exams, quizzes and problem sets. These measures, which are no different than their counterparts in resident courses, account for 72% of the overall student assessment.

The SCS program is most dependent on these traditional measures, at 98%. In fact, the program actually employs proctored “on-site” mid-term and final exams as the primary assessment method. The SOE and SOB programs each rely on traditional measures for about 60% of student assessment, but with different emphasis. The SOE program heavily favors paper/essay assessments while the SOB favors exams, quizzes, and problem sets.

The following sections will describe in greater detail the ways that these programs use more ‘authentic’ methods (online discussion, project/simulation/case study, reflections/portfolios, collaborative projects, and peer or self assessments) to measure student performance.

Online Discussion

Online discussion / chat is a prominent feature of both the SOE (15%) and SOB(10%) programs, but completely absent from SCS courses due to their totally individualized approach. Discussion is used in the SOE programs for two distinct purposes. In the IT department, discussion is used for interactions between group members in the process of completing team assignments. In the LE department, discussion is used more for communicating their individual ideas regarding the individual assignments they are working on. In the SOB program, discussion is used for either or both of these purposes depending on the course.

Projects / Simulations / Case Studies

Perhaps the most authentic form of assessment, projects, simulations, and case studies account for an extremely minor (1.5%) portion of SCS courses and a more significant share of student assessment in the SOE (10%) and SOB (18%) programs. In SCS, the only project requirement is ironically one of the most ‘authentic’, as students are required to produce an audio recording for a communications course. In the IT department of the SOE, the courses rely heavily on genuine instructional projects that are typically done as part of a team and often for actual clients. In the LE department, projects are less product-centered and are completed individually. In the SOB, students complete a wide array of analysis projects, business simulations, and analyses of business case studies in both individual and team formats.

Reflection / Electronic Portfolio

Another pair of ‘authentic’ assessment methods, reflection exercises and the electronic portfolio, are the sole province of the SOE, at 8% and 6.5% respectively. Reflection exercises are used in four of the IT department and three of the LE department courses. The IT department uses an electronic portfolio as a comprehensive assessment of the production abilities that students in online courses have attained.

Team, Peer, and Self-Assessment

Team assessment of collaborative activities, while completely absent from the SCS program, is a significant factor in the SOE (21%) and SOB (23%) programs – featured in half of the courses in each program. This team assessment takes the form of ‘group’ grades given for projects completed as a team in these courses, whether that is instructional design projects in IT or business simulation / case analysis in business courses. Peer assessment (peer evaluation) is used in conjunction with these team assessments in only three of the IT courses and one SOB course. Interestingly, though, the SOB course (Electronic Commerce) that uses peer evaluation employs it as the sole method of evaluation in the course.

Although it is an excellent tool, self-assessment does not figure prominently in any of the three programs, but is offered to a small degree in each of them. It is best represented in the SCS program, and takes the form of a self-test (ungraded) section in each lesson. In the SOE (IT department), self-assessment is offered in the form of downloadable quizzes. An ungraded quiz feature is offered in only one SOB course.

Assessment of Different Knowledge Domains

From the review of each course’s syllabus, the authors could not determine the assessment of any knowledge domain other than the cognitive one. None of the course outcomes are written in terms of desired affective changes or increased psychomotor or interpersonal skills, nor do the assessment schemes appear to be designed to measure these types of outcomes.

It is plausible that one could consider a number of the SOE (IT) and SOB courses to be likely candidates for the development and assessment of interpersonal skills due to the substantial amount of group work built into them. However, to the extent that this may be happening it is not an articulated outcome of these programs nor is it an object of their assessment schemes. Therefore the authors don’t consider there to be any knowledge domains addressed by any of these programs besides the cognitive domain.

Discussion

Are these three programs following the guidance of the online distance education literature?

Generally the authors conclude that the courses analyzed from these three programs are not following the advice from the literature regarding assessment in online distance education courses.  The reason for this conclusion is the fact that they display a relatively low overall usage of the more authentic assessment methods suggested by Meyen et al. (2002) and Kibby (2003) – collaborative projects / team assessments, project/simulation/case study, discussion/chat, reflection, portfolio, and peer evaluation.  If more authentic assessments supporting the needs of learners were provided in the courses, learners could have better chances to reflect priorities and challenges in the full range of their tasks (Grant, 1990), share their information, and construct new knowledge (Rovai, 2000).  However, these courses largely continue to rely on the same types of assessments – namely paper/essay or exam/quiz/problem set – that are found in traditional face-to-face courses mentioned by Kibby (2003) and Concord Consortium (2002).  The School of Continuing Studies courses are the most prominent examples of this, while the School of Education and the School of Business courses reflect the suggestions of the literature to a greater degree due to their greater use of authentic methods to assess student performance.  A notable exception to this trend is in the area of formative assessment, as each of the three programs uses formative assessment as a prominent feature of their overall assessment schemes as recommended by the literature (Meyen et al. 2002; Pennsylvania State University, 1998). Thus, students could have more opportunities to reflect their tasks from various perspectives based on the instructor’s practical feedback through formative assessments (Bransford et al., 2002).

Are there differences between programs regarding assessment tools?

The authors noticed several differences between the programs, which the authors can attribute to the nature of the context and the subject matter of the courses.  First, the SCS courses clearly demonstrated a reliance on traditional measures such as papers and tests, to the complete exclusion of more authentic assessments.  The authors attribute the lack of authentic assessments to the fact that the SCS courses are intentionally designed in a “self-study” context (courses could have one student), for which collaborative activities would be either impossible or irrelevant.  Of course this does not explain the lack of other authentic assessments; however, the fact that the SCS online courses’ heritage lies in the correspondence format very well could.

The SOE courses the authors studied show an extremely high rate of formative assessment suggested by Meyen and his colleagues (2002).  Even more significantly, a large part of this formative assessment took place within the framework of collaborative “design team” projects in the IT department, which according to the literature is a highly authentic and recommended method of assessing student performance.  The other side of that coin is the fact that the Language Education department’s courses prominently featured individual paper/essay activities, in sharp contrast to the IT department’s more authentic measures. 

Finally, as could be expected from a program geared toward practicing professionals in a business context, the SOB program relies more heavily on collaborative work and projects/simulations/case studies (Grant, 1990; Rovai, 2000) than the other two.  Such a finding is not surprising considering the clientele the program is designed to serve; what is surprising is that this authentic assessment still only makes up such a small percentage of the assessment in the course. Thus, it would be essential in the SOB program to include more teamwork tasks and assessments including projects and case studies for promoting collaborative learning, which can transfer across a wide range of situations rather than the memorization of factual information and content materials easily forgotten (Bennett, Dunne, & Carre, 1999; Tsui, 2000).

Conclusion and Recommendations for Future Study

This analysis of three programs at a large mid-west university indicated that the assessment schemes used by online distance education courses do not strictly follow the principles suggested in the literature.  The authors were encouraged, however, by the degree to which formative and authentic assessment methods were used.  The authors found that the nature of each program (its history, purpose, and learner characteristics) had a significant impact on the assessment methods used.

This study also raised interesting questions regarding the assessment schemes adopted by the developers of online distance courses.  In order to draw more substantial conclusions and paint a more accurate picture of the state of online course assessment, the authors recommend that a follow-on study incorporate the following measures:

With these improvements to this study, researchers would be able to make firmer conclusions regarding the efficacy of various assessment principles and methods in the online distance context, as well as determine the validity of the recommendations the authors found in the literature.


References

Bennett, N., Dunne, E., & Carre, C. (1999). Patterns of core and generic skill provision in higher education. Higher Education, 37, 71-93.

Bransford, J. D., Brown A. L, & Cocking, R. R. (2000). How People Learn, Washington D.C.:National Academy Press,

Bransford, J. D., Vye, N., & Bateman, H. (2002). Creating High-Quality Learning Environments: Guidelines from Research on How People Learn. In P. A. Graham & N. G. Stacey (Eds.), The knowledge economy and postsecondary education: Report of a workshop. Washington DC: National Academy Press.

Dewald, N., Scholz-Crane, A., Booth, A., & Levine, C. (2000). Information literacy at a distance: Instructional design issues. The journal of academic librarianship, 26(1). 33-44.

Freeman, M., & Mckenzie, J. (2002) SPARK, a confidential web-based template for self and peer assessment of student teamwork: benefits of evaluating across different subjects. British Journal of Educational Technology, 33, 551-569.

Gokhale., A. A. (1995) Collaborative Learning Enhances Critical Thinking. Journal of Technology Education, 7(1).  Retrieved 15 April 2003 from http://scholar.lib.vt.edu/ejournals/JTE/jte-v7n1/gokhale.jte-v7n1.html

Keegan, D. (1990).  Foundations of distance education, 2nd edition.  London: Routledge.

Kerka, M., & Wonacott, J (2003) Focus on assessment. VET in schools. Retrieved April 1, 2003, from http://online.curriculum.edu.au/the_cms/tools/new-display.asp?seq=5928

Kibby, M. (2003) Assessing students online. The University of New Castle. Retrieved March 3, 2002, from http://www.newcastle.edu.au/discipline/sociol-anthrop/staff/kibbymarj/online/assess.html

Kirkpatrick, D. L. (1998).  Evaluating training programs.  San Francisco: Berrett-Koehler.

Meyen, E.L., Aust, R. J., Bui, Y. N., & Isaacson, R. (2002). Assessing and monitoring student progress in an e-learning personnel preparation environment.  Teacher education and special education, 25 (2). 187-198.

Pennsylvania State University (2002). An emerging set of guiding principles and practices for the design and development of distance education. Pennsylvania State University. Retrieved March 3, 2002, from http://www.outreach.psu.edu/DE/IDE/

Robles, M. & Braathen, S. (2002).  Online assessment techniques. Delta Pi Epsilon Journal, 44 (1). 39-49.

Romiszowski, A. J. (1981).  Plugging the gaps in present approaches to instructional design.  In Designing instructional systems: decision making in course planning and curriculum design.  London: Kogan Page.

Rovai, A. P. (2000). Online and traditional assessments: what’s the difference? Internet and higher education, 3. 141-151.

Seels, B. B. & Glasgow, Z. (1990).  Psychological basis for instructional design. In Exercises in instructional design. Columbus, OH: Merrill.

Tsui, L. (2000). Effects of campus culture on students' critical thinking. The Review of Higher Education, 23(4), 421-441.

Concord Consortium (2002). E-learning model for online courses. The Concord Consortium. Retrieved March 3, 2002, http://www.concord.org/courses/cc_e-learning_model.pdf

Wiggins, G. (1990). The case for authentic assessment. ERIC Digest. Retrieved February 27, 2003, from http://ericae.net/db/edo/ED328611.htm


Appendix A: Course List

School of Education courses (10 courses)

Instructional Technology Foundations

Effective Writing in Instructional Technology

Instructional design and development

Computer-Mediated Learning

Evaluation and Change in the Instructional Development Process

Designing Instructional Systems

Learning and Cognition in Education

Instructional Issues in Language Learning

Reading and Learning Skills Development at Post-Secondary Level

Advanced Study in the Teaching of Writing in Elementary Schools

School of Business Online courses (14 courses)

Human Resources Management / Leading Change

Quantitative Analysis

Managing Accounting Information

Financial Management (MBA Course)

Financial Management

Business Law

Strategic Management and Business Planning

The United States in a Global Economy

Developing Strategic Capabilities

Capstone Course

Electronic Commerce

Thinking Strategically

Strategic Marketing Management

Operations Management

School of Continuing Studies Undergraduate Courses (16 courses)

Anthropology

Human Origins and Prehistory

Business

Basic Accounting Skills

The International Business Environment

Communication & Culture

Business and Professional Communication

English

Creative Writing

Fine Arts

Art Appreciation

Geography

World Regional Geography

Geology

Earth Sciences: Materials and Processes

Our Planet and Its Future

History

Colonial America

American Colonial History II

American History II

Health and Physical Education

Personal Health

Journalism

Introduction to Mass Communications

Political Science

Introduction to International Politics

Sociology

The Family


Online Journal of Distance Learning Administration, Volume X1, Number I, Spring 2008
University of West Georgia, Distance Education Center
Back to the Online Journal of Distance Learning Administration Content