A Comparison of Student Outcomes & Satisfaction Between Traditional & Web Based Course Offerings


Julio C. Rivera, Associate Professor
E-Mail: jrivera@uab.edu
M. Khris McAlister, Professor
E-Mail: Kmcal@uab.edu
Department of Accounting and Information Systems
School of Business
University of Alabama at Birmingham
Phone: (205) 934-8820
Fax: (205) 975-4429

Margaret L. Rice
Associate Profesor
Instructional Technology
The University of Alabama
Tuscaloosa, AL 35487
E-Mail: mrice@bamaed.ua.edu

Introduction

With the recent growth of the Internet and other distance technologies, web based course delivery has become an attractive option for expanding the educational opportunities available to students. Our institution, like others, is actively pursuing this means of delivery in order to expand its reach to new students and to facilitate the scheduling of existing students. During a recent academic term, our students had the opportunity to enroll in such a course. Unique circumstances resulted in the simultaneous offering of additional course sections in a traditional lecture/discussion format, as well as a web-enhanced format. This pilot study documents a comparative evaluation of the three course formats.

Review of Literature

Many research studies have shown that cognitive factors such as learning, performance, and achievement in distance education classes are comparable to those observed in traditional classes (Carr, 2000; Russell, 1999; Schoech, 2000; Sonner, 1999; Spooner, Jordan, Algozzine, & Spooner, 1999), however, perceptions and satisfaction levels of instructors and students of distance education have not shown the same consistency (Bower, 2001; Hara & Kling, 1999; Stocks & Freddolino, 1998). Factors such as accessibility to materials, other students, instructors, control of time, and cost can influence individuals' perceptions of distance education (Middleton, 1997). Petracchi (2000) found that students were pleased with the performance of their instructor, availability of materials, and performance of technological tools used for conducting the class, while Carr (2000) found that undergraduates enrolled in an introductory psychology course performed better in distance education courses, but were generally less happy with them. Students in the web based course consistently scored an average of five percentage points higher on the final exam than did those in the lecture course, but they consistently reported less satisfaction than the students in the lecture course. Carr surmised that one of the reasons for less satisfaction could be more time required to complete assignments. In a study of adult distance education students, Carter (2001) found that most students did not find that technological equipment used in distance education interfered with the instruction. Carnevale (2000) found that distance education students look for many of the same things found in traditional courses including a knowledgeable professor, interaction with the professor, and additional features that create a feeling of community within the class.

Spooner, Jordan, Algozzine, & Spooner (1999), reported student ratings for two courses taught in both a traditional and distance format by the same instructor. No differences in the overall course grade means of the students and in the students' ratings of the course, instructor, teaching, and communication method were found. However, conflicting results were found for the ratings of class organization. For one course, the distance education class was rated as more organized than the traditional class, while for the other course; the distance education class was rated as less organized. Schoech (2000) reported that the satisfaction, grades, and performance of students in a semester-long graduate social work course taught in a distance format were consistent with previous outcomes in traditional courses of similar content.

In addition to web based courses consisting of an instructor created web site, many instructors are now using course delivery platforms such as WebCT or Blackboard. Kendall (2001) reported on a study in which courses taught through traditional means were converted into units using WebCT software as the primary means of delivery. Results indicated overall satisfaction with the WebCT software and the organization and content of the units. Wernet, Olliges, and Delicath (2000) reported on a survey examining the satisfaction levels and perceptions of 39 social work students regarding the use of WebCT in social work education course. All students responded that they found the course materials on the course Web site helpful. Graduate and nontraditional students' responses indicated greater use of the course management tools and nontraditional students responded that they were not disadvantaged by, and preferred access to, Web based courses.

Sanders and Morrison-Shetlar (2002) hold that student attitudes toward the Internet and web based courses can influence the future use of the web based instructional materials and how educationally beneficial web based resources are for students. Instructor attitudes toward web based instruction also affect student's experiences with web based courses. The National Education Association (National Education Association, 2000) administered a survey in which 75% of the instructors surveyed indicated that they were positive about distance education. Inman and Kerwin (1999) also found that most instructors currently teaching a distance education course indicated they would be willing to participate in a distance education course again, however, when asked about the quality of the distance education course, nearly 50% of the instructors indicating they would participate in a distance education course again reported that the quality of the distance education course, when compared to a traditional course, was lower. Additionally, not one instructor indicated that the distance education course was of higher quality.

Purpose of the Study

The purpose of this pilot study was to compare the efficacy of the three class formats, as well as to learn from the experience. To this end, we compared student performance, student satisfaction, and instructor experiences. In addition the study documented the benefits and limitations of the three delivery alternatives. We believe this was a unique opportunity to compare the three formats.

Furthermore, due to the composition of the student population at our school, we believe that our students are more representative of those who might be interested in enrolling in web based classes. Most of the students in our institution work full or part-time and attend classes at night. Intuitively, it would seem that Web delivered classes would be very attractive to this demographic group. Thus we believe that this study may be particularly useful for other institutions in similar situations.

Methodology

Course Format Description

The course offered was an undergraduate introductory course in Management Information Systems. It is a survey course, and was selected for this reason. The publisher provided the text, web materials, and an extensive examination test bank. Two instructors taught the three sections, the traditional section covered by one instructor, and the other sections by a second instructor. All sections covered substantially similar material, and completed similar assignments. The course formats were as follows:

Sample

The sample consisted of students enrolled in the three sections of a junior level Introduction to Information Systems course. The three sections participating in this study comprised all of the sections offered during the academic term studied. This course was required of all students in the School of Business, thus insuring a broad cross-section of our business student population. The number of students enrolled were 41 in the traditional course section, 40 in the hybrid course section, and 53 in the web based course section. Most of the students were juniors.

Instruments

The instruments used in the study consisted of multiple-choice exams using questions selected by the instructors from the publisherís test bank and a researcher-developed questionnaire. The exams themselves contained a different mix of topics, thus precluding direct exam-to-exam comparison. Nevertheless, since the classes covered the same topics and questions were drawn from the same test bank, a direct comparison of the average of the exam scores was possible.

Student satisfaction was considered important for the long-term success of web based course offerings. For this purpose a researcher-developed questionnaire addressing the specifics of the course offerings was administered to the students in the three course sections; the questionnaire included both objective and open-ended questions. The questionnaire was composed of questions addressing overall student satisfaction and satisfaction with the course format, student motivation for selecting a particular course format, and whether students were willing to enroll in a similar course in the future. The questionnaire used to elicit these responses was simple and did not include provisions for calculation of validity and reliability statistics. In addition to this, students filled out the routine teaching evaluation survey administered to all classes.

Instructor experiences were also documented through discussion and anecdotal references. The goal was to document problems experienced, as well as potential solutions to them. Also instructor satisfaction with the course format and supporting materials were of interest.

Results

Student Performance

Student performance was evaluated by comparing exam scores between the two classes using duplicate exams, and as an aggregate using overall exam averages. The goal was to determine if there were significant differences in student performance between classes. Although two different instructors taught the three course sections studied, use of the same text, similar assignments, and test bank questions and format allow for direct comparison. It should be noted that the instructors had also taught the course for a number of years and had a long history of coordinating the content, topic sequence and projects in the different sections.

A direct comparison of exam scores for three of the four exams administered in the hybrid and a web based class was undertaken. The first of the four exams was excluded from this comparison due to problems encountered when administering this exam. These problems are discussed in a later section. As is evident from examining Table 1, there is little difference between average exam scores. Statistical analysis using t-tests indicates that there is no significant difference between the exam scores.

Table 1: Average Exam Scores

Course Section

Exam #2 Average

Exam #3 Average

Exam #4 Average

Hybrid Class

74.92

66.51

79.84

Web based class

74.26

65.76

82.12

Comparison of the average of the exam scores for all three classes yielded results similar to those above. Again, the scores from the first exam were dropped for the reasons stated above. Examination of the data in Table 2 clearly shows how close the results were. T-tests between paired sets of scores also indicate no significant difference between exam averages.

Table 2: Exam Score Average

Course Section

Exam Score Average

Traditional Class

74.85

Hybrid Class

73.35

Web based Class

73.97

Although exam results are but one out of several criteria normally used in evaluating student performance, in this situation they serve as a readily available measure. It appears from the data collected, that there was no significant difference in student performance, regardless of the class format. This is an encouraging result, and bodes well for the future of web based education.

Student Satisfaction

Although student performance is an important measure of the success of a web based curriculum, student satisfaction is important for the continued success of such a program. It was therefore important to gauge student feelings about the web based course. This was accomplished by asking the students how they felt about the course, as well as whether they might want to take a course offered in a similar format again. It was also of interest to ascertain the reason the students had selected a particular section and the attendant delivery format. Students were asked these questions as part of the researcher developed questionnaire.

First, it was important to establish why students had enrolled in a particular section of this course. The assumption here was that students would select the web based section deliberately. Unfortunately, the course schedule given to students did a poor job of differentiating the web based course from those offered in a regular format. In particular the web based section was listed in the schedule at a specific time and place, with only a cursory note indicating it was to be web based. It is quite probable that many students were unaware that they had enrolled in a web based course. Nevertheless, as shown in Table 3, most students selected a given section of the course based on the time and place convenience for that section.

Table 3: Student Course Section Selection (# Students)

Selection Criteria

Traditional Class

Hybrid Class

Web based Class

Time & Place Convenience

27

30

40

Instructor Preference

2

2

1

Other

9

2

9

Students were also asked if they were satisfied with the course. A five point Likert scale with a range from "Strongly Disagree" to Strongly Agree" was used to gauge student satisfaction. Table 4 shows the data collected from this question. Examination of the data show that students enrolled in the web based section were somewhat less satisfied with the course than other students. This was confirmed through the use of a Chi square test of independence, which showed the results as independent at the 0.079 level of significance. The data then, show evidence of different satisfaction levels due to differing course formats. It is quite likely, as will be discussed later, that problems encountered by students in the web class may have had some impact on their satisfaction level.

Table 4: Student Satisfaction by Class (# Students)

Section

Strongly Disagree - 1

2

3

4

Strongly Agree - 5

Traditional Class

4

4

9

13

8

Hybrid Class

2

4

3

18

7

Web based Class

10

8

13

16

3

Results from the teaching evaluation survey were also illuminating. Although the survey used is instructionally mandated, three of the questions in the survey seemed particularly relevant to this situation. These results are presented in Table 5. It is quite evident that the students in the web based class had a different instructional experience than those in the two other sections.

Table 5: Teaching Evaluations

Question

Traditional Class

Hybrid Class

Web based Class

Method of Presentation Easy to Follow

3.632

3.235

2.419

Improved Understanding of Concepts and Principles

3.421

3.059

2.545

Explained Difficult Material Clearly

3.632

3.059

2.484

Finally, students were asked if they would enroll in a similar course in the future. This was intended to draw on the studentsí experience, and determine future intentions. Presumably, dissatisfied students would not wish to enroll in the same format course again. As the data in Table 5 show, most students would be willing to enroll in a similar format course again. There is however, a substantial degree of dissent among those students enrolled in the web based class.

Table 6: Sign-up for Similar Course in the Future (# Students)

Section

Yes

No

Traditional Class

35

3

Hybrid Class

30

4

Web based Class

33

17

Analyzing the data presented above, it is evident that students were not completely satisfied with the web based course offering. There are a number of factors that may have contributed to this perception. Some of these will be addressed in the next section covering the instructional experience. From a student perspective, though, two factors come to mind. It is quite likely that students having less technological mastery would have a less than satisfactory experience. It is also likely that students who are not self-motivated would find web based courses less satisfactory. Both of these conjectures require further study not contemplated in the scope of this study.

Instructional Experience

It is important to preface the following observations by stating that this was the first time a web based class was offered in this discipline at this institution. Both instructors had used the web previously to enhance traditional class formats, but a complete web course had not been offered. This was therefore a learning experience, and although efforts were made to follow recommendations, mistakes were made. Several areas that adversely affected the conduct of the web based and hybrid classes were the adoption of a course delivery platform, consistent instructional support, and course material delivery difficulties.

The course delivery platform selected for this course was WebCT. This platform was selected to take advantage of the web based materials available from the book publisher. The web based course section, as well as the hybrid course section used this platform. The hybrid section did not use the publisher produced course materials, relying instead on instructor-supplied materials.

WebCT, in the form maintained by the institutionís instructional support staff, did not always work consistently. This was a source of frustration for both students and the instructor. For example, it was originally intended that all exams would be administered through WebCT in a proctored lab environment. After a dismal experience with the first exam, this approach was abandoned and the class reverted to traditional mark sense forms (Scantron). There were also problems with the grade delivery facilities and occasionally with the delivery of course materials. These experiences quickly reduced the instructorís and the studentsí confidence in the course delivery technology. It should be pointed out that these technologies are rapidly evolving and may be less problematic in other instructional environments.

Another aspect of adopting course delivery platforms is the extent to which they limit the instructorís ability to control the presentation and format of the class web site. As mentioned above, both instructors had experience with maintaining course web sites, but not with the use of course delivery platforms. In this regard, the WWW/hybrid instructor experienced a steep learning curve with the web delivery platform.

A frustrating aspect of this learning curve was the discovery of the limitations inherent in the use of a course delivery platform. WebCT and other platforms are designed to provide the instructor with the facilities to create a class web site with a number of supporting functions. Unfortunately, in order to deliver this functionality, they require that the instructor strictly adhere to their methods and procedures. This can be particularly frustrating for instructors who are accustomed to directly manipulating the contents of a course web site. In this particular case, certain functions were ultimately provided through a separate web site in order to reduce the amount of effort required in delivering them.

An area that was not well implemented in our situation was that of instructional support. Among the materials provided to the web based class were videotaped lectures available through the web in the form of streaming media. The support to tape and prepare these lectures was lacking, and on occasion lagged behind the class schedule. This lack of support also manifested itself in the sluggish resolution of student problems accessing the materials, and in the lab facilities available to students. Although these problems had some impact on all of the course sections, they were felt most severely in the web based section.

The web based class also suffered from problems in consistently accessing web materials through the Internet. These problems fell in several areas. In the case of streaming media, there were problems with studentís having the right software to receive it, as well as being able to maintain an Internet connection consistent enough to use it. There were also problems stemming from the studentsí choice of Internet Service Provider, and their peculiar ways of delivering Internet content. Although data were not collected to verify the extent and nature of these problems, anecdotal reports from frustrated students were abundant.

Finally, the students enrolled in the web based section were not uniformly knowledgeable in the use of the web and its technologies. Their knowledge and skills ranged from rudimentary to well advanced. This required the use of some of the limited class time to demonstrate the use of particular features. Unfortunately, this lack of knowledge also meant that some students could not adequately configure their computer systems. Ultimately these experiences probably had a negative impact on studentsí satisfaction with the web based course.

Limitations

As with any study there are situational and methodological limitations to be considered. Methodologically, this was a pilot study. Thus, any generalizations are left to the discretion of the reader. Other limitations that should be considered are the following:

Conclusions and Implications

As documented above, this was a learning experience. From an instructional perspective, it was at times frustrating. In the end though, it was a useful experience that revealed several problem areas and lay to rest some fears. There were a number of problems with the course delivery platform and web materials, which it appears ultimately, affected the studentsí level of satisfaction with the web based course. Surprisingly, overall student performance did not suffer.

Initial concerns regarding student performance appear to be groundless. As detailed previously, student performance was remarkably consistent from one section to another. Whether this can attributed to instructional performance is debatable. From an instructional perspective, this means that other WWW sections can be offered with little concern over how well students will do in comparison with courses taught conventionally. This finding is also in agreement with other distance learning studies.

For the WWW delivered course, improvement is necessary from both the studentsí and instructorís perspective. It is clear that the delivery problems, lack of support, and the studentís prior mastery of the Internet contributed to the lower satisfaction levels. Presumably future classes will benefit from some of the lessons learned here, and result in more satisfied students. Some of the things that require improvement are:

In summary, although this experience in offering a web based class was not ideal, it provides us with some experience to build on for the future. It appears that student satisfaction rather than performance are the issues to be concerned about.


References

Bower, B. (2001). Distance education: Facing the faculty challenge. Online Journal of Distance Learning Administration, 4(2). Retrieved April 6, 2002 from http://www.westga.edu/~distance/ojdla/summer42/bower42.html

Carnevale, D. (2000). Study assesses what participants look for in high-quality online courses. Chronicle of Higher Education, 47(9), A46. Retrieved April 6, 2002 from Academic Search Elite database.

Carr, S. (2000, March 10) Online psychology instruction is effective, but not satisfying, study finds. Chronicle of Higher Education, 46(27), pA48, 2/5p. Retrieved April 6, 2002 online from Academic Search Elite database.

Carter, A. (2001). Interactive distance education: Implications for the adult learner. International Journal of Instructional Media, 28, 249-261. Retrieved April 6, 2002 from Academic Search Elite database.

Hara, N. & Kling R. (1999). Studentís frustrations with a web-based distance education course. First Monday, 4(12). Retrieved April 6, 2002 from http://firstmonday.org/issues/issue4_12/hara/index.html

Inman, E. & Kerwin, M. (1999). Instructor and student attitudes toward distance learning. Community College Journal of Research & Practice, 23, 581-592. Retrieved April 6, 2002 from Academic Search Elite database.

Kendall, M. (2001). Teaching online to campus-based students: The experience of using WebCT for the community information module at Manchester Metropolitan University. Education for Information, 19(4) 325Ė346. Retrieved April 6, 2002 online from Academic Search Elite database.

McAlister, Khris M., Rivera, Julio C., and Hallam, Stephen F. (2000). Twelve questions to answer before offering a web based curriculum, challenges of information technology in the 21st century, Proceedings of 2000 IRMA International Conference, p. 1018.

Middleton, A. J. (1997). How effective is distance education? International Journal of Instructional Media, 24, 133-138. Retrieved April 6, 2002 from Academic Search Elite database.

Mock, R. L. (date?). Comparison of online coursework to traditionaliInstruction. Retrieved January 11, 2001 from the World Wide Web: http://beaverton.k12.mi.us/teachers/mock/masters/abstract.htm

National Education Association. (2000). Confronting the future of distance learning Ė placing quality in reach. Retrieved April 6, 2002 from http://www.nea.org/nr/nr000614.html

Petracchi, H. E. (2000). Distance education: What do our students tell us? Research on Social Work Practice, 10, 363-378. Retrieved April 6, 2002 online from Academic Search Elite database.

Russell, T. L. (1999). The "No Significant Difference" phenomenon. Raleigh: North Carolina University. Retrieved April 8, 2002 from the World Wide Web: http://teleeducation.nb.ca/nosignificantdifference/

Ryan, R. C. (2000). Student assessment comparison of lecture and online construction equipment and methods classes. THE Journal, (27)6.

Sanders, D. W., & Morrison-Shetlar, A. I. (2002). Student attitudes toward web-enhanced instruction in an introductory biology course. Journal of Research on Computing in Education, 33(3), 251-262.

Schoech, D. (2000). Teaching over the Internet: Results of one doctoral course. Research on Social Work Practice, 10, 467-487. Retrieved April 6, 2002 from Academic Search Elite database.

Sonner, B. (1999). Success in the capstone business courseóassessing the effectiveness of distance learning. Journal of Education for Business. 74(4), 243-248.

Spooner, F., Jordan, L., Algozzine, B. & Spooner, M. (1999). Student ratings of instruction in distance learning and on-campus classes. Journal of Educational Research, 92, 132-141. Retrieved April 6, 2002 from Academic Search Elite database.

Stocks, J. T. & Freddolino, P. P. (1998). Evaluation of a World Wide Web-based graduate social work research methods course. Computers in Human Services, 15, 51-69.

Wernet, S. P., Olliges, R. H., Delicath, T. A. (2000) Postcourse evaluations of WebCT (Web Course Tools) classes by social work students. Research on Social Work Practice, 10(4), 487, 18p. Retrieved April 6, 2002 online from Academic Search Elite database.


Online Journal of Distance Learning Administration, Volume V, NumberIII, Fall 2002
State University of West Georgia, Distance Education Center
Back to Journal of Distance Learning Administration Contents