A Framework for Design and Evaluation of Internet-Based Distance Learning Courses
Phase One - Framework Justification, Design and Evaluation
Russell K. Baker
University of Tampa
Introduction - Distance Learning (DL)
Defined in its most basic form, distance learning occurs when the student and the instructor are logistically separated. Considered from this prospective, distance learning is not a new concept to academia. Educational institutions have been providing distance learning courses in various formats for many years. Correspondence courses were offered as early as the mid-1800s (Sonner, 1999). As technologies developed, various types of distance learning have evolved along with the technologies. Universities have been providing directed and independent study distance learning courses utilizing videotapes and interactive television since these technologies became available. The advent of Internet technologies and their application to distance learning resulted in an explosive growth of distance-learning courses at the collegiate level. According to U.S. Department of Education reports, distance-learning enrollments at the university level increased 70 percent during the period between 1995 and 1998 (Boehie, 2000). This growth has continued as more online courses became available through a variety of educational venues.
Online distance learning, also called e-learning, has become a major consideration in curriculum development in higher education and is expanding into the K-12 arena. The Kentucky Migrant Technology Project has developed an online program to supplement its traditional classroom based program. The courses cover core curriculum subject areas including mathematics, English, sciences, social studies, arts, humanities, and practical living subjects for students in grades six through twelve (Abell, 2002). Virtual High School, an online consortium of over 200 high schools in 26 states, reports a 537 percent increase in student enrollment during the period from 1997 through 2002 (govhs.org, 2002). At all levels, these courses create significant problems for faculty and others concerned with evaluating the quality of instruction and content of these courses. The key question is whether students receive a quality education through online distance-learning formats. Other problems, like digital cheating (cheating using computer technology) and plagiarism, are also significant issues for e-learning. However, these issues are outside the scope of this paper.
Although numerous studies suggest the effectiveness of distance-learning courses to be comparable to that of traditional coursework, evaluations of e-learning course effectiveness have been based on comparison of final grades for students who took the online courses compared with grades of students who participated in classroom-based courses (Sonner, 1999). This limited basis of evaluation does not resolve the concern for the quality of education provided using internet-based delivery. “While technology has provided meaningful tools for tracking, sorting, and disseminating information, it has created unprecedented complexity, as well as a concern for the value and integrity of that information” (Karr, 2002). Additionally, decisions regarding distance learning are made in a context unfamiliar to many faculty and administrators. Resolving curriculum issues, a difficult challenge within itself, is compounded by the application of rapidly evolving technologies. Even accreditation organizations have difficulty resolving issues raised by distance learning. The Council for Higher Education Accreditation suggested the following issues need to be addressed regarding distance learning effectiveness (Eaton, 2000):
- What is an effective framework for distance learning?
- How can quality be evaluated and insured?
A significant body of research exists that addresses these issues for traditional curriculum development. As early as 1932, Ralph Tyler stated that teachers should:
"Formulate the course objectives, define the objectives in terms of student behavior, collect situations in which students are to indicate the presence or absence of each objective, and provide the method of evaluating the student’s reactions in the light of each objective" (Tyler, 1932).
These concepts, further developed in his 1949 book Basic Principles of Curriculum and Instruction, provide the basis for curriculum and course development through the establishment of objectives and the evaluation of effectiveness through measurement of student mastery of these objectives. Subsequent research developed the concept of course objectives. Taxonomies developed by Bloom, Krathwohl, Gagne, Anderson and others classify learning objectives by type and hierarchical order, providing a basis for framework development (Bonner, 1999). The most widely applied of these is Bloom’s Taxonomy. According to Bloom:
"It is intended to provide for classification of the goals of our educational system. It is expected to be of general help to all teachers, administrators, professional specialists, and research workers who deal with curricular and evaluation problems. It is especially intended to help them discuss these problems with greater precision "(Bloom, 1956).
The educational outcomes described in the taxonomy are discussed from a behavioral perspective and address the changes in individuals resulting from educational experiences. Thus, the taxonomy provides a means of classifying the behaviors which represent the desired outcomes of educational processes (Psychology of Learning, 2002). Specifics of the taxonomy are discussed later in this paper.
While widely applied to traditional classroom curriculum, the work of Bloom and Tyler has not been integrated into the venue of distance learning. One reason for this may be the significant differences between delivery methods employed in traditional classroom courses and those mandated by the technologies used in distance learning. Traditional delivery methods used to achieve Bloom’s learning objectives are predominantly unavailable to the online course instructor. Furthermore, traditional evaluation procedures prescribed by Tyler fail to address the unique circumstances imposed by e-learning. Lack of immediate feedback, time and space separation of students from faculty and each other, elimination of nonverbal (body language) communication and reduction of control of testing environments may weaken the evaluation process unless provisions are made to adapt feedback and testing to online distance learning.
Several studies and informational articles reviewed during preparation for this paper (dlnr.org, 2002; Osborne, 2002) partially address the delivery methods available to online course instructors in conjunction with Bloom’s taxonomy and Tyler’s principles. The works discuss some of the delivery methods available to online educators. While informational, the works did not address the following issues:
- Provide a more comprehensive list of online delivery methods
- Propose specific online delivery methods to Bloom’s cognitive learning hierarchy
- Incorporate feedback and evaluation methods appropriate to e-learning to verify course effectiveness
- Integrate the above into a comprehensive framework for development and evaluation of online distance learning courses
Thus, the purpose of this paper is derived from the necessity to address these issues.
The purpose of this paper is to propose a framework for the development and evaluation of online distance learning courses, based on integrating an adaptation of Tyler’s principles within the levels of cognitive learning in Bloom’s Taxonomy. The methodologies for delivery of course materials and evaluation of outcomes specific to online distance learning are discussed. Examples are suggested to achieve each learning level objective. The framework will then be functionally evaluated by applying it to a currently available online course.
Limitations of Framework Evaluation
The study had the following limitations:
1. The study was restricted to an analysis of K - 12 curriculum components.
2. The study is designed to focus on a distance learning component.
3. K-12 distance learning courses located on the web do not allow non-participant access to all courses. Consequently, analysis was limited to the sample courses provided.
4. Sample courses did not allow full access to all course components
In the Taxonomy of Educational Objectives: Handbook 1: The Cognitive Domain (1956), Bloom provides a theoretical framework for classification of behaviors resulting from educational processes and evaluation of the extent to which desired behaviors were learned by students (Bloom, 1956). The cognitive domain, predominant in a majority of educational courses, consists of learning that is demonstrated by recall of knowledge and intellectual skills including comprehension of information, organization of ideas, analysis and synthesis of data, application of knowledge, alternative evaluation and choice, and problem solving (dlrn.org, 2002). Bloom defined six levels of learning objectives within the cognitive domain. These levels represent a hierarchy of complexity of learning skills ranging from simple recall and fact recognition at the first level of the hierarchy to increasingly more abstract and complex mental levels culminating with evaluation reflected in the student’s application of learned behaviors. Bloom’s classifications of learning objectives in the cognitive domain as defined in Principles of Curriculum and Evaluation (Osborn, 2002) are:
1. Knowledge - remembering previously learned material
2. Comprehension - grasping the meaning of material
3. Application - using learned material in new and concrete situations
4. Analysis - breaking down material into its component parts and understand its organizational structure
5. Synthesis - assembling parts together to form a new whole
6. Evaluation - judging the value of material for a given purpose
Bloom viewed these objectives as the “intended behaviors which the student shall display at the end of some period of education” (Bloom, 1956) and a developmental process through the learning objectives hierarchy, each intended objective building on an achieved predecessor (Psychology of Learning, 2002). By applying specific verb terminology related to each of the learning objectives it becomes possible to define specific behaviors to evaluate successful attainment of a learning objective. Some of the verbs related to each of Bloom’s learning objectives include: (Osborn, 2002; dlrn.org, 2002)
1. Knowledge: arrange, define, duplicate, memorize, recognize
2. Comprehension: classify, describe, identify, report, restate
3. Application: apply, choose, illustrate, solve, write
4. Analysis: analyze, categorize, criticize, distinguish, test
5. Synthesis: assemble, collect, manage, organize, propose
6. Evaluation: argue, assess, choose, value, evaluate
A more comprehensive listing of verb examples is provided in the appendix. Incorporating these verbs into specific course objectives categorizes learning objectives within Bloom’s classifications and clarifies desired outcomes for both teacher and student. Consider the following two versions of the same course objective:
1. Discuss centralized and distributed database systems.
2. Distinguish between centralized and distributed database systems and analyze the advantages and disadvantages of each.
Significant ambiguity exists in the first statement. Does discuss mean define and fall into the Knowledge classification? Does it mean describe and fall into the Comprehension classification? Without the application of objective verbs as in the second statement, classification of the complexity of objectives is difficult. Additionally, application of the objective verbs clarifies exactly what is expected from the student.
Tyler's Basic Principles
In his Basic Principles of Curriculum and Instruction (1949), Tyler stated that “if we are to study an educational program systematically and intelligently we must first be sure as to the educational objectives aimed at.” The pioneer in the application of objectives to curriculum, Tyler defined four basic objective-centered principles: (Tyler, 1949, 1)
1. What educational purposes should the school seek to attain? (objectives)
2. What educational experiences can be provided that are likely to attain these purposes? (experiences)
3. How can these educational experiences be effectively organized? (organization)
4. How can we determine whether these purposes are being attained? (evaluation)
Tyler did not attempt to answer these questions. His purpose was to provide “a rational by which to examine problems of curriculum and instruction” (Tyler, 1949, 3). The rationale should be used by curriculum / course developers based on the needs of their particular institution and curriculum. These principles are used as a component of the framework for course development suggested later in this paper.
Learning objectives need to be developed for each educational experience. Tyler stated that objectives are necessary both to provide a focus for teaching and as a criteria for evaluation. In a 1981 interview conducted by Jeri Ridings Nowakowski, Ed.D., Tyler said that objectives were “…very important for people starting a program to reach new students and find our whether they were accomplishing their purposes)” (Nowakowski, 1981) and that they should be reflective of what the instructor hoped their students would be learning.
Initial objectives should be based on three sources. The needs and interests of the students, the competencies demanded by contemporary society, and the subject matter to be covered should be evaluated and screened to develop precise instructional objectives (Orlosky, 1978). For objectives to be useful, they must be stated in a manner such that both the kind of behavior desired from learners and the “content or area of life” that the behavior fits are both stated within the context of the objective statement. The objective statement To write clear and well-structured programs for database applications using Visual Basic programming language defines the kind of behavior desired, writing clear and well-structured programs, and also indicates the content area which the programs address, Visual Basic databases.
Although widely accepted, Tyler’s principles are not without criticism. Some researchers believe that specific goals are too confining for higher-order learning - “the difficulty of writing behavioral objectives for higher order capabilities meant that many of the objectives were individually quite trivial. In order to achieve the level of precise behavioral specification demanded, curricula became atomized into sometimes hundreds of objectives with the assumption being that the whole was simply the sum of the parts” (Kissane, 1999).
Tyler defines learning experiences as an interaction between the learner and the environmental conditions with which he reacts (Tyler, 1949, P63). The experience is based on active student behavior, not the behavior of the instructor. However, it is the instructor who provides the educational experience by establishing the environment in which learning can be achieved, providing a delivery method suitable to the targeted outcomes, and structuring the learning activities to produce the desired learner behavior.
"The teacher can provide an educational experience through setting up an environment and structuring the situation so as to stimulate the desired type of reaction " (Tyler, 1949, P65).
Tyler goes on to provide five general principles for establishing educational experiences:
1. Students must practice behavior required by the objective (practice).
2. Students must obtain satisfaction from the required behavior (rewards).
3. Students must be able to perform the required behavior (perform).
4. Different experiences should be used to accomplish the objective (variety).
5. The same experience usually brings about several outcomes (multiple outcomes)
The process of developing learning experiences is creative. No definite prescribed experiences are applicable to each particular objective. Learning experiences should be developed by the instructor based on the above criteria, the available methods of delivery of material, and the potential of the experiences of achieving the stated objectives.
"In order for educational experiences to produce a cumulative effect, they must be so organized as to reinforce each other" (Tyler, 1949).
Organization of learning experiences is necessary to achieve continuity and to identify major elements around which other elements are structured. Prior to evaluating organization, it must be determined which level of structural element is being developed or evaluated. Tyler's three levels of structural elements include:
1. The entire curriculum as a broad unit.
2. Sequential courses like social studies 1 followed by social studies.
3. Individual course design or lesson plans for individual courses.
Three guidelines are provided by Tyler for effectively organizing curriculum.
1. Continuity refers to the vertical reiteration of major curriculum elements
2. Sequence emphasizes the importance of having each successive experience build upon the preceding one but to go more broadly and deeply into the matters involved
3. Integration refers to the horizontal relationship of curriculum experiences.
These three guidelines provide the basic criteria to direct the design of effectively organized learning experiences.
Tyler’s approach to performance evaluation is based on initially establishing broad goals or objectives, classifying them into subordinate objectives, defining the objectives in behavioral terms, establish learning experiences in which achievement of objectives can be shown and practiced, develop measurement techniques that reflect the objectives, collect data on learner performance on the measurement, and compare the data against the previously stated behavioral objectives. Any discrepancies between performance and objective should then be given to the learner as feedback and the cycle would be repeated until desired outcomes were achieved (Worthen and Sanders, 1987).
"The process of evaluation is essentially the process of determining to what extent the educational objectives are actually being realized by the program of curriculum and instruction" (Tyler, 1949, 105).
According to Tyler, since it is the changes in the behavior of students we seek to evaluate, evaluations must target these changes. Additionally, a series of evaluations is necessary to determine behavioral changes. An initial evaluation is needed to establish baseline performance and subsequent evaluations measure progress. Any valid format for evaluation that is reflective of behavioral changes is appropriate. Tyler suggests several methods including testing, observations, interviews, sampling and collection of products (term papers, etc.) produced by students (Tyler, 1949, 107). The type of evaluation used must be appropriate to the skills necessary to achieve the desired student behavior. Hence, while paper and pencil (or select and click in the case of digital courses) are appropriate for a mathematics course, there are desired behaviors which, according to Tyler, are not easily appraised by written examinations.
Publications of behavioral theorists support the effectiveness of goal directed behavior. Kenneth Blanchard, co-creator with Paul Hersey of the Situational Leadership Theory, is a proponent of the 80-20 rule of goal setting. Blanchard states that 80 percent of really important results are derived from 20 percent of goals (Blanchard, 1981). This is the result of most goals being poorly defined and communicated. The most effective goals are designed, according to Blanchard, to fit the SMART acronym. Smart goals are those that are specific, measurable, attainable, relevant, and trackable. Specific goals are clear, well defined, concise, and absent of ambiguity. Measurable goals are stated to clearly indicate when the goal has been successfully achieved. Additional measurements should be provided for to determine progress. A goal is attainable when the individual attempting to accomplish the goal believes that can be accomplished.
"You want to stretch them, but you don't want to make the goals so difficult that they are unattainable and the individual gets demotivated" (Blanchard, 1984).
Relevance of a goal is the degree to which it actually changes performance. Relevant goals are focused on the desired behavior. Trackable goals have designed measurements and milestone points (deadlines) to determine progress towards the goal, timeliness, and final attainment of the goal. Goals can not be accomplished unless they are clearly communicated to and understood by those that are trying to accomplish them.
Blanchard's SMART goal concept was modified into a comprehensive framework for linking goals to performance by Edward Locke and Gary Latham. The framework suggests that designing goals that are challenging but attainable, specific, accepted by those attempting to accomplish the goal (ownership) and are beneficial to these individuals are more likely to lead to the desired performance than goals that do not fit these criteria (Locke and Latham, 1990). Locke stipulates that individual performance is affected by several moderators. These moderators include the complexity of the tasks involved in goal attainment, the quality and frequency of progress and performance feedback received by the individuals attempting to accomplish the goal, and situational or environmental factors beyond the control of goal setters.
Distance Learning Tools
Distance learning courses are by definition not face-to-face. Additionally, they may be either synchronous or asynchronous. The differences between traditional in-class courses and DL courses create several factors which need to be addressed in DL course design. First, traditional courses are face-to-face synchronous courses. This means that learning occurs with the instructor in front of the learner and instruction and learning occur simultaneously and in the same place. This personal interaction between the two gives the instructor the opportunity to provide feedback, direction, and to observe learning activities. It gives the learner the opportunity to solicit feedback and receive responses and directions in real time. Second, traditional courses place the learner with other learners. Learners are able to draw from each others experiences and interact in groups. Third, the requirement to attend class creates a responsibility for the learner and provides a source of accountability, possibly increasing his/her motivation to perform the required tasks.
To develop procedures for addressing these factors, an understanding of the tools available for distance-learning instructional methods and course design is needed. A variety of web-based tools and course design strategies including the following, are available (Ferguson, 2001):
1. Syllabus / course outline posting
While syllabus posting is available for both DL and non-DL courses, the absence of an instructor to review the syllabus with the learner requires an easily understandable and comprehensive syllabus.
2. Video classroom
Streaming video has replaced video tapes as the tool of choice for lecture delivery in DL courses. Video must be interesting, engaging, and worthwhile. Videos should not simply restate what is available in written materials. A primary consideration for streaming video is the bandwidth available to the learner. While high speed connections are usually available on campus, Students who rely on dial-up connections will be at a significant disadvantage when using streaming video. Provisions for downloadable or CD-Rom video files should be made to prevent this problem.
3. Course Notes
Course notes can be posted to the web to supplement video lectures and required readings.
4. Course Reference Materials, Readings, Cases
Supplemental reading materials can be posted to the web for students to download. Links to other websites are frequently provided in references.
5. Chat Rooms
Structured chat rooms conducted by the instructor provide group discussion on course activities and assignments. Using real-time chat, the instructor can ask questions during in a similar manner to the traditional classroom. Audio chat, a web-based tool that functions similar to teleconferencing, using a tool like Microsoft’s NetMeeting ® has an advantage in that it is more spontaneous than text-based chat and not dependent on the learners keyboard speed. Chat also allows the instructor to provide immediate feedback to learner questions, evaluate learner participation, and take attendance.
Email allows students to asynchronously communicate with their instructor. Learners can ask questions and send assignments to the instructor. The instructor can use email to send evaluated assignments back to the learner
7. Bulletin Boards, Group Discussion Boards, Digital Drop Boxes
These tools allow the learners to collaborate on projects, exchange ideas and participate in group activities.
8. Online Testing
Online testing procedures allow the instructor to design evaluation instruments comparable to any form of paper-based instrument. Multiple choice, true or false, matching, and fill-in-the-blank questions can be automatically graded and posted. Long or short answer essay questions can also be used. However, essay questions must be graded by the instructor.
9. Interactive Activities
Interactive activities provide a method of having the students practice desired behaviors. Click and drag techniques can be used to assemble components online (for example atoms into a molecule).
Specific provisions must be provided to insure students receive sufficient feedback. A frequent criticism of DL is its disembodied nature restricts feedback leaving learners feeling abandoned. Instructors must be trained to promptly respond to emails. Virtual office hours can be held using chat. Computer graded exams should have provisions for giving the student correct answers to the questions answered incorrectly. Provisions for both asynchronous and synchronous feedback should be provided in course design. Asynchronous feedback occurs when the individual requesting the feedback (the student) experiences a time delay before feedback is received. Synchronous feedback occurs when the feedback response immediately follows the question or request with no time delay.
11. Virtual Classroom
An online, interactive class session between students and instructor. Simulates much of the interaction found in traditional face-to-face classrooms. Frequently incorporates other web-based tools including audio chat, video classroom, whiteboarding, etc.
The ability to write and draw on an electronic board during a virtual classroom session.
Method of Analysis
A Framework for Design and Evaluation
Utilizing a combination of Blooms classifications and Tyler’s principles, it is possible to construct the following initial framework for design and evaluation of curriculum:
By incorporating the Blanchard and Locke concepts for successful goals and the Tyler concepts for Experiences, Organization, and Evaluations, the framework is expanded to include potential design and evaluation points for curriculum.
Behaviorally targeted evaluation
Appropriate evaluation type
Integrating Evaluation Questions into the Framework
To use the framework as a functional tool for design or analysis of curriculum components, specific questions were developed to address each of the above evaluation points. The following expansion of the above framework adds questions to address each evaluation point. The questions, designed to be answered yes or no, are based on the descriptions of each of the evaluation criteria by their respective authors.
Bloom Criterion - Which of the Bloom criterion is being addressed by this learning activity? To determine criterion category, compare objective statements with verb list in appendix. To stipulate category in course design, use verbiage appropriate to criterion from verb list in appendix.
Tyler Objectives - Is the objective statement written utilizing verbiage appropriate to the Bloom criterion? (Reference the Verb list in the appendix)
Specific - Are objectives stated in a clear, well defined, and concise manner
Are objective statements absent of ambiguity?
Measurable - Is the objective stated such that a specific measurement is provided to indicate goal achievement?
Attainable - Are the objectives attainable by the students with effort in the time allowed? (This is the same as Tyler’s Performance principle.)
Relevant - Are the objectives focused on the desired behavior?
Are the objectives appropriate to the Bloom criterion?
Trackable - Are milestone measurements with time deadlines indicated?
Challenging - Are the objectives difficult but attainable?
Communicated - Are objectives well communicated to students?
Delineated - Are the objectives clearly and obviously delineated as objectives of the learning component?
Ownership - If possible to evaluate, are the students in agreement with the stated objectives?
Tyler Experiences - Which distance learning tools are used to deliver the course? Evaluate specific tools using Tyler’s five general principles for establishing educational experiences based on the following questions:
Practice - Do online activities require students to practice the skills necessary to achieve the desired behavior? Is a feedback mechanism provided to correct student mistakes during practice activities?
Reward - Are rewards provided to students that motivate them to perform the tasks necessary to achieve the desired behavior? Are the rewards delivered to the students in a timely manner after the desired behavior has been achieved? Are the rewards withheld from students who do not achieve the desired behavior?
Performance - This is the same as attainable objectives discussed previously.
Variety - Are a variety of experiences (based on a variety of distance learning tools) used to enhance student learning? Do the distance learning methods implemented provide experiences appropriate to the Bloom Criterion category being designed or evaluated?
Outcomes - Are distance learning tools selected and implemented to restrict the possible outcomes to those that achieve the desired behavior? If student activities result in undesired outcomes, is a feedback mechanism provided to correct students?
Tyler Organization - Which level of structural element is being developed or analyzed?
Continuity - Are multiple opportunities provided for students to practice the skills necessary to achieve the desired behavior to be practiced? Are practice opportunities provided continuously throughout the course?
Sequence - Do successive experiences (activities) build upon the preceding one but to go more broadly and deeply into the matters involved? Do successive experiences evolve to achieve the desired level of complexity within Bloom’s hierarchy?
Integration - Do the activities used to develop the skills necessary exemplify the relationship of the desired behavior to other subjects/fields?
Tyler Evaluations -Behaviorally targeted evaluation - Do the evaluations used specifically evaluate the intended behavior?
Baseline evaluation - Is a baseline evaluation used to access students initial knowledge of the desired behavior?
Successive evaluations - Are a series of evaluations used to assess student progress?
Appropriate evaluation type - Are the skills and knowledge necessary to successfully complete the evaluations representative of the skills and knowledge necessary to achieve the intended behavior?
Appropriate Bloom Criterion Level - Are the evaluations appropriately structured to reflect mastery of the behavior at the desired Bloom criterion level?
Implementing a Framework Evaluation Scale
To use the framework as a tool for curriculum evaluation or design, a scale must be established to measure and assess the evaluation points. Under the premise that each of Tyler’s principles is equally important in the design or analysis of curriculum, each principle is subjectively assigned a value of 25% of the total weight. This value is equally divided among the evaluation point questions for each principle. Thus, while questions related to one principle may have a different value than questions in another principle, the total weighted value of one principle is equal to each of the others. Assign point values for each yes answer based on the category as indicated in the following table:
Objectives 2.5 each question
Experiences 2.5 each question
Organization 5.0 each question
Evaluation 5.0 each question
Analysis of Distance Learning Course
The Virtual High School (VHS) is a consortium of 87 high schools in 30 states that offer web-based distance learning high school courses (Kozma, 2000). Founded in 1996, the consortium currently offers over 200 courses in various subjects to over 4000 students. Sample lessons from several of these courses are available for review at www.govhs.org. The one selected for this analysis is Biotechnology: The Changing Face of Genetics. Designed as a twenty-six week course, only the following lessons are available in the sample:
Week 1 - So what are you bringing to the party?
Week 2 - Are you the master of your fate?
Week 3 - Just how alike are twins?
Week 4 - The Central Dogma in action
Week 10 - GEF Quest
Week 11 - Survey of Genetic Engineering
Week 14 - Redefining Natural Selection
Week 15 - Where are we headed?
Figure 1 - Sample Screen from the lesson "Just how alike are twins?"
Because of designed in restrictions of the sample courses, none of these lessons provides complete access to all lesson components. While specific content comparisons between online and traditional courses were not provided for this particular subject, several other virtual courses were compared against their traditional counterparts on the VHS website. Courses were compared for their material content and student performance on evaluation instruments. According to VHS, course content and student performance are comparable using either venue of course delivery (Kozma, 2000). The reader should bear in mind that this is based on VHS’s own internal evaluation.
Delivery Method And Content
The course uses various methodologies for delivery of content. The primary delivery method is via text and graphics integrated into the online class. The lesson has a primary content path called the Learning Space. Students are guided from screen to screen using typical next and back arrow icons used on most websites. The main topic discussion document is contained within this Learning Space. At the end of the main topic presentation students can enter the Course Room. The Course Room is the core of all of all course activities and communications. It contains a threaded discussion board allowing participants to post comments and questions regarding the lesson. All comments pertaining to that particular lesson are visible to all students for the duration of the course. This allows students to interact with each other. Students, student groups, and instructors can participate in discussions about course material, assignments, and other items that would normally be discussed in an actual face-to-face classroom. Students are provided numerous links within the course material to access additional information or more in-depth information regarding subject content. When students click on the buttons, expanded materials specific to the topic of the link are presented in a popup window. Introductory course materials state that some of the expanded materials are presented in digital video form. However, no digital videos were available in the sample lessons. These materials may also be accessed through the Media Center, a web component of each course containing digital materials posted by the instructor to augment the Learning Space material.
Students are also provided links in the Course Room called “Questions about Assignments” and “Questions about Technical Issues”. During lesson evaluation both links were clicked and neither allowed access to post or email a question. Again, this may be a limitation of the sample course rather than a defect in the technology. Students are offered an opportunity to solicit feedback prior to submitting homework. By selecting the “Request for Review” option, homework may be submitted for instructor feedback before being submitted for grading.
The Learning Space is the initial entry point for a particular lesson. The lesson selected for evaluation, Week 3 - Just how alike are twins, contains the following five sections:
Explanations - Adiscussion of the activities that make up the lesson (not objectives).
Milestones in Understanding Genetics - Readings and assignment from a section of Media Center. Media Center contents not available in sample.
The Minnesota Twins - Readings and assignment from a section of Media Center. Media Center contents not available in sample.
To be or not to be - Readings and assignment questions with links to detailed readings.
What is a twin - Readings and assignment from a section of Media Center. Media Center contents not available in sample.
In The News - Link to article on cloning with review question assignment.
Who, Where, or How - Link to review the Roslin Institute and review questions about institute's website.
The specific contents of each section of the lesson will be addressed, where available, during the application of the framework in the following section.
Applying the Framework
The individual components of the framework are applied to the lesson as follows:
Bloom Criterion - No objectives are defined for the lesson, consequently it is difficult to determine the desired Bloom criterion level. By using the questions asked of students in the individual lesson sections as learning objectives for the sessions, the criterion level for the lesson falls within the comprehension and knowledge categories.
The section- Just how alike are twins” asks the following questions:
1. How is cloning similar to the process by which twins “twinning” are formed?
2. For what scientific reason is the process of cloning being pursued?
3. What significant advance in cloning did “Dolly” represent at the time she was cloned? (www.govhs.org)
At best, these questions require the student to restate information provided in the readings in the student's own words. This is clearly within the comprehension criterion (see appendix). At the least they simply require recall and restatement of information which falls within the knowledge category.
Objectives - While no specific objectives are defined and none are discussed for the lesson as a whole, objectives are suggested in some sections of the lesson. As indicated in the above discussion of Bloom Criterion, questions are asked of the student in each lesson section. These questions are assumed to be the lesson section objectives throughout the analysis of the lesson objectives.
Specific (Question 1) - Section objectives, where available, are marginally specific. The section - Who, Where, or How” instructs the student to “While visiting find the answers to the following questions: 1. What is the Mission of the Roslin Institute? 2. What was the significant event that occurred there?”(www.govhs.org). Although not specifically delineated as learning objectives, this statement does provide the student with the desired learning outcome of the section. This is similar in structure to statements made in other sections. None of the sections contain a specific objective statement to the effect of “the objectives of this section are …”.
Specific (Question 2) - No. The absence of objective statements, either delineated or implied, obviously creates ambiguity.
Measurable - Assuming that finding the answers to the above questions are the objectives, they are clearly measurable. Once the student finds the answers to the questions, a measurement of completion can be made.
Attainable - Answers to the questions asked in the lessons were, where accessible, easily obtainable within the content of the lesson session. Since these questions were assumed to be the session objectives, the objectives are attainable by the students.
Relevant - The Bloom criterion for the lesson was determined based on using the lesson session questions as objectives. Consequently, the objectives obviously correspond to the criterion and are relevant. Since neither the desired Bloom criterion nor lesson objectives were stated in the lesson framework, this can not be accurately determined.
Trackable - No milestone measurements are provided for the individual lesson sections. However, each section is within itself a milestone for the overall lesson. Consequently, the lesson has trackable milestones in the form of the answers to the questions the students are required to submit for each lesson section.
Challenging - The answers to the questions were easily obtainable within the lesson session content. Additionally, the questions address low level Bloom criterion. Consequently, the objectives are not challenging.
Communicated - The goals for the lesson and lesson sections are not clearly communicated. The lesson goals are not stated anywhere in the accessible component of the web course. The session goals discussed previously are assumed to be the goals because they are the questions asked of the students regarding assignments.
Delineated - No delineation of objectives is evident in either the lesson or lesson sections. It was necessary to assume the objectives of each lesson session were the questions asked for homework.
Ownership - This metric is not possible to evaluate for the sample course.
Experiences - The distance learning tools used to present the lesson reviewed include course outline posting, course notes, and course reference materials. The course outline stated that video classroom and chat applications were utilized. However, these are not available in the sample lessons. No higher-level technologies (virtual classroom, whiteboarding) are used.
Significant use of asynchronous feedback is incorporated throughout the lesson sections. Students are provided email access to the instructor at several stages throughout the lesson, including the opportunity to submit assignments for review prior to submission for grading. However, no provisions for synchronous feedback are visible within the sample lessons.
Practice - Do online activities require students to practice the skills necessary to achieve the desired behavior? No. Students are asked simple questions whose answers come directly from the readings. Questions like “What is artificial twinning?” from the lesson section titled “What is a twin?” require the students to look up the answer in the readings but require no repetition of skill development or knowledge reinforcement, a requisite of practice.
Is a feedback mechanism provided to correct student mistakes during practice activities?
Yes. Asynchronous feedback is provided to the student before grading of questions. While this is not during practice activities, it is available to the student in time to correct mistakes made during the activities.
Reward - Are rewards provided to students that motivate them to perform the tasks necessary to achieve the desired behavior? The only rewards indicated in the sample course are the grades provided for the homework assignments.
Are the rewards delivered to the students in a timely manner after the desired behavior has been achieved? This information is unavailable in the sample course. According to the information provided on the Virtual High School website, student assignments are graded promptly.
Are the rewards withheld from students who do not achieve the desired behavior? This is difficult to determine in the sample lessons since behavioral objectives are not clearly stated. Based on grading policies discussed on the website, it can be safely assumed that students who do not successfully complete the assignments will receive a reduced grade.
Performance - This is the same as attainable objectives discussed previously,
Variety - Are a variety of experiences (based on a variety of distance learning tools) used to enhance student learning? No. The learning experiences are basically focused on distance reading, not utilizing all the technologies available (virtual classroom, whiteboarding, online testing)
Do the distance-learning methods implemented provide experiences appropriate to the Bloom Criterion category being designed or evaluated?
Since the questions asked suggest a low level Bloom criterion objective, the distance learning methods used are marginally appropriate. However, the lack of synchronous feedback and absence of advanced technology tools restricts the student to absorbing the information principally from reading. Internet based text reading is still reading and only one component of distance learning.
Outcomes - Are distance-learning tools selected and implemented to restrict the possible outcomes to those that achieve the desired behavior? No. The distance learning tools that provide students with immediate feedback are not used. Additionally, students are encouraged to use the asynchronous feedback facilities but not required to do so. Consequently, little in the way of control mechanisms are indicated in the sample course to direct student behavior.
If student activities result in undesired outcomes, is a feedback mechanism provided to correct students? Yes. Asynchronous feedback is provided as indicated above.
Organization - The individual weekly lesson and the lesson sections are being analyzed
Continuity - Are multiple opportunities provided for students to practice the skills necessary to achieve the desired behavior to be practiced? No. Throughout the lesson, individual questions are asked only once. However, while questions from subsequent sections do not reinforce previously learned knowledge, they do increase the amount of knowledge and build on the complexity of the knowledge from previous sections.
Are practice opportunities provided continuously throughout the course? Yes. Review questions are a component of each lesson section.
Sequence - Do successive experiences (activities) build upon the preceding one but go more broadly and deeply into the matters involved? Yes. See the explanation discussed in the Continuity section above.
Do successive experiences evolve to achieve the desired level of complexity within Bloom’s hierarchy? Yes, since only the lowest levels of complexity within Bloom’s hierarchy appear to be desired. However the question appears inappropriate when only knowledge and comprehension are being targeted.
Integration - Do the activities used to develop the skills necessary exemplify the relationship of the desired behavior to other subjects/fields? No. No indication of integration either within this course or with other courses was provided in the sample lesson evaluated.
Evaluations - While no evaluation instruments are available in the sample lesson, program documentation on the website discusses typical testing procedures consisting of several evaluations during the course of the class. The exact number of tests apparently varies depending on the course. Since no facility for online testing was either discussed or viewed in the sample course, it is assumed testing is either take home and open book or proctored. The following framework items are addressed assuming the testing is take-home, open book and emailed to the instructor.
Behaviorally targeted evaluation - Do the evaluations used specifically evaluate the intended behavior? Yes. Since the evaluation procedures for the sample lessons are not available, this can not be definitively determined. However, based on the discussions within the Virtual High School website and the low level of Bloom’s criterion targeted as the intended behavior, the assumption is made that the evaluations properly target the intended behavior.
Baseline evaluation - Is a baseline evaluation used to access students initial knowledge of the desired behavior? No. None is indicated within the course or Virtual High School information provided.
Successive evaluations - Are a series of evaluations used to assess student progress?
Yes. While none are available in the sample lesson, statements contained within the program description indicate that several evaluations are used throughout a course. Additionally, each lesson section requires the student to respond to a series of questions regarding the readings.
Appropriate evaluation type - Are the skills and knowledge necessary to successfully complete the evaluations representative of the skills and knowledge necessary to achieve the intended behavior? Yes. Again, this is based on the same assumptions stated above under Behaviorally Targeted Evaluation.
Appropriate Bloom criterion level - Are the evaluations appropriately structured to reflect mastery of the behavior at the desired Bloom criterion level? Yes. While none are available in the sample lesson, statements contained within the program description indicate that appropriate evaluations are used throughout a course.
Applying the Framework Metric
Applying the point value to the number of questions answered yes gives the following scores for each of the four sections:
A discussion of this result is discussed in the Conclusions section of this paper.
Conclusions and Recommendations
Several conclusions can be drawn about the framework based on this analysis. First, the framework provides an in-depth means of evaluating distance-learning curriculum components, verifying compatibility with Tyler’s principles and determining Bloom’s criterion level. While the purpose of this discussion is neither to verify nor vilify Tyler’s principles, applying these long established principles to the developing distance-learning curriculum provides both a means of evaluating the completeness of a distance-learning curriculum unit and a basis for comparing distance-learning curriculum units with their traditional counterparts. Obviously a significant number of additional evaluations using the framework are needed to substantiate its validity and reliability as an instrument for distance-learning evaluation, but the evaluation discussed herein supports its functionality with a minor modification to the evaluation scale.
Second, utilizing a purely binary evaluation measurement (Yes/No point assignment) appears to be inappropriate for many questions asked in the framework. For example, the framework question "Are rewards provided to students that motivate them to perform the tasks necessary to achieve the desired behavior?" can be answered in the affirmative if the only reward provided is the grade or if other reward mechanisms are incorporated. The question "Are a variety of experiences used to enhance student learning?" is not specific as to what constitutes a variety. Are two experiences a variety, does it require five, or is it situational? The Continuity question asking "Are multiple opportunities provided for students to practice the skills necessary to achieve the desired behavior to be practiced?" poses the same enigma. Do to two practice opportunities constitute multiple opportunities or does it require more? Rewording the questions and using a Likert scale metric, while being more subjective for the evaluator, can better differentiate between levels of conformance with the framework questions thus providing a more accurate assessment of the curriculum element being evaluated. The Continuity question can be reworded using a Likert scale as follows:
How many opportunities are provided for students to practice the skills necessary to achieve the desired behavior to be practiced?
1: 0 2: 1 3: 2-3 4: 4-5 5: more than 5
This would allow the evaluator to rate programs with higher practice requirements at higher levels, giving a more appropriate comparison among curriculum elements being evaluated.
Third, complexity of evaluations using the framework will increase at higher levels of Bloom’s classifications of learning objectives. The framework is designed to be applied to each classification level contained within the curriculum unit being evaluated. Since higher level classifications, those requiring synthesis or evaluation, usually contain some lower lever classifications as part of a developmental foundation, each subordinate level contained within the curriculum unit will need to be analyzed using the framework. Absent this hierarchical evaluation, there is no way of insuring the proper foundation is developed to achieve the high-level objectives. While this is not viewed by the author as a problem, it is a schedule and time-management issue that potential evaluators using the framework will need to address.
Conclusions regarding the lesson and lesson segments are more difficult to determine. Although the 59 point score of the evaluation appears low, no other courses or lessons were evaluated to establish a baseline for comparison. Drawing conclusions regarding the lesson segments is further complicated because of the restricted access to course components for sample lessons. As indicated during the discussion of the application of the framework, numerous assumptions were necessary to complete the evaluation because of the lack of access.
Based on the evaluation of the lesson and lesson segments using the framework, the following points were observed:
- Based on the framework analysis, the lesson analyzed earned a poor evaluation. The curriculum analysis using an objective-based framework was difficult because the lesson contained no objectives. The lack of defined objectives that fit the established criteria resulted in a fifteen point deduction, which is significant. The lack of defined objectives also causes difficulty in establishing evaluations to measure the attainment of the objectives.
- The lesson fails to completely utilize the available web-based distance-learning tools, restricting the variety of experiences students are exposed to resulting in a decreased score in the experiences category. The principle tool used is text posted for students to read. While chat and streaming video are discussed in the program description, none were observable in the sample lessons.
- No provisions for synchronous feedback were observed in the sample lessons. Using synchronous feedback for online courses simulates the feedback available to students in face-to-face courses creating a learning experience more typical of what students are used to.
- Online testing is not used. Online testing can give students immediate feedback regarding their attainment of the learning objectives. Student assignments are emailed when completed. This also results in delayed feedback on student performance.
- The lesson focuses on low level Bloom criterion. Utilizing more advanced web technologies would enable restructuring the lesson to address mid and high level Bloom criterion.
Proposed Continued Study
Phase Two of this project will consist of framework validation. While the functionality of the framework is established by the study herein, the effectiveness of the framework as a tool for evaluation of distance learning courses has not been validated. To validate the framework, additional evaluations using the previously-discussed Likert-based evaluation scale are recommended. Numerous universities offer distance learning courses. Because of the significantly greater number of courses available at the university level, university courses will be used for future studies. Proposed framework validation will be based on six courses or lessons, one each from each of Bloom’s criterion levels, using the revised metric.
Another appropriate area of study would be an analysis of the effectiveness of the various distance-learning tools and delivery methods in achieving learning objectives at the different Bloom criterion levels.
(2002) Bloom's Taxonomy (On-line). Available:http://www.coun.uvic.ca/learn/program/hndouts/bloom.html.
(2002) Distance Learning (On-line). Available:http://distancelearn.about.com/mbody.htm.
(2002) DLRN's Technology Resource Guide (On-line). Available:http://:www.dlrn.org/lybrary/dl/guide4.html.
(2002) Objectives in an Outcomes Focused Learning Environment (On-line). Available:http://www.eddept.wa.edu.au/cip/learntech/res/res2.htm.
(2002) Preparing for Clinical Teaching (On-line). Available:http://www.utexas.edu/phamarcy/general/experimental/practitioner/effective/prep.html.
(2002) Ralph Tyler and the Educationa Theory of the Day (On-line). Available:http://www.ittheory.com/tyler.htm.
(2002). Behavioural Objectives and Systematic Instruction. Psychology of Learning: improving pupil performance .
(2002). Distance Learning Master's Degree Significantly Improves Teaching Effectiveness. Business Wire 1-3.
Abel, M. , & Foletta, G. (2002). Kentucky Educators First to Web with Middle School and High School Online Learning. Mathematics Teacher 396-397.
Accetta, R. (2002). Clearing the fog. e-learning 30-32.
Blanchard, K., & Johnson, S. (1984). The One Minute Manager. New York: Berkley.
Blanchard, K., & Lorber R. (1984). Putting the One Minute Manager to Work. New York: Willian Morrow and Company.
Blanchard, K., Zigarmi, P., & Zigarmi, D. (1985). Leadership and the One Minute Manager. New York: Willian Morrow and Company.
Bloom, B. (1953). Taxonomy of Educational Objectives: The Classification of Educational Goals. New York: McKay.
Bloom, B. S. (1956). Taxonomy of Educational Objectives Handbook I: Cognitive Domain.. New York: McKay.
Boehle, S. , Dobbs, K. & Stamps, D. (2000). Two views of Distance Learning.. Training 34.
Bonner (1999). Choosing teaching methods based on learning objectives: An integrative framework. Issues in Accounting Education 11-39.
Cannon, M., Umble, K., Steckler A., & Shay S. (2001). "We're Living What We're Learning": Student Perspectives in Distance Learning Degree and Certificate Programs in Public Health. Journal of Public Heath and Management and Practice 49-59.
Cutshall, S. (2002). Going the Distance. Techniques 22-23.
Earls, A. (1999). E-Colleges: New England Institutions Marry Traditional Strengths to Cybertools. Connection: New England's Journal of Higher Education & Economic Development 28-32.
Eaton, J. (2000). Focus: Assuring Quality in Distance Learning. The Presidency.
Faux, T., & Hughes, C. (2000). A Comparison of Using the Internet Versus Lectures to Teach Social Work History. Research on Social Work Practice 454-466.
Ferguson, K. & Wijekumar (2000). Effective Design & Use of Web-Based Distance Learning Environments. American Society of Safety Engineers 28-32.
Grifith, V. (2001). Successful lessons from afar: Distance Learning. Financial Times 1-3.
Gunderson, L. (2000). Distance Learning Mushrooms at PCC the popular Online Offerings are Part of a Nationwide Trend Despite Criticisms Concerning Quality, Effectiveness and Motives. The Oregonian 1-3.
Hazari, S. (2002) Online Testing Methods for Web Courses (On-line). Available: http://sunil.umd.edu/documents/assmnt/onlinetest.htm.
Hofmann, D. W. (2002). Internet-Based Distance Learning in Higher Education. Techdirections 28-32.
Hong Kong, Kinshuk, Xiaoqin He, Ashok Patel, & Chris Jesshope (2001). Trends in Web-Based Learning Environment. Distance Education Report .
Jackson, G. (1990). Evaluating Learning Technology. Journal of Higher Education.
Kaczmarczyk, L. (2001). Accreditation and student assessment in distance education; Why we all need to pay attention. ITiCSE 2001; proceedings of the 6th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, June 25-27, 2001, Canterb 113-116.
Kagima, L. , & Housafus, C. (2001). Faculty central element in instructional technology integration. Journal of Family and Consumer Sciences 33-36.
Karr, S. (2002). Striving for Excellence. Financial Executive 69-71.
Kehoe, J. (2002) How to write Multiple Choice Questions (On-line). Available:http://www.robmccormack.com/qphelp/pqhowto5.htm.
Kizlik, R. (1999). Generation X wants to teach. International Journal of Instructional Media 193.
Kodama, M. (2001). Distance learning using video terminals. International Journal of information Management 227-243.
Kreitzberg, C. (1998). Distance Learning Isn't going the distance. IntenetWeek 1-2.
Lane, C. (2002) Bloom's Taxonomy of Learning (On-line). Available:http://www.uis.edu/~jenkins/Bloom.htm.
Loeding, B., & Wynn, M. (1999). Distance Learning planning, preparation, and presentation: Instructors' perspectives. International Journal of Instructional Media 181.
Moorhead, G., & Griffin, R. (1998). Organizational Behavior. Boston: Houghton Mifflin.
Mulqueen, W. (2002). Technology in The Classroom: Lessons Learned Through Professional Development. Technology in the Classroom 248-255.
Neo, L., & Eng. C. (2001). Getting It right: Enhancing On-line Learning for Higher Education using the Learner-driven Approach. Singapore Management Review 61-74.
Olosky, D., & Smith, B. (1978). Curriculum Development Issues and Insights. Chicago: Rand McNally.
Rice, M., & Miller, M. (2001). Faculty Involvement in Planning of the Use and Integration of Instructional and Administrative Technologies. Journal of Research on Computing in Education 328-336.
Sanders, D., & Morrison-Sheltlar, A. (2001). Student attitudes toward Web-enhanced instruction in an introductory biology course. Journal of research on Computing in Education 251-262.
Schermerhorn, J., Hunt, J., & Osborn, R. (2000). Organizational Behavior. New York: Prentice Hall.
Shea, T., Motiwalla, L., & Lewis, D. (2001). Internet-based distance education--the administrator's perspective. Journal of education for business 112-117.
Sonner, B. (1999). Success in the Capstone Business Course - Assessing the Effectiveness of Distance Learning. Journal of Education for Business .
Stallard, J., & Ray, C. (2001). Reflections on two careers in higher education. Information Technology, Learning, and Performance Journal 1-3.
Symonds, W. (2001). Giving It the Old Online Try. Business Week .
Tyler, R. (1949). Basic Principles of Curriculum and Instruction. Chicago: University of Chicago.
Wardrope, W. (2001). A communication-based response to distance learning in business communication. Business Communication Quarterly .
Wilson, J. (2002). The Power of Distance Learning Guest Editorial. Personal Experience with Distance Learning 638; 673.
Woit, D., & Manson, D. (2002) Enhancing Student Learning through On-line Quizzes (On-line). Available:http://www.sarg.ryerson.ca/sarg/papers/200003-sigce00/index.phtml/sarg.html.
Wright, V., Wright, K., & Barker, S. (2001). Considering teaching via the internet? Some practical tips to consider.. Athletic Therapy Today 25-28.
Ypsilandis, G. (2002). Feedback in distance education. Computer Assisted Language Learning 167-181.
Zolkos, R. (1999). Online education getting good grades. Business Insurance 40B.
Russell K. Baker is an Assistant Professor of Information and Technology Management at the University of Tampa, 401 W. Kennedy Blvd., Tampa, Florida 33606. Email: firstname.lastname@example.org
Online Journal of Distance Learning Administration, Volume VI, NumberII, Summer 2003
State University of West Georgia, Distance Education Center
Back to the Online Journal of Distance Learning Administration Contents