Closing the Loop on a Continuous Program Improvement Process


 

Vickie Booth
Georgia WebBSIT
Vickie@GAWebBSIT.org

Larry Booth
Clayton State University
LarryBooth@clayton.edu

 

Abstract

The WebBSIT, a Bachelor of Science in Information Technology, is a fully online degree offered through a consortium of five University System of Georgia institutions. This paper begins by summarizing the change management system developed for continuous program improvement. Analysis of data should drive improvement, closing the loop. The balance of this paper presents an outline for stakeholder participation, describing critical checkpoints in the process that must occur to close the loop on closing the loop.

Background of the Georgia WebBSIT

The Georgia Bachelor of Science in Information Technology (BSIT) degree program (WebBSIT, 2010) is offered collaboratively by five University System of Georgia (USG) institutions: Armstrong Atlantic State University; Clayton State University; Columbus State University; Georgia Southern University; and Southern Polytechnic State University.

The degree requires that students be admitted to one of the five collaborating institutions. The WebBSIT offers the lower division Information Technology core curriculum (18 hours) and all upper division courses (51 hours) entirely online. The program assumes that students have completed most of their general education courses before beginning.

Curriculum and Course Development

The WebBSITs focus was the development of an integrated curriculum rather than a set of discrete courses. The BSIT curriculum is built on nine core program outcomes. Each individual course addresses a subset of these program outcomes. Outcomes are mapped to courses using Blooms taxonomy of the cognitive domain (Bloom, 1956). Students are expected to demonstrate competency in each outcome at some level of mastery: developing, mature, or proficient.

Program outcomes are broad statements about the skills students should acquire as they move through the curriculum. Each course has more specific objectives designed to address the topics of the course. Course Architects create one or more course outcomes to support each program outcome that has been mapped to a course. Writing concise course outcomes is difficult. To make the job easier, over the years sets of action verbs (Rothwell Kazanas, 2008) have been associated with each level in Blooms Taxonomy. The WebBSIT used these action verb sets to develop a tool to assist Course Architects in the writing of course outcomes. See Appendix A.

Initially, WebBSIT courses were developed and offered using WebCT Vista version 3. In the summer of 2009, courses were migrated to WebCT/Blackboard Vista version 8. Vista 8 provides a Goals tool that allows program outcomes (goals) to be recorded. The Goals tool can record program outcomes and associated course objectives (course outcomes). A sample from one course has been provided. See Appendix B. Within this structure, content files, assessments and assignments can be associated with one or more goals. As part of the migration to Vista 8, Course Architects embedded the program outcome-course outcome-assessment hierarchy into each course. As a result, assessment data can now be collected and used to evaluate the curriculum. A sample from one course has been provided. See Appendix C.

Change Management System

The WebBSIT is using roles and business rules to enforce a change management system for the collaborative (Booth, Booth, Hartfield, 2009). The WebBSIT Operating Board is responsible for oversight of the curriculum, modification of program outcomes, and approval of course learning objectives that support program outcomes.

The Executive Directors role is that of project manager for course development. The Instructional Designer helps to ensure that courses map properly to online pedagogy. The Course Section Instructor role is that of content expert with online teaching expertise.

The Course Architect collaborates with the Operating Board and Executive Director to implement continuous course improvement. The Course Architect incorporates feedback from a variety of sources to initiate change. Minor updates and improvements to courses (new or updated content modules, assignments, or assessments) are the purview of the Course Architect. Changes that impact course outcomes or program outcomes must be referred to the Operating Board.

Critical Checkpoints

In any system where improvement depends on critical analysis of data, checkpoints should be designed into the process. A checkpoint serves a quality control function where the activities of various roles intersect. New perspectives are brought to bear and differences worked out.

A checkpoint is a decision nexus. Data about past performance is analyzed and decisions about change for the future are made. In a change management system, checkpoints document change. In most education environments, organizational boundaries where oversight normally occurs provide natural checkpoints. For example, individual teaching faculty evaluate student performance in courses and initiate instructional change. At the departmental level, individual courses are evaluated in light of the curriculum as a whole. At the college or school level, departmental performance is evaluated from the perspective of the college or schools contribution to the community and university.

From the standpoint of a change management system designed to improve curriculum and instruction, work must occur at several critical checkpoints.

Work at the College Level

Develop college outcomes in line with university outcomes.

Establish goals, objectives, and guidelines; an overall plan for achieving college outcomes.

Develop a feedback loop that takes into account departmental performance, faculty feedback, and departmental evaluation of program effectiveness. Revise college outcomes, goals, objectives, and guidelines as necessary. Changes should be documented so that the college can verify continuous improvement.

In the WebBSIT work at the college level is coordinated by the Governing Board. The Governing board members represent the Information Technology Deans of the several participating institutions. At this level, data from the WebBSIT program is evaluated based on the individual standards of each participating institution. Measures include: number of graduates, number of majors, number of enrollments, placement of graduates, retention and attrition rates, faculty-to-student ratios, and faculty work load.

Work at the Department Level

Develop program outcomes in line with college outcomes.

Map program outcomes to courses. Establish acceptable performance criteria.

For each course, develop core course objectives that support each program outcome mapped to the course. This is a departmental level, top down design, exercise because courses and their prerequisites flow together to create the curriculum as a whole. Courses do not exist in isolation.

Develop a feedback loop that takes into account course performance data, student feedback, and faculty evaluations of courses. Revise program outcomes, course outcomes, and acceptable performance criteria as necessary. Changes should be documented so that the department can verify continuous program improvement.

In the WebBSIT, work at the department level is coordinated by the Operating Board. A spreadsheet tool was developed to collate data collected across all courses and sections to provide an overall picture of student success in meeting program outcomes. Measures include: For each program outcome a comparison of student performance with established performance criteria, student evaluations of courses, section instructor evaluation of courses, Course Architect recommendations.

Work at the Faculty Level

Develop additional course objectives. This secondary set of course objectives encourages bottom-up evolution of the curriculum. For both core course objectives and secondary course objectives, develop instructional components designed to teach course objectives.

Develop assessments and corresponding rubrics for each course objective.

Create a spreadsheet for recording student performance based on assessments and rubrics. Note: be as discrete as possible. For example, if a test covers two or more objectives, the spreadsheet elements for recording the test should have a column for each objective. This is the most difficult part of the task because it will likely require each faculty member to critically evaluate their assessment procedures. Also, some assessments may not lend themselves well to discrete analysis.

A separate section of the grading spreadsheet should contain roll-up formulas that summarize overall student performance that can be compared to established performance criteria. While each faculty member may have individual and creative instructional components, assessments, and rubrics, the roll-up should be standardized so that departmental summaries of course objectives and program outcomes are easy to achieve.

Develop a feedback loop that takes into account student performance, student feedback, and peer evaluations of teaching effectiveness. Revise instructional components, assessments, and rubrics as necessary. Changes should be documented so that faculty can verify continuous course improvement. Proposed changes to program outcomes and/or core course objectives should be submitted to the Department for consideration by the faculty.

In the WebBSIT, work at the faculty level is coordinated by the Course Architect. The Course Architect relies on feedback from students, and section instructors. Constant Contact was used to create a survey tool to document student feedback on courses and instruction (See Appendix D). Section instructors embed notes to the Course Architect in the course itself. A spreadsheet tool was developed to help Course Architects extract and assemble assessment data for each course objective (See Appendix C).

Conclusions

It’s all well and good to have in place a continuous improvement plan. Ideally, such a plan would include checkpoints where changes supported by data are routinely recorded. Faculty are generally evaluated on teaching effectiveness yearly. Typical measures include: student grade analysis, student feedback via end-of-semester evaluations of courses and instruction, peer evaluations of teaching, and so on. A more effective measure would be an examination of what teaching faculty do with this data to improve instruction and courses. If faculty recorded the changes made to each course along with the supporting data, a yearly record of continuous improvement would accrue.

Significant changes to curriculum are generally approved by departmental, college, or school curriculum committees. Usually, a rationale for change is included in any proposal. Such rationale may be driven by outside influences such as accrediting bodies, professional standards, or community requirements. But when applicable, changes driven by analysis of data collected should be included. Again, a yearly record of continuous improvement would accrue. University-wide curriculum committees could have a positive impact on the collection of relevant data by simply requiring analysis of collected data in support of any proposal.

Closing the loop on closing the loop requires concerted effort to collect and analyze relevant data at pre-defined checkpoints. Just saying that continuous improvement is observed is not enough. Checkpoints create the habit of collecting and then using data to make informed decisions about the evolution of courses and curriculum. Checkpoints provide the documentation necessary to verify the effectiveness of continuous program improvement.

References


Bloom, B. S. (1956). Taxonomy of Educational Objectives Handbook 1: Cognitive domain. New York: Longman, Green Company.

Booth, L., Booth, V., Hartfield, F. (2009). Continuous course improvement, enhancements, modifications: Control Tracking. Online Journal Distance Learning Administration(Summer 2009).

Rothwell, W. J., Kazanas, H. C. (2008). Mastering the instructional design process: A systematic approach (4 ed.). San Francisco: John Wiley Sons.

WebBSIT. (2010). Home Page. Retrieved February 12, 2010, from http://www.webbsit.org/

Appendix


Appendix A: Writing Course Objectives

Developing Level of Mastery: Demonstrates an emerging level of knowledge and skills; can perform beginning skills and shows potential to perform independently.


Mature Level of Mastery: Demonstrates a refined level of comprehension; is able to apply appropriate skills and perform both independently and as a team member.


Proficient Level of Mastery: Demonstrates a superior level of knowledge and understanding; integrates and applies skills across multiple areas both independently and as a team member.

Sample Action Verbs to Use When Writing Measurable Course Objectives
Adapted rom Rothwell and Kazanas (2008, page 181)

Developing

Mature

Proficient

Knowledge

Comprehension

Application

Analysis

Synthesis

Evaluation

define

translate

interpret

distinguish

compose

judge

repeat

restate

apply

analyze

plan

appraise

record

discuss

employ

differentiate

propose

evaluate

list

describe

use

appraise

design

rate

label 

recognize

demonstrate

calculate

formulate

predict

 match

explain

dramatize

experiment

arrange

value

 memorize

indicate

practice

test

collect

revise

 name

identify

illustrate

compare

construct

score

 order

review 

operate

contrast

create

select

 recall

 sort

schedule

criticize

set up

choose

 reproduce

 classify

solve

diagram

organize

assess

 

 

sketch

inspect

manage

estimate

 

 

choose 

categorize

prepare

argue

 

 

schedule 

inventory

assemble 

assess 

 

 

 

question

 

 

Examples:

Developing
-Students will recognize and describe database design methodologies.
-Students will identify and explain database concepts.

Mature
-Students will use and apply web skills to plan a website.
-Students will demonstrate mature writing skills to produce written reports.
-Students will compare and contrast the concepts of query optimization.

Proficient
-Students will appraise and evaluate the issues and problems of multi-user databases.
-Students will assess the fundamental concepts of concurrency control.
-Students will formulate basic techniques for database security

Appendix B: Program Outcome-Course Outcome-Assessment Hierarchy


WebBSIT Program Outcome #5

Appendix C: Program Outcome-Course Outcome-Assessment Data



rse Outcome Data

Appendix D: Sample Survey-Student Opinion of Course and Instruction



Online Journal of Distance Learning Administration, Volume XIII, Number II, Summer 2010
University of West Georgia, Distance Education Center
Back to the Online Journal of Distance Learning Administration Contents