EVALUATING THE IMPACT OF A CHARACTER EDUCATION CURRICULUM
Cletus R. Bulach
Dept. of Educational Leadership
College of Education
State University of West Georgia
Carrollton, GA 30118
Paper presented at the Character Education Partnership Conference in Philadelphia, PA on 10-20-2000.
Dr. Bulach is an associate professor at the State University of West
Georgia and he is also the project evaluator for the Georgia Department
of Education Character Education grant
EVALUATING THE IMPACT OF A CHARACTER EDUCATION CURRICULUM
A B S T R A C T
A comprehensive research design is presented
that evaluates behaviors associated with sixteen character traits, the
climate/culture of the school, and how effective the character education
program is implemented. The design also incorporates qualitative data from
parents and students about the character education program. Instrumentation
for data collection purposes are described.
Character education programs are being implemented in many schools across the United States. Some schools have a required curriculum that is taught for 15 minutes during the day. Often the curriculum is focused on a specific character trait for the week or month. In other districts character education is part of the health curriculum, the social study's curriculum, or the responsibility of the counselor. Still others follow Lickona's Eleven Principles of Effective Character Education and infuse it throughout the school day. Further, they involve all faculty, staff, parents, and the community in the character education program. In short, there is a plethora of approaches to character education.
There are also a number of approaches to measuring the effect of a character education program. The most common approach is for school officials to collect student performance data on attendance, discipline, and academic performance. Currently, however, there is little research data that measures changes in behaviors associated with specific character traits. There is also little data on how effectively a school district has implemented a character education program.
An effective research design to measure the impact
of a character education curriculum would be very helpful. Certainly, some
character education programs should be more effective than others. Identifying
those that are more effective would be helpful to school districts in their
dissemination efforts. On the other hand, I also believe that many school
officials and teachers are spinning their wheels implementing ineffective
character education programs. The purpose of this article is
to introduce a research design that would identify effective character
education programs for dissemination and ineffective programs for elimination.
In addition to describing a research design, instrumentation will be presented
to collect quantitative and qualitative data.
The Research Design
The research design requires the collection
of data from students, teachers, and parents. Three different types of
data should be collected from students as follows:
If the character education program is effective, there should be improved behavior related to character traits, improved student performance, and the number of "likes" should be greater and the "dislikes" should be less.
Two different types of data should be collected
from teachers as follows:
If student behavior improves as a result of the character education program, the climate/culture of the school should improve. For example, if students are more respectful, dependable, responsible, honest, etc., the impact on the overall school climate should be an improvement. On the other hand, if the pre-data indicates a poor climate there may be a need to improve the climate before implementing the character education curriculum. Bulach and Malone (1994) found a positive correlation between the successful implementation of change and school climate. Trying to implement a character education program in schools with a poor climate could be a mistake. Regarding teacher perceptions of the implementation of the character education program, the pre-data should be used to identify the strengths and weaknesses of the program. This data could be used to improve the implementation process resulting in improved post-data scores.
Early in the development of this research design, there was some thought that parental data on the same 100 behaviors used to measure students would be desirable. However, on closer analysis, this idea was abandoned because it was believed that the data would not be valid. For example, very few parents would admit that their child cheated, abused drugs, lies, steals, etc. Instead it was decided that we would survey parents regarding what they liked and did not like about the school's character education program. Again, if the character education program is effective, the post-data measure should contain more "likes" and fewer "dislikes."
The character instrument (Bulach, 2000) consists of 99 behaviors for elementary students and 100 for middle and high school students. The extra behavior for the upper grades is a sexually oriented item. Students respond to each of the items on a five-point Likert scale ranging from "never" to "always." "Never" was scored as a "one" and "always" was scored as a "five." Thirty-eight of the items were stated negatively and had to be reverse scored. For example, one of the behaviors is: "Students think it is okay to do something as long as they don't get caught." The instructions tell them to choose the response which comes closest to the behavior that describes what you think other students do or think. The 16 character traits measured by the instrument are as follows: (1) respect for self, others and property, (2) honesty, (3) self-control/discipline, (4) responsibility/dependability/accountability, (5) cooperation, (6) integrity/fairness, (7) kindness, (8) perseverance/diligence/motivation, (9) compassion/empathy, (10) courtesy/politeness, (11) forgiveness, (12) patriotism/ citizenship, (13) tolerance of diversity, (14) humility, (15) generosity/charity, and (16) sportsmanship. For a graphical illustration of these traits, based on a sample school profile, see Appendix A. Scores approaching 4.0 indicate student agreement that behaviors associated with this trait are present at the school. Scores that approach 3.0 indicate some disagreement that they are present.
A Cronbach alpha of .96 indicates the instrument has internal consistency/reliability. The instrument has construct validity for those behaviors that were identified as indicative of a character value. While the instrument can be used to have teachers evaluate behaviors, it is believed that students are a more reliable and valid measure of what goes on in bathrooms, hallways, buses, etc. In related research, (Bulach and Butler, accepted--pending revisions) teachers tended to view students more positively than students saw themselves, although there was very little difference in their perceptions. Consequently, it is best to use the instrument to measure student perceptions of other students' behavior instead of teacher perceptions.
The Instructional Improvement Survey was used to measure climate and culture. The openness and trust behaviors in this instrument are modifications of the Group Openness and Trust Scale (Bulach, 1993). As a way of understanding the concepts of culture and climate, the analogy of an iceberg can be used. Climate is the part of an iceberg that is easily seen above the water and culture is the part of the iceberg below the water. You can't see it, and you do not know that it is there. However, the climate (the part that is observable) cannot exist without the culture (the part that cannot be seen). The culture variables are group openness, group trust, group cooperation, and group atmosphere, i.e., it is difficult to tell if people are receptive, trust, work together, or care about each other. The climate variables are sense of mission, parent involvement, teaching, discipline, leadership, assessment/time on task, and expectations, i.e., you can see if students obey rules; if the principal leads; how teachers teach, etc.
Reliability data on the Instructional Improvement Survey using the Cronbach alpha or measure of rationale equivalence was .92 for the eleven subscales. For a graphical illustration of the eleven climate/culture factors, based on a sample school profile, see Appendix B. The graph represents teacher responses from 18 different elementary schools. As can be seen from results, there is a need to improve the climate/culture of these schools.
The graph allows school officials to determine areas of strengths and weaknesses. Scores above 32 indicate areas of strength and those below, indicate areas which might be considered for improvement. Each area or subscale is measured by eight (8) behaviors. An agree response is scored as a four (4), which would produce a score of 32 (4 X 8 = 32). A disagree response is scored as a two (2) which would produce a score of 16 (2 X 8 = 16). A completely agree response is scored as a five (5) producing a score of 40. The areas of openness and trust have more than seven (7) behaviors, but they have been mathematically controlled to be equal with the other subscales.
The survey for qualitative data from students
and parents would be gathered by giving them two sentences to complete
as many times as they wish. The student sentences would be as follows:
- I like the students at this school because . . .
- I would like the students at this school better if . . .
The parent sentences would be the following:
- I like the character education program at Blank School because . . .
- I would like the character education program at Blank School better if . . .
This type of survey generates a wealth of rich data that can be very beneficial to school officials in tweaking their character education curriculum. Student and parent responses should be categorized according to types of responses for better analysis. For example, comments that are student related, teacher related, administration related, manners related, discipline related, etc. When the post-data are collected, the number of "like" and "like better" comments can be compared to see if there are more "like" and fewer "like better" comments. Further comparisons can be made according to the different categories of behavior. For example, there may be no difference in the overall responses, but there may be a difference in the "manners related" or some other category.
The Eleven Principles of Effective Character Education instrument was adapted from a document developed by Thomas Lickona. It consists of 45 practices grouped by 11 principles. For example, principle number seven is: "Character education should strive to develop students' intrinsic education." According to Lickona, if these 11 principles are followed, school officials will have an effective character education program. Teachers respond to each of the behaviors according to a five-point Likert scale ranging from strongly disagree to strongly agree.
A factor analysis of the instrument revealed that there are three factors measured by the instrument as follows: school community relations' behaviors, curriculum related behaviors, and internal relations' behaviors. Reliability of the instrument was +.90, as measured by the Cronbach alpha. The reliability of the school community relations' factor was +.88, the curriculum factor was +.91, and the internal relations' factor was +.93. There is little data on the construct validity of this instrument. However, it does provide school officials with a wealth of information about practices that could help or hinder the success of their character education program. For example, one of the eleven behaviors in the school community relations' factor is as follows: "Our school sends home communications and suggestions that help parents reinforce the same character qualities the school is trying to teach." One of the seventeen behaviors under the curriculum factor is as follows: "Our program is comprehensive across the curriculum; the character traits are regularly integrated into instruction in all subjects and at all grade levels." One of the seventeen behaviors in the internal relations factor is as follows: "We help students acquire a developmentally appropriate understanding of what the character traits mean in everyday behavior and to grasp the reasons why some behaviors are right and others are wrong."
Data from the 45 behaviors are scored according to the three factors and a graph (see Appendix D) allow school officials to see how effective they are implementing their character education program. Scores of 4.0 and above indicate agreement that the behaviors in those factors are being implemented. Scores below indicate areas needing improvement. The same procedures are used as with the climate data to investigate those behaviors that have caused a factor to be low or high. The data should be used to develop an improvement plan that more effectively implements the character education program. The scores at Sample Elementary School are typical of most character education programs, i.e., they are weakest in their efforts to involve the community.
In theory, if a character education program
is effectively implemented, student behaviors associated with the 16 character
traits should improve. This should result in an improvement in school
climate. Furthermore, since there is a positive relationship between school
climate and student achievement (Bulach, Malone, and Castleman, 1995),
student achievement should also improve. This is an untested theory, but
the instrumentation described in the foregoing provide the framework for
testing this theory. For those practitioners who are more interested in
results and not theory, the instrumentation provides a wealth of data that
can be used to develop school improvement plans targeted towards raising
student test scores.
Bulach, C. R., & Butler, J. (accepted--pending revision). A comparison of character values as perceived by teachers and students at differing grade levels. Journal of Curriculum and Supervision.
Bulach, C. R. (2000). Do moral citizens have character? A presentation to The Association for Moral Education at Glasgow, Scotland, on 7-10-2000.
Bulach, C. R., Malone, B., & Castleman, C. (1995). An investigation of variables related to student achievement. Mid-Western Educational Researcher, 8(2), 23-29.
Bulach, C. R., & Malone, B. (1994). The
relationship of school climate to the implementation of school reform.
SPECTRUM: Journal of School Research and Information 12(4), 3-9.
You are visitor number
This Web page is designed in January, 2001.
The copyrights are reserved.