Assessing Asynchronous Discussions: An Exploratory Hybrid Model



Camilla Gant, Ph.D.
Director & Associate Professor
Mass Communications
University of West Georgia

 

Abstract

Given the increasing import of asynchronous discussions to achieving learning objectives, this analysis proposes an exploratory assessment model that involves three levels of analysis, i.e., argument structure, disagreement, and interaction. The model promotes objective, systematic assessment; streamlines the assessment process; and clearly defines expectations for students, which realizes more meaningful and focused discussions.

Asynchronous Discussions

Computer mediated communication has redefined the dynamics of traditional class discussions. It has shifted learning in the classroom from an objectivist model in which learning flows from teacher to students, to a constructivist model in which learning is collaborative and the teacher facilitates the learning process (Coombs, 1993; Hazari, 2004). As a result of this shift, asynchronous discussions are increasingly becoming a critical component of web-enhanced, online, and hybrid (face-to-face/online) courses, and they are credited with fostering active and collaborative learning as well as higher order critical thinking (Edelstein & Edwards, 2002; Hazari, 2004; Markel, 2001). While scholars generally agree that asynchronous discussions are integrally linked to achieving learning objectives, a search of select distance learning and journalism/communication journals relevant to this analysis yielded few studies concerned with assessing asynchronous discussions.

Analytic & Holistic Assessment

According to Edelstein and Edwards (2002), assessment begins with defining the level and quality of participation that is expected of students. Assessment criteria, they argue, should involve objective scoring of criteria that reflects areas linked to the achievement of learning objectives. Drawing on the work of Popham (2002), Hazari (2004) identifies two common approaches to objective scoring-- analytic scoring and holistic scoring. Analytic scoring assigns points to individual criteria; whereas holistic scoring is more impressionistic, and criteria are scored as a whole without assigning points to each criterion. Perhaps Spatariu, Hartley, & Bendixen (2004) offer the most comprehensive literature review of analytic and holistic approaches to scoring online discussions. Based on a purposeful sample of literature, they classify approaches according to four coding systems-- levels of disagreement, argument structure, interaction, and content analysis (which is excluded in the subsequent review given its peripheral relevance to this analysis).

Levels of Disagreement Analysis

In their discussion of analysis concerned with levels of disagreement, Spatariu et al. (2004) center on a coding system used by Nussbaum, Hartley, Sinatra, Reynolds, and Bendixen (2002). This holistic coding system scores messages based on a 4-point scale in which a score of 1 reflects that a message agrees with a previous one and offers no new information; a score of 2 reflects that a message agrees with a previous one and offers some new information; a score of 3 reflects that a message offers a qualified disagreement; and a score of 4 reflects that a message exhibits outright disagreement. A limitation of this coding system, however, is that it does not assess the appropriateness of content or support for positions (Spatariu et al).

Argument Structure Analysis

Coding systems that focus on argument structure analysis (argument quality) are concerned with assessing a student’s point of view or contribution to discussions. Spatariu et al. (2004) reviewed two coding systems within this approach that are significant to this analysis. Drawing on the work of Inch & Warnick (2002), Spatariu et al. discussed a general model, which appears adaptable to analytic or holistic coding, and scores messages based on complexity of argument structure in terms of the number and nature of premises and claims. For example, the model defines Type I arguments as those that consist of one premise and one claim; Type II arguments as those that consist of one claim and multiple premises; Type III arguments as those that consist of proven claims as evidence for unproven claims; and Type IV arguments as complex arguments that consist of multiple premises and multiple claims.

The primary challenge of scoring discussions based on this model, however, is differentiating between premises and claims (Spatariu et al., 2004). In other words, distinguishing between verifiable, least arguable statements and expressed opinions and conclusions, which Inch & Warnick (2002) respectively define as a premises and claims. Also, Spatariu et al. note that a drawback of the general model is that it does not assess unstated inferences and assumptions of an argument.

In reviewing Bendixen, Hartley, Sas, and Spatariu’s (2003) coding system, Spatariu et al. (2004) note that it is a hybrid approach to argument structure analysis. First, messages are assigned an analytic score based on nature of evidence, i.e., Positive Evidence, which consists of established, supported facts, and/or causal logical reasoning; Negative Evidence, which consists of beliefs, opinions, or speculations; and Non-Scored, which consists of redundant, unrelated, or incomprehensible statements. Then messages are assigned a holistic score based on overall evidence, i.e., a Holistic 1 score indicates that a message consists of isolated statements; a Holistic 2 score indicates that a message excludes a clear argument, supporting evidence, or conclusion; and a Holistic 3 score indicates a message includes a clear argument, supporting evidence, and conclusion.

Interaction Analysis

Whereas the previous coding systems focus on analyzing the message, coding systems concerned with interaction analysis focus on analyzing the message as part of a larger discussion (Spatariu et al, 2004). Spatariu et al. reviewed two such models that are significant to this analysis. The first model, which involves analytic scoring, draws on the work of Schaeffer, McGrady, Bhargava, and Engel (2002). Somewhat similar to levels of disagreement analysis, this coding system scores messages based on their relatedness and agreement. On a 5-point scale, a score of -2 (counter) reflects that a message opposes a previous point and introduces a new element; a score of -1 (challenge) reflects that a message opposes a previous point, but does not introduce a new element; a score of 0 (unrelated) reflects that a message makes no clear reference to previous points; a score of +1 (acceptance) reflects that a message supports a previous point, but does not introduce a new element; and a score of +2 (enhancement) reflects that a message supports a previous point and introduces a new element.

The second model, which is a hybrid approach to interaction analysis, draws on the work of Järvelä and Häkkinen (2002). Within this coding system, messages are first assigned an analytic score based on stages of perspective-taking. Stage 0 is defined as Egocentric in which students present subjective perspectives and opinions, but do not advance discussion; Stage 1 is defined as Subjective Role-Taking in which students discriminate between subjective perspectives, but do not advance discussion; Stage 2 is defined as Reciprocal Perspective-Taking in which students acknowledge the value of others’ perspectives, and discussion advances with minimal perspectives; Stage 3 is defined as Mutual Perspective-Taking in which students coordinate perspectives, and discussion progresses from mutual experiences to more elaborated debate; and Stage 4 is defined as Societal-Symbolic Perspective in which students conceptualize subjective perspectives, and discussion demonstrates capacity of abstracting multiple mutual perspectives. Next, messages are assigned a holistic score based on the overall nature of discussion, i.e., Low Level, discussions that involve mainly separate comments and opinions; Progressive, discussions that involve generalizations, some joint knowledge building, and cross references, though not theory-based; and High-Level, discussions that involve shared, theory-based discussions, new points or questions, and rich cross-referencing.

An Exploratory Hybrid Assessment Model

Both analytic and holistic scoring have limitations. For instance, analytic scoring allows students to identify strengths and weaknesses; may be more time intensive; and may overlook the overall quality of discussion (Hazari, 2004). However, holistic scoring may present inverse issues. Consequently, an exploratory hybrid assessment model is proposed because it allows for interplay of the strengths of analytic and holistic scoring. Also, though two of the models described in the literature review involve hybrid assessment, both are limited to a specific genre of analysis, argument structure analysis or interaction analysis (Bendixen, Hartley, Sas, & Spatariu’s, 2003; Järvelä & Häkkinen, 2002). In contrast, the proposed hybrid assessment model, which is the product of the author’s experience of assessing hundreds of asynchronous discussions threads over the past five years, involves three levels of analysis-- argument structure analysis, levels or disagreement analysis, and interaction analysis (see Figure 1).

Figure 1 - Assessment Criteria for Analytical Reflection & Response Messages
COMM 3357 - Diversity & Mass Media

Analytical Reflection (1)

Evidence/Support

Response (2)

Writing

Format

45 Points

20 Points

20 Points

10 Points

5 Points

Present at least three discussion points based on a combination of premises and claims. Discussion points should advance overall understanding of subject matter.

Support discussion points with references to assigned readings, independent research, examples, and personal experience. Adhere to citation guidelines.

Discuss agreement/disagreement with classmates’ analytical reflections and provide explanation of logic; and/or respond to questions posed in classmates’ analytical reflections, and when appropriate, reference assigned readings, independent research, examples, and personal experience.

Responses should advance overall understanding of subject matter. Mere statements of agreement/disagreement will significantly compromise credit for this criterion.

Write in clear, concise, and grammatically acceptable terms.

Adhere to guidelines regarding length, subject lines, and text formatting.

Argument Structure &
Interaction Analysis

Disagreement &
Interaction Analysis

   

(1) An Analytical Reflection message is defined as a student’s general thoughts, reactions, and analyses of assigned readings, including, but not limited to, profound or controversial points, moments of epiphany, points that piqued interest or raise questions, methodological issues, and comparison/contrast observations. At some point, students are also required to pose at least one provocative question (one that invites a response).

(2) A Response message is defined a student’s response to a classmate’s analytical reflection message.

When scoring each criterion, particular consideration should be given to ensuring that scores reflect areas linked to the achievement of learning objectives. In other words, a critical question to address is how does the asynchronous discussion component of the course accomplish or relate to learning objectives? Strategic scoring, therefore, should result in interplay of the strengths of analytic and holistic scoring. That is, individual analytic scores should indicate the importance of each criterion relative to the overall holistic score; and the overall holistic score should indicate the relative accomplishment of learning objectives.

For example, the proposed hybrid model is an adaptation of one used in the author’s Diversity & Mass Media course, which involves controversial race, gender and media discourse. Therefore, the presentation of qualified arguments that advance overall discussion and understanding of the subject matter is a primary learning objective linked to the asynchronous discussion component of the course. This link is especially important given that conflict and emotions generally short-circuit such discourse in face-to-face discussions. Accordingly, note that the Analytical Reflection and Evidence/Support criteria comprise 65 of 100 points; and criteria that are less integrally linked to learning objectives, comprise few points, Response (20 points), and Writing and Format (15 points).

Application of Hybrid Assessment Model

Faculty can adapt this hybrid assessment model to their courses by defining and scoring assessment criteria that reflect their learning objectives and realize more meaningful and focused discussions. Analytic and holistic scoring promotes objective, systematic assessment; streamlines the assessment process; and clearly defines assessment criteria, and thus expectations, for students. The model is also of import because it advances research concerned with assessment, which is a “necessary precursor to research related to improving [asynchronous] discussions,” (Spatariu, Hartley, & Bendixen, 2004). Perhaps the best indicator of the value of this model is that student evaluations and performance affirm the author’s philosophy that when faculty clearly identify, define, and score assessment criteria for asynchronous discussions, students who sincerely engage in the learning process will put forth a good faith effort to satisfy and exceed the criteria.



References

Bendixen, L.D., Hartley, K., Sas, I.C., & Spatariu, A. (2003, April). The impact of epistemic beliefs and metacognition on on-line discussions. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.

Coombs, N. (2003). CMC: The medium and the messages. Electronic Journal of Communication, 3(2).

Edelstein, S. & Edwards, J. (2002). If you build it, they will come: Building learning communities through discussions. Online Journal of Distance Learning Administration, 5(1).

Hazari, S. (2004). Strategy for assessment of online course discussion. Journal of Information Systems Education, 15(4), 349 - 355.

Inch, E. S., & Warnick, B. (2002). Critical thinking and communication. Boston: Allyn & Bacon.

Järvelä, S., & Häkkinen, P. (2002). The levels of Web-based discussion: Using perspective-taking theory as an analysis tool. In H. Van Oostendorp (Ed.), Cognition in a digital world (pp. 77-95). Mahwah, NJ: Erlbaum.

Markel, S. (2001). Technology and education online discussion forums: It’s in the response. Online Journal of Distance Learning Administration, 4(2).

Nussbaum, E. M., Hartley, K., Sinatra, G.M., Reynolds, R.E., & Bendixen, L. (2002, April). Enhancing the quality of on-line discussions. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.

Popham, W. (2002). Classroom assessment: What teachers need to know (3 rd edition), Allyn & Bacon, Massachusetts.

Schaeffer, E. L., McGrady, J. A., Bhargava, T., & Engel, C. (2002, April). Online debate to encourage peer interactions in the large lecture setting: Coding and analysis of forum activity. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.

Spatariu, A., Hartley, K., & Bendixen, L. (2004). Defining and measuring quality in online discussions. The Journal of Interactive Online Learning, 2(4).

Camilla Gant, Ph.D., is Director & Associate Professor of Mass Communications at the University of West Georgia, Carrollton, GA 30118. Email: cgant@westga.edu.


Online Journal of Distance Learning Administration, Volume X, Number III, Fall 2007
University of West Georgia, Distance Education Center
Back to the Online Journal of Distance Learning Administration Content