Forum Quality or Quantity: What is Driving Student Engagement Online?


Cassandra S. Shaw
American Public University System
cshaw@apus.edu

Kathleen C. Irwin
American Public University System
kirwin@apus.edu

Abstract

The purpose of this study was to determine the relationship between forum quality and student engagement. It was hypothesized when the forum prompt was of expected quality it would be a driver of student engagement and examined the length of the forum prompt in relation to student engagement. The methodology adopted for this study was quantitative--a regression was performed for the regressor variables, collectively, with each dependent variable. In addition, a standard regression was performed for quality of forum prompt with each dependent variable, separately.  Data was collected over an eight-month period from May through December of 2015 from the following programs within the School of Business in an online university: accounting, business administration, government contracting, economics, entrepreneurship, finance, hospitality, human resource management, management, marketing, retail management, reverse logistics management, and transportation and logistics management. We examined the theory established by Salmon regarding the 5-stage model for forum development: (1) access and motivation, (2) online socialization, (3) information exchange, (4) knowledge construction, and (5) development. It was determined stages one and two are critical for student engagement as the more in depth the prompt the less likely students were to engage.

Forum Quality or Quantity: What is Driving Student Engagement Online?

In this current age of data collection, student engagement in the asynchronous, online environment is becoming a hot topic. With numerous companies on the forefront of data analysis promising to provide enlightenment to online universities everywhere, it is critical to have an understanding of what is driving the data. In an effort to examine some of this data, this study looked at a small portion of this environment—the discussion forums. Seeking to gain insight on whether quality or quantity drives student engagement in the discussion forums, researchers examined the data provided from eight months of forums in 2015 stretching across 13 different programs in the School of Business at an online university.

Literature Review

Best Practices

Although many publications have been released detailing “best practices” when it comes to formulating discussion forums, it was important to review the quantitative studies detailing what those best practices should be.  While it is possible to generate very useful analytics using tools provided by Learning Management System (LMS) software, it was only by doing the research manually have researchers been able to show the real learning around engagement and its roots.

In 2016, Casimiro conducted a study whereby he concluded the discussion forum prompts must be carefully constructed—both cognitively and interculturally, the social interaction among the students must be strong, and the student must have a strong learning community supporting the process of discourse. In a study conducted by Mason in 2011, the following tasks were concluded to increase engagement in the online forum environment:

  1. Introducing the forum task formally (via lab or video) to ensure all students understand the task and to eliminate any infrastructure barriers,
  2. Provide instructions, specifically detailing the task’s relationships to the module and to the class.
  3. Provide a motivational explanation of the forum.
  4. Encourage students to add a photograph to their introductions.
  5. Provide links to real life websites and articles or blogs to link the task more closely with real life.
  6. Increase quantity of engagement.

In 2013, another study conducted at the University of South Africa examined student online participation in discussion forums over a three-month period. The researcher grouped reasons for poor participation into themes: technical problems, unclear expectations, and providing feedback (Mokoena, 2013).

Gilly Salmon

At the forefront of e-learning theory is Dr. Gilly Salmon, creator of the 5-stage model. In this model, Dr. Salmon discussed the following theory, presented in Figure 1.



Figure 1. Salmon 5 Stage Model.

The five stages of this model are (1) access and motivation, (2) online socialization, (3) information exchange, (4) knowledge construction, and (5) development. The amount of interactivity along the five stages was directly related to the learning experienced by the student. The more involved the facilitator, the more engaged the learner, according to this theory. (Salmon, 2016). This model provided a theoretical framework with which to examine the forum process from the student engagement perspective. In stage 1, the student should be provided access and be motivated to engage in the forum. Stage 2 requires interaction on the part of the facilitator in order for socialization to occur on the part of the student. In this stage, the facilitator helps students to overcome any barriers to engagement, including cultural, social, or technical barriers (Salmon, 2016). The third stage encourages students to interact with other students and the learning material in the course. The fourth stage requires learners to discuss in the forum topics using both researched materials and material from their own personal experiences. Finally, in the last stage, students are encouraged to reflect on what has been learned. The model is a good example of a quality initiative in forum development and execution. In comparison, quality can be viewed quite differently depending on the perspective of the researcher.

Defining Quality

Quality has been measured and discussed over the past decade by various researchers from different perspectives. These include looking at the level of controversy (Burstall, 2000), the grammatical efficacy (Edelstein & Edwards, 2002), the length of the prompt (Biesenbach-Lucas, 2003), the tone of the prompt (Grady, 2003), or by the level of interaction between students (Nandi, Hamilton, & Harland, 2012).

Looking at public forums was another way to measure quality in forums. This broadened the landscape in which we have to examine forum quality. In a study done in 2014, Jung-Tae, Min Chul, and Hae Chang defined quality, as “a high quality thread in an online forum is a thread that possesses posts in which the majority of readers have highly rated or recommended” (Jung Tae, et al, p. 521). In this study, the researchers go on to discuss features from surface threads, some of which closely mirror the factors measured in this study, such as length of posting.

Using a rubric, as shown in Figure 2, researchers of this current study defined quality over seven categories and three levels rendering a quantitative score for each forum reviewed.

Figure 2. Quality score rubric.

Disciplines

The disciplines represented in the study were accounting, business administration, government contracting, economics, entrepreneurship, finance, hospitality, human resource management, management, marketing, retail management, reverse logistics management, and transportation and logistics management (Table 1).

Table 1

Number of Courses per Discipline and Course Level as a Percentage of the Total Number of Courses Studied (N = 7871)



Accordingly, studies have been done focusing solely on students within these disciplines and their discussion forum activities. One specific study by Hsu in 2011 focused on hospitality students and concluded students with specific learning styles were more likely to engage in discussion forums (Hsu, 2011). Another study conducted by Weil, McGuigan, Kern, and Hu in 2013 looked at the use of discussion forums to engage students in the examination of case studies in accounting classes (Weil, et al, 2013). In the past decade, countless studies have been conducted in the various disciplines focused on discussion forums, but none looked at these specific disciplines together. In the School of Business, these disciplines share resources, thus it is important to gain an understanding of how they are performing as a unit as well as individually.

Methodology

This research focused on level of course, week of forum, length of forum description, quality score (of the forum prompt), and forum appropriateness as regressor variables for student engagement, student presence, and forum grade.

It was hypothesized when the forum prompt was of expected quality it would be a driver of student engagement versus the forum quantity requirement (length of prompt). A regression was performed for the regressor variables, collectively, with each dependent variable. A standard regression was also performed for quality of forum prompt with each dependent variable, separately.

The population was defined as an online School of Business. The “participants” were each discipline within the School of Business and a random selection of courses, selecting one course from the 100 and 200 level, one course from the 300 and 400 level, and one course from the 500 and 600 level, when applicable, for each discipline. No 700 or above course levels exist. Eight months of course runs were studied, with each course running for eight weeks (Table 1 ). Within each course selected, the weeks 1, 3, and 5 forums were used for data collection of the following: course prefix and numbers, week of forum, forum grade (of student), student presence (posting initial response to forum prompt), student engagement (posting replies), level of course, length of forum description (the prompt), and quality score (of the forum prompt).

During the study, each forum was assessed for quality and a quality score calculated (Figure 2). Quality assessment consisted of weighting the following categories of the forum prompt:

Forums with a score of 79% or lower were flagged as “Needs Improvement;” forums with a score of 80% to 89% were flagged as “Standard;” forums with a score of 90% to 100% were flagged as “Outstanding.” The average forum prompt quality score across all disciplines was 75%; results were summarized in Table 2 .

Table 2

Mean Quality Score of Forum Prompt by Discipline per Course Level

Results

Student Engagement

Scores on student engagement were regressed from the following variables: level of course (coded 100, 200, 300, 400, 500, 600), week of forum (coded as week 1, week 3, week 6), length of forum description prompt (coded as a word count), quality score (of forum prompt, see Figure 2), and forum appropriateness (coded -1 = below level, 0 = on level, +1 = above level). The total number of observations N was 7871; there were no missing data. Preliminary data screening included examination of normal probability plot of residuals. Univariate distributions were not normal but included no extreme outliers producing an S-shape with long tails, meaning more variance than would be expected in a normal distribution (Figure 3 ).



Figure 3. Normal probability plot of residuals for student engagement.

A multiple linear regression was performed; the regressors included:  Level of course, Week of Forum, Length of Forum Description, Quality Score, and Forum Appropriateness. The rationale for this order of entry were factors emerged earlier in student engagement were entered in earlier steps. Results for this multiple regression were summarized in Figure 4 .

Figure 4. Regression analysis of student engagement.

To assess whether there were any multivariate outliers, the standardized residuals from this regression were plotted against the standardized values (Figure 5 ). No indication of a pattern, trend, or heteroscedasticity was present in this graph of residuals, nor were there any outliers; thus, it appears the assumptions required for multiple regression were reasonably well met.

Figure 5. Residuals from regression plotted against standardized values

The overall regression, including all five variables, was not statistically significant, R = 0.13, R2 = 0.017, adjusted R2 = 0.02, F(5, 7865) = 27.53, p = 1.01. Student engagement could not be regressed well from this set of five variables, which explained approximately only 27% of the variance in student engagement scores accounted for by the regression.

To assess the contributions of individual regressors for student engagement, the t ratios for the individual regression slopes were examined. All of the regressors were significant except for the quality score of student engagement; these included week of forum, t(7865) = 3.438, p = .0006; length of forum description, t(7865) = -3.378, p = .0007; forum appropriateness, t(7865) = -2.943, p = .0033. The nature of the relation of week of forum was positively related to student engagement, as expected; the higher the week of the forum, the higher student engagement. Further, the nature of the relation of length of forum description was negatively related to student engagement; higher lengths of forum had lower student engagement, which was as expected. The nature of the relation of forum appropriateness was negatively related to student engagement; the higher the forum appropriateness (or above level), the lower student engagement, which was as expected.

The other regressor variables (level of course and forum quality score) were not significantly related to student engagement when level of course and week of forum were statistically controlled; their partial slopes were not significant. Overall, student engagement scores were not highly related from this set of variables; the strongest unique regressor contributions were week of forum and length of forum description with a smaller contribution from forum appropriateness. Neither level of course nor forum prompt quality score were significant to student engagement in this regression; apparently, the information they contributed to the regression was redundant with other variables.

Standard regression was performed of forum quality score as a single regressor and was statistically significant, R = 0.02, R2 = 0.000597, adjusted R2 = 0.00047, F(1, 7869) = 4.70, p = 0.03. Student engagement could not be regressed well from this single variable, with approximately 0.047% of the variance in student engagement accounted for by the forum quality. To assess the contributions of the individual regressor of forum quality for student engagement, the t ratio for the individual regression slope was examined. Quality score was significant for student engagement, t(7869) = -2.17, p = .03. The nature of the relation of quality score (of the forum prompt) was not as expected; the higher the quality score, the lower student engagement.

Student Presence

Scores on student presence were regressed from the following variables: level of course (coded 100, 200, 300, 400, 500, 600), week of forum (coded as week 1, week 3, week 6), length of forum description prompt (coded as a word count), quality score (of forum prompt, see Figure 2), and forum appropriateness (coded -1 = below level, 0 = on level, +1 = above level). The total N for this sample was 7871; no cases were dropped as all data was present. Preliminary data screening included examination of normal probability plot of residuals. Bivariate distributions were not normal and included one extreme outlier producing a flat line, meaning no variance than would be expected in a normal distribution was shown, not all slopes had the expected signs, and there were no bivariate outliers.

Linear multiple regression was performed; each regressor variable was entered in one step in an order determined by the statistician, as follows: Step 1, Level of course; Step 2, Week of Forum; Step 3, Length of Forum Description; Step 4, Quality Score; and Step 5, Forum Appropriateness. The rationale for this order of entry was factors emerge earlier in student presence were entered in earlier steps. Results for this sequential multiple regression are summarized in Figure 6.

Figure 6. Regression analysis of student presence.

The overall regression, including all five regressors, was not statistically significant, R = 0.10, R2 = 0.01, adjusted R2 = 0.01, F(5, 7865) = 17.15, p = 6.56. Student presence could not be regressed well from this set of five variables, with approximately 1.0% of the variance in student presence accounted for by the regression.

To assess the contributions of individual regressors for student presence, the t ratios for the individual regression slopes were examined. Two of the five regressor variables were significant of student presence; these were week of forum, t(7865) = 3.70, p = .0002 and length of forum description, t(7865) = -2.09, p = .0370. The nature of the relation of week of forum was positively related, as expected; the higher the week of the forum, the higher student presence. However, the nature of the relation of length of forum description was negatively related to student presence; higher lengths of forum prompt had lower student presence, which was contrary to expectations.

The other three regressor variables (level of course, forum quality score, and forum appropriateness) were not significantly related to student presence when level of course and week of forum were statistically controlled; their partial slopes were not significant. Overall, student presence scores were not highly regressed from this set of variables; the strongest unique contribution was week of forum with a smaller contribution from length of forum description.  The level of course, forum prompt quality score, nor forum appropriateness were significant of student presence in this regression; the information they contributed to the regression was redundant with other variables.

Forum Grade

Scores on student forum grade were regressed from the following variables: level of course (coded 100, 200, 300, 400, 500, 600), week of forum (coded as week 1, week 3, week 6), length of forum description prompt (coded as a word count), quality score (of forum prompt, see Figure 2), and forum appropriateness (coded -1 = below level, 0 = on level, +1 = above level). The total N for this sample was 7871; no cases were dropped as all data was present. Preliminary data screening included examination of normal probability plot of residuals. Bivariate distributions were not normal but included no extreme outliers producing an S-shape with long tails, meaning more variance than would be expected in a normal distribution was shown; not all slopes had the expected signs, and there were no bivariate outliers (Figure 8 ).

Figure 8. Normal probability plot of residuals for forum grade.

Linear multiple regression was performed; each regressor variable was entered in one step in an order determined by the statistician, as follows: Step 1, Level of course; Step 2, Week of Forum; Step 3, Length of Forum Description; Step 4, Quality Score; and Step 5, Forum Appropriateness. The rationale for this order of entry were factors emerged earlier in forum grade were entered in earlier steps. Results for this sequential multiple regression were summarized in Figure 9 .

Figure 9. Regression analysis of forum grade.

The overall regression, including all five regressors, was not significant, R = 0.14, R2 = 0.02, adjusted R2 = 0.02, F(5, 7865) = 31.90, p = 2.70. Forum grade (of student) could not be regressed well from this set of five variables, with approximately 2.0% of the variance in forum grade scores accounted for by the regression.

To assess the contributions of individual regressors for forum grade, the t ratios for the individual regression slopes were examined. None of the five regressors were significant of forum grade, at the 1% level of significance. Overall, forum grades were not highly regressed from this set of variables; the information contributed to the regression was redundant with other variables.

Standard regression was performed of forum quality score as a single regressor and was not significant, R = 2.06, R2 = 4.24, adjusted R2 = -8.47, F(1, 7869) = 0.33, p = 0.57. To assess the contributions of the individual regressor of forum quality for forum grade, the t ratio for the individual regression slope was examined. Quality score was not significant of forum grade, t(7869) = 0.58, p = .56. The nature of the relation of quality score (of the forum prompt) was as expected; the higher the quality score, the higher forum grades.

Discussion

The research studied  level of course, week of forum, length of forum discussion prompt, quality score (of forum prompt), and forum appropriateness to regress student engagement, student presence, and student forum grade. The research question was: Forum quality or quantity: What is driving student engagement online?

The regression models did not reject the null hypothesis forum quality did not have an effect on student engagement, student presence, and forum grade. However, when controlling for level of course and week of forum, the length of forum description and forum appropriateness were significant for student engagement. When controlling for level of course and week of forum, the length of forum description was significant for student presence. Further, when controlling for level of course and week of forum, no regressor variables were significant for forum grade.

Student Engagement

As expected, the week of the forum (week 1, week 3, week 6), length of the forum description (the prompt), and the forum appropriateness (forum at appropriate level of course) were statistically significant. While length of forum description and forum appropriateness resulted in negative slopes, those results were as expected. It was expected the further the student progressed in a course, the student would remain engaged. For instance, a student in week 6 of an 8-week course would be more likely to stay engaged in the course. This might occur for a couple of reasons: it is too late to drop and the student needs to pass the course or the student needs the course to progress in their program. A student’s engagement level will vary depending on the week of the course.

When students faced forums of an appropriate length, they engaged more. The reverse was true as well, as indicated in Figure 10 and Figure 11 . As expected, if the forum prompt goes “on and on” and does not get to the point of what the student needs to post, the student, as indicated by the results, engaged less. This could indicate students need a straightforward, to the point, forum prompt with clarity of the forum requirements. This supports the quality definition presented by Biesenbach-Lucas who looked at the length of the forum prompt in 2003.

Traditional forum development has a tendency to require the same repetitive responses from students, thereby making it difficult for students and faculty to provide varying and meaningful comments in postings. However, when a forum has been developed with appropriate Bloom’s level, quality, and drives the response to be different for each student, the posted content would therefore be more interesting, which would drive engagement by both student and faculty.

Figure 10. Average length of discussion prompt by discipline in School of Business.

Figure 11. Average student engagement by discipline in School of Business.

In reviewing Figure 10 , BUSN (Business), MKTG (Marketing), ECON (Economics), and TLMT (Transportation and Logistics Management) had low word counts for the forum prompt. Based on statistical analysis, this should result in higher student engagement (Figure 11 ). However, those results were split – BUSN and MKTG had the highest student engagement with ECON and TLMT on the lower end of student engagement. Course level by discipline or the nature of the discipline could account for the differences. Also contrary to statistical findings was ACCT (Accounting) was the second highest for length of forum description yet had high student engagement. As this Accounting program is a highly accredited program, this could mean students know they do not have the option of dropping a course and because of the accreditation requirements length of forum prompts may be more involved, thus resulting in a longer forum prompt.

In this study, the range of length of the forum discussion prompt was 14 words in a marketing 300-level course to 1,509 words in a human resource management 600-level course with a mean of 250 words; Figure 10 showed the average length of forum discussion prompt by discipline. This fits the regression data analysis results because management courses had the lowest student engagement scores with the highest length of forum discussion (Figures 10 and 11). Further, while marketing had the lowest word count, it had the highest student engagement, further supporting statistical significance.

When the forum prompt was written above the course level (forum appropriateness), students engaged less. This most likely translated to students not comprehending what was being asked of them, at course level. For instance, if a student was in a 200-level course but the prompt was written above at a 400-level course, the critical thinking required to answer the prompt did not fit the cognitive level of the student. Thus, the forum prompt was “out of range” of the student’s ability to appropriately answer and therefore engaged less. Interestingly, the quality of the forum prompt was significant for student engagement, p = 0.030 when using a standard regression analysis.

Student Presence

It was expected the week of the forum (week 1, week 3, week 6) and length of the forum description (the prompt) were statistically significant. While forum appropriateness resulted in negative slopes, those results were as expected. As with student engagement, it would be expected the further the student progresses in a course, the student would remain present. For instance, a student in week 6 of an 8-week course would be more likely to stay present in the course. This might occur for a couple of reasons: it is too late to drop and the student needs to pass the course or the student needs the course to progress in their program. In previous studies, this would be similar to measuring student engagement (Casimiro, 2016; Mason, 2011).

When a forum prompt was lengthy, student presence was lower, as represented by a negative slope in the results. In this study, the range of length of the forum discussion prompt was 14 words in a marketing 300-level course to 1,509 words in a human resource management 600-level course with a mean of 250 words, with the average by discipline shown in Figure 10. This fits the regression data analysis results because management courses had the lowest average student engagement score (too close to 1 for no engagement) with the highest average length of forum discussion. Further, while marketing had the average lowest word count, it had the highest student average engagement (closer to 2 for being engaged).

Interestingly, forum appropriateness and quality of the forum prompt were not significant for student presence, p = 0.98 and p = .69, respectively. However, it may be possible students posted their initial post (student presence) no matter the forum appropriateness or quality of the forum prompt because students know their initial posting is for a grade.

Forum Grade

Forum grade was not affected by these regressor variables. It would be expected the further the student progresses in a course, the higher the student would score due to learning, growth, and ingesting new content. For instance, a student in week 6 of an 8-week course would be more likely to have higher grades than in the first week; however, this study provided results to the contrary. This would be a good opportunity for future research to further investigate the impact of student engagement on student performance.

It was further expected when the forum prompt was written above the course level (forum appropriateness), students would score lower on forum grade due to the nature of students not comprehending what was being asked of them, at course level. For instance, if a student is in a 200-level course but the prompt was written above at a 400-level course, the critical thinking required to answer the prompt did not fit the cognitive level of the student. Thus, the forum prompt was “out of range” of the student’s ability to appropriately answer and it was therefore expected to have significance on students’ forum grades. This contradictory unexpectedness may be due to faculty recognition of a forum prompt being “above level” and was therefore lenient on the grading. Interestingly, the quality of the forum prompt was not significant for forum grade, p = 0.79. In the literature, the researchers found content regarding the levels of the answers of students, but not research regarding the level of the prompts (Casimiro, 2016).  This would be a good area for future research.

Recommendations and Conclusions

Many questions arise from this study, including, “Why did forum quality not have an effect on student engagement, student presence, and/or forum grade?” It would be expected when the forum prompt was written with care (no spelling/grammar errors), visually appealing, the prompt asked a multi-layer question, and provoked critical thinking, student presence and student engagement, at least, would be effected. The question remains, “Why?”

As we conclude this study, we found unexpectedly the quality of the forum prompt had no significance on student engagement, student presence, and forum grade when using a multiple regression analysis. However, when using a standard regression, forum quality was significant for student engagement, yet produced a negative slope indicating the higher the quality score the lower the student engagement.  Since quality was based on items such as depth of discussion and critical thinking, it is possible students are not as prepared for the higher quality forums. Due to this conclusion, it is recommended forum prompts be clear, concise and follow the guidelines outlined by Salmon in the 5-stage model for setting up forum engagement. Specifically, stages 1 & 2 should be done with more consistency—ensuring students are secure in their understanding of the task and are socialized within the forum environment.


References

Biesenbach-Lucas, S. (2003). Asynchronous discussion groups in teacher learning classes: perceptions of native and non-native students. Journal of Asynchronous Learning Networks, 7(3), 24-46.

Burstall, J. (2000). Learning communities for social change in forums on the web. Australian Journal of Adult Learning, 40(1), 33-52.

Casimiro, L. (2016). Cognitive engagement in online intercultural interactions: Beyond analytics. International Journal of Information and Education Technology, 6(6), 441-447.

Edelstein, S., & Edwards, J. (2002). If you build it, they will come. Building learning communities through threaded discussions. The Online Journal of Distance Learning Administration, 5(1), 21-34.

Grady, D. (2003). Mapping online discussions with lexical scores. Journal of Interactive Learning Research, 14, 209-229.

Hsu, L. (2011). The perceptual learning styles of hospitality students in a virtual learning environment: The case of Taiwan. Journal of Hospitality, Leisure, Sports and Tourism Education, 10(1), 114-127.

Jung Tae, L., Min Chul, Y., & Hae Chang, R. (2014). Discovering high quality threaded discussions in online forums. Journal of Computer Science and Technology, 29(3), 519-531.

Mason, R. (2011). Student engagement with, and participation in, and e-forum. Educational Technology & Society, 14(2), 258-268.

Mokoena, S. (2013). Engagement with and participation in online discussion forums. The Turkish Online Journal of Educational Technology, 12(2), 97-105.

Nandi, D., Hamilton, M., & Harland, J. (2012). Evaluating the quality of interaction in asynchronous discussion forums in fully online courses. Distance Education, 12(2), 5-31.

Salmon, G. (2016). The five stage model. Retrieved from Gilly Salmon: http://www.gillysalmon.com/five-stage-model.html

Weil, S., McGuigan, N., Kern, T., & Hu, B. (2013). Using asynchronous discussion forums to create social communities of practice in financial accounting. Pacific Accounting Review, 25(1), 30-57.

Online Journal of Distance Learning Administration, Volume XX, Number 3, Fall 2017
University of West Georgia, Distance Education Center
Back to the Online Journal of Distance Learning Administration Contents