Measuring Student Engagement in an Online Program


Paula Bigatel
pmb6@psu.edu

Vicki Williams
Pennsylvania State University
vqw@psu.edu

Abstract

In an effort to measure the effectiveness of faculty development courses promoting student engagement, the faculty development unit of Penn State’s Online Campus conducted a pilot study within a large online Bachelor of Science in Business (BSB) program. In all, 2,296 students were surveyed in the spring and summer semesters of 2014 in order to seek their perspectives on (1) the extent of their engagement in the courses and (2) the degree to which their instructors promoted their engagement. The survey comprised three sub-scales: the first and third sub-scales addressed instructional design aspects of the course, and the second sub-scale addressed attitudes and behaviors whereby the instructors promoted student engagement. The results showed a significant difference on the second sub-scale (sig = 0.003) at the .05 level, indicating that students rated instructors with professional development higher on instructor behaviors that engaged them in their courses than those instructors who received no professional development. There were no significant differences found for the first and third sub-scales indicating that the instructional design aspects of the courses under investigation were not influenced by instructors’ professional development. Qualitative data showed that three quarters of the students who had instructors whose background included professional development geared to encouraging student engagement felt that their courses had engaged them. Future research will focus on increasing the response rate and exploring in more depth both the instructional design and qualitative aspects of student engagement.

Introduction

How do we know if our professional development programs for faculty are effective? Does student engagement increase when faculty complete professional development courses aimed at promoting student engagement in the online learning environment? The present study addresses these questions in an effort to think about ways to measure whether and to what extent faculty are applying the material taught in two professional development courses offered by the Faculty Development Unit at the Penn State World Campus. The mission of this unit is to support faculty in best teaching practices in order to positively impact student success. Research affirms that engaging learners in highly interactive learning environments, where they can engage with content, instructors, and peers, leads to student retention and success (Robinson & Hullinger, 2008; Wyatt, 2011).

Motivated by such positive student outcomes associated with engaging students in the learning process, the World Campus Faculty Development Unit developed and offered faculty training in strategies that promote student engagement. To measure faculty effectiveness in engaging students, a survey instrument was emailed to students asking them to rate their level of engagement in the courses. The purpose of this study was to find out whether students whose instructors had been trained in strategies to promote student engagement scored higher on measures of student engagement than students whose instructors had not received such training. This study measured level and quantity of student engagement along with the quality of student engagement as indicated by students’ responses to several qualitative, open-ended questions.

OL 2000 Effective Online Teaching and OL 2700 Online Presence Course Descriptions

Both faculty development courses referred to in this study aim at helping faculty develop and apply strategies for promoting active participation on the part of students as well as the instructors, facilitating online learning, and fostering a sense of community among students and instructors.  The emphasis for both courses is on what the instructor can do to help students feel connected to the online learning experience such that learning is enhanced. The rationale for providing such professional development opportunities is that students will be more likely to persist in their courses if they feel supported in their learning and satisfied with their learning experiences, see value in the course content, and have a clear sense that instructors possess a desire for students to succeed.

The following research questions guided our study:

Research Question 1: Are faculty who have completed OL 2000 (Effective Online Teaching) and/or OL 2700 (Online Presence) using strategies to engage their students to a greater extent than faculty who have not taken either of these faculty professional development courses?

Research Question 2: Do students who have taken courses from faculty who have completed OL 2000 and/or OL 2700 score higher on measures of student engagement?

Definition of Student Engagement

In this study, student engagement is defined in broad terms as the time and physical energy that students expend on activities in their academic experience (Kuh, 2003). More specifically, engagement involves the student’s efforts to study a subject, to practice, to obtain feedback, to analyze, and to solve problems. Furthermore, in the present study, the term “engagement” is used interchangeably with the term “interaction.”

Literature Review

Introduction

A high level of student engagement is associated with a wide range of educational practices including purposeful student–faculty contact, active and collaborative learning, and positive factors such as student satisfaction, persistence, achievement, and learning (Kuh et al., 2006). By encouraging student engagement, institutions of higher education can have a positive impact on student success; that is, academic institutions can affect the academic and social integration that leads to a commitment to graduate (Tinto, 2004).

Because of the physical distance involved in studying online, students in this environment have a greater chance of becoming psychologically distanced from the learning experience than do students in face-to-face contexts. Furthermore, since a large proportion of online learners are adults (non-traditional students typically between 25-54 years old) who have jobs and family obligations (NCES, 2009); they may be at a greater risk of dropping out of courses because of constraints on their time. Based on data collected from 114,000 students at 104 institutions, the National Online Learners Priorities Report (Noel-Levitz, 2013) indicated that 81% of the respondents were adult learners. Ensuring that adult learners complete their degrees can be more challenging. Many instructors  who teach face-to-face classes may be accustomed to teaching  traditional students and when they begin teaching online, they are confronted with a much different student audience. According to Allen & Seaman (2014), online enrollments have increased such that the majority of all higher education students are taking at least one online course. At the same time academic leaders are becoming increasingly concerned about student retention. Allen and Seaman (2014) surveyed academic leaders from 4,726 colleges and universities in degree-granting institutions. Based on 2,831 responses, a total of 41% of chief academic officers reported that they agreed that retaining students was a greater problem for online courses than for face-to-face courses. Since most online learners are adults, different approaches to address the needs of this audience are required to ensure degree completion.

Moreover, satisfaction with courses is associated with persistence. In the National Adult Learners Satisfaction-Priorities Report (2013), data was collected from 18,538 students at 45 four-year institutions and 5,826 students from 26 two-year institutions. Adult learners in this study rated what mattered most to them (importance) along with how satisfied (satisfaction) they were in the teaching and learning process. In the analysis of data, “teaching and learning process” ranked very high in importance and certain aspects of the teaching and learning process ranked very high in satisfaction i.e., clear understanding of what learners are expected to learn in their courses; course content is closely related to their lives and work goals; and frequency of interactions with instructors is adequate (p. 8).  With respect to the teaching and learning process, students ranked instructors’ timely feedback on academic progress low indicating clear dissatisfaction with instructors who delayed feedback or provided little or no useful feedback. Ultimately, according to the 2013 National Adult Learners Satisfaction-Priorities Report, satisfied students are more likely to be successful students i.e., students who complete courses. Research indicates that more satisfied students have higher graduation rates, which is a testimony to persistence and retention (Noel-Levitz, & CAEL, 2013; Noel-Levitz, 2013).

Adult Learners and Engagement

According to statistics provided by the U.S. Department of Education’s National Center for Educational Statistics indicate adult learners are one of the fastest growing populations of students (NCES, 2009). Therefore, in terms of institutional success, it is important to fully meet the needs of this group in order to ensure retention. Some of the characteristics that define adult learners are that they are highly motivated, seek relevance and meaning in their learning, and bring life/work experiences that they can share with others to the educational environment (Cercone, 2008). Knowledge about adult characteristics should be applied to teaching strategies to make the learning experience not only satisfying but also relevant to their personal and professional lives.  In terms of adult students’ assessments of educational quality, student engagement (e.g., opportunities for interaction with peers, content and faculty) was ranked highly in relationship to satisfaction and perceived learning in both face-to-face (Carini, Kuh, & Klein, 2006; Kuh, 2003) and online learning environments (Lehman & Conceição, 2014; Watwood, Nugent, & Deihl, 2009). Satisfaction and perceived learning were also positively linked with persistence (Rabe-Hemp, Woollen, & Humiston, 2009), which provides justification for evaluating adult learner satisfaction with their academic experiences and perceived learning because students that are more satisfied with their learning experiences and the level to which they are learning are less likely to drop out.

Moreover, in a study that looked at the external and internal factors that influenced adult learners’ decisions to drop out of online courses, it was also found that adult learners were less likely to drop out when they were satisfied with the courses and when the courses were relevant to their lives (Park & Choi, 2009). The authors recommended that “course design strategies and learners’ motivation should be prioritized at the course development stage in order to make the course participatory and interesting and to keep learners engaged” (p. 215).

In a “persistence model for online student retention,” Lehman and Conceição (2014) present a student-centered model incorporating strategies that faculty can implement to facilitate online student persistence and retention in terms of both instructional design and instructional delivery:

When creating the learning environment, the instructor uses consistency, variety, relevance, and content prioritization to help students stay motived throughout the course. When planning for the teaching process, the instructor sets up clear expectations, personalizes the interactions, and incorporates feedback throughout the course to create a sense of presence and engage students. (Lehman & Conceição, 2014, p. 89)

The indications are that for online learning to be effective, faculty must shift from a teacher-centered learning approach to one that puts students at the center of learning. The student-centered approach takes into consideration student characteristics such as level of self-awareness, level of self-efficacy, educational goals, and motivations. Based on information of this nature, instructors can provide appropriate support for students to help them in achieving their goals by incorporating effective intrinsic and extrinsic motivators into the course design and delivery. Most importantly, “the online classroom depends on student interaction and dialogue” (Spellman, 2007, p. 73), which means interaction with peers, faculty and the course content are necessary because it reminds students that they are actually working with people and can help alleviate the dissonance inherent to online learning (Knowlton, 2000).

 In another study focused on measuring online student engagement, Robinson and Hullinger (2008) drew similar conclusions in regard to the need for instructors to create purposeful course designs that promote interaction, participation, and communication in the online environment. There seems to be a consensus, therefore, that students need to feel a connection to other students and be supported by instructors during the learning process if an increase in retention is to occur.

Importance of Building an Online Learning Community

Higher education attrition rates are considerably higher in online learning environments than in face-to-face environment (Allen & Seaman, 2014; Patterson & McFadden, 2009; Rovai, 2003; Willging & Johnson, 2004). Thus, it is imperative to explore issues relating to online retention in order to find ways to encourage students to persist. Willging and Johnson (2004) looked at a variety of factors—categorized as personal, job-related, and program-related reasons—that influenced the decisions of students studying in the online environment to drop out. In the category of program-related reasons, the lack of one-to-one interaction with the instructors and other students ranked high among the reasons the students reported for dropping out. In a study focused on the importance of building a learning community, Rovai and Wighting (2005) noted that feelings of alienation and a limited sense of community both related to low student persistence in distance education programs.

The Community of Inquiry (CoI) model addresses the importance of establishing connectedness between students and instructors with the intention of improving the retention rate of online students. This model constitutes a theoretical framework that explains the online learning experience in terms of interactions between three overlapping kinds of presence: social, teaching, and cognitive (Garrison, Anderson, & Archer, 2001). Social presence is the basis for collaborative learning and refers to the extent to which students are able to project themselves socially and emotionally into the learning environment and the extent to which they perceive others in that environment as “real people” (Richardson & Swan, 2003). Teaching presence refers to the ways and the extent to which instructors facilitate discourse in the learning environment and direct instruction. Cognitive presence refers to the extent to which learners actively construct knowledge and meaning through a process of reflection and discourse with each other (Garrison, Anderson & Archer, 2001). Research supports the significance of social presence for retention (Boston et al., 2010; CCRC, 2013) and likewise the significance of teaching/cognitive presence for student learning (Anderson et al., 2001; Baker, 2010; Shea & Bidjerano, 2009). Research has also found that teaching presence is a significant predictor of students’ perceptions of learning, satisfaction, and sense of community (Gorsky & Blau, 2009; Russo & Benson, 2005). Tu and McIsaac (2002) examined social presence and its importance in establishing a sense of community among online learners. They concluded that “social presence is a vital element influencing online interaction” (p. 146). Pollard et al. (2014) looked at instructor social presence (ISP) and found that ISP and social presence (among students) were significant predictors of both a sense of community and a positive learning environment. There is also evidence that a strong sense of community is significantly associated with perceived cognitive learning and satisfaction with online programs, resulting in fewer drop outs (Rovai, 2002; Shea, Li, & Pickett, 2006). Furthermore, there is a positive correlational evidence to suggest that teaching presence in an online course is an important factor in fostering a sense of classroom community (Garrison & Cleveland-Innes, 2005; Shea, Li, & Pickett, 2006; Shea et al., 2006). In general, there is considerable evidence that engaging students and instructors in online learning communities may boost student retention and persistence whether this engagement is achieved through promoting social, teaching, and/or cognitive presence.

Participants

In the present study, students were surveyed about their learning experiences in the online Bachelor of Business (BSB) program offered through the Penn State World Campus. Students enrolled in one or more of the 64 BSB courses offered in the spring 2014 semester along with students enrolled in one or more of the 39 BSB courses in the summer 2014 semester were emailed the student engagement survey. In all, 2,296 surveys were sent. The sample used was a convenience sample, and permission to survey the students in this program was given by the Director of the BSB Program.

There was an even ratio of female to male respondents. In terms of age, 41% of the respondents were between the ages of 35 and 45 and 33% were between the ages of 25 and 34. Only 9% of the respondents were traditional-age students. Most of the respondents worked full-time (64%), and 91% had taken more than one online course.

The Student Engagement Survey

The Student Engagement survey comprised a total of 23 questions under three sub-scales:

  1. Student Engagement Activities (9 items)
  2. Instructor Attitudes and Behaviors (9 items)
  3. Thinking Skills (5 items)

 See Appendix A for the 23 items. Reliability of the survey was good for the survey and subscales (Table 1).

1.)

Scales

Cronbach’s Alpha

Cronbach’s Alpha Based
On Standardized Items

N of Items

Student Engagement

.925

.927

23

Student Engagement
Activities (subscale 1)

 

.802

 

.803

 

9

Instructor Attitudes
and Behaviors (sub-scale 2)

 

.937

 

.938

 

9

Thinking Skills (sub-scale 3)

.856

.863

5

Table 1: Reliability Statistics on Student Engagement Survey and Subscales

The survey items were based on Chickering and Erhmann’s (1996) seven principles of good practice in undergraduate education applied to the online learning environment and on Building from Content to Community: [Re]Thinking the Transition to Online Teaching and Learning (Watwood, Nugent, & Deihl, 2009). The survey items asked students how frequently they engaged in a variety of activities and the extent to which their instructor demonstrated engagement-related attitudes and behaviors in each of the courses in which they were enrolled (sub-scales 1 and 2). Furthermore, questions were asked related to the level of academic challenge presented by the course being rated (sub-scale 3), as measured by Bloom’s taxonomy (Anderson & Krathwohl, 2001) in terms of levels of learning from simple cognitive activities (e.g., memorizing) to more complex cognitive activities (e.g., analysis, evaluation, and creation).The rating scale ranged from 1 (Not at all/very little or Never/rarely) to 4 (Very often or very much).

Survey Administration

Students enrolled in the BSB program were sent the survey in an electronic format in their course learning management system via email. Each student was asked to respond to questions related to the course in which they were enrolled so that researchers could associate student responses to the course and the instructor who taught the course.

Both the students and instructors were informed of the purpose of the study via an email sent from each course in this study. The students’ responses were kept anonymous and no identifiable data could be traced back to their user identification numbers.

Further, given that the professional development courses were offered by the World Campus’s Faculty Development Unit, the research team was able to determine which instructors had taken them. Sixty-nine percent of the instructors had taken professional development courses with the Faculty Development Unit.

Results

In total, 2,296 surveys were emailed to students in the 2014 spring semester and the 2014 summer sessions. Two hundred students responded yielding a 9% response rate. Of the 200 students who completed the survey, only 159 student scores could be used to answer the two research questions, because only these students completed the question asking them to indicate the course and section in which they were enrolled. The course and section numbers were needed to match each student score with the corresponding instructor.

In order to determine whether professional development had an impact on student engagement, the three sections were subtotaled, yielding three sub scores: Sub score 1: engagement activities, Sub score 2: instructor behaviors, and Sub score 3: student thinking skills. These sub scores were used in the subsequent analysis.

Table 2 shows the mean scores and standard deviations for each sub-scale. This table groups student responses into two groups; one group represented the responses from students who had instructors with professional development (Yes) and the second group represented the responses from students who had instructor with no professional development (No).

                 

Professional Development?

N

Mean

Std. Deviation

Std. Error

Sub score 1
Engagement Activities

Yes

110

22.85

5.811

.554

No

49

23.92

6.506

.929

Total

159

23.18

6.033

.478

Sub score 2
Instructor Behaviors

Yes

110

28.82

7.246

.691

No

49

24.61

9.615

1.374

Total

159

27.52

8.252

.654

Sub score 3
Student Thinking Skills

Yes

110

15.32

3.692

.352

No

49

14.39

4.157

.594

Total

159

15.03

3.852

.306

Table 2: Mean and Standard Deviations for the Three Sub-Scales of the Student Engagement Survey

To answer research question 2, the research team looked at the total student engagement score for all sub-scales as well as student scores for Sub score 1 and Sub score 3 separately. A comparison of mean scores for each group of students (students who had instructors with professional development and those students who had instructors without professional development) for the total student engagement scale and sub-scales 1 and 3. No significant differences were found for the total student engagement scores or for the Sub score-1 and Sub score-3 scales, as shown in Table 3.

ANOVA

 

 

Sum of Squares

df

Mean Square

F

Sig.

Between Groups

559.734

1

559.734

2.365

.126

Within Groups

37157.637

157

236.673

 

 

Total

37717.371

158

 

 

 

Table 3. Comparison of means of total scores of student engagement and instructors with/without professional development, yes or no

To answer research question 1, the second sub-scale was used, as it directly related to behaviors and attitudes that instructors demonstrated in promoting student engagement. A one-way ANOVA was performed (Table 4).

The results showed a significant difference on the second sub-scale (sig = 0.003) at the .05 level, indicating that students rated instructors with professional development higher on instructor behaviors that engaged them in their courses than those instructors who received no professional development.

ANOVA

 

Sum of Squares

df

Mean Square

F

Sig.

Sub score 1 Engagement Activities

Between Groups

39.023

1

39.023

1.073

.302

Within Groups

5712.046

157

36.382

 

 

Total

5751.069

158

 

 

 

Sub score 2
Instructor Behaviors

Between Groups

599.677

1

599.677

9.267

.003

Within Groups

10159.996

157

64.713

 

 

Total

10759.673

158

 

 

 

Sub score 3
Student Thinking Skills

Between Groups

29.346

1

29.346

1.990

.160

Within Groups

2315.496

157

14.748

 

 

Total

2344.843

158

 

 

 

Table 4. Comparison of means of student sub scores and PD
Qualitative Data Results

The tables below offer representative student responses to each qualitative question.

Define what it means to you to be engaged in a course.

The majority of students felt that engagement meant interacting both with their peers and with the instructor, participation in online discussions, and involvement with the subject matter. Often, the students mentioned that timely, meaningful instructor feedback was a part of these interactions.

“Interaction between students and professor as well as student to student. Student to student helping each other out and engaging in problem-solving. Also, getting to know each other as if we were in a real classroom situation.”

“To be engaged in a course to me means that the student along with the instructor, are proactive in discussions and different types of feedback. The students should be actively engaged in the discussions to learn from each other and to hear different opinions from others.”

“Engaged means to me that I am learning from the instructor, the material, and most importantly from fellow students. The exchange of ideas and experience is most helpful.”

“An instructor, engaged = active participation and timely response/feedback with the end goal of helping a student succeed./ For a student, engaged = active and timely participation in an effort to understand the course material for more reasons than to pass an exam or get an A on a test, and to feel supported by the instructor to learn the material.”

Table 5: Interaction and Active Participation/Involvement

Another common theme that emerged from student responses indicated how important relevance and real-world application was to the students in all their interactions.

“Engaged means not only to interact with students but to have a clear understanding of the concepts and apply them to real world situations.”

“Relevant course material that can be applied to business situations.”

“This course has all the right stuff, just lacks us thinking more about current events. I really think this course could embody engagement if we didn’t spend some weeks thinking about the basic concepts by themselves and instead thought about them as applicable to the real world and what is going on around us.”

“It means learning the course content and seeing ways to utilize that information in my regular life and work.”

“To be interested in the material being presented and to be able to apply it in a practical way not only in class but in a real work situation.”

“To actively participate in the course work. More importantly to gain an understanding of real world situations.”

“Engage in a course means to me using real-world, relatable examples that allow a student to not only research unfamiliar topics, but also being able to interact with the professor on a weekly basis.”

Table 6: Real-World Applications

Yet another theme that emerged related to student interest and motivation to learn course content. Some students included this theme with the use of real-world applications inherent in their courses. Instructor characteristics are implied in that the students want to be inspired or engaged by instructors who demonstrate enthusiasm and interest in what they are teaching.

“For me to be engaged in a course I need to be interested in it as well as challenged by the material. It means I’m actively putting effort into the course and do so wanting to know more.”

“For me, engaged is when I’m intrigued and motivated by the lesson. The instructor plays a large role when it comes to engaging students, especially in an online environment.”

“To be motivated and intrigued by the course. To inspire curiosity, which leads to researching and finding answers.”

“To be actively interested in the material being presented. To have the opportunity to discuss the information in a way that knowledge is being imparted. Be able to apply the information and knowledge in real-world situations in a way that adds value.”

“To be ‘engaged’ in a course is to have a desire both to actively participate in the course work and desire to extract more than is offered through the standard assigned course work. Please note, the word ‘DESIRE.’ The instructor can easily encourage students to extract more information on a small amount of enthusiasm and encouragement. If the instructor has a cold feel when interacting with students this often closes the door for ‘the desire to learn more.”

“Not only involve yourself because it’s a requirement, but do so because you want to. I was engaged in this course because the teacher was and it was inspiring to me to want to do well.”

Table 7:  Motivation and Interest

Does this course meet your definition of engaged learning?

The students were asked whether the course they had taken met their definition of engaged learning. Just over three-quarters of the students felt that their courses met their definition of engaged learning. Most of the students who answered yes to this question explained that the instructor was present, gave timely, meaningful, or personalized feedback, and interacted/participated with the students.

“Yes, both of my courses meet my definition of engaged. Instructors responded in a timely manner. Also, as opposed to simply grading my assignments, both of these instructors provide personalized feedback on my assignments and tests.”

“This course does meet my definition of ‘engaged’ because it presents information in many different ways, which I like. We have the online videos, the textbook reading, comprehensive problems, and team group assignments which all help to present information in an interesting way. It keeps me interested and helps me practice what I’m learning in real-world ways that I can easily use in my life.”

“Well, we are required to do forum discussions each week and have a group project going on that we have to check daily. Also, there are discussion forums before each exam that allows us to post what we are struggling with so the professor and classmates can comment on to help with the understanding of those confusing concepts.”

“The instructor has been very helpful in answering questions. Her attitude in e-mails and course discussions shows me she truly wants the class to do well and grades everything back in a timely manner.”

“The instructor provides meaningful and personalized feedback to my assignments which assist me in determining whether or not I’m on the right track. It’s not about just getting a good grade and degree for me I genuinely want to learn and have the experience I would have if I were attending class on campus.”

Table 8: Students’ Definitions of Engaged Learning

However, some of students who answered ‘yes’, did so not because the instructor had participated in the course, but because of applicability of the course material to their real-world experiences in business or because the course challenged them to think critically.

“I feel pretty engaged in this class. We have a lot of discussion and group work that create a sense of working together. The one aspect I would recommend improving is professor feedback regarding group discussions. There have been a couple of disputed points in discussions and we never receive feedback from the professor regarding these elements. Sometimes you just want to know which perspective/opinion was the correct one. Overall though, I feel fairly engaged in this class.”

“Yes and no. The course content was great. Assignments and discussion topics have been very relevant and have pushed for a deeper level of thought and application of the materials. However, the instructor has not provided meaningful feedback on some assignments, on very few discussions, and on those items where feedback was given, it was often very late in the process, and only after inquired and prompting by the student. Some assignment directions were not clear, or misleading, and the clarification came so late in the game that it placed an unfair burden on the student to make the required changes in the required time frame.”

“I said yes because the three projects/assignments/labs which required students to get out there, were really good (if you challenged yourself). The rest of the work could have used more effort from the professor. What is the point in having discussion boards if we do not discuss?”

Table 9: Engagement Definitions Not Related to Instructor Attitudes and Behaviors

For students who said that their courses did not meet the definition of engaged learning, the overwhelming consensus was that their instructors were not present in the courses and were late returning graded assignments. The students noted that these factors interfered with their ability to improve based on timely and meaningful feedback.

“It has been over six weeks since we turned in an individual assignment that uses the same format and concepts as a group assignment that is a major portion of our grade. We have still not received grades or any type of feedback on the individual assignment. We have received multiple videos and emails from the professor stating he has our assignments graded but he never delivers. The lies and lack of feedback from the professor has completely ruined this class and all that it could have been. I have lost my motivation and at this point I just don’t care anymore as it’s clear the professor doesn’t either.”

“I almost never hear from my teacher. I don’t even know his/her name. I get no feedback. I get no instruction. The book is my core source of instruction, which not only makes the content horribly uninteresting, but takes quite a bit of work to translate. My quizzes are administered and managed by software, so they don’t resemble the exams at all. There are no projects or discussions or tutorials. This class is a joke and a waste of my money.”

“The instructor has not provided a single word of instruction. There is not one slide, not one lecture, not one e-mail containing instruction. There is no meaningful actionable feedback on submitted assignments. We take quiz modules and interact with one another according to the syllabus. It is a miserable, hollow, wasteful experience. Please fix this class!”

“We did have discussion posts but nothing that pertained to the class. We had a single group project. For comprehensive problem five, both the teacher and tutor had told the class ahead of time that this was an assignment that most students have difficulty with. If that is the case and instructional staff knows that, I would think that they would have had a blackboard eluminate session or at least more interactive session than what we got, which was ‘Read the text very carefully.’ In online courses so much depends on the student’s reading comprehension and we accept that going in but we could use some teacher time …. out here, we often feel alienated and as though we are left to our own devices.”

“I felt the instructor was not present in the process. There was no interaction in the discussions despite the opportunity of several teachable moments. The course is poorly designed and delivered and does not reflect current trends of diversity in the workplace. There were several opportunities to have valuable instructor led discussion about issues happening now in the work environment and those opportunities were missed.”

Table 10: Lack of Timely Feedback and Instructor Presence

Instructor Professional Development and Student Engagement

One hundred and fifty-four records contained information that allowed the research team to match qualitative data to instructors who had taken one or more of the professional development courses. Of the 154 records, 107 instructors had taken one or more of the professional development courses whereas 47 had not taken either of the courses. Three quarters of the students who reported that their courses had engaged them were taking courses taught by instructors who had taken one or more of the professional development courses. The students’ explanations regarding the factors that had contributed to their engagement fell into three categories:

The students’ explanations lend some support to the effectiveness of professional development efforts that encourage engagement strategies, thus answering Research Question 1 in a positive way.

 A quarter of the students who had instructors with professional development were not engaged in their courses. Most of the reasons for this lack of engagement related to lack of instructor participation and feedback on assignments and/or discussion posts. On the other hand, the students with instructors who had not taken either of the professional development courses and who reported being disengaged in their courses attributed their lack of engagement to reasons similar to those given by students who were disengaged in courses where the instructor had taken one or more of the professional development courses.

Discussion

Student engagement is a strong predictor of student persistence and degree completion. As a result, training instructors in strategies to encourage student engagement is a valid goal for any faculty development program. To determine whether and to what extent instructors who have had such training are effective in engaging students was the object of this study. According to the accounts of the students surveyed in the online business program, it was evident that most instructors with relevant professional development applied strategies to engage the students. Both quantitative and qualitative data supported the fact that many trained instructors had a positive impact on student engagement. This impact was clearly indicated where engagement scores for Sub score 2: Instructor Behaviors were significantly higher for instructors who had professional development in comparison to those who did not (Research Question 1). The qualitative data also provided evidence that trained instructors were more actively involved in the courses and provided more timely and meaningful feedback than did untrained instructors.

One issue that needs to be corrected in further iterations of this survey is that the students did not always record the course and section in which they were enrolled despite the fact that their recruitment email gave them this information. Without the course and section number, we could not match the instructor who taught the course to the student’s comments in order to determine whether the course had been taught by an instructor who had or had taken one or more of the professional development courses. To resolve this issue in future surveys, the survey will be constructed such that the student must enter this information before the survey will allow him/her to move on to the next question.

Furthermore, as our response rate was very low, in the next iterations of this survey, the research team will send out several reminders to students to complete the survey. We may even consider offering an incentive such as a gift card. With a greater response rate, we would be able to offer a more definitive conclusion pertaining to the positive impact that trained instructors have in their courses as a result of the actions they take to engage their students. Moreover, a greater response rate with additional programs included in this study might reveal some course design aspects that need to be improved such that student engagement is enhanced. We intend to explore more fully the impact of the design/content aspects of the courses, which are addressed in the first and third subscales of the survey. Neither of these subscales showed a significant difference between instructors with professional development and those without. These subscales reflected aspects of the course design over which instructors often have no control, but they may have influenced students’ negative responses that were attributed to the instructor behaviors. Moreover, additional qualitative questions might provide further insights into improvements that need to be made to the design of courses, the selection and presentation of content, and the nature and number of student activities/assignments. Such data would be helpful to administrators of programs, instructor course authors, and instructional designers. It may be that despite weaknesses in course design, what instructors do to engage students is more important when it comes to student persistence and degree completion. Further research is needed to determine the veracity of this supposition.

Conclusion

With improvements made to the survey and administration procedures, the research team involved in this study plans to target additional programs in different disciplines as well as programs at the graduate level. We not only want to see if similar patterns emerge such that professional development for instructors is perceived as effective in promoting student engagement, but also confirm that strategies employed by trained instructors are effective despite discipline or level of program. Although some instructors who had taken either of the professional development courses did successfully engage students, it would be interesting to determine the extent to which this occurred using a larger sample. On this point, we do not know whether the non-trained instructors were able to engage the students because these instructors were experienced in the face-to-face and/or online context, because they were trained in pedagogy elsewhere, or for other reasons. There is also the possibility that some instructors are natural-born teachers. Further research might explore instructor perceptions about whether or not they are engaging students so that we can compare student versus instructor perceptions. Examining instructor perceptions would give insights into the value of professional development, another measure of the effectiveness of our faculty development program.

Acknowledgement

We wish to acknowledge Nai-Fen as the data analyst who contributed to the analysis of the data used in this study.


References

Allen, I. E., & Seaman, J. (2014). Grade change: Tracking online education in the United States. Babson Research Group. Retrieved from http://www.onlinelearningsurvey.com/reports/gradechange.pdf.

Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. New York, NY: Addison Wesley Longman.

Anderson, T., Rourke, L., Garrison, D. R., & Archer, W. (2001). Assessing teaching presence in a computer conferencing context. Journal of Asynchronous Learning Networks, 5(2), 1-17.

Arbaugh, J. B., Cleveland-Innes, M., Diaz, S. R., Garrison, D. R., Ice, P., Richardson, J. C., & Swan, K. P. (2008). Developing a Community of Inquiry instrument: Testing a measure of the COI framework using a multi-institutional sample. Internet and Higher Education, 11(3-4), 133-136.

Baker, C. (2010). The impact of instructor immediacy and presence for online student affective learning, cognition, and motivation. The Journal of Educators Online 7(1). Retrieved from http://www.thejeo.com/Archives/Volume7Number1/BakerPaper.pdf

Boston, W., Diaz, S. R., Gibson, A. M., Ice, P., Richardson, J., & Swan, K. (2010). An exploration of the relationship between indicators of the Community of Inquiry framework and retention in online programs. Journal of Asynchronous Learning Networks, 14(1), 3-19.

Carini, R. M., Kuh, G. D., & Klein, S. P. (2006). Student engagement and student learning: Testing the linkages. Research in Higher Education, 47(1), 1-32.

Cercone, K. (2008). Characteristics of adult learners with implications for online learning design. AACE Journal, 16(2), 137-159.

Chickering, A. W., & Erhmann, S. C. (October, 1996). Implementing the seven principles of good practice in undergraduate education: Technology as lever.American Association of Higher Education Bulletin, 3-6.

Cobb, S. C. (2011). Social presence, satisfaction, and perceived learning of RN-to-BSN students in web-based nursing courses. Nursing Education Perspectives, 32(2), 115-119. Retrieved from http://www.nlnjournals.org/doi/abs/10.5480/1536-5026-32.2.115

Community College Research Center (CCRC) (2013). Creating an effective online instructor presence, Teachers College, Columbia University, 525 West 120th Street Box 174, New York, NY 10027. Retrieved from http://eric.ed.gov/?id=ED542146

Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7-23.

Garrison, D. R., & Cleveland-Innes, M. (2005). Facilitating cognitive presence in online learning: Interaction is not enough. American Journal of Distance Education, 19 (3) (2005), 133–148.

Gorsky, P., & Blau, I. (2009). Online teaching effectiveness: A tale of two instructors. The International Review of Research in Open and Distance Learning, 10(3). Retrieved from http://www.irrodl.org/index.php/irrodl/index.

Harrington, S. J, & Floyd, K. S. (2009). Enhancing engagement and the value of the course to the student through course organization and active learning. Online Classroom, November 2009.

Knowlton, D. S. (2000). A theoretical framework for the online classroom: A defense and delineation of a student-centered pedagogy. New Directions for Teaching and Learning, 84 5-14.

Kuh, G. D. (2003). What we’re learning about student engagement from NSSE: Benchmarks for effective educational practices. Change, 35(2), 24-32.

Kuh, G. D., Kinzie, J., Buckley, J. A., Bridges, B. K., & Hayek, J. C. (2006). What matters to student success: A review of the literature.
Commissioned Report for the National Symposium on Postsecondary Student Success: Spearheading a Dialog on Student Success. Retrieved from http://nces.ed.gov/npec/pdf/kuh_team_report.pdf.

Lehman, R. M., & Conceição, S. C. O. (2014). Motivating and retaining online students. San Francisco, CA: Jossey-Bass.

National Survey of Student Engagement (NSSE) (2013). Promoting high-impact educational practices: Maximizing educational gains. Bloomington, IN: Indiana University Center for Postsecondary Research.

Noel-Levitz (2013). 2013 national online learners priorities report. Coralville, Iowa: Author. Retrieved from www.noellevitz.com/Benchmark.
Park, J. H., & Choi, H. J. (2009) Factors influencing adult learners’ decision to drop out or persist in online learning. Educational Technology & Society, 12(4), 207-217.

Patterson, B., & McFadden, C. (2009). Attrition in online and campus degree programs. Online Journal of Distance Learning Administration. Retrieved from https://www.westga.edu/~distance/ojdla/summer122/patterson112.html.

Pollard, H., Minor, M., & Swanson, A. (2014). Instructor social presence within the Community of Inquiry framework and its impact on classroom community and the learning environment. Online Journal of Distance Learning Administration. Retrieved from https://www.westga.edu/~distance/ojdla/summer172/Pollard_Minor_Swanson172.html.

Rabe-Hemp, C., Woollen, S., & Humiston, G. S. (2009). A comparative analysis of student engagement, learning, and satisfaction in lecture hall and online learning settings. The Quarterly Review of Distance Education, 10(2), 207-218.

Richardson, J., & Swan, K. (2003). Examining social presence in online courses in relation to students’ perceived learning and satisfaction. Journal of Asynchronous Learning Networks, 7(1), 68-88.

Robinson, C. C. & Hullinger, H. (2008). New benchmarks in higher education: Student engagement in online learning. Journal of Education for Business, 84(2), 101-109.

Rovai, A. P. (2002). Sense of community, perceived cognitive learning, and persistence in asynchronous learning networks. Internet and Higher Education, 5, 319-332.

Rovai, A. P. (2003). In search of higher persistence rates in distance education online programs. The Internet and Higher Education, 6, 1-16.

Rovai, A. P., & Wighting, M. J. (2005). Feelings of alienation and community among higher education students in a virtual classroom. Internet and Higher Education, 8, 97-110.

Russo, T., & Benson, S. (2005). Learning with invisible others: Perceptions of online presence and their relationship to cognitive and affective learning. Educational Technology & Society, 8(1), 54-62.

Shea, P., Swan, K., Fredericksen, E., & Pickett, A. (2002). Student satisfaction and reported learning in the SUNY learning network. Elements of Quality Online Education, Needham, MA: Sloan-C.

Shea, P., Li, C. S., & Pickett, A. (2006). A study of teaching presence and student sense of learning community in fully online and web-enhanced college courses. The Internet and Higher Education, 9(3), 175-190.

Shea, P., & Bidjerano, T. (2009). Community of Inquiry as a theoretical framework to foster “epistemic engagement” and “cognitive presence” in online education. Computers and Education, 52(3), 543-553.

Spellman, N. (Spring, 2007). Enrollment and retention barriers adult students encounter. The Community College Enterprise, 63-79.

Tinto, V. (1997). Colleges as communities: Exploring the educational character of student persistence. Journal of Higher Education, 68, 599-623.

Tinto, V. (2004). Student retention and graduation: Facing the truth, living with the consequences. The Pell Institute for the Study of Opportunity in Higher Education, Occasional Paper 1, 1-16.

Tinto, V. (2006). Research and practice of student retention: What next? Journal of College Student Retention, 8(1), 1-19.

Tu, C. H., & McIsaac, M. (2002). The relationship of social presence and interaction in online classes. The American Journal of Distance Education, 16(3), 131-150.

Watwood, B., Nugent, J.,& Deihl, W. (2009). Building from content to community: [Re]Thinking the transition to online teaching and learning. Virginia Commonwealth University CTE White Paper. Retrieved from, https://www.azwestern.edu/learning_services/instruction/center_teaching_effect/resources/downloads/Online%20Teaching%20and%20Learning%20White%20Paper%20CTE%20Virginia%20Commonwealth%20University.pdf

Willging, P. A., & Johnson, S. D. (2004). Factors that influence students’ decision to dropout of online courses. Journal of Asynchronous Learning Networks, 8(4), 105-118.

Wyatt, L. G. (2011). Nontraditional student engagement: Increasing adult student success and retention. The Journal of Continuing Higher Education, 59(1), 10-20.

Zepke, N., & Leach, L. (2010). Improving student engagement: Ten proposals for action. Active Learning in Higher Education, 11(3), 167-177.



Online Journal of Distance Learning Administration, Volume XVIII, Number 2, Spring 2015
University of West Georgia, Distance Education Center
Back to the Online Journal of Distance Learning Administration Contents