Determinants of Self-Reflective Learning and its Consequences in Online Master Degree Programs
Touro University Worldwide
Touro University Worldwide
Touro University Worldwide
Based on recent studies of self-reflective learning and its effects on various learning outcomes, this study examined the concept of self-reflective learning in the context of the Robust Learning Model (RLM), which is a learning model designed for improving the educational effectiveness of online degree programs. Two models were introduced to assess the efficacy of the self-reflective learning in the first course of the RLM with a spiral curriculum. The first model was aimed at examining the relative strengths of the various learning activities and the quality of faculty feedback in predicting the self-reflective learning at the end of the first course. The second model examined the lasting effect of the self-reflective learning at the first course on final program educational effectiveness. Findings from the first model indicated that self-reflective learning at the end of the first core course in the program curriculum was indeed directly influenced by student performance on threaded discussion, problem-based learning, and the signature assignment as well as by the quality of faculty feedback. In the second model's findings, there was a long, unique, and lasting effect of the self-reflective learning ability acquired in the first course on the student performance in the program, capstone, the student overall GPA, and reduction in the student time-to-degree. The implications of this overall study were discussed that may contribute to future research.
Self-reflective learning has gained growing recognition as an instrumental component of effective learning and learning outcomes (Altahhan, 2016; Arabi, 2013; Ellman & Fortin, 2012; Hinett, 2002; McPherson & Zimmerman, 2011; Moon, 2004; Pai, 2016; Palmer & Zajon, 2010; Siverman, 2008; Travers, Morisano, & Locke, 2015; Zimmerman, 2002). Self-reflective learning is defined as the student self-assessment of his/her performance relative to a standard of comparison (self-imposed goals, self-experience, peers, and outside groups) while using this comparison to improve his/her learning and to foster better learning outcomes and deeper learning (Altahhan, 2016; Arabi, 2013; Lonie & Desai, 2015; McPherson& Zimmerman, 2011; Neumann & Neumann, 2010, 2016; Tisngos Bosnic-Anticevich, Lonie, & Smith., 2015; Zimmerman, 1998). Moon (2004) argued that self-reflective learning is a necessary condition to improve one's learning outcomes. Palmer & Zajonc (2010) and Silverman (2008) suggested that the increase in online learner-centered education would increase the emphasis on self-reflective learning. Despite this assertion, self-reflective learning seemed to be ignored by most online learning models. Tsingos et al. (2015) reviewed the research on self-reflective learning and found 53 different studies using the self-reflective approach to enhance the learning performance. Most of these studies included only health disciplines, and 26 of these studies proposed or actually developed an assessment approached to the impact of self-reflective learning strategies. None of the aforementioned studies included online learning and online degree programs.
Self- Reflective Learning in the Robust Learning Model
TOne exception to the lack of presence of self-reflective learning in online education is the Robust Learning Model (RLM). The Robust Learning Model (Neumann & Neumann, 2010, 2016) and revised Robust Learning Model with Spiral Curriculum (Neumann, Neumann, & Lewis, 2017) included self-reflective learning as part of the overall online learning experience. Neumann & Neumann (2010) developed, implemented, and assessed the Robust Learning Model as the foundation for building multiple online degree programs as well as online universities. Neumann and Neumann (2016) initially engendered the RLM as the basis of the development of a forward-looking and sustainable online university. They developed a "multi-factorial model based on the basic belief that successful learning outcomes depend on multiple factors employed together in a holistic approach" (Neumann & Neumann, 2010, p. 28), which can be used to manage an entire university. The RLM was developed to enable systematic applications to all degree programs. The pedagogy of the programs was one of the basic factors developed in the RLM and included multiple levels: university learning outcomes, degree Program Learning Outcomes (PLOs), and course-level Student Learning Outcomes (SLOs). The main focus was on consistency across programs and courses, alignment of mission and goals of the university with those of the program, alignment of degree program learning outcomes with course level learning outcomes, and hierarchical and exhaustive at a rigor commensurate with the degree level. A faculty committee assessed the degree to which the totalities of the course SLOs rigorously lead to the attainment of the PLOs.
The courses were developed as a module-based approach across all degree programs. At the course level, students engaged in a variety of learning activities as follows:
Case Assignment (Problem-Based Learning)
Self- Reflective Essay
In the self-reflective essay, students reflected upon what they learned or how they performed in the course by comparing those self-assessments with their own expectations or goals. Students were to address five items ranging from how they felt the course improved their knowledge, skills, and abilities to a self-report of whether course SLOs were achieved. The University used content analysis to assess the achievement of the SLOs in the Self Reflective Essays.
According to Neumann & Neumann (2010), the last component in each course expressed throughout the various active learning engagements was the faculty feedback. The standards of conduct for faculty were:
Responsiveness (24-hour turnaround on email and 72-hour turnaround for grading assignments).
Flexibility with students on course and assignment deadlines.
Constructive and supportive feedback and communications with students.
Providing timely and constructive feedback (including text-based and audio) on students’ assignments for each module.
Grading of all assignments and submitting final grades.
Engaging students in meaningful learning through discussions with their peers and faculty.
Recently, the RLM was revised by introducing the spiral curriculum into the degree program's pedagogy. Harden (1999), building on the work of Bruner (1960), defined the following four steps as the main characteristics of the Spiral Curriculum:
1. Topics were revisited throughout the curriculum with increased complexity.
2. There were increasing levels of difficulty and/or depth throughout the curriculum.
3. New learning was related to previous learning.
4. The learner’s competence increased throughout the curriculum until the overall PLOs are achieved.
The first course in each degree program was designed as the main point of introduction to the subject matter. One criterion was to introduce the students, at a minimum, to 50% of the PLOs of the degree program. A degree map was designed for each program where subsequent courses were sequentially built where the PLOs are then further developed, practiced, mastered, and finally integrated. Specialty courses were the last part of the curriculum map and are organized into concentrations. In the final course (the Capstone) students were required to demonstrate the full mastery and integration of course PLOs. The result was a revised RLM with Spiral Curriculum, or in short, RLM-SC. Although all courses in the degree programs were important, the first and last courses played a critical role. The first course laid the framework and foundations of the degree program and key concepts. In the last course, students had to demonstrate understanding, comprehension, application, and integration of all the competencies and learning outcomes required for successful completion of the degree program.
Neumann et al. (2017), tested the full RLM-SC that was predicated on the pedagogical principle by repeating and increasing the complexity level of the program knowledge, comprehension, application, and mastery. Theend result was a fully interlinked curriculum with a cause-and-effect relationship from the first course through the final Capstone as well as various educational outcomes. The components of the first course in RLM-SC (Threaded Discussion, Case Assignment, Signature Assignment, Self- Reflective Essay, and the Quality of Faculty Feedback) were the independent variables that can predict an array of educational effectiveness outcomes (Retention Rates, Mastery of Competencies in the Capstone Course, Degree Completion, Time-To-Degree, and Overall GPA).
Self-reflective learning in the first course in the online master degree program's curricula was the dominant predictor of most of the educational effectiveness indicators in several multivariate models (Neumann et al., 2017).
The present study investigated the extent to which the curriculum’s first course's self-reflected learning was determined by the learning engagements and faculty feedback that preceded it. In other words, did the student performance on the first course's threaded discussion, problem-based learning, and an integrative signature assignment together with the faculty feedback influence the self-reflective learning? Further, which which of these aforementioned dimensions had a strong effect on self-reflective learning? The conceptual framework is summarized in Figure 1.
Figure 1.-Determinants of Self-reflective learning in the first course in RLM-SC
The second part of this study was to assess the unique contribution of self-reflective learning, by itself, on student performance in the integrative capstone course (the culminating learning experience where students are required to demonstrate the attainment of all PLOs), overall program GPA, and overall program time-to-degree (see conceptual map in Figure 2).
Figure 2. Separating the Unique Contribution of Self-Reflective Learning on Educational Effectiveness
All students in non-clinical online master degree programs who could have graduated under the revised RLM-SC within three years from their starting date until the end of fall 2016 were included in this study for a total of 397 cases. Although this was the total population for the revised learning model, it could serve as a sample for future students of the university as the university student demographics has been quite stable over time.
The variables for the first (and crucial) course in the degree program were measured as follows:
1. Threaded Discussion Performance (TDP) was assessed by the average grade that the instructor assigned to each of the student's required 16 unique postings throughout the course (twice per week). The grades (A-F) assessed the extent to which the student demonstrated: (a) the complete understanding, comprehension, and application of the key concepts and quality constructive feedback to other postings, (b) used citations to support opinions, interpretations, and facts, and (c) expressed new ideas in an articulated and concise form. The letter grades are then transformed to the regular (0 to 4) scale.
2. Case Study Performance (CSP) was assessed by a rubric comprised of six areas. The first area was completeness where the case analysis needed to be complete in all aspects and reflected all requirements. The second area was an understanding of the topics and issues covered in the case. The third area was analysis, evaluation and recommendations where the rubric examined the extent to which the case analysis: (a) represented an insightful and thorough analysis of all issues identified in the case, (b) made powerful connections among the various concepts, and (c) supported opinions with strong arguments and evidence while presenting a critical and objective interpretations. The fourth area was the extent to which research was incorporated into the case analysis. The fifth area was the accuracy and clarity of the case analysis. The last area was the completeness of the presentation, citations and bibliography. The CSP was assigned an A to F grade with the corresponding numerical interpretation.
3. The Signature Assignment Performance (SAP) was assessed by direct evidence that all the Student Learning Outcomes (SLOs) for the course (including skills and competencies) were fully demonstrated at all levels (knowledge, comprehension, application, critical thinking, communication and integration). The SAP was assigned an F to A grade with the corresponding numerical interpretation.
4. The Quality of Faculty Feedback (QFF) was a main differentiating factor in the Robust Learning Model. All university faculty members were trained on effective feedback and the Director of Quality Assurance regularly assessed their performance. In the revised RLM-SC, the role of the faculty was as crucial as in the traditional RLM. QFP performance was measured by four major factors:
A. Timeliness of the feedback on all required course assignments meaning if the feedback on threaded discussions were received within 24 hours, and the feedback on the case study and signature assignment were received within 72 hours.
B. The constructiveness of the feedback needed for effective student positive reinforcements.
C. The meaningful substance and clarity of the feedback.
D. The extent to which the feedback guided students in how to strengthen their learning efficacy.
If the assessment by an independent expert of each of the four items was "Yes" then the QFF score was 1, otherwise the QFF score was 0.
5. The Self Reflective Essay (SRE) was submitted at the end of each course. The students wrote self-reflective essays responding to 5 statements (items) that included:
A. Descriptions of how the student felt he/she improved their knowledge, skills, abilities, and self through the course.
B. Evaluation of the work the student did during the session and explanations of ways he/she could have performed better.
C. Topics that the student did not understand or applied suggestions students provided about how to improve the course materials on those topics.
D. Student's measured the future effects of what he/she learned in the course.
E. A student’s self-report of whether or not he/she achieved the course outcomes (Student Learning Outcomes).
Each of the five items was scored separately for the quality of the response. A score of 1 indicates a complete response, a fraction score for a non-complete response, and 0 score for missing the purpose of self-reflection. A self-reflective score was then created by summing up all the individual scores.
Three educational effectiveness indicators were used in this study:
1. Capstone Course Performance (CCP) was the student performance in their Capstone Integrative course at the end of their master degree program (see description above). Students in this course should have demonstrated an understanding, applied and integrated all the programs PLOs upon course completion. A rubric was developed to assess the depth to rigor to which students demonstrate the attainment of the degree PLOs acquired through core, advanced, and concentration courses. CCP was graded from F to A with its corresponding 0-4 numerical interpretation.
2. The GPA measure was the cumulative grade point average on a 0 to 4 range based on the A through F grade assessment.
3. Time-to-Degree was measured as the time between student's initial enrollment and degree completion.
The assessment of determinants of self-reflective learning was performed by a regression model where the measure of self-reflective learning was the dependent variable and the student academic performances in threaded discussion, case study, and signature assignment as well as the quality of the feedback were the dependent variables. In addition to beta coefficients, and R2 values, the contribution of each predictor to the explained variation was assessed by using the "fair-share" method (Neumann & Neumann, 1989). In each regression model, the fair-share partitioning of the explained variation (R2) "is equal to the sum of multiplication of each independent variable's weight with the correlation between the same independent variable and the dependent variable. Thus, unique contribution of each independent variable to the explained variation is simply the multiplication of the beta weight with the corresponding correlation coefficient. The fair share method is applicable only if the signs of the beta weight and correlation coefficient are the same" (Neumann & Neumann, 1989, p. 231).
The models included the educational effectiveness indicators (Capstone Course Performance, GPA, or Time-to-Degree) and were assessed by a two-stage multiple regression analysis needed to assess the unique contribution of self-reflective learning. In stage one, the four independent variables used to predict self-reflective learning were entered to the model. In the second stage, the self-reflective learning was entered to the model. The differences in R2 between stage one and stage two was the unique contribution of self-reflective learning on the specific dimension of educational effectiveness.
Table 1 presented the regression models including the first course's predictors of the self-reflective learning at the end of that course. Overall, the predictability (R2) of the self-reflective learning was quite strong (0.58; p < 0.01). In other words, the student performance on the threaded discussion, the case analysis, and the integrative signature assignment together with the quality of faculty feedback shaped and enhanced the student skill of self-reflective learning. All of the four components had a significant effect on the student self-reflective ability (each beta coefficient was statistically significant at p < 0.01). The student performance on the course signature assignment contributed 0.24 out of the total explained variation of 0.58. The student performance on the case study (problem-based learning) as well as the quality feedback on all assignments, is vitally important in determining the student’s self-reflective learning performance. Clearly, each of the determinants had a distinct role in developing the student's self-reflective learning.
The capstone course was the last course in the online master degree program. As presented in Table 2, the model without the self-reflective learning explained 59% of the capstone performance's variation. The dominant predictors for this model were the quality of faculty feedback and student's case study performance, both in the first core course of the degree program. The addition of the student's self-reflective learning in the first course increased the R2 from 0.59 (the model without SRL) to 0.79 (the model with SRL). The increment between the two models in the explained variation was significant (p < .01). The dominant determinants in the model with SRL were self-reflective learning and the quality of the faculty feedback.
The first course’s indicators predicted the program GPA in Table 3. The Model without SRL resulted in a relatively small R2 of 0.29 with quality of faculty feedback and student's performance on signature assignment having the largest effects on GPA. The R2 was substantially enhanced from 0.29 to 0.74 when SRL was included in the model. The added explanatory power by one variable was quite remarkable. In the second model, both self-reflective learning (the most dominant predictor) and signature assignment had a strong effect on the overall GPA.
The Unique Contribution of SRL to Explaining Time-to-Degree (Educational Effectiveness Indicator)
Time-to-Degree was the dependent variable in Table 4. The model without SRL resulted in a strong level of predictability (R2 = 0.48; p < 0.01). The Quality of Faculty Feedback was the most dominant predictor on reducing time-to-degree in the first model. Adding the SRL into the second model increased the predictability of the model from 0.48 to 0.63, with an increase that is also statistically significant (p < 0.01). The dominant predictors on reducing time-to-degree in the second model were self-reflective learning and quality of faculty feedback. Just to iterate, all independent variable were measured in the first courses of the degree program.
This study provided strong evidence about the importance of self-reflective learning in an online degree program in higher education. Self-reflective learning is a skill that can be developed and nurtured as part of the curriculum starting with the first course in a degree program. Engagement in a variety of learning activities and assignments (problem identification, problem solving, analytical tools, projects, discussions, critical thinking),enhanced self-reflective learning when a component of quality faculty feedback was added to each of those activities. This study further augmented the conclusions reached by Neumann & Neumann (2016). That is, engagement in a variety of learning activities and assignments improved self-reflective learning when the feedback received from the professor and/or other learners was immediate (less than 24 hours), constructive, substantial and (in the case of professor feedback) guided students in how to strengthen their learning efficacy. It was clear that self-reflective learning had become a cornerstone of the "learning-to-learn" ability (Neumann & Neumann, 2016). Learning-to-learn is the ability to persist in learning through student awareness of his/her learning needs, effective search for information, efficient time management, and the proactive use of support to overcome challenges. Students with high learning-to-learn ability will find greater success than students without this ability.
The self-reflective learning (gained in the first course of the curriculum) played a substantial role and impacted various program educational effectiveness indicators. This initial learned ability made uniquely strong contributions to enhancing capstone course performance and overall program GPAwhile reducing substantially the time-to-degree. In each of the program educational outcomes, self-reflective learning was by far the most dominant predictor. To further understand the role of self-reflective learning we examined the educational effectiveness indicators for two groups (students who exhibited a very high level of self-reflective learning in the first course compared to all others). The differences were striking. Students who measured above 4 (out of 5) on the SRL measure were defined to exhibit a very high level of self-reflective learning. Those with a high SRL received a mean grade of 3.94 on the final capstone compared to 3.24 grade received by those who scored below 4 on SRL. The overall mean degree program GPA for the very SRL score group was 3.94 while the rest of the students' mean GPA was 3.52. Finally, students with a very high SRL score graduated from their master degree program within 1.2 years while the other group graduated in 2.02 years.
Clearly, the overall quality of educational experience of the students in their first course (including their level of engagement and performance in a variety of learning activities) made a significant impact on their self-reflective learning ability at the end of the first course. In turn, the acquired self-reflecting learning from the first course made a lasting and pronounced effect on the overall program performance.
The results of this study have implications for faculty, leadership, and policy makers who are actively looking for ways to effectively and efficiently develop or improve their online degree programs. Clearly, the paths for improving online degree program's educational effectiveness received unequivocal input from this study.
Altahhan, A. (2016, July). Self-reflective deep reinforcement learning. In Neural Networks (IJCNN), 2016 International Joint Conference on (pp. 4565-4570). IEEE.
Arabi, E. (2013). Enhancing Learning Through Discussion Forums and Blogs: Personalized Learning & Self-Reflection. In ICERI2013 Proceedings (pp. 5454-5459). IATED.
Bruner J. S. (1960). The process of education. Cambridge, MA: Harvard University Press
Ellman, M. S., & Fortin, A. H. (2012). Benefits of teaching medical students how to communicate with patients having serious illness: comparison of two approaches to experiential, skill-based, and self-reflective learning. The Yale journal of biology and medicine, 85(2), 261.
Harden, R. M. (1999). What is a spiral curriculum? Medical teacher, 21(2), 141-143.
Hinett, K. (2002). Improving learning through reflection – part one. In the higher education academy.
Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.604.5051&rep=rep1&type=pdf
Lewis, S. & Ewing, C. (Winter 2016) Assuring student learning outcomes achievement through faculty development: An online university example.Online Journal of Distance Learning Administration, 19(4). Maynes, N. & Hatt, B. E. (2013). Online learning: Growing into our capacity for flexible learning. Journal of Transformative Entrepreneurship, 1(2), 83-92.
Lonie, J. M., & Desai, K. R. (2015). Using transformative learning theory to develop metacognitive and self-reflective skills in
pharmacy students: A primer for pharmacy educators. Currents in Pharmacy Teaching and Learning, 7(5), 669-675.
McPherson, G. E., & B. J. Zimmerman. “Self-Regulation of Musical Learning: A Social Cognitive Perspective on Developing Performance Skills.” In Vol. 2 of MENC Handbook of Research on Music Learning, edited by R. Colwell and P. R. Webster, 130-175. New York: Oxford University Press, 2011.
Moon, Jennifer A. Reflection in Learning and Professional Development: Theory and Practice. Reprint, New York: RoutledgeFalmer, 2004.
Neumann, Y., & Finaly-Neumann, E. (1989). An organizational behavior model of students' evaluation of instruction. Higher Education, 18(2), 227-238.
Neumann, Y. & Neumann, E. (2010). The Robust Learning Model (RLM): A Comprehensive approach to a new online university. Journal of College Teaching & Learning, 7(1), 27-36.
Neumann, Y. & Neumann, E. (May 3, 2016). Lessons about online learning. Inside Higher Ed.
Retrieved from https://www.insidehighered.com/views/2016/05/03/what-weve-learned-after-several-decades-online-learning-essay
Neumann, Y., Neumann, E.F., & Lewis, S. (February 2017). The Robust Learning Model with a Spiral Curriculum: Implications for the Educational Effectiveness of Online Master Degree Programs. Working Paper Series on Online Learning, 2017-1.
Pai, H. C. (2016). An integrated model for the effects of self-reflection and clinical experiential learning on clinical nursing performance in nursing students: A longitudinal study. Nurse Education Today, 45, 156-162.
Palmer, P. J., & Zajonc, A. (2010). The heart of higher education: A call to renewal. San Francisco: Jossey-Bass
Silverman, E. (2008). Ongoing self-reflection. American Journal of Speech-Language Pathology, 17(1), 92.
Tsingos, C., Bosnic-Anticevich, S., Lonie, J. M., & Smith, L. (2015). A model for assessing reflective practices in pharmacy education. American journal of pharmaceutical education, 79(8), 124.
Travers, C. J., Morisano, D., & Locke, E. A. (2015). Self-reflection, growth goals, and academic outcomes: A qualitative study. British Journal of Educational Psychology, 85(2), 224-241.
Zimmerman, B. J. (Spring 2002). Becoming a self-regulated learner: An overview. Theory into Practice, 41(2), 64-70.
Online Journal of Distance Learning Administration, Volume XX, Number 1, Spring 2017
University of West Georgia, Distance Education Center
Back to the Online Journal of Distance Learning Administration Contents