Continuous Improvement in Online Education: Documenting Teaching Effectiveness in the Online Environment Through Observations

Jennifer W. Purcell
Kennesaw State University
jpurce10@kennesaw.edu

Heather I. Scott
Kennesaw State University
hscott18@kennesaw.edu

Deborah Mixson-Brookshire
Kennesaw State University
dmixson@kennesaw.edu

Abstract

Teaching observations are commonly used among educators to document and improve teaching effectiveness. Unfortunately, the necessary protocols and supporting infrastructure are not consistently available for faculty who teach online. This paper presents a brief literature review and reflective narratives of educators representing online education at multiple organization levels within a comprehensive university.  Each vignette presents strategies for implementing observations among online faculty, including considerations for teaching faculty, their peers and supervisors, and online education administrators.


Introduction

Booth, Booth, and Hartfield (2009) concluded that iterative cycles of feedback from multiple stakeholder audiences is necessary for the continuous improvement of online courses and pedagogy.  Although their concluding thoughts are seemingly superficial and expected, the reminder cannot be overstated.  Indeed, our experiences, observations, and conversations with colleagues suggest a tendency among educators to limit course and teaching feedback to student performance indicators and student course evaluations.  Although we commonly find course evaluations that address quality assurance in online course design, it is critical that we distinguish active teaching techniques from static course design (Tobin, Mandernach, & Taylor, 2015). 

Teaching observations serve multiple functions for educators (Chism, 1999).  The observations may serve as a formative evaluation of the faculty members’ teaching strategies.  This use of teaching observation encourages reflection and self-assessment as a measure of professional development.  In addition to utilizing the teaching observation as a data point for self-evaluation, teaching observations may also be utilized by academic units to identify professional development and training needs among the faculty.  For example, online teaching observations could affirm the faculty member’s utilization of technology in the course or provide an opportunity for such recommendations.  Suggestions for online teaching improvement could reference the variety of available technologies, including screen capture and video recording software, educational social media tools, and expanded communications technologies that integrate with existing learning management systems (LMS).  Such formative feedback has potential for improving online teaching effectiveness.  Additionally, research suggests teaching online influences instructor’s teaching strategies in traditional classroom settings.  Therefore, teaching observations across multiple deliveries may provide a more holistic understanding of instructors’ strengths and weaknesses related to teaching (McQuiggan, 2012).

Research on online teaching suggests a differentiation between teaching personas exists among instructors who teach through varied delivery modes.  For example, Richardson, Koehler, Besser, Caskurlu, and Mueller (2015) note that instructors may have distinct presentations of themselves in social and instructional elements of the course.  Whether intentional or not, these distinctions may have a negative influence on student-instructor interaction as the student attempts to discern acceptable norms of engagement.  Baran and Correia (2014) advocate for targeted professional development, such as that identified via online teaching observations, to enhance teaching effectiveness.  Despite initial reluctance to pilot online content delivery, faculty members routinely express confidence and satisfaction with online teaching (Dietrich, 2015).  It is important for current online educators to document their success and refine their practice in order to improve and sustain quality online education.  Online teaching evaluations can be conducted by a single expert or a team of colleagues depending upon an instructor’s preference and the availability of reviewer resources.  Depending upon the context, motivations, and intended purpose for the observation, observers may be peers, supervisors, or a combined team.  Regardless of the composition of the observing team, clear communication between the instructor being observed and the observers will help to ensure the process is constructive (Tobin et al., 2015).

Educator Perspectives

This paper presents reflective narratives of three educators who teach online and have varying levels of responsibility for online learning administration.  These narratives include perspectives from a tenure-track faculty member, an academic program director, and an assistant dean who oversees distance learning for an entire college and advises institution-wide policy related to distance learning.  The purpose of the multiple perspectives is to present the unique insights representing various constituent groups whose buy-in is necessary for implementing a sustainable culture of online teaching observations.

Faculty Level Perspective

Multiple professional development workshops offered by my university for teaching online helped prepare me to develop and teach courses online.  These sessions were optional; however, I was hired with the understanding that I would further expand online course offerings in my department.  This intentionality in the hiring selection process matched the department’s strategic goals with my experience and interest.  As a former hybrid and online learner in both of my graduate programs, I naturally gravitated toward online teaching opportunities and welcomed the opportunity to develop and teach online courses.  I personally experienced the advancement opportunities and benefits afforded by distance education programs, so I value the access provided by online courses to student populations who would otherwise be enable to pursue a college degree.  Fortunately, online teaching is valued at my institution as evidenced by support units, incentives, and recognition.  I imagine the quality of my online teaching would be different without such tremendous institutional support.  For example, we have an internal process for reviewing online courses based upon the Quality Matters rubric.  I have developed three courses that were refined based upon the feedback I received during the course review.  Additionally, I serve as a peer reviewer and have learned from colleagues’ course designs.  

During my first year, my department chair completed a teaching observation in one of my traditional, face-to-face courses, which was helpful in affirming my teaching practice and identifying areas for improvement.  In subsequent years, I found myself teaching predominantly online and began exploring online teaching observation protocols.  I requested an observation of my online teaching because I wanted feedback on my teaching, to document what I did well in the online environment, and to be intentional in seeking out additional professional development to improve my online teaching practice.  At the time, there was no formal process for online teaching observations that mirrored the existing protocol for traditional teaching observations, so I collaborated with my program director to develop and pilot an instrument and protocol to facilitate the process.  My goal was to addresses the aforementioned needs I had as an online educator and to also pilot a process that could be shared with colleagues and potentially adopted by my program and department.

To date, the protocol and instrument my program director and I developed has been disseminated among colleagues at my institution, with colleges in my discipline at national conference, and with fellow online educators a national conference for online learning.  Most importantly, the feedback I received through the online teaching observation was helpful in developing strategies to improve my teaching practice and documenting areas of excellence.  I believe these observations should be faculty-driven and formative.  I also think there are strategies for program directors and senior academic administrators to help cultivate cultures of evaluation that support online teaching observations even among faculty who are reluctant to complete an online teaching observation.  Additionally, the information collected during online teaching observations may serve a variety of purposes that support student learning by informing practices at the program and college level.  In no way should the process be punitive and any data collected should require consent to opt-in.

Academic Program Level Perspective

In rapidly growing online learning programs adjunct faculty support is being incorporated at greater rates than ever before. This incorporation and the rapid growth of programs highlight the need for ensuring quality online instructors and consistency amongst program course offerings. A critical starting point is course development. Online course development done well creates a strong base for a program. Through the use of Quality Matters, our institution has a robust and quality program that supports the creation of master courses that can become a shared space for course development for adjunct and full-time instructors to engage in developing enhanced instructional opportunities.  The collegial setting in which Quality Matters takes place provides a respected and meaningful place for exploring well regarded teaching practices. As a program director, I see this space as a critical component to providing learners with a quality experience that supports learners at a variety of levels. The programmatic consistency offered in this type of program model provides learners with a certain familiarity and comfort with online learning.

A strong augment for this type of course development is the continued practice of assessment of program and instruction quality. The evaluative process of learning tools and instructors keep a program director in tune with areas of achievement and areas for program growth and improvement. The practice of teaching observations in the online space can support the validity of online programs and highlights the strong pedagogical practices that a number of online instructors engage in with their online instruction. The evaluation process and resulting findings can also become an opportunity for program benchmarking and developing measures for growth. A well thought out an implemented assessment plan can provide meaningful data rather than relying on memory or anecdotal notions about successful online learning ventures.

As a program recalibrates its focus, a historical account or overview of effective instructional methods determined in part by observational review of teaching practices can assist with defining or refining a program’s focus. Programs have the opportunity to determine what their core values are as it relates to instruction and can develop opportunities to share what some instructors find success with practicing in their online learning environments and what areas may be challenging. When such observations are performed at a programmatic level, programs may begin to see patterns for success or growth.

College and University Level Perspective

Administrators of online education programs are constantly looking for innovative ways to conduct classroom teaching observations for their faculty members that allow for developmental and formative feedback.   This feedback is imperative to facilitate the online faculty members’ professional and personal development and to provide them with strategies to enhance their online teaching skills.  Moreover, this feedback often contributes to an overall improved classroom experience for the students.  Thus, formative observations can provide faculty members with not only positive feedback that encourages their best practices but also constructive criticism that promotes enhancements to their online teaching capabilities.

As administrators, we have encountered challenges in conducting teaching observations of the online course in our college, particularly in the areas of implementation and identifying knowledgeable and experienced online observers from among our teaching faculty.  The primary difficulty regarding implementation is that online faculty members are often reluctant to invite observers into their classrooms when the institution does not require observations.  Additionally, faculty members sometimes view the classroom observations process negatively, fearing them to be a high stakes assessment. Identifying knowledgeable and experienced online faculty observers is a challenge because they may have experience teaching online, but many faculty would like a guide provided to ensure a fair and accurate observation for their fellow faculty peers. Conversely, administrators may not have online teaching experience but have observation experience which can be an additional challenge to ensure the observer is knowledgeable and experienced.  Although some faculty members in our college actively seek out opportunities to improve through a developmental and formative teaching observation process, we must ensure the process – and faculty perception of the process – is collegial and supportive.

Promoting a culture of observation and continuous improvement requires an administrator to engage in assessment activity at all levels within the college.  If college administrators engage actively in assessments to enhance quality and provide continuous improvement across programs, faculty will see the positive impact of implementing assessments to enhance teaching and the student experience.  To ensure the feedback provided though online teaching observations is both meaningful and constructive, administrators of online education programs should provide training to ensure those conducting the observations are experienced and knowledgeable regarding the online environment and effective teaching in that environment.   For example, training workshops can be offered throughout the year to provide faculty members the opportunity to observe the evaluation process from multiple perspectives.  Through a cultural shift involving assessment, the workshops could be a part of the promotion and tenure training offered for faculty and administrators to engage faculty in the continuous improvement of teaching through sharing the value of formative assessments.

Discussion

Online teaching observations are valuable tools for documenting and improving teaching effectiveness.  Faculty and administrator buy-in is necessary for institutionalizing the process, and advocates for online teaching observations are advised to consider a multi-pronged approach that engages teaching faculty, academic program directors, and senior administrators.  To encourage faculty members to participate in the online classroom teaching observation process, the following strategies can be implemented:

  1. Convey to faculty the value of a developmental and formative observation through
    • The testimonials of faculty who have participated in the process
    • The benefits as documented in the literature on teaching practices
    • An explanation of the importance of this feedback as part of the evidence provided in promotion and tenure portfolios
  2. Provide faculty members with a detailed instrument that will be used to observe the online classroom in order to allow them the opportunity to review it and ask questions
  3. Provide faculty members with the option to choose one or more observers from a list of available, knowledgeable, and experienced online evaluators
  4. Promote the formative evaluation as a supplement to the KSU Quality Matters internal review process

Furthermore, when implementing an online teaching observation process, the following should be taken into consideration:

  1. Online teaching faculty should have the opportunity to share their perspectives regarding online teaching and the observation instrument to garner their buy-in for the process
  2. Online program administrators should evaluate the opportunities and challenges of implementing an online classroom teaching observation process from all perspectives
  3. Online program administrators should consider their organizational structure and culture before institutionalizing observations
  4. Online program administrators should ensure proper protocols and infrastructures are in place to support the process
  5. Data ownership should be clear and respect faculty members’ willingness, or lack thereof, to contribute to ongoing data collection

While the online teaching observation process is valuable for full-time faculty, it can also prove useful as a means to provide formative feedback to part-time, temporary, and adjunct faculty members.   The process may be time intensive, but the formative observations for part-time, adjunct, and temporary faculty could be a tool to ensure quality of material delivery and provide feedback promoting enhanced teaching capabilities. Providing a formative assessment to part-time, adjunct, and temporary faculty can be, a part of full-time faculty professional service to the department and can be included in annual reviews.  The following should be considered to ensure effective means of assessment:

  1.  Chair or director of department provide an annual formative assessment to all part-time, temporary, and adjunct faculty teaching online
  2. Program coordinator or director within the department provides an annual formative assessment to all part-time, temporary, and adjunct faculty teaching online
  3. All full-time permanent faculty with online teaching experience within the department are required as part of their professional service to the department to engage in a certain number of formative assessments for the part-time, temporary and adjunct faculty employed within their department who teach online
  4. Provide guidelines and training for peer-to-peer formative observation assessments for part-time, temporary, and adjunct faculty members

Providing part-time, temporary, and adjunct faculty members with a formative assessment annually is an opportunity for positive and constructive feedback to enhance their teaching capabilities.

Personal experiences and anecdotal data prompted our exploration of online teaching evaluation, and there is opportunity to further explore the motivations and barriers to online teaching observation through empirical research.  Additionally, case studies that delve into the unique contexts and experiences across institution sizes and type would provide insight for practitioners seeking to implement online teaching observation instruments and protocol.  We also encourage professional learning communities to support the continuous improvement of online teaching.  Such learning interventions provide peer support and promote cultures of reflection, sharing, learning, and growth.

Conclusion

Online teaching observations are a necessary element of online education and distance learning administration.  With the end goal of providing professional development and strategies to improve teaching, the online teaching observation process, whether required or optional must be viewed positively by faculty if it is to be embraced and implemented successfully.   Online program administrators would be prudent to cultivate a culture of continuous improvement among their faculty and secure their support as a first-step in the implementation process.  Additionally, practitioner-oriented scholarship that explores strategies, challenges, and successes in creating systemic, organizational change to support online teaching observation would be helpful for both faculty and administrators seeking to implement the recommendations provided in this paper. 

Our first-hand experience of the positive impact of online teaching observations for faculty and their students motivated us to advocate for the process in our college and champion the work more broadly within our institution.  Mostly recently, our center for teaching excellence began promoting an online teaching observation process that validated and reiterated the efforts underway in our college.  We encourage willing faculty to model best practices by participating in online teaching observations as both an observer and teacher and sharing their experiences as a means of promoting a culture of continuous improvement and encouraging colleagues to reflect on their own online teaching practice.

References

Baran, E., & Correia, A. (2014). A professional development framework for online teaching. Techtrends: Linking Research & Practice to Improve Learning, 58(5), 95-101.

Booth V., Booth L., & Hartfield, F. (2009). Continuous course improvements, enhancement, and modifications: Control and tracking. Online Journal of Distance Learning Administration, 12(2).

Chism, N. V. N. (1999). Peer review of teaching: A sourcebook. Bolton, MA: Anker.

Dietrich, D. C. (2015). Observations of a Reluctant Online Instructor: Transitioning from the classroom to the computer. College Teaching, 63(3), 93-98.

McQuiggan, C. A. (2012). Faculty Development for Online Teaching as a Catalyst for Change. Journal of Asynchronous Learning Networks, 16(2), 27-61.

Nicolson, M., & Harper, F. (2014). Online Peer Observation: An Exploration of a Cross-Discipline Observation Project. International Journal of Teaching and Learning in Higher Education, 26(2), 251-259.

Richardson, J. C., Koehler, A. A., Besser, E. D., Caskurlu, S., Lim, J., & Mueller, C. M. (2015). Conceptualizing and Investigating Instructor Presence in Online Learning Environments. International Review of Research in Open and Distributed Learning, 16(3), 256-297.

Tobin, T. J., Mandernach, B. J., & Taylor, A. H. (2015). Evaluating Online Teaching.  San Francisco, CA: Jossey-Bass.

Online Journal of Distance Learning Administration, Volume XX, Number 4, Winter 2017
University of West Georgia, Distance Education Center
Back to the Online Journal of Distance Learning Administration Contents