Using the Technology Adoption Model to Assess Faculty Comfort with the Learning Management System
Lisa Anne Bove
University of North Carolina Wilmington
University of North Carolina Wilmington
When faculty need to use technology that is not familiar to them, it may take more time to integrate into to their teaching. The purpose of this study was to gather data on the ease of use and usefulness of the Blackboard LMS in anticipation of transitioning to a new LMS. A survey with the Technology Adoption Model (TAM) usability and perceived ease of use questions was sent to faculty in the Fall semester prior to the beginning of training for the transition to a new learning management system. Participation was voluntary and no questions were required.
Most participants completed the full usefulness and ease of use Likert-style questions, but many declined to fill out other questions about years teaching, title, age, or type of employment and comfort level. We obtained a 29 % response rate. Of those who received the survey, 48.5 % were Professors (full, assistant and associate) and 51.5 % were lecturers. Faculty averaged 13 years teaching with an average of 10 years at UNCW and most were between the ages of 44 and 65.
There was no significant association (α> 98%) between years teaching, title, age, or type of employment and usefulness or ease of use. Comfort level, however, demonstrated a negative association with both usefulness and ease of use. Faculty who identified themselves with lower comfort levels also reported that Blackboard was less useful and less easy to use. Based on the information gleaned in this survey, a full implementation plan including communication, opportunities for faculty buy-in for the plan, published timelines, and multiple options for training will be used to implement the new system.
When faculty need to use technology that is not familiar to them, it may take more time to integrate into to their teaching. Faculty willingness to adopt new technology in their practice varies (Buchanan, Sainter, & Saunders, 2013). In order to minimize the impact of a new Learning Management System (LMS) on teaching time, the University planned to develop multiple methods of training and practice for faculty. The campus has been with the current LMS, Blackboard, for the last nine years. The purpose of this study was to gather data on the ease of use and usefulness of the Blackboard LMS in anticipation of transitioning to a new LMS.
Learning Management Systems (LMS) provide a variety of tools and functions that include but are not limited to group chats, threaded discussions, document sharing, assignments, quizzes, grading and course evaluations to support teaching and learning. LMSs have evolved to support complex tasks such as peer evaluation and group work (Freire, Arezes, Campos, Jaxcobs & Soares, 2012). Freire et al. (2012) stated usability should be studied from the user perspective not the systems perspective. As with most technology, integration into practice is not dependent on just availability, but how the technology is embraced and utilized by the end user (Fathema, Shannon, & Ross, 2015).
Many prior LMS studies found that functions within an LMS were not equally used by the users (Akpinar, Bal, & Simsek, 2004; Jaschik & Lederman, 2014; Panda & Mishra, 2007; Weaver, Spratt & Nair, 2008). Fathema and Sutton (2013) found document uploading and grade posting were the most frequently used feature of Blackboard LMS. They reported that specific challenges included system problems and design flaws which reduced the overall utilization of the LMS by faculty. Barriers to adoption included lack of training, knowledge and skills, inadequate technical support and instructional design for elearning (Pajo & Wallace, 2001; Panda & Mishra, 2007). Weaver et al. (2008) reported that when using an LMS, system quality is important to both the students and faculty.
Technology Acceptance Model
Davis (1989) developed the Technology Acceptance Model (TAM) which has been used in numerous studies across various specialties and can be used to measure diffusion through acceptance of new technology (Lee, Hsieh, & Hsu, 2011). TAM questionnaires have been subjected to factor analysis to provide support for the content and construct validity of the subscales of usefulness (10 items) and ease of use (10 items) and demonstrated acceptable results (King & He, 2006).
TAM has been widely used as the theoretical basis for many empirical studies of user technology acceptance and is useful in predicting end-users’ acceptance of an e-learning system in an organization (Davis Bagozzi, & Warshaw, 1989; Arbaugh, 2002; Wu, Tsai, Chen, & Wu, 2006). TAM was derived to apply to any specific domain of human-computer interactions (Davis Bagozzi, & Warshaw, 1989). TAM postulates that two salient constructs, perceived usefulness and perceived ease of use, determine technology acceptance and are key antecedents of behavioral intentions to use information technology. Perceived usefulness is the “degree to which an individual believe that a particular system would enhance job performance” (Davis, 1989, p. 320). Perceived ease of use is the “degree to which an individual believes that using a particular system would be free of effort” (Davis Bagozzi, & Warshaw, 1989, p. 320). Along with the two constructs, two additional constructs contribute to actual use: attitudes toward using and behavioral intention to use. Perceived ease of use is directly related to perceived usefulness.
TAM postulates that if a user finds the technology easy to use then the technology is perceived as useful. If the technology is perceived as useful, then the end user will have a positive attitude toward using the technology and will lead to an intention to use the technology which leads to actual use or adoption (see Figure 1).
Figure 1. Technology Acceptance Model (TAM) (Davis, Bagozzi & Warshaw, 1989, p. 985)
Fathema et al. (2015) conducted a study on LMS adoption and found that system quality (e.g., usability, availability, reliability, adaptability, and response time) had a significant positive effect on the perceived ease of use and perceived usefulness of an LMS, which indicated that faculty place importance on the quality issues. The results also indicated that there was a positive effect of facilitating conditions (e.g., adequate guidance of LMS use, specialized instructions) on attitudes towards using an LMS.
A survey was sent to all University faculty at a university in the south who were identified by the Registrar's Office as teaching online or face-to-face Fall 2018 semester courses. There were a total of 688 full-time faculty and 305 part-time faculty who received the survey. Participation in the survey was voluntary, and we obtained a 29 % response rate. Of those who received the survey, 48.5 % were Professors (full, assistant and associate) and 51.5 % were lecturers. Faculty averaged 13 years teaching with an average of 10 years at UNCW and most were between the ages of 44 and 65 (see table 1).
Demographics. * Survey questions were not required, so not all participants completed every question.
The survey with the TAM usability and perceived ease of use questions was sent to faculty in the Fall semester prior to the beginning of training for the transition to a new learning management system. Participation was voluntary and no questions were required. Participants were asked to follow a URL to the survey from their recruitment email. The survey was completed on line using Qualtrics TM survey tool.
Questions included fill-in style for years teaching, age, and radio style for title, rank, type and number of courses, comfort level with the LMS. Likert-style questions were used for ease of use and usability based on the TAM. A final open-ended question allowed participants to describe advantages and/or challenges with the current LMS.
The data was collected anonymously, without specific names and with limited identifying characteristics such as years teaching, level, and experience with Blackboard. Survey results were compared only in aggregate. No individual information identifying faculty was used in the analysis. Data was maintained on a university owned password protected PC.
Faculty were emailed a survey link to the voluntary, anonymous electronic survey using the Qualtrics TM survey tool. The email included a description of the purpose of the survey, methods to maintain confidentiality and their role in the study. The survey link was open for two weeks in mid-October prior to the majority of the University sponsored new LMS training events. IRB approval was obtained prior to the start of the surveys.
Responses were uploaded into IBM SPSS Statistics 25® for analysis. Once the data was collected, descriptive statistics were used to compare the survey findings. Responses to the open-ended question on perceived advantages and challenges was only analyzed for themes. Data was aggregated to remove any identifying characteristics and included in a report for the LMS migration team so that issues could be addressed.
A factor analysis of usefulness and of ease of use showed 98%, demonstrating that this the TAM questions were a good scale. There was no significant association (α> 98%) between years teaching, title, age, or type of employment and usefulness or ease of use. Comfort level, however, demonstrated a negative association with both usefulness and ease of use. Faculty who identified themselves with lower comfort levels also reported that Blackboard was less useful and less easy to use. See Figures 1 and 2.
No survey questions were required, so there was missing data. Most participants completed the full usefulness and ease of use Likert-style questions, but many declined to fill out other questions about years teaching, title, age, or type of employment and comfort level. Table 2 depicts the descriptive statistics for the TAM survey. Overall mean for perceived usefulness was 2.25. For the perceived usefulness questions, none of the indicators were above 2.25. Only two indicators, ‘Using Blackboard improves my job performance’ and ‘Using Blackboard allows me to accomplish more work that would otherwise be possible’ had the highest mean both at 2.41.
Descriptive statistics for the TAM survey questions ease of use and usability)
At the end of the survey, participants were asked if they had additional information to share. Many of the comments support the data in Table 2. For example, the data above suggests that although faculty do not perceive Blackboard useful even though it has been used for nine years, they still find it “clunky”. Another participant stated “Bb’s inline grading is less functional”. A third participant stated “The biggest negative about blackboard is being unable to use it effectively on my table and phone.” Many participants who find Blackboard useful stated, “I am comfortable with Blackboard.” and “The main challenge would be that I now feel comfortable using Blackboard.”
With regards to ease of use, the highest means were with the following statements, ‘I feel it takes a lot of effort to become skillful with Blackboard’ and ‘interacting with Blackboard takes a lot of mental effort’. Both of these statements were supported through the qualitative data. One participant stated, “Blackboard is cumbersome and not user friendly or intuitive for students”. Another stated “When I was in the process of learning Blackboard, I believe A LOT of time was wasted because of having to figure out how to use the software verses being able to focus on building course content.” Another participant stated, “I feel I am more proficient in [new system] which I have been using for about five months than with Blackboard which I have been using for over ten years.”
The results of study showed that usefulness and ease of use are associated with faculty comfort with the technology. When Blackboard was first implemented nine years prior to this study, faculty were given training on how to use it. Ongoing training, however, was not offered, nor was Blackboard training part of new faculty orientation. Faculty were expected to learn how to use the LMS on their own, or bring those skills with them when they joined this university. The Dreyfus model of skill acquisition shows adult learners moving from novice, to advanced beginner, to competence, to proficiency, and potentially to expertise (Dreyfus, 2004). In order to move through these stages of learning, however, the student applies situational and non-situational experiences and instruction to the activity. Without any ongoing formal training opportunities, faculty may not be able to become more competent or even proficient with Blackboard and will just ‘get by’, therefore, will also not see the perceived usefulness in the technology. Increased training can assist with demonstrating how an LMS can assist faculty with time management and ability to connect with students (e.g. perceived usefulness). Some of the comments were directly related to the limitations of the technology (e.g. ability to use on mobile devices). This implies that faculty should have a stake in deciding the next LMS to ensure it meets the minimum needs of the faculty to ensure ease of use.
According to the latest study by Inside Higher Ed, faculty members teaching on line course is increasing with 44 percent in the current report as opposed to 30 percent in 2013 (Jaschik & Lederman (Eds), 2019). Thirty-three percent of faculty in this survey identified themselves as early adopters, whereas 55% only adopt after seeing their peers use the technology effectively and the rest reported they are not included to use the technology at all. Adoption was similar across age groups, gender and rank. Faculty were evenly split when asked if their college provided technical support for teaching online, but most did not feel they received adequate time or incentives for online course development (Jaschik & Lederman (Eds), 2019).
Faculty were surveyed while in the year of the transition to the new learning management system. Faculty who did not want to change LMSs may have been more positive about Blackboard. Since the survey questions were not required, many faculty did not respond to all the questions, so missing data may have skewed results. The survey focused on perceived ease of use and usefulness, but did not ask faculty about how they learned Blackboard, just their comfort level. In addition, all faculty were surveyed whether or not they were teaching fully online or just using Blackboard for basic activities such as syllabus and grading.
Of note, this University suffered a month long closure the semester this survey was collected. The decision was made to send the survey as planned since the training for the new learning management system schedule was not changed. This may have impacted survey response rate and overall ‘attitudes’ of respondents.
Based on the information gleaned in this survey, changes will be made to how the new LMS will be rolled out. Training will be offered for all levels of users, from novices on the new system to LMS experts. A full implementation plan including communication, opportunities for faculty buy-in for the plan, published timelines, and multiple options for training will be used to implement the new system. In addition, faculty will be asked to evaluate their experiences with training and with the new LMS in order to modify (if needed) ongoing training and orientation for new faculty.
Akpinar, Y., Bal, V., and Simsek, H. (2004). An e-learning content development system on the Web: BULMS. In Proceedings of the Fifth International Conference on Information Technology Based Higher Education and Training. (pp. 239-243). Istanbul, Turkey.
Arbaugh, J. B. (2002). Managing the on-line classroom: a study of technological and behavioral characteristics of web-based MBA courses. Journal of High Technology Management Research, 13, 203-223.
Buchanan, T., Sainter, P., and Saunders, G. (2013). Factors affecting faculty use of learning technologies: Implications for models of technology adoption. Journal of Computers in Higher Education, 25, 1 – 11. DOI 10.1007/s12528-013-9066-6.
Davis, F. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, 13(9), 319-340.
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: A comparison of two theoretical model. Management Science, 35(8), 982-1003.
Dreyfus, S. (2004). The five-stage model of adult skill acquisition. Bulletin of Science, Technology & Society, 24 (3), 177-181. DOI: 10.1177/0270467604264992. Fathema, N., Shannon, D., & Ross, M. (2015). Expanding The Technology Acceptance Model (TAM) to Examine Faculty Use of Learning Management Systems (LMSs) In Higher Education Institutions. Journal of Online Learning & Teaching, 11(2).
Fathema, N., Shannon, D., & Ross, M. (2015). Expanding The Technology Acceptance Model (TAM) to Examine Faculty Use of Learning Management Systems (LMSs) In Higher Education Institutions. Journal of Online Learning & Teaching, 11(2).
Fathema, N., Sutton, K. (2013). Factors influencing faculty members’ Learning Management Systems adoption behavior: An analysis using the Technology Acceptance Model. International Journal of Trends in Economics Management & Technology, Vol. II(vi), pg20-28.
Freire, L., Arezes, P., Campos, J., Jacobs, K., & Soares, M. M. (2012). A literature review about usability evaluation methods for e-learning platforms. Work 41, 1038-1044. doi: 10.3233/WOR-2012-0281- 1038
King, W. & He, J. (2016). A meta-analysis of the technology acceptance model. Information & Management, 43, 740–755.
Jaschik, S. & Lederman, D. (2014). The 2014 Inside Higher Ed Survey of faculty attitudes on technology: A study by Gallup and Inside Higher Ed. Washington, DC: https://www.insidehighered.com/news/survey/online-ed-skepticism-and-self-sufficiency-surveyfaculty-views-technology.
Jaschik, S. & Lederman, D. (Eds.) (2019). 2018 survey of faculty attitudes on technology: A study by Inside Higher Ed and Gallup. Retrieved from https://www.mediasite.com/wp-content/uploads/2018/11/2018-Faculty-Survey-Mediasite.pdf.
Lee, Yi-Hsuan, Hsieh, Yi-Chuan & Hsu, Chia-Ning. (2011). Adding Innovation Diffusion Theory to the Technology Acceptance Model: Supporting Employees' Intentions to use E-Learning Systems. Journal of Educational Technology & Society, 14(4), 124-137.
Pajo, K. & Wallace, C. (2001). Barriers to the Uptake of Web-based Technology by University Teachers. The Journal of Distance Education, 16(1), 70-84.
Panda, S., & Mishra, S. (2007). E-Learning in a Mega Open University: Faculty attitude, barriers and motivators. Educational Media International, 44(4), 323-338. doi: 10.1080/09523980701680854.
Weaver, D., Spratt, C., & Nair, S. (2008). Academic and student use of a learning management system: Implications for quality. Australasian Journal of Educational Technology, 24(1), 30-41.
Wu, J. P., Tsai, R. J., Chen, C. C., & Wu, Y. C. (2006). An integrative model to predict the continuance use of electronic learning systems: Hints for teaching. International Journal on E-Learning, 5(2), 287-302.