Quality Assurance in Large Scale Online Course Production
Embry-Riddle Aeronautical University
The course design and development process (often referred to here as the “production process”) at ERAU-Worldwide aims to produce turnkey style courses to be taught by a highly-qualified pool of over 800 instructors. Given the high number of online courses and tremendous number of live sections running at any given time, maintaining quality was a significant concern. The model of faculty instructors each producing and delivering their own online course would make achieving consistency in design and delivery difficult. While this production model is common at many schools, it is known to be ineffective (Bates, 2000, Laird, 2004, Chao, Saj, & Hamilton, 2010), so at ERAU-Worldwide it was decided to centralize the process using a collaborative course production team, administered through the Instructional Design and Development (IDD) department. Over time, a process evolved that ensures healthy collaboration among production team members and meeting quality standards based on sound learning, teaching and instructional design theoretical foundations, both factors cited as key influences on the success of distance learning efforts (Chao, Saj, & Hamilton, 2010). Several ERAU-Worldwide courses have won industry awards, including Quality Matters and Blackboard’s Exemplary Course Program, and the influence of the external course review and awards process on the establishment of course design and development quality standards will be addressed below in detail.
This paper will focus on the purpose (quality assurance) to which we strive for achieving excellence; the philosophy (theoretical foundations) which informs our practice; the physical resources (production team, infrastructure, administration, and organizational culture) that allow us to achieve our goals and, at the same time, may also pose tangible constraints; and the process that enables us to realize our product: high-quality online courses that meet student demand and fulfill learner needs.
With the tremendous demand for online learning at ERAU-Worldwide, client (student) needs drive all our efforts and providing students with the highest quality, accredited education is our top priority. The production process model below that lies at the center of the illustration will be presented in a detailed format customized to the ERAU-Worldwide online course production process later in this of the paper, after reviewing the each of the supporting (theoretical) and environmental elements (student needs and demand, quality assurance, organizational culture, and physical resources) that influence the course design/production process.
Figure 2. ERAU-Worldwide Online Course Design and Production Model
This paper will conclude with best practices that can be applied to the course design process used at any institution, even smaller organizations working with much more limited resources.
A. History and Challenge
Online learning began at ERAU-Worldwide in 1993 as part of a hybrid program that combined video and a homegrown online bulletin board system for interaction. WebCT was the first fully-online learning management system, adopted in the late 1990’s, followed by Blackboard in 2000. As mainstream online learning grew rapidly in popularity, there was tremendous concern for the quality of the educational experience (van Damme, 2002, Sloan Consortium, 2004, Newton, 2007, Abdous, 2009), and this is especially true in the case of ERAU-Worldwide, given the current and future potential size and geographical breadth of the student body. For ERAU-Worldwide, the challenge was to produce, deliver, and maintain online courses with the highest quality possible to thousands of primarily non-traditional students all over the world. ERAU-Worldwide has not been alone in coping with this problem. Higher education institutions typically do not have the resources needed to produce online learning to meet demand efficiently and effectively (Sims, Dobbs, & Hand, 2002, Royal, 2007). Online learning development frequently suffers from a lack of resources, particularly infrastructure, policy and support mechanisms, and is conducted under pressure to quickly meet growing demand. As a result student attrition rates are high and faculty blame the administration for failure to provide adequate resources. In addition, faculty are subject matter experts, but not always instructional design experts, and having a lack of instructional design expertise, especially specific to online learning, is seen as a significant cause of failure in an online learning program. Unfortunately, many educational institutions still follow this model of faculty as subject matter expert, instructional designer, content producer, learning management system developer, and student support, from admissions and technology troubleshooting (Frith and Kee, 2003, Escoffery et al., 2005, Tellen-Runnels, et al., 2006, Meyer & Barefield, 2010). This is sometimes referred to as the “lone wolf” (Laird, 2004) or the “lone ranger” (Chao, Saj, & Hamilton, 2010) model of course production and delivery. This approach has proven to be not scalable and does not lend itself to the diffusion of innovative practice in an organization (Bates, 2000, p. 2, in Chao, Saj, & Hamilton, 2010).
Adding to the pressure, by 2005, quality assurance standards for online learning had been implemented by most accrediting bodies and many of the other significant educational organizations that monitor and evaluate educational providers, from ERAU’s accrediting agency, the Southern Association of Colleges and School to the United Nations Educational, Scientific and Cultural Organization (Abdous, 2009).
ERAU-Worldwide recognized, and wisely so, that the “lone wolf” approach (Laird, 2004) was not going to be sufficient for their model of delivery and that upfront investment in sufficient resources would be required to convert to a centralized approach. Restauri (2004) contends that a second model, one that creates this centralization, utilizes a collaborative team approach to instructional design, and is much more effective and efficient at producing quality in online learning and receives better faculty buy-in once proven. Instead of the faculty member performing all the roles above, a division of labor is established in which experts trained in each specialty come together and apply best practices and a sound instructional design process to collaboratively produce and maintain online courses, all in an environment that effectively supports the information technology infrastructure. (Frith and Kee, 2003, Escoffery et al., 2005, Tellen-Runnels, et al., 2006, Daniel, 2009, Chao, Saj, & Hamilton, 2010, Meyer & Barefiled, 2010, Parscal & Riemer, 2010). This is referred to by Laird (2004) as the integration model: all institutional resources cooperate to produce online learning aligned with the organization’s mission and goals. Research (Paolucci & Gambescia, 2007) has shown this centralized model to be the most cost-efficient while producing the highest quality courses in which faculty can satisfactorily focus on teaching (Meyer & Barefield, 2010). This approach is being used on a large scale with success at other universities, such as Royal Roads University in British Columbia, Canada (Chao, Saj, & Hamilton, 2010). A similar centralized solution, adopted in 2002 in its earliest rendition at ERAU-Worldwide, came in a multifaceted form involving several key components: a collaborative production team with clearly identified division of labor; a flexible instructional design process; a master template and course shells management strategy; a system of checks and balances created through a highly constructive reviews process; and responsive maintenance, all with quality at the heart of the model.
C. The Art and Science of Course Design
For many, being confounded by the sometimes overwhelming stresses imposed on the production process by external factors, a formulaic approach was long ago abandoned and the resulting philosophy towards course design and development became a mix of art and science (Kobeleva & Strongman, 2012). Instructional design is a “creative, active and iterative” (Gustafson & Branch, 2002, pg. 11) process that is both complex and organic. But yet, instructional design is the “linking science” that uses orderly procedures and research-based approaches in an attempt to solve the problem that is designing effective instruction (Tennyson, 2010, pg. 1; Rothwell and Kazanas, 2004). In fact, it was once said that instructional design “ill-structured problem solving” (Jonassen, 2002, p. 117, in Dicks and Wright, 2008) and that the efficacy of any instructional design model is clearly in doubt given the general lack empirical testing. Willis and Wright (2000) declared instructional design to be more an art than a science in that it is more than “the correct application of technical recipes” (p.5, Willis and Wright, 2000). Ultimately, course design and development is likely both an art and a science: a non-linear, collaboration-dependent team creation (Kenny et al., 2005, Botturi, 2008). Regardless, any sound practice must be based upon sound theory. While this paper is not an exhaustive review of instructional design theory and its underlying theoretical foundations, a solid review of the instructional design, instructional strategy, and learning theories that influence the course design philosophy at ERAU-Worldwide is appropriate at this juncture.
Educational theory in modern times began first with learning theories – how do humans learn? Then, the next logical progression in theoretical development for education was the formulation of teaching, or instructional, theories. Finally, as the design of teaching and learning came into its own as a viable profession, so too did instructional design theories. Each theory built off of its predecessors, but took into account the current events, societal trends, and technological innovations of their time. This section takes a look at each of the three theoretical categories (learning, teaching, and instructional design) and discusses how they have influenced our philosophy of online teaching and learning and informed our practical application of instructional design in our course production process at ERAU-Worldwide.
A. Learning and Instructional Theory
Until the constructivist movement took a central place in schools of educational thought in the latter part of the 20th century, behaviorism, attributed primarily to Skinner (1958) and its offshoots, namely cognitivism, dominated educational philosophy and informed most instructional design. Behaviorism(Skinner, 1958) focuses on practice, stimulus and response, reinforcement, reward and punishment, and task-based learning as key to effective teaching and learning. The attributes of behaviorist theory still applicable to today’s instructional design include: goal identification, needs assessments, chunking of learning content/learning objects and providing feedback and reinforcement. Keller’s Personalized System of Instruction (Keller, 1968), a behaviorist-based system designed to interlock and sequentially order tasks, contributed the concepts of learning objectives and self-paced modules. (Burton, Moore, & Magliaro, 1996). Over time as we moved into the post-modern period in human history, the popular view of learning and teaching became more humanistic, and Keller’s approach is symptomatic of this change. Slowly moving away from the mechanistic views of behaviorsim, the first characteristic shift was that of putting the learner’s needs at the center of the process, and this became evident with the rise of cognitivism. Common attributes of learner-centered, cognitivist-based approaches included guided discovery; activating prior knowledge; encoding, storage and retrieval; reasoning and problem-solving. Considered a cognitivist, Reigeluth’s innovative (1979) elaboration theory provided a systems approach to constructing learning by providing a broad view of a concept, then providing opportunities for detailed examination of subparts, always linking them back to the greater view in which the subparts live. Sequencing lessons, providing reviews, using concrete, relevant examples, and transitioning from lower-order skills to higher-order skills during the progression of the course are all features adopted from elaboration theory that fit our model today. Finally, Reigeluth believed that meaningful context was required in order for students to assimilate new learning, which was another innovative precursor to constructivist thought. Another cognitivist, Robert Gagné (1985), offered an influential theory of instruction focusing on three areas: the taxonomy of learning outcomes (cognitive, affective, psychomotor), the conditions of learning (performance verbs), and the nine events of instruction, which further broke down and ordered the learning process with a greater focus on recalling prior learning and requiring performance of learning to demonstrate evidence of integration with existing knowledge. The Gagné approach to learning outcomes is closely related to Bloom’s Taxonomy (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956), which is the standard system for developing learning objectives used in the ERAU-Worldwide IDD department. Evident that by the late 1980’s educators had moved to the far opposite end of the spectrum away from behaviorist beliefs, models such as Keller’s (1987) ARCS Model of Motivational Design began to see popular acceptance. ARCS gave us an affective-heavy framework that sought to bolster student self-esteem in learning through techniques of gaining attention (A), providing relevance (R), boosting confidence (C), and engendering satisfaction (S). The structure of ERAU-Worldwide’s online courses include opportunities for each of these through our engaging module overviews, content and assessments related to aviation and aerospace, and numerous opportunities for students to interact and share their expertise with one another.
While, as demonstrated above, our current philosophy of instructional design draws from all of these earlier learning and teaching theories, the IDD department, as a whole, consider ourselves to be basically constructivists in philosophy and practice. The related theory of social constructivism also greatly influences our approach to design. Based upon the works of Dewey, Montessori, Kolb, and others from throughout the 20th century, constructivism posits that learning results from the scaffolding of new information and experiences onto existing mental constructs to formulate new understanding (Cooperstein & Kocevar-Weidinger, 2004). Social constructivism, developed from the works of Piaget (1926) and Vygotsky (1962, 1978), maintains that the social and developmental contexts within which the learning occurs affects and influences how information is integrated into existing mental constructs. (Piaget, 1926, Vygotsky, 1962, 1978). Jonassen's (1994) constructivist learning environments (CLE) provides guidelines for creating instruction based upon constructivist principles. These include using authentic tasks, using case-based learning, provide opportunities for collaborative interaction, and encouraging critical reflection, all part of the ERAU-Worldwide IDD model.
With a shift towards virtual learning have come efforts to create new theories of instruction, which often heavily concentrate on the social aspects of learning. One of the most widely discussed prospects is connectivism (Siemens, 2005), a newly emerging theory which considers the impact of technology and digital social learning environments on the learning process. All new theory development takes much criticism and time to become accepted, and these new efforts at devising instructional theories that suit evolving virtual learning environments and learners are no exception. Regardless of the lack of research evidence or yet-attained scholarly legitimacy, they admittedly do affect our thinking about what is good instructional design theory for online learning.
B. Instructional Design Theory
Similar to teaching and learning, instructional design likewise appears to be more a socially-constructed process rather than a formula to be employed (Campbell et al., 2005, p.244, as cited in Botturi, 2008). This author shares the viewpoint that course design and development is essentially both an art and a science: a non-linear, communication and collaboration-dependent team creation (Kenny et al., 2005, Botturi, 2008), and elaborates to contend that it depends on standards-based guidelines and a flexible procedural framework. However, this viewpoint has not always been so, and like learning and teaching theories, current, popular instructional design theory still retains many influential vestiges from older theories. Additionally, many instructional design models are broadly focused on the important steps, or the process, required to construct a learning environment, but there is much less built in to address learning and teaching theories and their application (content development) in the instructional design process. This is especially true with regards to identifying which teaching and learning strategies work best in differing subject matter areas taught in the online learning. (Häkkinen, 2002, Naidu, 2003, Royal, 2007). A brief discussion regarding the various types of instructional design models will be followed by an assessment of the appropriateness of instructional design models in today’s online learning and those which influence instructional design at ERAU-Worldwide.
A widely-accepted definition of instructional design states that it is a “systematic process that is employed to develop education and training programs in a consistent and reliable fashion” (Gustafson & Branch, 2007, pg. 11). With the advent of constructivism and then social constructivism in the 1990s and, sometime related, sometime unrelated rise computers, software, and computer-mediated learning, a significant change in the evolutionary direction of instructional design occurred. This change is often referred to in terms of the linearity of the instructional design process. Characteristics of the older, traditional objective-rational ID models Willis (2009a) included: (1) an ordered, linear process where (2) planning is “top-down” and “systematic”; (3) objectives are the basis for instructional development; (4) subject matter experts are at the heart of the instructional design process; (5) sequentialized skills and tasks are central to lesson planning; (6) “preselected knowledge” is the focus of teaching strategies; (7) summative evaluation is required, whereas formative evaluation is rare; and (8) objective data are paramount in the pursuit of teaching and learning. Strict process linearity and many of these other characteristics are fundamental components of the traditional instructional design models, which include the Instructional Development Learning System (IDLS) (Esseff & Esseff, 1970), ADDIE (Analyze, Design, Develop, Implement, and Evaluation) model and the Dick, Carey & Carey (1978, 2005) Systems Approach model. The DC&C model, one that has been refined over time and now reflects many more of the current trends in instructional design, was one of the first semi-linear models. While derived from ADDIE and still rooted in objectivist/behaviorist-based principles (Moallem, 2001), the DC&C model acknowledges the need to revise design throughout the development process, not just after the fact. It also differs in that it assumes development of assessment instruments prior to formulation of instructional objectives (a practice not endorsed at IDD), but more importantly, views the process from a systems perspective, acknowledging the interrelated parts and their effects on one another.
Since the traditional instructional design models are viewed to be linear, rigid, costly, slow, and based upon outdated pedagogical/andragogical philosophies (Tennyson, 1997, Reigeluth, 1999 , Gordon & Zemke, 2000, Häkkinen, 2002, Wallace, Hybert, Smith, & Blecke, 2002, Royal, 2007), they may be useful to inform practice to a degree, but are rarely directly applicable in today’s course design and development environment (Kenny, Zhang, Schwier, & Campbell, 2004). In reality, the prescription of a formulaic, systematic instructional design approach does not reconcile well within a highly-creative and sometimes turbulent academic culture (Moore & Kearsley, 2004; Magnussen, 2005) characteristic of institutions of higher learning. Recognizing this, newer, non-linear instructional design models contain attributes of classical models such as ADDIE, but they also integrate the notions of flexibility and adaptability, allowing collaborative teams to adjust the order and processes used for producing online learning (Tripp and Bichelmeyer, 1990, Tennyson, 1999, Merrill, 2002, Sims and Jones, 2003, Royal, 2007). Characteristics of the newer, constructivist-interpretivist instructional design models (Willis, 2009a) commonly include: (1) an instructional design process is “recursive, nonlinear, and sometimes chaotic”; (2) planning which is organic, developmental, reflective, and collaborative; (3) learning objectives that frequently emerge and evolve throughout design and development; (4) subject matter experts and instructional design experts don’t generally exist as one person; (5) learning is designed in meaningful, relevant, and authentic contexts; (6) formative evaluation is as critical as summative evaluation; and (7) subjective data may be more valuable than objective data. Applying constructivist principles to the instructional design process has resulted in what is referred to as Constructivist Instructional Design (C-ID). Wiwat Puntai (2007) has proposed three principles of C-ID: (1) the process is flexible and inclusive of all stakeholders; (2) the process is non-linear and often has multiple recursivity points and with a continual refinement cycle; and (3) the process is reflective and this reflection drives progress, as opposed to strict procedural rules.
Beginning with Tripp and Bichelmeyer’s (1990) RapidPrototyping model, virtually every new instructional design model proposed within the last two decades falls in this constructivist-interpretivist, or C-ID, category, to one degree or another. The differences amongst the models are essentially their structural framework (waterfall, circular, or sprial are three common types) and to their tendency to focus more emphasis on either process or content development. Sometimes the models are more concerned with the process of producing courses, rather than actual design of instruction (the content developed for teaching and learning), and sometimes it is the other way around, but they all call themselves instructional design models.
Process elements include project management, communication flow, task delegation, employment of cognitive and productivity tools. (Botturi, 2008) Instructional designers develop process procedures using tools such as flowcharts, workflow symbols, etc. They concentrate on the physical interaction with courseware and other technology, development procedures and standards for learning objects, observing and adhering to policy, and coordinating with stakeholders. The tools used to facilitate these actions are sometimes referred to as “mediating artifacts” (Kobeleva & Strongman, 2012), which in our case at ERAU-Worldwide include timeline and process flowcharts, alignment and course mapping documents, textbook tracking tools, task monitoring spreadsheets, maintenance records, and review documentation.
Quality concerns, which will be addressed in detail below, are, however, generally related to the content development component of instructional design. Two primary areas of attention when addressing the quality of instructional design are physical structure and dialogical structure. Physical structure refers to learning objectives, activities, assessments, navigation, directional text, etc. Dialogical structure refers to the patterns of communication and interaction between students and the instructor. (Moore & Kearsley, 2005, Kobeleva & Strongman, 2012). Accreditors aren’t nearly as concerned with how we get our courses produced, but significantly more so with their content and student learning outcomes.
C. Millennial-Net Generation of ID Models
Using the term “millennial” (from “Millennial Generation”) combined with “net” (from “Net Generation”) to describe the evolution of instructional design theory occurring between the 1990s and 2000s, there appears to be a trend towards a divergent evolution, a split into two types of models – process-oriented models and content-oriented models. Whereas, ADDIE and earlier instructional design models tended to address both the process and content development within their singular framework, newer models frequently do not do this well. What they do appear to have in common are their trends towards non-linearity, cyclicality, recursivity, and adaptability. But, as the diagram below shows, some models, based upon their graphical descriptions, focus more intently on how to produce online learning and other forms of instruction while others concern themselves with the content that is produced. Please note that this is a broadly generalized and superficial categorization, and each and every model does address, so some degree, both process and content development.
Millennial-Net Generation of ID Models
Degree of recursion
Selection of design tools
Authentic, alternative assessments
Content evaluation and refinement
- Willis (2000) Reflective, Recursive Design and Development (R2D2)
- Sims and Jones’ (2003) Three-PhaseDesign (3PD)
- Crawford’s (2004) Eternal, Synergistic Model
- eLab Fast-Prototyping Design Model (Botturi, Cantoni, Lepori & Tardini, 2008)
- Katherine Cennamo (2009) Layers of Negotiation Model
- Tripp and Bichelmeyer’s (1990) RapidPrototyping
- Reigeluth’s (1999) Simplifying Conditions Method (SCM)
- Tennyson’s (1999) ISD4
- Davidson-Shivers & Rasmussen’s (2006) Web-Based Instructional Design (WBID) Model
- Kranch (2008) I3D Model
- SeelsandGlasgow (1998) Model
- SmithandRagan (1999)
- Merrill’s (2001) FirstPrinciplesofInstruction and (2002) PebbleinthePond
- Hall, Watkins, & Eller’s (2003) Web-Based Design for Learning Model
- Morrison, Ross & Kemp (2004) KempModel
- Snyder (2009) Model of Instructional Design for Adult Learning Community
Figure 3. Millennial-Net Generation of ID Models
As newer models confirm an ever-increasing trend towards greater collaboration among stakeholders in the instructional design process, a recent study (Dicks & Wright, 2008) on this topic revealed that instructional designers reflect heaviest upon and assign priority to finding ways to understand and communicate with their faculty/subject matter experts. These efforts appeared to be focused in two areas: (1) building relationships and (2) creating order and logical progression through cognitive tools. The investigators determined that social skills such as establishing rapport and credibility, fostering a sense of collaboration and compromise, identifying needs and design goals, and coaching “subtly" were deemed critical by the instructional designers in order to build healthy relationships within the design team (Dicks & Wright, 2008). Secondly, in the cognitive tools area, the following were listed: role playing (putting yourself in the shoes of the learner); visualization through storyboards, flowcharts, concept maps, multimedia, etc.; using analogy to tie online learning to past experience (“think of the modules as a binder”); using examples and non-examples; and using alignment tools and formative feedback instruments (Dicks & Wright, 2008).
This focus on collaboration in emerging instructional design theory is at the heart of the next topic. But first, to summarize the theoretical foundations that influence and inform our process, as with learning and instructional theories, classical instructional design models stem from a behavioristic view of teaching and learning, whereas the newer models offer an essentially heuristic approach that makes it more responsive to practical application in the workplace (Botturi, 2008). Based upon reviews of current research on the application of instructional design models (and particularly the traditional instructional design models), even though they remain the central focus of instructional designer preparation programs, their direct, intentional use in the workplace is limited, with any application being modified to fit the situational context (Botturi, 2008). As well, the linear nature of traditional models do not work effectively or efficiently for collaborative teams working together to develop online learning in an atmosphere of constantly evolving and emerging technologies and organizational circumstances (Tripp and Bichelmeyer, 1990, Sims and Jones, 2003, Royal, 2007). With increased emphasis on the social constructivist approaches to teaching and learning, and instructional design theory to support the development of an online learning community is needed (Snyder, 2009). In conclusion, no one, current instructional design model will work effectively for all types of learning environment development needs (Dick, Carey, & Carey, 2005, Gustafson & Branch, 2002, Irlbeck, et al., 2006, Royal, 2007). With so many moving parts, i.e. ever changing technological, organizational situational and educational theoretical factors, is it ever possible to develop and apply a functional, procedural-based process model that also effectively addresses pedagogical/andragogical needs? This author believes, from experience in the field, that the answer is yes, if we view it rather as a somewhat amorphous and fluid procedural framework in which shifting factors have room to move and adjust, innovate and evolve. And so we return to the notion that instructional design lives somewhere in a very grey area between free-form art and formulaic science. While likely a combination of both, instructional design models will continue to serve more as a compass and road map, rather than a content development tool and strict procedural process (Kenny et al., 2005). A graphical representation of the current ERAU-Worldwide instructional design model is presented again, this time with more detail, in the Production Process topic further below, and it, too, does not fully, accurately represent our process or content development, but attempts to in a rather simplified format.
D. Production Team
There was (and sadly still is) a time when an unwitting faculty member, wrangled into the assignment of producing and delivering their own online course, found themselves with no expertise in this area and no support with which to rest upon. Very few “stars”, capable of doing it all, on time and flawlessly, shone on the virtual horizon. Sadly, however, many institutions either cannot or simply won’t invest in building effective course design and development teams. Likewise, instructional designers rarely (and almost never effectively) work in isolation since they typically lack the subject matter expertise (Bates & Poole, 2003, Liu et al., as cited in Kenny et al., 2005, Botturi, 2008). Experts in the field now believe collaborative course development is the best possible way to design quality online courses with the evidence showing that designing a high-quality online course requires various sources of expertise rarely possessed by one person (Chao, Saj, & Hamilton, 2010, Kidney, Cummings, & Boehm, 2007; Oblinger & Hawkins, 2006; Wang, Gould, & King, 2009). Bringing together experts in each of the critical components of online course development, including subject matter experts, instructional designers, and information technologists, to work together collaboratively has been proven to consistently create quality products (Oblinger & Hawkins, 2006). The 3PD model (Sims and Jones, 2003) uses the concept of “lean teams” which are responsible for producing online learning with limited human capital. How lean the team is typically depends on the organizational culture and investment in resources to support the process. Other universities are now adopting this collaborative team approach to course production; one example is Royal Roads University (RRU) in British Columbia, Canada (Chao, Saj, & Hamilton, 2010). With a structure that anchors course production and maintenance to a single, central instructional design unit, similar to ERAU-Worldwide’s IDD department, RRU uses a systematic and collaborative approach to manage over 600 courses with each course being designed by a faculty course developer and instructional designer to align with program outcomes and institutional standards (Chao, Saj, & Hamilton, 2010).
With the instructional designer now often situated as the linchpin of the collaborative online course production process, their role has seemingly influenced and shaped educational practice at all levels of the post-secondary institution. Some have asserted that this places the instructional designer in the role of change agent, not just with the influence of technology integration but with a shifting focus to the development of proper, measureable learning objectives, alignment with course, program and institutional outcomes, proper addressment of copyright, Section 508 and universal design elements, and the implementation of appropriate alternative forms of assessment (Chao, Saj, & Hamilton, 2010, Moore & Kearsley, 2004; Magnussen, 2005, Campbell, Schwier, & Kenny, 2007).
Instructional designers should ideally be involved in the course production process from beginning to end (Kobeleva & Strongman, 2012, Caplan, 2004). Instructional designers’ primary responsibilities are to coordinate communication and project planning between stakeholders while following guidelines and maintaining standards, then facilitate or lead the design and development of quality online instruction (Kenny, Zhang, Schwier, & Campbell, 2004; Chao, Saj, & Hamilton, 2010). Kenny et al. (2004) discern four main competencies areas for Instructional designers, which are slightly refined here, along with related tasks, to be:
Figure 4. ERAU-Worldwide Instructional Designer Roles and Responsibilities – The Four Competency Areas
Similar to the Dicks & Wright (2008) study evaluating collaboration in the course design process, Chao, Saj, & Hamilton study results (2010) regarding factors that hindered collaboration included: (1) introducing all the quality guidelines at once (felt to be overwhelming), especially in a compressed production cycle; (2) information and task overload; (3) not clearly specifying roles and responsibilities; and (4) not developing a shared vision. Study results regarding factors that facilitate collaboration naturally are the opposite: (1) developing a healthy rapport between faculty member and instructional designer; (2) having an initial conversation that sets expectations and outlines the process and requirements, specifically deadlines; and (3) maintaining mutual respect (Chao, Saj, & Hamilton, 2010).
Following this collaborative model and lessons learned, course design at ERAU-Worldwide is coordinated through the IDD department, but the process is not handled solely by the department, rather, production is a group effort among members of a substantial interdepartmental team. It truly “takes a village.” The “production coordinator” (PC), with instructional design as a primary role and responsibility, serves as the central point in our collaborative formation, and this individual is responsible for prepping for all production work and initiating cooperation with numerous stakeholders, most notably the faculty “course developer” (CD), who serves as the subject matter expert, and other departments across the ERAU Worldwide system. The chart below shows the categories of production team members and the number of members (as of June 2012) in each production team member category, along with brief descriptions of roles and responsibilities.
ERAU-Worldwide Online Course Production Team Member Categories
Number of Members*
Primary Role in the Course Production Process
Production Coordinator/Instructional Designer (PC)
Coordinates and facilitates course production in partnership with the faculty Course Developer and provides instructional design expertise throughout the process.
Course Developer (CD)
Is responsible for the content of online courses, serving as the subject matter expert, and lead course monitor. Teaches the first section of the course, at a minimum.
Academic Department Chairs
Identifies faculty developers and works with the Executive Director to determine the production and teaching schedule for the CDs. Approve course planning documents and completes an academic review of the course as part of the last stage of the course production process.
Executive Director, Instructional Design And Development (IDD)
Schedules the production of all online courses and supervises the work of production coordinators and instructional designers. Is responsible for managing course developer contracts. Manages the front-end of Blackboard for Worldwide online, provides the production coordinator with technical support and helps with media production.
Develops media for online courses including videos, animations, graphics and other digital media as needed.
Creates course shells in Blackboard and manages the back-end operations associated with the learning management system. Coordinates with IDD to manage LMS upgrades, which normally require some course maintenance.
eLearning (LMS) Support
Populates sections of courses, manually creates new sections after auto-generation cutoff date, provides technical assistance to instructors, and assists in revisions to live sections. Relays student and instructor feedback to PC regarding course corrections or other maintenance needed.
Department of Online Instruction-Faculty Contracts and Faculty Quality
Assigns instructors to courses and provides orientation and monitoring assistance to instructors. Handles and resolves queries from students regarding instructor performance and coordinates plagiarism cases. Relays student and instructor feedback to PC regarding course corrections or other maintenance needed.
Figure 5. Production Team Member Data
*as of June 2012
There can be a natural tension when fostering collaboration within a group of creative professionals and academics (Griffin & Moorhead, 2012). Opinions on how to design a course, which formatting looks best, what kinds of activities are best suited to a particular subject area, and so on, are always diverse. Over time, the production team, as a metaphorical organism, must evolve to achieve a type of homeostasis, balancing out the creative desires and need for academic freedoms of all members within the unit (Eckel, 2010). New group members, particularly instructors that are perhaps coming from institutions where course design and delivery are left almost entirely up to the individual, can find the structure of online course development and delivery at ERAU-Worldwide disconcerting. So much is already done by the time they teach a course that instructors, in the past, have felt left out of the design process and somewhat resentful. This has been overcome by establishing expectations during the hiring process and via instructor professional development. However, regarding the course design part itself, the most significant relationship, and the one with the greatest propensity for conflict, is that of production coordinator and faculty course developer. In our model of collaboration, the role of the course developer lies approximately midway between the do-it-all “lone wolf” (Laird, 2004) faculty developers and the subject matter expert purists found in corporate or government training development organizations. The ERAU-Worldwide course developer provides both the subject matter content and activities for within the courses. The production coordinator serves in the role of project manager, shepherding the course production process from start (scheduling and prep) to finish (delivery), and works closely with the course developer to guide and at times, coach, them through the process, particularly during the instructional design planning phase. All the instructional design and project management knowledge in the world won’t help a bit, if the relationship between the production coordinator and course developer is rocky. A major focus must remain on developing healthy, functional production team relationships and preventing power struggles at all costs. Often this requires the instructional designer to serve in a somewhat subservient role, while continuing to provide their expertise in a gentle way. Some pairings simply don’t have the right chemistry, even though the individuals may work wonderfully with others. If a production team is not working, sometimes a simple adjustment in the makeup of team members (and probably attitude) is all it takes to make the project succeed. Keep the team focus on the end product and your client: students!
E. Quality Assurance
Quality assurance (QA) models are influenced by numerous situational and environmental factors, including accreditation, technology, and competitiveness, to name a few. QA appears to be a core value, and underlies many policy decisions in a higher education organization (Abdous, 2009, Newton, 2007). While support for and views with regards to how online learning should be administered, particularly with the course design component, are diverse, Myer and Barefield (2010) have found one significant factor that rises above the others and pushes institutions to seek ways to increase quality: accreditation. According to the Council for Higher Education Accreditation (CHEA), accreditation provides a means for institutions to self-regulate educational quality through self-examination and peer review using a system of standards and outcomes for public disclosure (CHEA, 2012). CHEA goes further to (2002) define QA as a “planned and systematic review process of an institution or program to determine that acceptable standards of education, scholarship and infrastructure are being maintained and enhanced.” As is standard in accreditation guidelines, accountability, control, and improvement are commonly considered to be the main aims of QA (Abdous, 2009). Any administrative model for online learning should have a comprehensive QA approach in four main areas in order to achieve accountability, control, and improvement: (1) instructor professional development, (2) instructional design and course development, (3) quality assurance itself, and (4) assessment (Parscal & Riemer, 2010). For the instructional design element of online course production, these aims can be achieved through the establishment and application of minimum standards and continuous reviews to ensure quality throughout the course development process. (Merisotis & Phipps, 2000, Chao, Saj, & Hamilton, 2010)
Peer evaluations, assessment and outcomes alignment, continuous improvement methods, student satisfaction, and performance indicators are all leading approaches to measuring QA (Bogue, 1998, Abdous, 2009). The development of QA models for online learning have begun to emerge, and in alignment with current theoretical trends, their development has evolved into non-linear, often cyclical structures, such as the process-oriented one proposed by Abdous (2009). This model identifies QA tasks organized within the instructional design process structure with three phases: (1) planning and analysis; (2) design, prototype and production; and (3) post-production and delivery. QA markers include clearly delineating the instructional design process for all development team members during each phase. During the first phase (planning and analysis), the use of a flowchart for procedures, a timelines for the production process, and preparation of templates and QA checklists are considered appropriate actions. During Phase 2 (design, prototype, and production), reviews and QA checklists are used during development to assure adherence to standards. Finally, during Phase 3 (post-production and delivery), end-user feedback is collected and results in making applicable updates the content. In another QA model for online courses, Mihai (2009) bases an E-Modules Lifecycle and QA Mechanisms model upon Abdous (2009), but goes a step further by delineating the stakeholders most involved in each stage of the instructional design QA process.
External organizations affiliated with distance learning in one form or another, such as membership associations, technology providers, and of course, accrediting agencies, often establish their own QA initiatives. Quality Matters (QM) was established specifically for this purpose. The QM rubric was developed using national standards of best practice, is firmly rooted in research literature, and integrates accepted instructional design principles (Parscal & Riemer, 2010). The QM rubric by which online courses and programs are evaluated addresses eight areas: (1) course overview and introduction, (2) learning objectives (competencies), (3) assessment and measurement, (4) instructional materials, (5) learner interaction and engagement, (6) course technology, (7) learner support, and (8) accessibility (QM website, http://www.qmprogram.org/rubric). Popular learning management system providers, such as Blackboard, also promote their vision of quality through evaluation tools such as the Exemplary Course Program rubric, which addresses: (1) goals and objectives, (2) content presentation, (3) learning engagement, (4) interaction and collaboration, (5) assessment expectations and design, (6) learner support, and (7) templates (Blackboard ECP rubric website, http://goo.gl/JSXCd).
Many of the QM and Blackboard (among other evaluation providers) criteria are addressed via the standard template used in all ERAU-Worldwide online courses, but care has been taken to ensure that all other criteria are covered during the individual course design phase of the process. The Quality Matters (QM) program provided much of the baseline for the development of our standardized template and course design standards during the development and implementation of the “Gold” template in 2007, described in detail further below. As part of the review process for QM, the IDD department submitted two courses: WEAX 201 Meteorology I, receiving 79 out of 80 points, and ENGL 221 Technical Report Writing, receiving 80 out of 80 points. The results were excellent (either perfect or near perfect scores were achieved), so there was confidence that the model was indeed of the highest quality possible. Our quality approach has been further validated with numerous Exemplary Course Awards from Blackboard, including: ENGL 221 Technical Report Writing (2008), MBAA 514 Strategic Marketing Management in Aviation (2011), DAV 733 Globalization and the Aviation Environment (2011), ENGL 222 Business Communication (2012), DAV 714 The Legal Environment of Aviation (2012), DAV 712 Aviation Safety Management Systems (2012), and DAV 713 The Economic Environment of Aviation (2012).
Whatever the quality guidelines are, the way they are implemented and the communication that occurs around them are both critical elements to gaining acceptance from the stakeholder community. Study results (Chao, Saj, & Hamilton, 2010) regarding helpfulness of quality guidelines found the following approaches to be positive in gaining broad institutional acceptance of quality guidelines: (1) review guidelines at beginning and keep them in mind throughout the process; (2) use them as a checklist at the end of development; and (3) adapt to the needs of each course. At ERAU-Worldwide, we can confirm that these approaches work well, and what our quality guidelines are and how we integrate them is the focus of the next topic.
Establishing and Maintaining Standards
A. Six Quality Standards for Course Content
Based upon a critical review and compilation of evaluation criteria from numerous reputable sources, including Quality Matters, Blackboard, Southern Regional Education Board, and Southern Association of Colleges and Schools (SACS), the accrediting body for ERAU, the IDD department (2007) holds the following standards as the basis of all course design requirements:
- Standard 1 - Basic Design: Course navigation, organization, and statements of expectations have clarity and alignment; style guidelines are adhered to systematically throughout the course.
- Standard 2 - Learning Objectives: Objectives are formulated as a subset of course and program outcomes and express measurable expectations student assessment.
- Standard 3 - Interactive Learning: Course design supports interactive learning essential for student motivation, intellectual commitment, and personal development.
- Standard 4 - Instructional Materials: Comprehensive, current, and accurate instructional materials align with course outcomes and are prepared by qualified persons.
- Standard 5 - Course Technology: Course technology supports interactive learning and provides fully accessible modes of delivery, resources, and student support for its use.
- Standard 6 - Learning Assessments: Formative and summative assessments align with objectives, content, and online learning context.
Each standard has an accompanying, comprehensive subset of course design objectives. Many of the objective requirements have been integrated into the master template that is the foundation for each and every course at ERAU-Worldwide, discussed next.
B. Master Template
Research in online learning has consistently shown that students learn quicker and with greater learning outcomes when the framework in which the instruction is provided (the online course “shell” as it is called at ERAU-Worldwide) is predictable, consistent, and accurate (Meyer & Barefiled, 2010, Parscal & Riemer, 2010). Using a standardized template greatly increases each of these values. In addition, the use of course templates streamlines the course production process (Meyer & Barefiled, 2010, Parscal & Riemer, 2010). Course templates also support quality assurance by setting various standards such as those related to navigation and structure, style, and instructional requirements (Henry et al, 2008).
The current online course instructional design model at ERAU-Worldwide was originated in 2007, with the introduction of such a master template, nicknamed the “Gold” template. This template increased consistency in course structure, navigation, and function. Template structure included standard, set areas of the course including “Start Here” (home to the course syllabi, instructor profile, student policies, online learning orientation, accessibility information and disability support), “Announcements”, “Modules”, “Resources” (course-specific an academic-general), “Discussion Board”, “Email”, “Help”, “My Grades”, and more. Each section contains specific elements designed to make learning easier through logical content and support resource organization and flow. Navigation refers to the order in which these sections are listed on the course menu, and the order in which elements in each of the sections are set. Standardized functions include specific tasks all students must complete, tools used commonly for specific activities, rules regarding feedback, and so on. Remaining course design objectives not yet met via the master template, are addressed through alignment procedures, formatting standards, and activity and assessment design, which are addressed next.
Alignment is a critical component to achieving quality in online learning, or any educational program (Sims, Dobbs, & Hand, 2002). The alignment component is so critical, that it normally represents one of the major benchmarks for accreditation reviews (SACS, 2000). In order to achieve program outcomes, there needs to be course-level outcomes accompanied with subsets of learning objectives dispersed into consistently structured units of learning. In ERAU-Worldwide online courses, these are referred to as “modules.” Finally, learning objectives are individually mapped to unit-level activities (content presentation, practice, and assessment activities), containing both graded and ungraded tasks, through which students either work towards competency or demonstrate actual competency of the learning objectives.
Figure 6. ERAU-Worldwide Instructional Alignment
As with course design standards guidelines, the process of creating and documenting alignment must be flexible and adaptable to both the course developer needs and course subject matter requirements. Each production coordinator has this flexibility with regards to how the alignment process occurs, with careful consideration as to the developer’s needs and the requirements of the subject matter. Commonly used tools in achieving alignment include:
- Course Guide: Academic department-level document for every course in the ERAU-Worldwide system, this document provides the learning outcomes required for each course, regardless of whether the course in online, on-ground, or blended.
- Outcomes and Objectives Planning and Alignment Tracking Matrix: Course developers plan subsets of learning objectives for each course outcome, then tentatively map them to modules in concurrence with the completion of the Preliminary Course Map/Schedule Planning document below. As course development proceeds, changes in the assignment of learning objectives to modules are reflected in the matrix, ensuring an accurate alignment map at the completion of the course production process.
- Preliminary Course Map/Schedule Planner: A course developer compiles all learning objectives in this high-level planning document that allows the developer to rough sketch out each module, including the module titles and the activities that will be designed to meet the learning objectives.
- Individual Module Template: Once the two planning documents above are approved by the academic department chair, then the module titles, learning objectives and activity titles are transferred over into an individual module template. Course developers then flesh out each activity. The production coordinator emphasizes in the course production kick-off meeting that the developer need not concern themselves with providing the “directional text” explaining to students where to go and what to do, from a course navigational/functional perspective, but rather simply to provide the content and specific activity task descriptions. The production coordinator will provide all directional text, much of it standardized. Finally, developers provide any grading guidelines/rubrics and assess the time-on-task for each activity, providing a time total for each module. Undergraduate courses have a different time-on-task (replacing traditional so-called “contact hours”) requirement (typically 5-8 hours per module) than graduate courses (typically 8-12 hours per module).
Basic course formatting requirements are followed by each production coordinator in the development of the course content in the Blackboard learning management system. There are three categories:
- Text Style: Specific font sizes, colors, and types are provided for headings and body content
- Web Design: Alternate tags, file formats, file sizes, hyperlink, image, embedded document construction and behaviors, etc.
- Directional Text: Standard navigation and functionality instructions for students to complete tasks using the built-in learning management system tools
The IDD team has been careful (and through some trial and error) not to overextend the style requirements, making them too complex, constraining or overly taxing. There is room for each individual production coordinator to be creative and innovative, within reason.
Strict copyright rules are followed in all online courses at ERAU-Worldwide. No non-original documents are embedded in any course without explicit written permission from the document author and sometimes, publisher. Proper, American Psychological Association-formatted citations and references are provided for all content. Copyright dates are listed on the Course Entry page and Module Menu page for each course, identifying the course as belonging to ERAU.
F. Checks and Balances: Course Development Reviews
For ERAU-Worldwide Instructional Design and Development department, the process of critical self-examination as a path to QA is manifested through a system of checks and balances called the reviews process. Reviews are typically tied to milestones (MS) in the production process, as illustrated in Figure 10 in the next part of this paper’s section and their related stages of the process. Course development reviews are conducted according to the following schedule, in the order shown below. The process appears strenuous, but once the first two reviews (Planning Documents Reviews / Modules 1 and 2 Reviews) are completed and revisions made to satisfaction of all involved production team members, generally, the remaining reviews are quite smooth. Depending on the delivery timeframe and other factors, this process can be flexible, with some reviews happening simultaneously in cases where time is tight.
Planning Documents (Outcomes & Objectives Alignment; Preliminary Course Map/Schedule Planner) Reviews (Stage 3/Milestone 2)
- Lead Instructional Designer Review: Checks for alignment, wording of learning objectives, may make activity suggestions to meet learning objective needs
- Academic (Department Chair) Review: Checks and approves planning documents, ensures alignment of proposed learning objectives and activities/assessments with course outcomes and program outcomes.
Modules 1 and 2 Reviews (Stage 4/Milestone 3)
- CD Review: Checks content of Modules 1 and 2
- Lead Instructional Designer Review: Checks primarily for alignment and adherence to style guidelines and time-on-task per module; may also recommend some activity/assessment/evaluation (rubric) setup changes at this point
- Academic (Department Chair) Review: Checks and approves overall structure of course this far, including alignment, learning objectives, activities and assessments
Module 3-6 Reviews (Stage 4/Milestone 4)
- CD Review: Checks content of Modules 3-6
Final Reviews (inclusive of Modules 7-9/12) (Stages 5 and 6/Milestones 5 and 6)
- CD Review: Checks content of Modules 7-9/12, Discussion, Exams, and Resources areas, Grade Center, plus additional documentation (syllabus, instructor guidance, etc.)
- Peer Production Review: Checks for technical issues (hyperlinks) and spelling/grammatical errors
- Lead Instructional Designer Review: Checks primarily for alignment and adherence to style guidelines and time-on-task per module
- Director Review: Does final check; updates any new or newly-revised template items; checks Grade Center setup
- Academic (Department Chair) Review: Checks and approves overall structure of course, including alignment, learning objectives, activities and assessments
Depending on the nature and complexity, reviews may also be part of post-production tasks, such as initial delivery and pilot course revision and regular maintenance and updates.
The concept of a peer review or any other review, for that matter, in which criticism is sometimes constructive and sometimes not, will find strong resistance in any bastion of autonomy such as a post-secondary educational institution. But, we have also found ways to mitigate resistance and use reviews to maximize our potential for quality, unlike any other tool available to us in our instructional design arsenal. Some of the review guidelines that have been found to decrease resistance and therefore increase the success of reviews being accepted and useful include:
- Begin with a review of the course syllabus and any instructor guidance documentation to gain an overview of the course, then refer back to them regularly to make sure that the content in the course matches those documents.
- Chart review comments in table format and provide a clear pathway to the location under comment.
- The reviewer should not correct any mistakes during a review; this could be a mistake that is duplicated in other areas of the course that only the production coordinator might be familiar with.
- Don't use superlatives or derogatory remarks, for example "this is the most confusing rubric I have ever seen" or "I have seen better composition from my 6th grader."
G. Triple-Constraint Project Management Triangle
The time-cost-scope triple-constraint triangle illustrates a common problem in managing project quality (PMI, 2009), often referred to as the Iron Triangle or the “Pick two: fast, good, or cheap” conundrum. In this triangle, scope can be equated with quality. While the IDD department strives to always produce the highest quality online learning possible, time and cost frequently moderate attempts. In the end, with ever-increasing demand for IDD departmental services, a course is often produced with less “sparkle” in order to save time and money and increase efficiency, but it is always ensured that the product is spotless and adheres to stringent quality standards. Yet, the pursuit of excellence is never-ending, and course improvements and upgrades are addressed when discussing the maintenance phase of the course lifecycle below.
A. Course Lifecycle
The online course production process at ERAU-Worldwide is part of a larger system, referred to here as the online course lifecycle. The online course lifecycle is composed of four distinct phases, beginning with course production scheduling and ending with course maintenance. In reality, the lifecycle is often a looping process, with almost every course repeating the cycle at least every 1-3 years.
Figure 7. ERAU-Worldwide Online Course Lifecycle
B. Course Production Scheduling
This phase includes: (1) identifying a course for production; (2) scheduling production and delivery dates; (3) assigning a production coordinator and course developer; (4) negotiating the course developer contract; and (5) providing course developer training. Calculating production demand and determining prioritization are important components of this phase. The following chart illustrates the diversity and spread of course production projects in the IDD department for July 2011 through June 2012. This does not include any essential maintenance tasks.
Category of Production
Percentage of Total Projects
New Gold course developments
Significant updates/ redevelopments
New ground master templates
New, unique templates for the Singapore program
Undergraduate compressions (to reduce number of weeks from 12 to 9)
Self-paced continuing education courses for Office of Professional Education
Total Production Projects
Figure 8. 2011-2012 Course Production
Once a new course or significant redevelopment is identified, it is added to the production schedule. It is an ongoing challenge to complete work in a timely fashion and balance the workload. There appears to be a wave attribute to online course production workload. Scheduling is tied to course delivery terms, and there appears to be a tendency for certain months to be far busier with regards to course production deliveries than others. A number of factors influence the production scheduling process, not the least of which are academic departments, vying, to some degree, to have their courses scheduled for work, whether it is new production, redevelopments, textbook updates, or other updates. How we prioritize scheduling is examined next.
Online Course Production Queuing Procedure
The impetus to begin a new course development, or significant redevelopment, usually begins at the academic department level, and with the department chair. The Director of IDD, coordinating with the academic departments, schedules production work for the team of production coordinators using the following inquiry list to help in the prioritization of work. This list was developed by former the former department director, Thomas Cavanagh in 2007, and still guides the prioritization and queuing of course production.
- Is the course missing from online offerings, therefore preventing a complete degree/specialization/certificate from being offered online? (enrollment implications)
- Is it a royalty course? (financial implications) These have been phased out for the most part, with developers no longer receiving royalties.
- Does the course support a new or high-priority program? (new enrollment implications)
- Are there external influences requesting the course be redeveloped? (e.g., student complaints, new/changing requirements)
- Does the production request originate from or have the direct approval of the academic Department Chair? (Reflecting departmental priorities)
- Is an approved developer assigned to the course (approved by both academic department and Online)?
- Has the developer completed the required course developer training?
- Is the developer available (workload issues for both full-time and adjunct faculty)?
- Is there a new edition of the textbook being released? (to synchronize redevelopment with the new edition)
Before delving into how a course is produced, it is prudent here to explain the process by which we manage the variety of course shells that are used in conjunction with the delivery of a single course.
A. Course Shell Management
The master template described in the above section forms the foundation upon which each of the master courses is built. Once a master course is completed and approved, it is ready for delivery to students. However, in order to ensure that the master course remains “pristine” and free from possible corruption, a protocol is in place that ensures its protection. This protocol is referred to here as “course shell management”.
Figure 9. ERAU-Worldwide Course Shell Management
The chart above shows the relationship, as well as the process flow by which live course sections are generated. “Shell” refers to nothing more than an empty course in Blackboard. The term “production shell” refers to the master course shell, in which “production” occurs. At the beginning of the production process, a new empty shell, which will become the production shell, is requested from the IT department, the responsible unit for administering the back-end of the Blackboard learning management system. Each new production shell comes with a unique code that identifies it as the production shell for a particular course. The master course template, described in the previous section, lives in its own shell, called the “IDD_Master_Template.” The PC uses the copy function in Blackboard to duplicate the master course template into the new empty production shell. After course development is completed and approved, the production shell is copied into what is called the “builder shell”. An automated process copies the builder shell into new live section shells approximately 90 days prior to a term start date. If the course production work is not completed by that time, then an announcement is placed into the builder shell prior to the auto-copy date to communicate to instructors that the course is still being worked on and will be updated in the live section, so that instructors do not complete their customization work prior to the final version of the course being copied into the live section shells. Then, when course production is completed, the eLearning Support department manually copies over the builder shell into the live sections. Should anything break in the builder shell during this process, as sometimes happens, the course remains safe in its pristine production shell and a new builder shell can be created.
B. Course Production Procedures
The ERAU-Worldwide course production process model has evolved and refined over time and currently is working very well for departmental needs, given the constraints and environmental conditions, to meet production demand. New gold productions follow a process similar to the flowchart below, which is a detailed version of the heart of the Online Course Design and Production Model presented at the beginning of this part of the article.
Figure 10. ERAU-Worldwide Online Course Production Process Model with Detail
While this author is not proposing a new instructional design model, the model above is intended to only illustrate the production process used at ERAU-Worldwide as accurately as possible. This representation gives a snapshot of our current instructional design process and is modeled upon existing, accepted instructional design theoretical frameworks. While it is primarily designed based loosely upon the Rapid Prototyping model (Tripp & Bichelmeyer, 1990), the ERAU- Worldwide IDD model draws from best practice elements of many instructional design models, both linear and non-linear. The semi-linear, overlapping process illustrated above may also be organized into stages, providing another perspective of the process:
- Stage 1: Course production prep, course developer contract, and materials (textbook, case studies, etc.) adoption
- Stage 2: Course production kick-off meeting
- Stage 3: Planning documents completion and approvals
- Stage 4: Module, media, and assessment development in Blackboard with reviews at intervals
- Stage 5: Additional course documents (syllabus, rubrics, instructor memo, etc.) and final developer review
- Stage 6: Final production peer review
- Stage 7: Final academic review and course production delivery
How much time does the course production process take? Ideally, given that any one production coordinator is working on multiple projects simultaneously, the ideal time frame for a new Gold course production is no less than six months. Developers, too, have regular teaching responsibilities and other non-course production professional activities to attend to, and therefore they must also fit their course production duties into their hectic schedules.
In reality, a full 6-month timeframe is not always possible with production coordinators having been given new course developments with deadline timeframe as little as two months. The following chart illustrates time requirements, organized by hours per task category, for a single, typical course production. Keep in mind that many factors affect these time requirements, from content development delays, administrative delays, review delays, technology upgrades, etc. This data comes from a record of a single Gold course production, AVS 1000 Private Pilot Online Ground School, produced between August and October of 2008. The total production time equaled approximately 260 hours, which is about 6 ½ weeks of full-time work.
Category of Production Tasks
Approximate Time Required
Course Production Preparation: Documentation setup (folders, planning documents, module templates) and course shell request and template importation
Email Communication (239 emails)
Meeting Prep and Follow-Up (reviewing submitted content, preparing discussion agendas, compiling meeting summaries and updating status documentation
Media Development: Manual Flight Computer Tutorials
Production Work in Blackboard
Reviews Facilitation and Revisions
Total Production Time
Figure 11. Average Production Coordinator Work Hours for Standard Gold Template Course Development
This phase includes the following tasks required of the production coordinator: (1) copying production shell into builder shell and completing copy checks; (2) notification of eLearning Support that course is ready for delivery; eLearning Support then copies manually into live sections if schedule auto-generation date is passed; and (3) notification of appropriate Faculty Quality Manager in the Department of Online Instruction and subsequent notification of instructors that the course is ready for preparation/customization.
This phase includes: (1) course evaluations and other feedback; and (2) course repairs and updates for textbook changes. Feedback may come from instructors, students, eLearning Support, Faculty Quality Managers, course developers, and any other end-user or support staff. Course maintenance falls into two categories: the “must haves”, or essential updates, and the “nice to haves”, or non-essential updates. Corrections that, left unmade, would harm or interfere with student learning fall into the essential update category, while all other changes and improvements, usually aesthetic in nature, fall into the non-essential update category and are made during redevelopments or major updates at a later time. Non-essential improvements in older courses are made only during redevelopments or major updates, and extensive time is not spent “going backwards” in an attempt to keep all courses 100% consistent. The chart below provides an example of essential, “must-have” maintenance data. Test corrections, hyperlink replacements, activity direction corrections/updates, and content (subject matter) corrections, in that order, took the greatest percentage of both time and instances.
Category of Essential Maintenance*
Number of Instances
Percentage of Total Number of Instances
Total Time (in hours, rounded) Required Per Category
Percentage of Total Essential Maintenance Time
Activity Direction Corrections/Updates
Textbook Update-Related Corrections
Grade Center Corrections/Adjustments
New Content Additions
Student Support Updates
Figure 12. Essential Maintenance Requirements for a Single Production Coordinator Between April 2011-March 2012
*The results are ordered greatest to least with regards to Total Time Required for essential maintenance.
Adopting Emerging Technologies in the Production Process
The use of Web 2.0 communication tools and cloud applications forms much of the basis of the collaboration among production team members, and at every stage of the production process. While the planning documents and individual module templates have traditionally been developed using a word-processing program, and email has and still serves as the primary communication tool for the ongoing exchange of simple messages, much of the “backstage” work on course development is now being facilitated through online tools such as: Doodle (meeting scheduling); JoinMe or ERAU’s own Saba Centra EagleVision (Web conferencing); Google Docs, Drive, and Sites (Worldwide Textbook Forecasting Spreadsheet, IDD Production Schedule, and IDD Production Team Website); Microsoft’s OneNote and SkyDrive (Individual Module Templates); PB Wiki (Course Maintenance Wiki); and Blogger (IDD Production Blog “Resources for the Instructional Design and Development of Learning Environments” or “RIDDLE”). Experimentation with new tools is encouraged to enhance collaboration and increase efficiency, but the tolerance of the course developer in accepting newer technologies for completing their tasks is always considered, and typically the production coordinator will use whichever tool(s) the developer is most comfortable with. On the far, cutting edge, Second Life has been experimented with for learning and teaching, but has been primarily used for the development of videos and other multimedia for use in courses. The ERAU-Worldwide Island in Second Life is designed as a virtual airport, and has made an excellent backdrop for airport operations simulations.
Summary of Best Practices
No matter the size of the institution, the following course design best practices may be followed regardless of budgetary or other physical resource constraints that may exist. If anything, administration and organizational culture may be the biggest hurdles to overcome, but with diplomacy, consensus building, and time, these strategies are accomplishable.
- Set a baseline of standardization achieved via a master course template and promote course design standards across the institution.
- Focus on developing healthy, functional production team relationships; prevent power struggles at all costs. Often this requires the instructional designer to serve in a somewhat subservient role, while continuing to provide their expertise in a gentle way. If a production team is not working, adjust the members of the team as needed. Some pairings simply don’t have the right chemistry, even though the individuals may work wonderfully with others.
- Plan for course quality. Alignment via the development of sound learning objectives should be the main aim of planning, followed closely by creating varied interaction points throughout a course structure, developing a variety of assessment tools, and evaluating time/effort-on-task for activities, adjusting appropriately for the level of the course.
- Enact a system of checks and balances: use reviews at multiple stages in the course production process. If reviews are not already part of the process, then provide evidence for why reviews are critical and form an advisory committee of a cross-section of stakeholders to identify a path of least-resistance (and there will always be resistance) to achieving a system. Ensure that reviews are constructive and respectful by providing guidelines to achieving such.
- Provide instruments for easy and timely student and instructor feedback to encourage technical maintenance and subject matter accuracy and track maintenance work.
Abdous, M. (2009). E-learning quality assurance: A process-oriented lifecycle model. Quality Assurance in Education, 17(3), 281-295. http://www.emeraldinsight.com/journals.htm?articleid=1801284
Bates, A. W. (2000). Managing technological change: Strategies for college and university leaders. San Francisco, CA: Jossey-Bass, Inc.
Bates, T. W., & Poole, G. (2003). Effective teaching with technologies in higher education. San Francisco, CA: Jossey-Bass.
Blackboard. (2012). Exemplary Course Program Rubric. Washington D.C.: Author. Retrieved online April 21, 20122 from: http://goo.gl/JSXCd
Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: the classification of educational goals; Handbook I: Cognitive Domain New York, Longmans, Green, 1956.
Bogue, G. (1998), “Quality assurance in higher education: the evolution of systems and design ideals”, New Directions for Institutional Research, Vol. 25 No. 3, pp. 7-18.
Botturi, L. (2006). Design models as emergent features: An empirical study in communication and shared mental models in instructional design. Canadian Journal of Learning and Technology, 32(2). Retrieved June 3, 2012, from the World Wide Web: http://cjlt.csj.ualberta.ca/index.php/cjlt/article/viewArticle/50
Botturi, L., Cantoni, L., Lepori, B., & Tardini, S. (2006). Fast Prototyping as a Communication Catalyst for E-Learning Design. In M. Bullen & D. Janes (eds.), Making the transition to e-Learning: Strategies and Issues. Hershey, PA: Idea Group.
Burton, J. K., Moore, D. M., & Magliaro, S. G. (1996). Behaviorism and instructional technology. In D. H. Jonassen (Ed.), Handbook of Research for Educational Communications and Technology (pp. 46-73). New York: Macmillan.
Campbell, K., Schwier, R. A., & Kenny, R. F. (2005). Agency of the instructional designer: Moral coherence and transformative social practice. Australasian Journal of Educational Technology, 21(2), 242–262. Retrieved June 3, 2012, from http://www.ascilite.org.au/ajet/ajet21/campbell.html
Campbell, K., Schwier, R., & Kenny, R. (2007). The critical, relational practice of instructional design in higher education: An emerging model of change agency. Educational Technology Research & Development, 57(5), 645–663.
Caplan, D. (2004). The development of online courses. In T. Anderson & F. Elloumi (Eds.), Theory and practice of online learning (pp. 33–60). Athabasca, Canada: Athabasca University.
Cennamo, K. (2009). Design as Knowledge Construction: Constructing Knowledge of Design. In J. Willis (Ed.), Constructivist Instructional Design: Foundations, Models, and Examples (C-ID) (pp. 357-378). Charlotte, NC: Information Age Publishing.
Chao, I. T., Saj, T., & Hamilton, D. (2010). Using collaborative course development to achieve online course quality standards. International Review of Research in Open and Distance Learning 11(3), 106–126. Retrieved June 3, 2012 from http://www.irrodl.org/index.php/irrodl/article/view/912/1645
Commission on Colleges, Southern Association of Colleges and Schools (SACS). (2000). Best practices for electronically offered degree and certificate programs. Retrieved April 17, 2012 from http://www.sacscoc.org/pdf/commadap.pdf
Cooperstein, S. E., & Kocevar-Weidinger, E. (2004). Beyond active learning: A constructivist approach to learning. Reference Services Review, 32(2), 141–148.
Council for Higher Education Accreditation (CHEA) (2001), Glossary of Key Terms in Quality Assurance and Accreditation, Retrieved on February 16, 2012 from http://www.chea.org/international/inter_glossary01.html
Council for Higher Education Accreditation (CHEA). (2012). Informing the public about accreditation. Retrieved June 1, 2012 from: http://www.chea.org/public_info/index.asp
Daniel, S. J. (2009). Is e-learning true to the principles of technology? Presented at the World Conference on E- Learning in Corporate, Government, Healthcare, and Higher Education. Retrieved April 21, 2012 from http://www.editlib.org/p/33043
Davidson-Shivers, G. V. & Rasmussen, K.L. (2006). Web-Based Learning: Design, Implementation, and Evaluation. New Jersey: Pearson.
Dick, W., & Carey, L.(1978). The Systematic Design of Instruction (1st ed.). Glenview, IL.: Scott, Foresman.
Dick, W., Carey, L., & Carey, J. O. (2005). The Systematic Design of Instruction (6th ed.). Boston, MA: Allyn & Bacon.
Dicks, D. & Ives, C. (2008) Instructional designers at work: A study of how designers design. Canadian Journal of Learning and Technology, 34(2), Spring. Retrieved online April 26, 2012 from http://cjlt.csj.ualberta.ca/index.php/cjlt/article/view/495/226
Eckel, N. (2010) Designer-SME collaboration. 27(1005) Alexandria, VA: American Society for Training & Development Press
Educational Technology Cooperative, Southern Regional Education Board (SREB). (2006). Checklist for Evaluating Online Courses. Retrieved from http://publications.sreb.org/2006/06T06_Checklist_for_Evaluating-Online-Courses.pdf
Escoffery, C., Leppke, A. M., Robinson, K. B., Mattler, E. P., Miner, K. R., & Smith, I. (2005). Planning and implementing a public health professional distance learning program. Online Journal of Distance Learning Administration, 8(1).
Esseff, Peter J. and Esseff, Mary Sullivan (1998) . Instructional Development Learning System (IDLS) (8th ed.). ESF Press. pp. 1–12.
Frith, K. H., & Kee, C. C. (2003). Effect of communication on nursing student outcomes in a Web-based course. Journal of Nursing Education, 42, 350-358.
Gagné, R. (1985). The Conditions of Learning (4th.). New York: Holt, Rinehart & Winston.
Gordon, J., & Zemke, R. (2000). The attack of ISD. Training, 37(4), 42-53.
Griffin, R. W., & Moorhead, G. (2012). Organizational behavior: Managing people and organizations (10th ed.). Mason, OH: South-Westen/Cengage.
Gustafson, K. L., & Branch, R. M. (2002). Survey of instructional development models (4th ed.). ERIC Clearinghouse on Information and Technology. (ERIC Document Reproduction Service No. ED477517).
Gustafson, K. L., & Branch, R. M. (2007). What is instructional design? In: R.A. Reiser & J. A. Dempsey (Eds.), Trends and issues in instructional design and technology (pp. 16-25). Saddle River, NJ: Merrill/Prentice-Hall.
Häkkinen, P. (2002). Challenges for design of computer-based learning environments. British Journal of Educational Technology, 33(4), 461-469.
Hall, R. H., Watkins, S. E., & Eller, V. M. (2003). A model of web-based design for learning in Moore, M. G. and Anderson, W. G. (eds.). Handbook of distance education Lawrence Erlbaum Associates, Mahwah, NJ. 367-375.
Henry, B., Marcella, K.B., Kurzweil, D., & Davis, S. (2008). Using templates to build courseware to enhance ease-of-use for faculty and usability for learning. In Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2008 (pp 87-91). Chesapeake, VA: AACE. Retrieved on February 16, 2012 from http://www.editlib.org/p/29584.
Irlbeck, S., Kays, E., Jones, D., & Sims, R. (2006, August). The phoenix rising: Emergent models of instructional design. Distance Education, 27(2), 171–185.
Jonassen, D. H. (1999). Designing constructivist learning environments. In C. M. Reigeluth (Ed.), Instructional design theories and models: A new paradigm of instructional theory, Volume II, pp. 215-239). Mahwah, NJ: Lawrence Erlbaum Associates.
Jonassen, D. (2002). Integration of problem solving into instructional design. In R. A. Reiser & J. V. Dempsey (Eds.), Trends and Issues in Instructional Design and Technology (pp. 107-120). Upper Saddle River NJ: Merrill Prentice Hall.
Keller, F. S. (1968). Goodbye teacher…Journal of Applied Behavior Analysis 1, 79-89.
Keller, J. M. (1987). Development and use of the ARCS model of instructional design. Journal of Instructional Development, 10(3), 2–10.
Keller, J. M. (1999). Motivation in cyber learning environments. Educational Technology International, 1(1), 7 – 30.
Kenny, R.F., Zhang, Z., Schwier, R.A., & Campbell, K. (2005). A review of what instructional designers do: Questions answered and questions not asked. Canadian Journal of Learning and Technology, 31(1), 9-16.
Kidney, G., Cummings, L., & Boehm, A. (2007). Towards a quality assurance approach to e-learning courses. International Journal on E-Learning, 6(1), 17-30.
Kranch, D. A. (2008). Getting It Right Gradually: An Iterative Method for Online Instruction Development. Quarterly Review of Distance Education, 9(1), 6. IAP-Information Age Publishing, Inc. PO Box 79049, Charlotte, NC 28271-7047. Tel: 704-752-9125; Fax: 704-752-9113; e-mail: infoage@ infoagepub. com; Web site: http://www.infoagepub.com/products/journals/qrde/order.html. Retrieved from http://eric.ed.gov/ERICWebPortal/recordDetail?accno=EJ875086
Kobeleva, P., & Strongman, L. (2012). Instructional design: Theory and practice. Lower Hutt, New Zealand: The Open Polytechnic of New Zealand.
Laird, P. (2004). Integrated solutions to e-Learning implementation: Models, structures and practices at Trinity Western University. Online Journal of Distance Learning Administration, VII(III), Fall.
Liu, M., Gibby, S., Quiros, O., & Demps, E. (2002). Challenges of being an instructional designer for new media development: A view from the practitioners. Journal of Educational Multimedia and Hypermedia, 11(3), 195–219.
Magnusson, J. L. (2005) Information and communications technology: Plugging Ontario higher education into the knowledge society. Encounters on Education, 6(Fall), 119–135.
Merisotis, J. P., & Phipps, R. A. (2000). Quality on the line: Benchmarks for success in internet-based distance education. Retrieved from http://www.ihep.org/Publications/publications-detail.cfm?id=69
Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43-59. Retrieved June 5, 2012 from: http://mdavidmerrill.com/Papers/firstprinciplesbymerrill.pdf
Merrill, M. D. (2002). A pebble-in-the-pond model for instructional design. Performance Improvement, 41(7), 39-44.
Meyer, J. D., & Barefield, A. C. (2010). Infrastructure and administrative support for online programs. Online Journal of Distance Learning Administration, 13(3). Retrieved on April 10, 2012, from https://www.westga.edu/~distance/ojdla/Fall133/meyer_barfield133.html
Mihai, A., (2009) “Teaching European Studies Online: The Challenge of Quality Assurance”, In EuropeanJournal of Open, Distance and E-learning, [online], http://www.eurodl.org/?article=377
Moallem, M. (2001, July). Applying constructivist and objectivist learning theories in the design of a Web-based course: Implications for practice [Electronic version]. Educational Technology & Society, 4(3), 113-125. Retrieved on April 10, 2012, from http://www.ifets.info/
Moore M. G., & Kearsley, G. (2005). Distance education: A systems view (2nd ed.). Belmont, CA: Wadsworth.
Morrison, G. R., Ross, S. M. & Kemp, J. E., (2004). Designing effective instruction, 4 th Edition, NY: John Wiley & Sons.
Naidu, S. (2003). Designing instruction for e-learning environments. In M. G. Moore & W. G. Anderson (Eds.), Handbook of distance education (pp. 349-365). Mahwah, NJ: Erlbaum.
Newton, J. (2007), in Bollaert, L., Brus, S., Curvale, B.B., Harvey, L., Helle, E., Jensen, H., Komljenovic, J., Orphanides, A. and Sursock, A. (Eds), “Embedding quality culture in higher education, a selection of papers from the First European Forum For Quality Assurance, November 23-25, 2006, hosted by the Technisch Universitaet Muenchen, Muenchen, Germany”, pp. 14-20, Retrieved April 12, 2012, from http://www.eua.be/fileadmin/user_upload/files/Publications/EUA_QA_Forum_publication.pdf
Oblinger, D., & Hawkins, B. (2006). The myth about online course development, Educause Review, January/February, 14-15.
Paolucci, R., & Gambescia, S. F. (2007). Current administrative structures used for online degree program offerings in higher education. Online Journal of Distance Learning Administration, X(III),Fall.
Quality Matters. (2011). Rubric for online and hybrid courses. Retrieved from http://www.qmprogram.org/rubric
Parscal, D., & Riemer, D. (2010). Assuring quality in large-scale online course development. Online Journal of Distance Learning Administration, 13(2), Retrieved from https://www.westga.edu/~distance/ojdla/summer132/parscal_riemer132.html
Piaget, J. (1926). The language and thought of the child. New York: Harcourt, Brace, Jovanovich.
Project Management Institute (2009) A Guide to the Project Management Body of Knowledge: PMBOK Guide. Chapter 1. Newtown Square, PA: Project Management Institute.
Reigeluth, C. M. (1979). In search of a better way to organize instruction: The elaboration theory. Journal of Instructional Development, 2 (3), 8-15.
Reigeluth, C.M. (1999). What is instructional-design theory and how is it changing? In C. M. Reigeluth (Ed.), Instructional-design theories and models (Volume II): A new paradigm of instructional theory (pp. 5-29). Mahwah, NJ: Erlbaum.
Restauri, S. L. (2004). Creating an effective online distance education program using targeted support factors. TechTrends, 48(6), 32-39.
Rothwell, W., & Kazanas, H. C. (2004). Mastering the Instructional design process: A systematic approach (3rd ed.). San Francisco, CA: John Wiley.
Royal, C. (2007). Exploring the use of instructional design models for web-based instruction in higher education: A modified Delphi study. Retrieved April 10, 2012, from ProQuest database.
Seels, B., & Glasgow, Z. (1998). Making Instructional Design Decisions. (2nd ed.) Upper Saddle River, NJ: Merrill
Shelton, K., (2011). A review of paradigms for evaluating the quality of online education programs. Online Journal of Distance learning Administration, 4(1). Retrieved April 21, 2012 from
Siemens, G. (2005, January). Connectivism: A learning theory for the digital age. International Journal of Instructional Technology & Distance Learning, Retrieved April 12, 2012, from http://www.itdl.org/Journal/Jan_05/article01.htm
Sims, R., Dobbs, G., & Hand, T. (2002). Enhancing quality in online learning: Scaffolding planning and design through proactive evaluation. Distance Education, 23(2), 134-148.
Sims, R., & Jones, D. (2003). Where practice informs theory: Reshaping instructional design for academic communities of practice in online teaching and learning. Information Technology, Education and Society, 4(1), 3-20.
Skinner, B. F. (1958). Teaching machines. Science 128 (967-77), 137-58.
The Sloan Consortium. (2004). Elements of quality online education: Into the mainstream. Retrieved from http://www.sloan-c.org/intothemainstream_wisdom
Smith, P. and Ragan, T. (1999). Instructional design (2nd ed.). New York: John Wiley & Sons, Inc.
Snyder, M. M. (2009). Instructional-design theory to guide the creation of online learning communities for adults. TechTrends, 53(1), 48-56.
Tallen-Runnels, M. K., Thomas, J. A., Lan, W. Y., Cooper, S., Ahern, T. C., & Shaw, S. M. (2006). Teaching courses online: A review of the research. Review of Education Research, 76(1), 93-35.
Tennyson, R. D. (1997). A system dynamics approach to instructional systems development. In R. D. Tennyson, F. Schott, N. Seel & S. Dijkstra (Eds.). Instructional design: International perspectives. Volume 1: Theory, research, and models. Mahwah, NJ: Erlbaum.
Tennyson, R. D. (1999). Instructional development and ISD4 methodology. Performance Improvement, 38(6), 19-27.
Tennyson, R. D. (2010). Historical reflection on learning theories and instructional design. Contemporary Educational Technology, 1(1), 1–16.
Tripp, S., & Bichelmeyer, B. (1990). Rapid prototyping: An alternative instructional design strategy. Educational Technology Research & Development, 38(1), 31-44.
van Damme, D. (2002), “Trends and models in international quality assurance and accreditation in higher education in relation to trade in education services”, paper presented at The OECD/US Forum on Trade in Educational Services, Washington, DC, May 23-24, Retrieved April 12, 2012, from http://www.oecd.org/dataoecd/51/29/2088479.pdf
Vygotsky, L. (1962). Thought and language. Cambridge, MA: MIT Press.
Vygotsky, L.S. (1978). Mind and society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.
Wallace, G. W., Hybert, P. R., Smith, K. R., & Blecke, B. D. (2002, August). Designing for the ISD life cycle. Performance Improvement, 41(7), 23-27.
Wang, H., Gould, L. V., & King, D. (2009). Positioning faculty support as a strategy in assuring quality online education. Innovate, 5(6). Retrieved from http://innovateonline.info/index.php?view=article&id=626&action=login
Willis, J., & Wright, K. E. (2000). A general set of procedures for constructivist instructional design: The new R2D2 model. Educational Technology, 40(2), 5-20.
Willis, J. (2009). Basic Principles of a Recursive, Reflective Instructional Design Model: R2D2. In J. Willis (Ed.), Constructivist Instructional Design (C-ID): Foundations, Models, and Examples (pp. 283-312). Charlotte, NC: Information Age Publishing.