Vicki Blakney Eveland eveland_vb@mercer.edu is an Associate Professor of  Marketing, Stetson School of Business and Economics, Mercer University.  William S. Sekely is an Associate Professor in the Department of Management and Marketing at the University of Dayton.

INTRODUCTION

Response-order effects occur when the order in which alternatives are presented in a multiple choice question influences choice. Because of its potential impact on test results, psychologists and educators have been interested in the problem of response-order bias for many years, and there are a number of studies that focus on experiments with various lists. However, most of these types of studies have limited relevance to survey problems. [Schuman and Presser 1981] Marketing researchers have long been aware that position or response-order bias may influence an individual's answer to a multiple choice question, but the issue has received limited empirical examination. Only a few studies have examined and documented response-order bias as it relates to survey questions. [Payne 1951, Becker 1954, Belson 1966, Mueller 1970, Brook and Upton 1974, Schuman and Presser 1981, Krosnick and Alwin 1987, Narayan and Krosnick 1996, and Knauper 1999]

Two studies [Whipple and McManamon 1992 and Sekely and Blakney 1994] found evidence that response-order bias should be of particular concern to one particular type of survey, that of the measurement of readership and receipt of trade magazines. Whipple and McManamon (1992) found that an aided recall, paired comparison approach to measuring trade magazine readership resulted in a consistent bias toward the visual stimuli presented in the first position. Sekely and Blakney (1994) expanded on the research of Whipple and McManamon (1992) by examining both recall and evaluative types of questions. Their research found that general questions often used by advertisers to measure business-to-business publication readership are prone to response order bias. This was true even if lists of magazines were fairly short and if respondents were allowed to select as many alternatives as apply.

Response order effects are assumed to arise from the difficulty respondents face in keeping in mind all the alternatives presented [Crano 1977)], and have been found to be more common when the list of alternatives is long [Schuman and Presser 1981]. Krosnick and Alwin (1987) even suggest that one way of minimizing primacy effect is to shorten the list of alternatives, but they do not indicate how short the list should be. Current theory states that shorter lists would be less susceptible to primacy problems, however there is no guidance as to how short a list should be. Primacy effects in psychology have been uncovered with lists of 13 and 16. Sekely and Blakney (1994) found that even lists as short as 7 can produce response-order bias. The purpose of this research is to replicate the Blakney and Sekely study with a shorter list to determine how short a list of alternatives must be to avoid response-order bias.

RESEARCH EVIDENCE

Generally, two types of response-order effects have been documented: primacy effects and recency effects. Primacy effects occur when an item placed at the beginning of a list has a higher likelihood of being selected over other items in the list. Recency effects occur when items at the end of a list are more likely to be selected. [Krosnick and Alwin 1987]

Krosnick and Alwin (1987) offered a theory to explain and help predict response-order effects in the aided recall techniques used by Whipple and McManamon and those used in this study. Based on their review of the literature, they suggest that when items are presented visually, primacy effects should be expected for several reasons. First, items presented early are given importance for later judgments because they establish a standard of comparison. Second, items seen first are processed at a deeper level because they are not in competition with other alternatives. Finally, many individuals will, in many cases, select the first acceptable answer or satisfice to minimize cognitive costs, rather than diligently search a list for an optimal answer.

Krosnick and Alwin (1987) tested their theory by asking respondents to list the three most desirable qualities for a child by observing a show card list of 13 total qualities. They expected and found strong evidence to support their theory. However, their research also raised a series of additional questions. First, does the length of the list play a role in producing a primacy effect? As mentioned earlier, response-order effects are usually assumed to arise from the difficulty respondents face in keeping in mind all the alternatives presented. [Crano 1977] According to Schuman and Presser (1981), primacy effects are more common when the list of alternatives is long. They found consistent evidence for primacy effects in choices of five favorite types of radio programs from a cardlist of 16. The reasoning behind the Krosnick and Alwin (1987) theory implies that a longer list might be more susceptible to primacy effects also. The longer the list, the more likely the first answer becomes a standard for comparison; the more alternatives given, the greater likelihood of stronger competition between alternatives given later in the list; and the longer the list, the greater the cognitive cost of optimizing rather than satisficing. Whipple and McManamon (1992) found primacy effects when paired comparisons were used. Would a short cardlist approach to readership measurement also produce primacy effects?

The type of question is also an issue. Rugg and Cantril (1944) attribute response-order effects with questions concerning issues about which respondents have "uncrystallized" opinions. McFarland (1981) examined the influence of question order and found that the more general the content of a question and the more vague a response it requires, the greater the possibility of order effects. Specifically, he found that recall is more easily influenced by question order than evaluation. The same possibility could be true with regard to response position. A very general type of question has a small risk of consequences when a respondent selects an acceptable answer rather than an optimal answer. On the other hand, an evaluation type of question requires a higher level of cognitive processing and may carry a heavier cost associated with selecting a non optimal alternative. Whipple and McManamon (1992) asked respondents about the receipt and readership of three trade magazines and found that both questions produced a primacy effect. However, neither of these questions required a higher level of cognitive processing. Sekely and Blakney (1994) found evidence of response-order bias with general readership questions, but found no significant evidence of a primacy effect from specific evaluative questions related to magazine readership and advertising usage.

TRADE MAGAZINE READERSHIP AND EVALUATION MEASURES

Although the accuracy of magazine readership measures is often debated, there has been very little published research on the inaccuracies that may result from response order effects. [Whipple and McManamon 1992] Obtaining good readership figures for business-to-business publications is particularly difficult because there is no syndicated research in the business-to-business media field. [(Morelli 1986] This forces potential advertisers to rely on readership studies that are funded by individual publishers or to conduct their own readership studies.

Whipple and McManamon (1992) examined order bias produced in a typical publisher-funded study that measured readership of trade publications through paired comparison questioning. The primary purpose of their study was to determine receipt and readership of the sponsor's publication compared to each of its three major competitors. This information is relevant if you are a magazine publisher, but it is less helpful to an advertiser trying to make a decision among many trade publications which best serve his/her media needs. Therefore, advertisers who plan to use trade publications extensively may prefer to conduct their own readership and usage studies.

Sekely and Blakney (1994) examined response order effects that might be produced in a typical advertiser-sponsored study that measures readership and usage of the business-to-business publications in one industry. Like the Whipple and McManamon (1992) study, their study used recent magazine covers as response stimuli. Unlike the Whipple and McManamon (1992) study, individuals who received the trade publications of interest were asked to select magazines they regularly read from a list of the publications directed toward the industry.

Objectives of the Study

To accomplish the objectives of this study, two hypotheses were tested. First, would measuring readership produce a primacy effect even if the list is relatively short and respondents are not limited in the number of alternatives they may choose?

H1 - The relative order in which a magazine was listed would not impact on the number of respondents who indicated that they read the magazine on a regular basis.

Second, would a more specific evaluative question be less likely to produce a primacy effect?

H2 - The relative order in which a magazine was listed would not impact on the number of respondents who indicated that they used a magazine to obtain information relevant to the purchase of supermarket equipment.

RESEARCH METHODOLOGY

A magazine readership survey similar to the one used by Sekely and Blakney (1994) was sent to supermarket buyers responsible for purchasing ovens and big shop equipment. Each survey contained a general question in which respondents selected trade magazines they read on a regular basis. It also contained a more specific evaluation question in which respondents identified trade magazines they would use to help them make equipment purchases. The survey provided a list of four magazine alternatives, and respondents were free to select any number of the magazine alternatives listed.

Questionnaire Design

Each questionnaire contained the following parts:

Identification of four trade magazines by name and cover photograph that were red by respondents on a regular basis;

Identification of four trade magazines by name and cover photograph that were used by respondents when looking for ideas for equipment purchase.

Classification Questions:

Type of food service business in which respondent worked;

Major area of responsibility of respondent.

The questionnaires were presented using an independent research designation to avoid any bias by connecting the survey with any magazine or equipment manufacturer. The magazines were listed in two columns in identical order. The first column had a question asking respondents to indicate which of the magazines they read on a regular basis and the second had a question asking respondents to indicate which of the magazines they would use to obtain information about purchasing supermarket equipment. A small picture of the most recent cover of each magazine was placed between the two lists to refresh the respondents as to how the magazines looked. The order of the magazines was rotated, so that one-fourth of the questionnaires had each magazine in each position in the list.

Data Collection

The survey was mailed to a random sample of 1282 various supermarket department managers responsible for equipment purchases. Two hundred thirty-two usable responses were returned for a response rate of 18.1 percent.

The response rate was somewhat lower than hoped. However, a number of studies done on response rates of industrial and commercial respondents have shown wide variability of response rates from these groups. While there have been some studies conducted with response rates greater than fifty percent, a great many have produced response rates from under ten percent to twenty-five percent. [Duhan and Wilson 1990, London and Dommeyer 1990, Jobber and Sanderson 1985, Swan et al. Ferrell and Krugman 1983, and Jones 1980] Additionally, the present study deals with the reaction of individuals to the placement of response items and their likelihood of being influenced by the ordering of the responses. This is an internal reaction and is not as likely to be impacted by non-response bias as are studies comparing differences between groups of respondents.

RESULTS

The overall results indicate that a relatively short list of alternatives and no limit on the selection of alternatives measuring magazine readership in the manner described earlier does not produce a primacy effect. Specific findings are summarized in the following sections.

Differences in Readership by Position

Supermarket equipment buyers did not report significantly higher readership of magazines listed in the first position in the survey (See Table 1.). The overall readership average for all magazines in all positions was 65.67 percent, compared to an average of 66.75 percent when the magazine was in the first position. Chi-square analysis was run on the position differences in readership. Chi-square is a statistical test used in this instance to determine whether some observed pattern of frequencies corresponds to an expected pattern of frequencies. This showed no significant differences related to position. The relative order in which a magazine was listed had no significant impact on the number of respondents who indicated that they read the magazine on a regular basis. H1 was therefore not rejected.

Differences in Equipment Searching by Position

Supermarket equipment buyers also did not show a position bias when evaluating a magazine's value for purchasing decisions. The percentage of respondents indicating that they used a magazine for equipment information when it was in the first position was not significantly higher than the overall average in any of the magazines. The overall usage indication average for all magazines in all positions was 34.41 percent, compared to an average of 32.51 percent when the magazine was in the first position. Chi-square analysis was run on the position differences in usage which showed that there were no significant differences caused by position of the magazine in the list. Table 2 shows the results of the supermarket equipment buyer survey with respect to usage differences.

The results indicate that the more specific, evaluative question did not produce a primacy effect. There was no significant difference in usage indication if the magazine were listed in any particular position in the list for either group of respondents. The relative order in which a magazine was listed did not impact the number of respondents who indicated that they used a magazine to obtain information relevant to the purchase of supermarket equipment. Therefore, H2 was not rejected.

DISCUSSION

General questions often used by advertisers to measure business-to-business publication readership have been shown to be susceptible to primacy effects. Current theory states that shorter lists would be less susceptible to primacy problems, however there is no guidance as to how short a list should be. As a result, many advertisers, when trying to evaluate the readership of a trade magazine or the selection of specific trade magazines for different uses, routinely rotate the magazine response positions to help reduce the effect of the position on whether or not the respondent has false recall. Primacy effects in psychology, indicating higher selection of items in the first position, have been uncovered with lists of 13 and 16. Other research indicates that even lists as short as 7 can produce response-order bias, when the question is very general and does not require very much thought. This study’s findings indicate that primacy effects are not of a concern in lists of four or fewer, even with very general questions. Thus, researchers concerned with trying to reduce false selection results, should probably rotate position when there are seven or more responses and the question has very low involvement or requires very little thinking on the part of the respondent, but not worry about it when there are relatively few responses.

In situations where the questions are more specific or thought provoking, the concern for response position is less, even in situations involving more response choices. The current study found no position bias when four responses were presented and the response required some thought or interruption. This is consistent with the Sekely and Blakney study, which showed no bias with as many as seven response options. Thus, advertisers who are attempting to answer more specific questions, should not be concerned with rotating responses until the response selections reach a relatively high number.

SOURCES

Becker, S. L. "Why An Order Effect?" Public Opinion Quarterly, 18 (1954): 271-278.

Belson, William A. "The Effects of Reversing the Presentation Order of Verbal Rating Scales," Journal of Advertising Research 6, 4 (1966): 30-37.

Blunch, Niels J. "Position Bias in Multiple Choice Questions," Journal of Marketing Research 21, 2 (1984): 216-220.

Brook, D. and G. J. Upton. "Bias in Local Government Elections Due to Position on the Ballot Paper," Applied Statistics 23 (1974): 414-419.

Crano, William D. "Primacy Versus Recency in Retention of Information and Opinion Change," The Journal of Social Psychology 101 (1977): 87-96.

Duhan, Dale F. and R. Dale Wilson. "Prenotification and Industrial Survey Responses," Industrial Marketing Management 19 (1990): 95-105.

Ferrell, O. C. and D. Krugman. "Response Patterns and the Importance of the Follow-Up Duplicate Questionnaire in a Mail Survey of Advertising Managers," European Research (Oct. 1983): 157-163.

Herzog, A. Regula and Jerald G. Bachman. "Effects of Questionnaire Length on Response Quality," Public Opinion Quarterly 45 (1981): 549-559.

Jobber, David and Stuart Sanderson. "The Effect of Two Variables on Industrial Mail Survey Returns," Industrial Marketing Management 14 (1985): 119-121.

Jones, Wesley H. "Sample Composition Bias and Response Bias in a Mail Survey: A Comparison of Inducement Methods," Journal of Marketing Research 17, no.1 (1980): 69-76.

Knauper, Barbel. "The Impact of Age and Education on Response Order Effects in Attitude Measurement," Public Opinion Quarterly 63, 3 (1999): 347-370.

Krosnick, Jon A. and Duane F. Alwin. "An Evaluation of a Cognitive Theory of Response-Order Effects in Survey Measurement," Public Opinion Quarterly 51, 2 (1987): 201-219.

Krosnick, Jon A. and Sowmya Narayan. "Education Moderates Some Response Effects in Attitude Measurement," Public Opinion Quarterly 60, 1 (1996): 58-89.

London, Sandra J. and Curt J. Dommeyer. "Increasing Response to Industrial Mail Surveys," Industrial Marketing Management 19 (1990): 235-241.

McFarland, Sam G. "Effects of Question Order on Survey Responses," Public Opinion Quarterly 45 (1981): 208-215.

Morelli, Thomas L. "Business-to-Business Readership Research." Madison Avenue 28, 1 (1986): 101-02.

Mueller, J. E. "Choosing Among 133 Candidates," Public Opinion Quarterly, 34 (1970): 395-402.

Payne, Stanley L. The Art of Asking Questions. Princeton: Princeton University Press, 1951.

Rugg, D. and H. Cantril. "The Wording of Questions." In H. Cantril ed., Gauging Public Opinion. Princeton: Princeton University Press, 1944.

Schuman, Howard and Stanley Presser. Questions and Answers in Attitude Surveys. New York: Academic Press, 1981.

Sekely, William S.and Vicki L. Blakney. "The Effect of Response Position on Trade Magazine Readership and Usage." Journal of Advertising Research 34, 6 (1994), 53-59.

Swan, J. E.; D. E. Epley; and W. L. Burns. "Can Follow-Up Response Rates to a Mail Survey be Increased by Including Another Copy of the Questionnaire?" Psychological Reports 47 (1980) 103-106.

Whipple, Thomas W. and Mary K. McManamon. "Primacy Order Effects in the Measurement of Trade Magazine Receipt and Readership." Journal of Advertising Research Sept./Oct. (1992): 24-29.