Sample Size Calculator

This Sample Size Calculator is presented as a public service of Creative Research Systems
survey software. You can use it to determine how many people you need to interview in order to
get results that reflect the target population as precisely as needed. You can also find the level of
precision you have in an existing sample.

Before using the sample size calculator, there are two terms that you need to know. These are:
confidence interval and confidence level. If you are not familiar with these terms, click here. To
learn more about the factors that affect the size of confidence intervals, click here.

Enter your choices in a calculator below to find the sample size you need or the confidence
interval you have. Leave the Population box blank, if the population is very large or unknown.

Top of Form
Determine Sample Size
Confidence Level: 95% 99%
Confidence Interval:
Population:

Sample size needed:

Top of Form

Find Confidence Interval

Confidence Level: 95% 99%

Sample Size:

Population:

Percentage:
 Confidence Interval:

Sample Size Calculator Terms: Confidence Interval & Confidence Level

The confidence interval (also called margin of error) is the plus-or-minus figure usually
reported in newspaper or television opinion poll results. For example, if you use a confidence
interval of 4 and 47% percent of your sample picks an answer you can be "sure" that if you had
asked the question of the entire relevant population between 43% (47-4) and 51% (47+4) would
have picked that answer.

The confidence level tells you how sure you can be. It is expressed as a percentage and
represents how often the true percentage of the population who would pick an answer lies within
the confidence interval. The 95% confidence level means you can be 95% certain; the 99%
confidence level means you can be 99% certain. Most researchers use the 95% confidence level.

When you put the confidence level and the confidence interval together, you can say that you are
95% sure that the true percentage of the population is between 43% and 51%. The wider the
confidence interval you are willing to accept, the more certain you can be that the whole
population answers would be within that range.

For example, if you asked a sample of 1000 people in a city which brand of cola they preferred,
and 60% said Brand A, you can be very certain that between 40 and 80% of all the people in the
city actually do prefer that brand, but you cannot be so sure that between 59 and 61% of the
people in the city prefer the brand.

Factors that Affect Confidence Intervals

There are three factors that determine the size of the confidence interval for a given confidence
level:

 Sample size
 Percentage
 Population size

Sample Size

The larger your sample size, the more sure you can be that their answers truly reflect the
population. This indicates that for a given confidence level, the larger your sample size, the
smaller your confidence interval. However, the relationship is not linear (i.e., doubling the
sample size does not halve the confidence interval).

Percentage
Your accuracy also depends on the percentage of your sample that picks a particular answer. If
99% of your sample said "Yes" and 1% said "No," the chances of error are remote, irrespective
of sample size. However, if the percentages are 51% and 49% the chances of error are much
greater. It is easier to be sure of extreme answers than of middle-of-the-road ones.

When determining the sample size needed for a given level of accuracy you must use the worst
case percentage (50%). You should also use this percentage if you want to determine a general
level of accuracy for a sample you already have. To determine the confidence interval for a
specific answer your sample has given, you can use the percentage picking that answer and get a
smaller interval.

Population Size

How many people are there in the group your sample represents? This may be the number of
people in a city you are studying, the number of people who buy new cars, etc. Often you may
not know the exact population size. This is not a problem. The mathematics of probability proves
the size of the population is irrelevant unless the size of the sample exceeds a few percent of the
total population you are examining. This means that a sample of 500 people is equally useful in
examining the opinions of a state of 15,000,000 as it would a city of 100,000. For this reason,
The Survey System ignores the population size when it is "large" or unknown. Population size is
only likely to be a factor when you work with a relatively small and known group of people (e.g.,
the members of an association).

The confidence interval calculations assume you have a genuine random sample of the relevant
population. If your sample is not truly random, you cannot rely on the intervals. Non-random
samples usually result from some flaw in the sampling procedure. An example of such a flaw is
to only call people during the day and miss almost everyone who works. For most purposes, the
non-working population cannot be assumed to accurately represent the entire (working and non-
working) population.

Sample Size Formulas for our Sample Size Calculator

Here are the formulas used in our Sample Size Calculator:

Sample Size

Z 2 * (p) * (1-p)
ss =
c2

Where:

Z = Z value (e.g. 1.96 for 95% confidence level)

p = percentage picking a choice, expressed as decimal
(.5 used for sample size needed)
c = confidence interval, expressed as decimal
(e.g., .04 = ±4)

Correction for Finite Population

ss
new ss =
ss-1
1+
pop

Where: pop = population

Significance in Statistics & Surveys

"Significance level" is a misleading term that many researchers do not fully understand. This
article may help you understand the concept of statistical significance and the meaning of the
numbers produced by The Survey System.

This article is presented in two parts. The first part simplifies the concept of statistical
significance as much as possible; so that non-technical readers can use the concept to help make
decisions based on their data. The second part provides more technical readers with a fuller
discussion of the exact meaning of statistical significance numbers.

Part One: What is Statistical Significance?

In normal English, "significant" means important, while in Statistics "significant" means
probably true (not due to chance). A research finding may be true without being important.
When statisticians say a result is "highly significant" they mean it is very probably true. They do
not (necessarily) mean it is highly important.

Take a look at the table below. The chi (pronounced kie like pie) squares at the bottom of the
table show two rows of numbers. The top row numbers of 0.07 and 24.4 are the chi square
statistics themselves. The meaning of these statistics may be ignored for the purposes of this
article. The second row contains values .795 and .001. These are the significance levels and are
explained following the table.
Significance levels show you how likely a result is due to chance. The most common level, used
to mean something is good enough to be believed, is .95. This means that the finding has a 95%
chance of being true. However, this value is also used in a misleading way. No statistical
package will show you "95%" or ".95" to indicate this level. Instead it will show you ".05,"
meaning that the finding has a five percent (.05) chance of not being true, which is the converse
of a 95% chance of being true. To find the significance level, subtract the number shown from
one. For example, a value of ".01" means that there is a 99% (1-.01=.99) chance of it being true.
In this table, there is probably no difference in purchases of gasoline X by people in the city
center and the suburbs, because the probability is .795 (i.e., there is only a 20.5% chance that the
difference is true). In contrast the high significance level for type of vehicle (.001 or 99.9%)
indicates there is almost certainly a true difference in purchases of Brand X by owners of
different vehicles in the population from which the sample was drawn.

The Survey System uses significance levels with several statistics. In all cases, the p value tells
you how likely something is to be not true. If a chi square test shows probability of .04, it means
that there is a 96% (1-.04=.96) chance that the answers given by different groups in a banner
really are different. If a t-test reports a probability of .07, it means that there is a 93% chance that
the two means being compared would be truly different if you looked at the entire population.

People sometimes think that the 95% level is sacred when looking at significance levels. If a test
shows a .06 probability, it means that it has a 94% chance of being true. You can't be quite as
sure about it as if it had a 95% chance of being be true, but the odds still are that it is true. The
95% level comes from academic publications, where a theory usually has to have at least a 95%
chance of being true to be considered worth telling people about. In the business world if
something has a 90% chance of being true (probability =.1), it can't be considered proven, but it
is probably better to act as if it were true rather than false.

If you do a large number of tests, falsely significant results are a problem. Remember that a 95%
chance of something being true means there is a 5% chance of it being false. This means that of
every 100 tests that show results significant at the 95% level, the odds are that five of them do so
falsely. If you took a totally random, meaningless set of data and did 100 significance tests, the
odds are that five tests would be falsely reported significant. As you can see, the more tests you
do, the more of a problem these false positives are. You cannot tell which the false results are -
you just know they are there.

Limiting the number of tests to a small group chosen before the data is collected is one way to
reduce the problem. If this isn't practical, there are other ways of solving this problem. The best
approach from a statistical point of view is to repeat the study and see if you get the same results.
If something is statistically significant in two separate studies, it is probably true. In real life it is
not usually practical to repeat a survey, but you can use the "split halves" technique of dividing
your sample randomly into two halves and do the tests on each. If something is significant in
both halves, it is probably true. The main problem with this technique is that when you halve the
sample size, a difference has to be larger to be statistically significant.

The last common error is also important. Most significance tests assume you have a truly random
sample. If your sample is not truly random, a significance test may overstate the accuracy of the
results, because it only considers random error. The test cannot consider biases resulting from
non-random error (for example a badly selected sample).

To summarize:

 In statistical terms, significant does not necessarily mean important.

 Probability values should be read in reverse (1 - p).
 Too many significance tests will turn up some falsely significant relationships.
 Check your sampling procedure to avoid bias.

Part Two - The Exact Meaning of Statistical Significance

Numbers
The preceding discussion recommends reading probability values in reverse (1 - p). Doing so
will normally lead to correct decision making, but it is something of an over-simplification from
the technical point of view. A more complex, technically correct discussion is presented here.

Unfortunately, statistical significance numbers do not directly tell us exactly what we want to
know. They tell us how likely we would be to get differences between groups in our sample that
are as large or larger than those we see, if there were no differences between the corresponding
groups in the population represented by our sample. In other words, these numbers tell us how
likely is our data, given the assumption that there are no differences in the population. What we
want to know is how likely there are differences in the population, given our data.

Logically, if we are sufficiently unlikely to get a difference found in our sample, if there were no
difference in the population, then it is likely that there is a difference in the population. We used
this logic in the first part of this article when we said that you can interpret significance numbers
by considering 1-p as the probability that there is a difference in the population (where p is the
significance number produced by the program). For example, if the significance level is .05 then
you could consider the likelihood that there is a difference in the population to be 95% (1-.05).

While this logic passes the common sense test, the mathematics behind statistical significance do
not actually guarantee that 1-p gives the exact probability that there is a difference is the
population. Even so, many researchers treat 1-p as that probability anyway for two reasons. One
is that no one has devised a better general-purpose measure. The other is that using this
calculation will usually lead one to a useful interpretation of statistical significance numbers.

In some non-survey fields of research, the possibility that 1-p is not the exact probability that
there is a difference in the population may be more important. In these fields, the use of
statistical significance numbers may be controversial.

Survey Design
Knowing what the client wants is the key factor to success in any type of business. News media,
government agencies and political candidates need to know what the public thinks. Associations
need to know what their members want. Large companies need to measure the attitudes of their
employees. The best way to find this information is to conduct a survey.

This chapter is intended primarily for those who are new to survey research. It discusses options
and provides suggestions on how to design and conduct a successful survey project. It does not
provide instruction on using specific parts of The Survey System, although it mentions parts of
the program that can help you with certain tasks.

The Steps in a Survey Project

1. Establish the goals of the project - What you want to learn

2. Determine your sample - Whom you will interview

3. Choose interviewing methodology - How you will interview

4. Create your questionnaire - What you will ask

5. Pre-test the questionnaire, if practical - Test the questions

6. Conduct interviews and enter data - Ask the questions

7. Analyze the data - Produce the reports

This page covers the first five steps. The Survey System's Tutorial Chapters 1 and 2 cover
entering data and producing reports.

Establishing Goals
The first step in any survey is deciding what you want to learn. The goals of the project
determine whom you will survey and what you will ask them. If your goals are unclear, the
results will probably be unclear. Some typical goals include learning more about:

 The potential market for a new product or service

 Ratings of current products or services

 Employee attitudes

 Customer/patient satisfaction levels

 Reader/viewer/listener opinions

 Association member opinions

 Opinions about political candidates or issues

 Corporate images

These sample goals represent general areas. The more specific you can make your goals, the
easier it will be to get usable answers.

Selecting Your Sample

There are two main components in determining whom you will interview. The first is deciding
what kind of people to interview. Researchers often call this group the target population. If you
conduct an employee attitude survey or an association membership survey, the population is
obvious. If you are trying to determine the likely success of a product, the target population may
be less obvious. Correctly determining the target population is critical. If you do not interview
the right kinds of people, you will not successfully meet your goals.

The next thing to decide is how many people you need to interview. Statisticians know that a
small, representative sample will reflect the group from which it is drawn. The larger the sample,
the more precisely it reflects the target group. However, the rate of improvement in the precision
decreases as your sample size increases. For example, to increase a sample from 250 to 1,000
only doubles the precision. You must make a decision about your sample size based on factors
such as: time available, budget and necessary degree of precision.

The Survey System (and this Web site) includes a sample size calculator that can help you decide
on the sample size (jump to the calculator page for a general discussion of sample size
considerations).

Avoiding a Biased Sample

A biased sample will produce biased results. Totally excluding all bias is almost impossible;
however, if you recognize bias exists you can intuitively discount some of the answers. The
following list shows some examples of biased samples.
The consequences of a source of bias depend on the nature of the survey. For example, a survey
for a product aimed at retirees will not be as biased by daytime interviews as will a general
public opinion survey. A survey about Internet products can safely ignore people who do not use
the Internet.

Quotas

A Quota is a sample size for a sub-group. It is sometimes useful to establish quotas to ensure that
your sample accurately reflects relevant sub-groups in your target population. For example, men
and women have somewhat different opinions in many areas. If you want your survey to
accurately reflect the general population's opinions, you will want to ensure that the percentage
of men and women in your sample reflect their percentages of the general population.

If you are interviewing users of a particular type of product, you probably want to ensure that
users of the different current brands are represented in proportions that approximate the current
market share. Alternatively, you may want to ensure that you have enough users of each brand to
be able to analyze the users of each brand as a separate group. If you are doing telephone or Web
page interviewing, The Survey System's optional Sample Management or Internet Module can
help you enforce quotas. They let you create automatically enforced quotas and/or monitor your
sample during interviewing sessions.

Interviewing Methods
Once you have decided on your sample you must decide on your method of data collection. Each
method has advantages and disadvantages.

Personal Interviews
An interview is called personal when the Interviewer asks the questions face-to-face with the
Interviewee. Personal interviews can take place in the home, at a shopping mall, on the street,
outside a movie theater or polling place, and so on.

Advantages
 The ability to let the Interviewee see, feel and/or taste a product.
 The ability to find the target population. For example, you can find people who have seen a film
much more easily outside a theater in which it is playing than by calling phone numbers at
random.

 Longer interviews are sometimes tolerated. Particularly with in-home interviews that have been
arranged in advance. People may be willing to talk longer face-to-face than to someone on the
phone.

Disadvantages
 Personal interviews usually cost more per interview than other methods. This is particularly true
of in-home interviews, where travel time is a major factor.
 Each mall has its own characteristics. It draws its clientele from a specific geographic area
surrounding it, and its shop profile also influences the type of client. These characteristics may
differ from the target population and create a non-representative sample.

Telephone Surveys
Surveying by telephone is the most popular interviewing method in the USA. This is made
possible by nearly universal coverage (96% of homes have a telephone).

Advantages
 People can usually be contacted faster over the telephone than with other methods. If the
Interviewers are using CATI (computer-assisted telephone interviewing), the results can be
available minutes after completing the last interview.
 You can dial random telephone numbers when you do not have the actual telephone numbers
of potential respondents.

 CATI software, such as The Survey System, makes complex questionnaires practical by offering
many logic options. It can automatically skip questions, perform calculations and modify
questions based on the answers to earlier questions. It can check the logical consistency of
answers and can present questions or answers choices in a random order (the last two are
sometimes important for reasons described later).

 Skilled interviewers can often elicit longer or more complete answers than people will give on
their own to mail, email surveys (though some people will give longer answers to Web page
surveys). Interviewers can also ask for clarification of unclear responses.

 Some software, such as The Survey System, can combine survey answers with pre-existing
information you have about the people being interviewed.
Disadvantages
 Many telemarketers have given legitimate research a bad name by claiming to be doing research
when they start a sales call. Consequently, many people are reluctant to answer phone
interviews and use their answering machines to screen calls. Since over half of the homes in the
USA have answering machines, this problem is getting worse.
 The growing number of working women often means that no one is home during the day. This
limits calling time to a "window" of about 6-9 p.m. (when you can be sure to interrupt dinner or
a favorite TV program).

 You cannot show or sample products by phone.

Mail Surveys
Advantages
 Mail surveys are among the least expensive.
 This is the only kind of survey you can do if you have the names and addresses of the target
population, but not their telephone numbers.

 The questionnaire can include pictures - something that is not possible over the phone.

 Mail surveys allow the respondent to answer at their leisure, rather than at the often
inconvenient moment they are contacted for a phone or personal interview. For this reason,
they are not considered as intrusive as other kinds of interviews.

Disadvantages
 Time! Mail surveys take longer than other kinds. You will need to wait several weeks after
mailing out questionnaires before you can be sure that you have gotten most of the responses.
 In populations of lower educational and literacy levels, response rates to mail surveys are often
too small to be useful. This, in effect, eliminates many immigrant populations that form
substantial markets in many areas. Even in well-educated populations, response rates vary from
as low as 3% up to 90%. As a rule of thumb, the best response levels are achieved from highly-
educated people and people with a particular interest in the subject (which, depending on your
target population, could lead to a biased sample).

One way of improving response rates to mail surveys is to mail a postcard telling your sample to
watch for a questionnaire in the next week or two. Another is to follow up a questionnaire
mailing after a couple of weeks with a card asking people to return the questionnaire. The
downside is that this doubles or triples your mailing cost. If you have purchased a mailing list
from a supplier, you may also have to pay a second (and third) use fee - you often cannot buy the
list once and re-use it.

Another way to increase responses to mail surveys is to use an incentive. One possibility is to
send a dollar bill (or more) along with the survey (or offer to donate the dollar to a charity
specified by the respondent). If you do so, be sure to say that the dollar is a way of saying
"thanks," rather than payment for their time. Many people will consider their time worth more
than a dollar. Another possibility is to include the people who return completed surveys in a
drawing for a prize. A third is to offer a copy of the (non-confidential) result highlights to those
who complete the questionnaire. Any of these techniques will increase the response rates.

Remember that if you want a sample of 1,000 people, and you estimate a 10% response level,
you need to mail 10,000 questionnaires. You may want to check with your local post office about
bulk mail rates - you can save on postage using this mailing method. However, most researchers
do not use bulk mail, because many people associate "bulk" with "junk" and will throw it out
without opening the envelope, lowering your response rate. Also bulk mail moves slowly,
increasing the time needed to complete your project.

Computer Direct Interviews

These are interviews in which the Interviewees enter their own answers directly into a computer.
They can be used at malls, trade shows, offices, and so on. The Survey System's optional
Interviewing Module and Interview Stations can easily create computer-direct interviews. Some
researchers set up a Web page survey for this purpose.

Advantages
 The virtual elimination of data entry and editing costs.
 You will get more accurate answers to sensitive questions. Recent studies of potential blood
donors have shown respondents were more likely to reveal HIV-related risk factors to a
computer screen than to either human interviewers or paper questionnaires. The National
Institute of Justice has also found that computer-aided surveys among drug users get better
results than personal interviews. Employees are also more often willing to give more honest
answers to a computer than to a person or paper questionnaire.

 The elimination of interviewer bias. Different interviewers can ask questions in different ways,
leading to different results. The computer asks the questions the same way every time.

 Ensuring skip patterns are accurately followed. The Survey System can ensure people are not
asked questions they should skip based on their earlier answers. These automatic skips are more
accurate than relying on an Interviewer reading a paper questionnaire.

 Response rates are usually higher. Computer-aided interviewing is still novel enough that some
people will answer a computer interview when they would not have completed another kind of
interview.

Disadvantages
 The Interviewees must have access to a computer or one must be provided for them.
 As with mail surveys, computer direct interviews may have serious response rate problems in
populations of lower educational and literacy levels. This method may grow in importance as
computer use increases.

Email Surveys
Email surveys are both very economical and very fast. More people have email than have full
Internet access. This makes email a better choice than a Web page survey for some populations.
On the other hand, email surveys are limited to simple questionnaires, whereas Web page
surveys can include complex logic.

Advantages
 Speed. An email questionnaire can gather several thousand responses within a day or two.
 There is practically no cost involved once the set up has been completed.

 You can attach pictures and sound files.

 The novelty element of an email survey often stimulates higher response levels than ordinary
“snail” mail surveys.

Disadvantages
 You must possess (or purchase) a list of email addresses.
 Some people will respond several times or pass questionnaires along to friends to answer. Many
programs have no check to eliminate people responding multiple times to bias the results. The
Survey System’s Email Module will only accept one reply from each address sent the
questionnaire. It eliminates duplicate and pass along questionnaires and checks to ensure that
respondents have not ignored instructions (e.g., giving 2 answers to a question requesting only
one).

 Many people dislike unsolicited email even more than unsolicited regular mail. You may want to
send email questionnaires only to people who expect to get email from you.

 You cannot use email surveys to generalize findings to the whole populations. People who have
email are different from those who do not, even when matched on demographic characteristics,
such as age and gender.

 Email surveys cannot automatically skip questions or randomize question or answer choice
order or use other automatic techniques that can enhance surveys the way Web page surveys
can.

Many email programs are limited to plain ASCII text questionnaires and cannot show pictures.
Email questionnaires from The Survey System can attach graphic or sound files. Although use of
email is growing very rapidly, it is not universal - and is even less so outside the USA (three-
quarters of the world's email traffic takes place within the USA). Many “average” citizens still do
not possess email facilities, especially older people and those in lower income and education
groups. So email surveys do not reflect the population as a whole. At this stage they are probably
best used in a corporate environment where email is common or when most members of the
target population are known to have email.

Internet/Intranet (Web Page) Surveys

Web surveys are rapidly gaining popularity. They have major speed, cost, and flexibility
advantages, but also significant sampling limitations. These limitations make software selection
especially important and restrict the groups you can study using this technique.

Advantages
 Web page surveys are extremely fast. A questionnaire posted on a popular Web site can gather
several thousand responses within a few hours. Many people who will respond to an email
invitation to take a Web survey will do so the first day, and most will do so within a few days.
 There is practically no cost involved once the set up has been completed. Large samples do not
cost more than smaller ones (except for any cost to acquire the sample).

 You can show pictures. Some Web survey software can also show video and play sound.

 Web page questionnaires can use complex question skipping logic, randomizations and other
features not possible with paper questionnaires or most email surveys. These features can
assure better data.

 Web page questionnaires can use colors, fonts and other formatting options not possible in
most email surveys.

 A significant number of people will give more honest answers to questions about sensitive
topics, such as drug use or sex, when giving their answers to a computer, instead of to a person
or on paper.

 On average, people give longer answers to open-ended questions on Web page questionnaires
than they do on other kinds of self-administered surveys.

 Some Web survey software, such as The Survey System, can combine the survey answers with
pre-existing information you have about individuals taking a survey.

Disadvantages
 While growing every year, Internet use is not universal. Internet surveys do not reflect the
population as a whole. This is true even if a sample of Internet users is selected to match the
general population in terms of age, gender and other demographics.
 People can easily quit in the middle of a questionnaire. They are not as likely to complete a long
questionnaire on the Web as they would be if talking with a good interviewer.

 If your survey pops up on a web page, you often have no control over who replies - anyone from
Antartica to Zanzibar, cruising that web page may answer.

 Depending on your software, there is often no control over people responding multiple times to
bias the results.

At this stage we recommend using the Internet for surveys mainly when your target population
consists entirely or almost entirely of Internet users. Business-to-business research and employee
attitude surveys can often meet this requirement. Surveys of the general population usually will
not. That said, Internet surveys did about as well, and in some cases better, than other methods in
predicting the outcome of the 2012 U.S. presidential election.
Even when Internet users may not closely match your target population, a Web page survey may
be your best choice if you want to show video or both sound and graphics. A Web page survey
may be the only practical way to have many people view and react to a video.

In any case, be sure your survey software prevents people from completing more than one
questionnaire. You may also want to restrict access by requiring a password (good software
allows this option) or by putting the survey on a page that can only be accessed directly (i.e.,
there are no links to it from other pages).

Scanning Questionnaires
Scanning questionnaires is a method of data collection that can be used with paper questionnaires
that have been administered in face-to-face interviews; mail surveys or surveys completed by an
Interviewer over the telephone. The Survey System can produce paper questionnaires that can be
scanned using Remark Office OMR (available from CRS). Other software can scan
questionnaires and produce ASCII Files that can be read into The Survey System.

Advantages
 Scanning can be the fastest method of data entry for paper questionnaires.
 Scanning is more accurate than a person in reading a properly completed questionnaire.

Disadvantages
 Scanning is best-suited to "check the box" type surveys and bar codes. Scanning programs have
various methods to deal with text responses, but all require additional data entry time.
 Scanning is less forgiving (accurate) than a person in reading a poorly marked questionnaire.
Requires investment in additional hardware to do the actual scanning.

Summary of Survey Methods

Your choice of survey method will depend on several factors. These include:
Questionnaire Design
General Considerations

The first rule is to design the questionnaire to fit the medium. Phone interviews cannot show
pictures. People responding to mail or Web surveys cannot easily ask “What exactly do you
mean by that?” if they do not understand a question. Intimate, personal questions are sometimes
best handled by mail or computer, where anonymity is most assured.

KISS - keep it short and simple. If you present a 20-page questionnaire most potential
respondents will give up in horror before even starting. Ask yourself what you will do with the
information from each question. If you cannot give yourself a satisfactory answer, leave it out.
Avoid the temptation to add a few more questions just because you are doing a questionnaire
anyway. If necessary, place your questions into three groups: must know, useful to know and
nice to know. Discard the last group, unless the previous two groups are very short.

Start with an introduction or welcome message. In the case of mail or Web questionnaires, this
message can be in a cover page or on the questionnaire form itself. If you are sending emails that
ask people to take a Web page survey, put your main introduction or welcome message in the
email. When practical, state who you are and why you want the information in the survey. A
good introduction or welcome message will encourage people to complete your questionnaire.

Allow a “Don't Know” or “Not Applicable” response to all questions, except to those in which
you are certain that all respondents will have a clear answer. In most cases, these are wasted
answers as far as the researcher is concerned, but are necessary alternatives to avoid frustrated
respondents. Sometimes “Don't Know” or “Not Applicable” will really represent some
respondents' most honest answers to some of your questions. Respondents who feel they are
being coerced into giving an answer they do not want to give often do not complete the
questionnaire. For example, many people will abandon a questionnaire that asks them to specify
their income, without offering a "decline to state" choice.

For the same reason, include “Other” or “None” whenever either of these is a logically possible
answer. When the answer choices are a list of possible opinions, preferences, or behaviors, you
should usually allow these answers.

On paper, computer direct and Internet surveys these four choices should appear as appropriate.
You may want to combine two or more of them into one choice, if you have no interest in
distinguishing between them. You will rarely want to include “Don't Know,” “Not Applicable,”
“Other” or “None” in a list of choices being read over the telephone or in person, but you should
allow the interviewer the ability to accept them when given by respondents.

Question Types
Researchers use three basic types of questions: multiple choice, numeric open end and text open
end (sometimes called "verbatims"). Examples of each kind of question follow:
Rating Scales and Agreement Scales are two common types of questions that some researchers
treat as multiple choice questions and others treat as numeric open end questions. Examples of
these kinds of questions are:

Question and Answer Choice Order

There are two broad issues to keep in mind when considering question and answer choice order.
One is how the question and answer choice order can encourage people to complete your survey.
The other issue is how the order of questions or the order of answer choices could affect the
results of your survey.

Ideally, the early questions in a survey should be easy and pleasant to answer. These kinds of
questions encourage people to continue the survey. In telephone or personal interviews they help
build rapport with the interviewer. Grouping together questions on the same topic also makes the
questionnaire easier to answer.

Whenever possible leave difficult or sensitive questions until near the end of your survey. Any
rapport that has been built up will make it more likely people will answer these questions. If
people quit at that point anyway, at least they will have answered most of your questions.

Answer choice order can make individual questions easier or more difficult to answer. Whenever
there is a logical or natural order to answer choices, use it. Always present agree-disagree
choices in that order. Presenting them in disagree-agree order will seem odd. For the same
reason, positive to negative and excellent to poor scales should be presented in those orders.
When using numeric rating scales higher numbers should mean a more positive or more agreeing
answer.

Question order can affect the results in two ways. One is that mentioning something (an idea, an
issue, a brand) in one question can make people think of it while they answer a later question,
when they might not have thought of it if it had not been previously mentioned. In some cases
you may be able to reduce this problem by randomizing the order of related questions.
Separating related questions with unrelated ones can also reduce this problem, though neither
technique will eliminate it.

The other way question order can affect results is habituation. This problem applies to a series of
questions that all have the same answer choices. It means that some people will usually start
giving the same answer, without really considering it, after being asked a series of similar
questions. People tend to think more when asked the earlier questions in the series and so give
more accurate answers to them.

If you are using telephone, computer direct or Internet interviewing, good software can help with
this problem. Software should allow you to present a series of questions in a random order in
each interview. This technique will not eliminate habituation, but will ensure that it applies
equally to all questions in a series, not just to particular questions near the end of a series.

Another way to reduce this problem is to ask only a short series of similar questions at a
particular point in the questionnaire. Then ask one or more different kinds of questions, and then
another short series if needed.

A third way to reduce habituation is to change the “positive” answer. This applies mainly to
level-of-agreement questions. You can word some statements so that a high level of agreement
means satisfaction (e.g., “My supervisor gives me positive feedback”) and others so that a high
level of agreement means dissatisfaction (e.g., “My supervisor usually ignores my suggestions”).
This technique forces the respondent to think more about each question. One negative aspect of
this technique is that you may have to modify some of the data after the results are entered,
because having the higher levels of agreement always mean a positive (or negative) answer
makes the analysis much easier. However, the few minutes extra work may be a worthwhile
price to pay to get more accurate data.

The order in which the answer choices are presented can also affect the answers given. People
tend to pick the choices nearest the start of a list when they read the list themselves on paper or a
computer screen. People tend to pick the most recent answer when they hear a list of choices
read to them.

As mentioned previously, sometimes answer choices have a natural order (e.g., Yes, followed by
No; or Excellent - Good - Fair - Poor). If so, you should use that order. At other times, questions
have answers that are obvious to the person that is answering them (e.g., “Which brands of car
do you own?”). In these cases, the order in which the answer choices are presented is not likely
to affect the answers given. However, there are kinds of questions, particularly questions about
preference or recall or questions with relatively long answer choices that express an idea or
opinion, in which the answer choice order is more likely to affect which choice is picked. If you
are using telephone, computer direct, or Web page interviewing, have your software present
these kinds of answer choices in a random order.

Other General Tips

Keep the questionnaire as short as possible. We mentioned this principle before, but it is so
important it is worth repeating. More people will complete a shorter questionnaire, regardless of
the interviewing method. If a question is not necessary, do not include it.

Start with a Title (e.g., Leisure Activities Survey). Always include a short introduction - who you
are and why you are doing the survey. If you are asking about different brands, it is often a good
idea to give the name of the research company rather than the client (e.g., XYZ Research Agency
rather than the manufacturer of the product/ service being surveyed). Many firms create a
separate research company name (even if it is only a direct phone line to the research
department) to disguise themselves. This is to avoid possible bias, since people rarely like to
criticize someone to their face and are much more open to a third party.

In some cases, though, it may help to mention the client. If you are surveying members of an
organization, the members may be more likely to respond if they think the organization is asking
their opinions on how it can best meet their needs. The same could be true when you are
surveying users of a particular service.

Reassure your respondent that his or her responses will not be revealed to your client, but only
combined with many others to learn about overall attitudes.

Include a cover letter with all mail surveys. A good cover letter or invitation to take a Web page
survey will increase the response rate. A bad one, or none at all, will reduce the response rate.
Include the information in the preceding two paragraphs and mention the incentive (if any).
Describe how to return the questionnaire. Include the name and telephone number of someone
the respondent can call if they have any questions. Include instructions on how to complete the
survey itself.

The most effective cover letters and invitations include the following elements: Ask the recipient
to take the survey. Explain why taking it will improve some aspect of the recipient's life (it will
help improve a product, make an organization better meet their needs, make their opinions
heard). Appeal to the recipient's sense of altruism ("please help"). Ask the recipient again to take
the survey.

Number mail questionnaires on each page and include the return address on the questionnaire
itself, because pages and envelopes can be separated from each other. Envelopes should have
return postage prepaid. Using a postage stamp often increases response rates, but is expensive,
since you must stamp every envelope - not just the returned ones.

You may want to leave a space for the respondent to add their name and title. Some people will
put in their names, making it possible for you to recontact them for clarification or follow-up
questions. Indicate that filling in their name is optional. If the questions are sensitive in nature,
do not have a space for a name. Some people would become suspicious and not complete the
survey.

If you hand out questionnaires on your premises, you obviously cannot remain anonymous, but
keep the bias problem in mind when you consider the answers.

If the survey contains commercially sensitive material, ask a "security" question up front to find
whether the respondent or any member of his family, household or any close friend works in the
industry being surveyed. If so, terminate the interview immediately. They (or family or friends)
may work for the company that commissioned the survey - or for a competitor. In either case,
they are not representative and should be eliminated. If they work for a competitor, the nature of
the questions may betray valuable secrets. The best way to ask security questions is in reverse
(i.e., if you are surveying for a pharmaceutical product, phrase the question as "We want to
interview people in certain industries - do you or any member of your household work in the
pharmaceutical industry?). If the answer is "Yes" thank the respondent and terminate the
interview. Similarly, it is best to eliminate people working in the advertising, market research or
media industries, since they may work with competing companies.

After the security question, start with general questions. If you want to limit the survey to users
of a particular product, you may want to disguise the qualifying product. As a rule, start from
general attitudes to the class of products, through brand awareness, purchase patterns, specific
product usage to questions on specific problems (i.e., work from "What types of coffee have you
bought in the last three months" to "Do you recall seeing a special offer on your last purchase of
Brand X coffee?"). If possible put the most important questions into the first half of the survey. If
a person gives up half way through, at least you have the most important information.

Make sure you include all the relevant alternatives as answer choices. Leaving out a choice can
give misleading results. For example, a number of recent polls that ask Americans if they support
the death penalty "Yes" or "No" have found 70-75% of the respondents choosing ”Yes.” Polls
that offer the choice between the death penalty and life in prison without the possibility of parole
show support for the death penalty at about 50-60%. Polls that offer the alternatives of the death
penalty or life in prison without the possibility of parole, with the inmates working in prison to
pay restitution to their victims’ families have found support for the death penalty closer to 30%.

So what is the true level of support for the death penalty? The lowest figure is probably truest,
since it represents the percentage that favor that option regardless of the alternatives offered. The
need to include all relevant alternatives is not limited to political polls. You can get misleading
data anytime you leave out alternatives.

Do not put two questions into one. Avoid questions such as "Do you buy frozen meat and frozen
fish?" A "Yes" answer can mean the respondent buys meat or fish or both. Similarly with a
question such as "Have you ever bought Product X and, if so, did you like it?" A "No" answer
can mean "never bought" or "bought and disliked." Be as specific as possible. "Do you ever buy
pasta?" can include someone who once bought some in 1990. It does not tell you whether the
pasta was dried, frozen or canned and may include someone who had pasta in a restaurant. It is
better to say "Have you bought pasta (other than in a restaurant) in the last three months?" "If
yes, was it frozen, canned or dried?" Few people can remember what they bought more than
three months ago unless it was a major purchase such as an automobile or appliance.

The overriding consideration in questionnaire design is to make sure your questions can
accurately tell you what you want to learn. The way you phrase a question can change the
answers you get. Try to make sure the wording does not favor one answer choice over another.

Avoid emotionally charged words or leading questions that point towards a certain answer. You
will get different answers from asking "What do you think of the XYZ proposal?" than from
"What do you think of the Republican XYZ proposal?" The word "Republican" in the second
question would cause some people to favor or oppose the proposal based on their feelings about
Republicans, rather than about the proposal itself. It is very easy to create bias in a questionnaire.
This is another good reason to test it before going ahead.

If you are comparing different products to find preferences, give each one a neutral name or
reference. Do not call one "A" and the second one "B." This immediately brings images of A
grades and B grades to mind, with the former being seen as superior to the latter. It is better to
give each a "neutral" reference such "M" or "N" that do not have as strong a quality difference
image.

Avoid technical terms and acronyms, unless you are absolutely sure that respondents know they
mean. LAUTRO, AGI, GPA, EIEIO (Life Assurance and Unit Trust Regulatory Organization,
Adjusted Gross Income, Grade Point Average and Engineering Information External Inquiries
Officer) are all well-known acronyms to people in those particular fields, but very few people
would understand all of them. If you must use an acronym, spell it out the first time it is used.

Make sure your questions accept all the possible answers. A question like "Do you use regular or
premium gas in your car?" does not cover all possible answers. The owner may alternate
between both types. The question also ignores the possibility of diesel or electric-powered cars.
A better way of asking this question would be "Which type(s) of fuel do you use in your cars?"
The responses allowed might be:

If you want only one answer from each person, ensure that the options are mutually exclusive.
For example:

This question ignores the possibility of someone living in a house or an apartment in the suburbs.

Score or rating scale questions (e.g., "If '5' means very good and '1' means very poor how would
rate this product?") are a particular problem. Researchers are very divided on this issue. Many
surveys use a ten-point scale, but there is considerable evidence to suggest that anything over a
five point scale is irrelevant. This depends partially on education. Among university graduates a
ten point scale will work well. Among people with less than a high school education five points
is sufficient. In third world countries, a three-point scale (good/acceptable/bad) may be all some
respondents can understand.

Giving a verbal or written label to each point on a scale, instead of just the endpoints, will
usually yield higher-quality data, though this may not be practical when there are more than five
points on the scale.

Another issue on which researchers differ is whether to use a scale with an odd or even number
of points. Some like to force people to give an answer that is clearly positive or negative. This
can make the analysis easier. Others feel it is important to offer a neutral, middle option. Your
interviewing mode can make a difference here. A good interviewer can often get an answer, but
in a self-administered interview, such as a Web page survey, a person who is frustrated by being
unable to give a middle answer may leave a question blank or quit the survey altogether.

Be sure any rating scale labels are meaningful. For example:

A question phrased like the one above will force most answers into the middle category,
resulting in very little usable information.

If you have used a particular scale before and need to compare results, use the same scale. Four
on a five-point scale is not equivalent to eight on a ten-point scale. Someone who rates an item
"4" on a five-point scale might rate that item anywhere between "6" and "9" on a ten-point scale.

Do not use negative numbers when asking for ratings. Some people do not like to give negative
numbers as answers. A scale of -2 to +2 is mathematically equivalent to a scale of 1 to 5, but in
practice you will get fewer people picking -2 or -1 than would pick 1 or 2. If you want 0 to be the
midpoint of a scale when you produce reports, you can weight the answers after data collection
to get that result.

Be aware of cultural factors. In the third world, respondents have a strong tendency to exaggerate
answers. Researchers may be perceived as being government agents, with the power to punish or
reward according to the answer given. Accordingly they often give "correct" answers rather than
what they really believe. Even when the questions are not overtly political and deal purely with
commercial products or services, the desire not to disappoint important visitors with answers that
may be considered negative may lead to exaggerated scores.

Always discount "favorable" answers by a significant factor. The desire to please is not limited
to the third world. Unfortunately, there is no hard and fast rule on how much to do this. It
depends on the situation. One factor to consider is the survey mode. People tend to pick the most
positive answer on a scale more often when answering telephone surveys than other types of
surveys, regardless of the details of the question.

The desire to please translates into a tendency to pick agreeing answers on agreement scales.
While logically the percentage that strongly agrees that "X is good" should exactly equal the
percentage that strongly disagrees that "X is bad," in the real world, this is unlikely to be true.
Experiments have shown that more people will agree than disagree. One way to eliminate this
problem is to ask half your respondents if they agree that "X is good" and the other half if they
agree that "X is bad." You could then reverse the answers given by the second group. This is
extra work, but it may be worth it if it is important to get the most accurate percentage of people
who really agree with something.

People sometimes give answers they feel will reflect well on them. This is a constant problem for
pre-election polls. More people say they will vote than actually will vote. More people say they
go to museums or libraries than actually do. This problem is most significant when your
respondents are talking directly to a person. People give more honest answers when answering
questions on a computer. Mail surveys are in-between.
Because people like to think of themselves as normal or average, the range of answer choices
you give when asking for a quantity or a frequency can affect the results. For example if you ask
people how many hours of television they watch in a day and you offer the choices:

you will get fewer people picking 4 or more than if you offered the choices:

The first list of choices makes 4 hours sound extreme, while the second list of choices makes it
seem typical.

In personal interviews it is vital for the Interviewer to have empathy with the Interviewee. In
general, Interviewers should try to "blend" with respondents in terms of race, language, sex, age,
etc. Choose your Interviewers according to the likely respondents.

Leave your demographic questions (age, gender, income, education, etc.) until the end of the
questionnaire. By then the interviewer should have built a rapport with the interviewee that will
allow honest responses to such personal questions. Mail and Internet questionnaires should do
the same, although the rapport must be built by good question design, rather than personality.
Exceptions to this rule are any demographic questions that qualify someone to be included in the
survey. For example, many researchers limit some surveys to people in certain age groups. These
questions must come near the beginning.
Do not have an interviewer ask a respondent's gender, unless they really have no idea. Have the
interviewer fill in the answer themselves.

Paper questionnaires requiring text answers, should always leave sufficient space for handwritten
answers. Lines should be about half-an-inch (one cm.) apart. The number of lines you should
have depends on the question. Three to five lines are average.

Leave a space at the end of a questionnaire entitled "Other Comments." Sometimes respondents
offer casual remarks that are worth their weight in gold and cover some area you did not think of,
but which respondents consider critical. Many products have a wide range of secondary uses that
the manufacturer knows nothing about but which could provide a valuable source of extra sales
if approached properly. In one third world market, a major factor in the sale of candles was the
ability to use the spent wax as floor polish - but the manufacturer only discovered this by a
chance remark.

Always consider the layout of your questionnaire. This is especially important on paper,
computer direct and Internet surveys. You want to make it attractive, easy to understand and easy
to complete. If you are creating a paper survey, you also want to make it easy for your data entry
personnel.

Try to keep your answer spaces in a straight line, either horizontally or vertically. A single
answer choice on each line is best. Eye tracking studies show the best place to use for answer
spaces is the right hand edge of the page. It is much easier for a field worker or respondent to
follow a logical flow across or down a page. Using the right edge is also easiest for data entry.

The Survey System lets you create a Questionnaire Form with the answer choices in two
columns. Creating the form that way can save a lot of paper or screen space, but you should
recognize doing so makes the questionnaire a little harder to complete. It also slows the data
entry process when working with paper questionnaires.

Questions and answer choice grids, as in the second of the following examples, are popular with
many researchers. They can look attractive and save paper, or computer screen space. They also
can avoid a long series of very repetitive question and answer choice lists. Unfortunately, they
also are a bit harder than the repeated lists for some people to understand. As always, consider
whom you are studying when you create your questionnaire.

Look at the following layouts and decide which you would prefer to use:
An alternative layout is:

The second example shows the answer choices in neat columns and has more space between the
lines. It is easier to read. The numbers in the second example will also speed data entry, if you
are using a paper questionnaire.

When using a grid like the above example the way you lay out the choices will affect the results.
It is not clear whether it is always best to make the leftmost column the most positive response or
the most negative response, but there is a tendency for people to pick the left side of the grid
more than the right side, regardless of whether the left side is positive or negative. So any time
you use a grid you should discount the left side responses to some degree. This tendency does
not interfere with comparing the answers in different rows in the grid, but does affect absolute
statements such as "45% agree that the company has a good vacation policy."

Surveys are a mixture of science and art, and a good researcher will save their cost many times
over by knowing how to ask the correct questions.

Additional Tips for Web Surveys

One principle is to consider good Web page design when creating your survey pages. Do not use
too many colors or fonts. They are distracting. On the other hand, bolding, italicizing, and
changing the colors of key words, used appropriately, can make your questions easier to
understand. Using color and/or a smaller font size to make instructions distinct from question
text can make your questionnaire easier to follow.

Always specify a background color, even if it is white (usually a good choice). Some browsers
may show a background color you do not expect, if you do not specify one. Background images
usually make text harder to read, even when they make a page more attractive at first glance.

Use graphics sparingly. Some home Internet users still connect via modems, and graphics slow
download times. Remember that showing a large graphic at a small size on a Web page does not
reduce the time needed to download the graphic. Create or modify the graphic to a file size that
is no bigger than you need. If your sample consists of people at work, you may use more
graphics, since those people usually have faster connections, but even they appreciate faster
downloads.

Use video only if that is what you are testing (e.g., a commercial). Make sure you do not require
people to scroll horizontally to view part of the survey page. Most people find horizontal
scrolling annoying. Question text wraps to fit the available space, but you can make a grid that is
wider than some screens. As of January 2013 about 9% of people still use 1024x768 screen
resolution. You may want to design your pages to be up to 980 pixels wide (leaving room for the
browser edges and a scrollbar). In any case, you should not ask opinions on any graphic wider
than that, since some people will have to scroll to see it.

Smartphones are a special consideration. Many newer ones have screens with 1024x768 or
higher resolutions, but their screen sizes are so small that font sizes that are perfectly readable on
a computer-sized screen may be unreadable without zooming in on a phone. If you think that
many of your respondents will be taking your survey on a smartphone or smaller tablet, you may
want to use large fonts for both labels and text input.

Include an introduction or welcome page. Explain the reason for the survey (as far as you can
without compromising the survey). Put instructions at the point they are needed, instead of
grouping them on the first page.

Make sure your page and question layout are consistent. Do not put answer choices on the right
for some questions and on the left for others. Use color consistently. For example, always use the
same color to represent an instruction, which is not part of a question per se. Use a different color
(or bolding) any time you want to highlight words within questions.

Recognize that requiring that questions be answered will likely increase the number of people
who drop out of a survey in the middle. If you do require answers, consider doing so only on key
questions. Whenever you require an answer make sure the available options include all possible
answers, including “don’t know,” “decline to state,” or “not applicable,” if there is any chance
that these may represent some people’s answers.

Consider your sample when designing the pages. Using answer grids and presenting answer
choices in two or more columns can look attractive, save space and help avoid vertical scrolling.
Unfortunately, these formats are a bit harder for some people to understand than a simple vertical
list of answer choices. If you think your target population may have some trouble understanding
how to fill out the survey, use these formats sparingly.

Allow space for long replies to comment type questions. Some people will type in longer
answers on a Web page than they would write on a paper questionnaire or say to an interviewer.
Drop-down lists save space on the screen, but be careful using them. Lists that require scrolling
to see some choices can bias the results. Use them only if there is only one possible choice a
person can make. One example is state of primary residence. If you present a list of choices that
people have to think about, and only some of the choices are initially visible, there will be a bias
in favor of those initially visible choices.
Researchers have been looking into the issue of whether it is better to present a survey in one or
more long scrolling pages or in a series of separate pages that do not need scrolling. Research has
not yet provided a clear answer. There is some evidence that grouping several similar questions
on a page may result in answers that are more similar than if the questions were on different
pages. Some people may dislike scrolling down a long page, while others may dislike the brief
wait between questions when each is on a different page. Having your questionnaire split into
multiple pages has the advantage that if someone quits partway through, at least you have the
answers they have already given. You will also need to split your survey into multiple pages, if
you want some people to not see certain questions, or if you want the answers given for some
questions to affect those shown for later questions.

Sometimes researchers like to announce the start of a new section in a survey. Using separate
section break pages that announce the start of a section, but do not include any questions will
likely lead to some people quitting the survey at one of those pages. If you want to announce a
new section, include at least one question on the page. The one exception to this rule is if the
section start includes some instructions, and you have decided to randomize the order of the
questions in the section. In that case, the section instructions would have to be one their own
page, since you do not know which question would follow.

Some researchers like to show progress bars in a web survey. A fast moving progress bar (20%,
40%, 60%...) might encourage people, but a slow moving one (2%, 4%, 6%) will discourage
people and likely lead to people quitting the survey.

When you have finished creating the survey and have it up on your Web site, test it thoroughly.
Make sure that all the pages look as you wish and that all skips, randomizations and other logic
work as you intend. A test on your own PC or a paper copy of the questionnaire does not
guarantee that the copy on the Web will look and act the same. We also recommend you monitor
the live results as your survey progresses (good Web survey software allows this). Doing so can
help you spot any problems that did not appear during your testing. Despite their best efforts
most researchers occasionally miss something.

Pre-test the Questionnaire

The last step in questionnaire design is to test a questionnaire with a small number of interviews
before conducting your main interviews. Ideally, you should test the survey on the same kinds of
people you will include in the main study. If that is not possible, at least have a few people, other
than the question writer, try the questionnaire. This kind of test run can reveal unanticipated
problems with question wording, instructions to skip questions, etc. It can help you see if the
interviewees understand your questions and give useful answers.

If you change any questions after a pre-test, you should not combine the results from the pre-test
with the results of post-test interviews. The Survey System will invariably provide you with
mathematically correct answers to your questions, but choosing sensible questions and
administering surveys with sensitivity and common sense will improve the quality of your results
dramatically.
Correlation

Correlation is a statistical technique that can show whether and how strongly pairs of variables
are related. For example, height and weight are related; taller people tend to be heavier than
shorter people. The relationship isn't perfect. People of the same height vary in weight, and you
can easily think of two people you know where the shorter one is heavier than the taller one.
Nonetheless, the average weight of people 5'5'' is less than the average weight of people 5'6'', and
their average weight is less than that of people 5'7'', etc. Correlation can tell you just how much
of the variation in peoples' weights is related to their heights.

Although this correlation is fairly obvious your data may contain unsuspected correlations. You
may also suspect there are correlations, but don't know which are the strongest. An intelligent
correlation analysis can lead to a greater understanding of your data.

Techniques in Determining Correlation

There are several different correlation techniques. The Survey System's optional Statistics
Module includes the most common type, called the Pearson or product-moment correlation. The
module also includes a variation on this type called partial correlation. The latter is useful when
you want to look at the relationship between two variables while removing the effect of one or
two other variables.

Like all statistical techniques, correlation is only appropriate for certain kinds of data.
Correlation works for quantifiable data in which numbers are meaningful, usually quantities
of some sort. It cannot be used for purely categorical data, such as gender, brands purchased, or
favorite color.

Rating Scales

Rating scales are a controversial middle case. The numbers in rating scales have meaning, but
that meaning isn't very precise. They are not like quantities. With a quantity (such as dollars), the
difference between 1 and 2 is exactly the same as between 2 and 3. With a rating scale, that isn't
really the case. You can be sure that your respondents think a rating of 2 is between a rating of 1
and a rating of 3, but you cannot be sure they think it is exactly halfway between. This is
especially true if you labeled the mid-points of your scale (you cannot assume "good" is exactly
half way between "excellent" and "fair").

Most statisticians say you cannot use correlations with rating scales, because the mathematics of
the technique assume the differences between numbers are exactly equal. Nevertheless, many
survey researchers do use correlations with rating scales, because the results usually reflect the
real world. Our own position is that you can use correlations with rating scales, but you should
do so with care. When working with quantities, correlations provide precise measurements.
When working with rating scales, correlations provide general indications.

Correlation Coefficient
The main result of a correlation is called the correlation coefficient (or "r"). It ranges from -1.0
to +1.0. The closer r is to +1 or -1, the more closely the two variables are related.

If r is close to 0, it means there is no relationship between the variables. If r is positive, it means

that as one variable gets larger the other gets larger. If r is negative it means that as one gets
larger, the other gets smaller (often called an "inverse" correlation).

While correlation coefficients are normally reported as r = (a value between -1 and +1), squaring
them makes then easier to understand. The square of the coefficient (or r square) is equal to the
percent of the variation in one variable that is related to the variation in the other. After squaring
r, ignore the decimal point. An r of .5 means 25% of the variation is related (.5 squared =.25). An
r value of .7 means 49% of the variance is related (.7 squared = .49).

A correlation report can also show a second result of each test - statistical significance. In this
case, the significance level will tell you how likely it is that the correlations reported may be due
to chance in the form of random sampling error. If you are working with small sample sizes,
choose a report format that includes the significance level. This format also reports the sample
size.

A key thing to remember when working with correlations is never to assume a correlation means
that a change in one variable causes a change in another. Sales of personal computers and
athletic shoes have both risen strongly in the last several years and there is a high correlation
between them, but you cannot assume that buying computers causes people to buy athletic shoes
(or vice versa).

The second caveat is that the Pearson correlation technique works best with linear relationships:
as one variable gets larger, the other gets larger (or smaller) in direct proportion. It does not work
well with curvilinear relationships (in which the relationship does not follow a straight line). An
example of a curvilinear relationship is age and health care. They are related, but the
relationship doesn't follow a straight line. Young children and older people both tend to use
much more health care than teenagers or young adults. Multiple regression (also included in the
Statistics Module) can be used to examine curvilinear relationships, but it is beyond the scope of
this article.

Statistics Reports
All editions of The Survey System include the most commonly used survey statistics, including:

 Percents
 Medians
 Means
 Standard Deviations
 Standard Errors
  Chi-squares
 Differences between Proportions
 t-tests

These are enough for most researchers to establish significant differences between data points.
However, for those researchers with more sophisticated statistical needs, such as university
dissertations or academic research publications, the Advanced Statistics Module adds the
following techniques:

 ANOVA - Analysis of Variance (One-way, Two-way, Repeated Measures)

 Correlation (Pearson and Partial)
 Multiple Regression (Standard and Stepwise)
 Descriptive Statistics

Descriptive Statistics tables can use banners to show answers given by different groups of
people. They are ideal for showing how many people live in each zip code, for example.