Surveys

Surveys are used to gather information from people.

When should we use surveys?

• Use it when information that is required is held by an identifiable and dispersed group of people.
  
  
• Use it to help decision-making, by turning disparate qualitative data into useful quantitative information.
  
  
• Use it only when the time and effort are available to complete the survey.
  
  
• Use it, rather than Brainstorming or Nominal Group Technique to gather real data about what a diverse group of people thinks (rather than an opinion of what they think).
  

How Should Surveys Be Interpreted?

Survey is a general term for a number of methods of collecting from people, which includes questioning them individually and collectively, in person, over the phone or on paper. The decision on what type of survey to use is driven by such factors as accessibility of the people and the response required. For example, in-depth personal interviews can give much useful information, but may be impractical when there are many people to ask.

There are two (2) main approaches to surveys, interviews and questionnaires as shown in Table I below. Interviews involve one (1) or more interviewers asking questions from (1) one of more respondents. In contrast, a questionnaire contains written questions and answers. The most noticeable difference is that with a questionnaire, the person asking the questions and the respondent answering them may never met.

Table I. Interviews vs. Questionnaires
Interviews	Questionnaires
Two-way. Can ensure interviewee understands questions and vice versa	One-way. Comments cannot be queried.
Flexible. Can be changed on the fly to follow interesting information	Fixed. Questions cannot be changed.
Interviewer does the writing, as well as questioning.	Respondent does the writing.
May gather a large amount of information in one interview.	Gathers limited information.
Best for open, in-depth questions, exploring ideas and feelings.	Best for closed, structured questions, checking facts and opinions.
Personal. Can be awkward when asking sensitive questions.	Impersonal. Replies may be anonymous.
Can only be used with limited number of respondents.	Can be used with many respondents.
Predictable number of respondents.	Unpredictable number of respondents - can often be lower than expected.
Tends to give qualitative data. Can be difficult to analyze.	Mostly quantitative data. Ease of analysis depends on questionnaire design.

A problem with asking people questions is that not only will different people answer differently, but changes in the wording, sequence and environment of questions are likely to result in different responses from the same person. The design of a questionnaire and the execution of an interview must therefore be done with great care.

The two (2) main types of questions that can be asked in surveys - closed questions and open questions - are contrasted here in Table II.

Table II. Closed and Open Questions
Interviews	Questionnaires
Narrow focus, aiming to find specific detail.	Broad focus, aiming to explore general area.
Start with: Which, What, Where, When, How many, How much.	Start with: Why, How, Tell me about.
Short reply, often one word (e.g. yes, no, ten, screws).	Long reply, often of uncertain length.
Limited to set of possible replies, often given in multiple-choice questions.	Wide variety of possible replies.
Good for quantitative (numeric) data for use in Pareto Charts, etc.	Good for qualitative data, for general understanding.
Control is with person asking questions.	Control is with person being questioned.
Can miss important points. Can be leading question.	Can drift off the point. Can be insensitive.
Can be used to ‘set the scene’ for open question.	Can be used to capture points missed with closed question.
Particularly useful in questionnaires.	Particularly useful in interviews

 

How Should Surveys Be Conducted?

The major steps in conducting a sample survey are:

1. Overall planning
2. Design and selection of samples
3. Design of questionnaires
4. Collection of raw data
5. Compilation and analysis of statistics and dissemination of survey results

Step 1: Overall planning

• The objectives of a survey should be clearly specified.
  
  
• In conducting a survey, professional input is often required. It is thus advisable to seek the opinions of professional statisticians right from the start, since rectification is normally very difficult to make when the survey is completed and its results are found invalid.
  
  
• The population of the survey, which refers to the totality of objects under study, should be clearly defined.
  
  
• It should be ensured that respondents of the survey are able to provide the information required.
• Adequate resources such as manpower, time and finance should be secured before deciding to proceed with the survey.

Step 2: Design and selection of samples

• Probability sampling methods should be used as far as practicable so that scientific inference can be drawn from the sample results about the entire population. Non-probability sampling methods, such as haphazard sampling, should be avoided. Such methods are not scientific and bias normally exists in the survey results.
  
  
• There are different types of probability sampling and survey-takers should understand the practical steps of various kinds of such methods. It is not absolutely necessary to adopt a sample design where the probability of selection of every individual respondent is equal.
  
  More importantly, the probability of selection of different respondents must be known.In case of unequal probabilities of selection, it is necessary to ensure that proper weighing methods are applied to survey results to give unbiased population estimates.
  
  
• A good sampling frame, from which sampling units are to be selected, should be acquired.
  
  
• In case households are taken as sampling units with individuals as the inquiry units, a proper method by which individuals are selected within the household should be adopted.
  
  
• The sample size depends on the characteristics of the population, the choice of method of sampling and the desired level of precision of estimates. It can be worked out using established statistical methodology.
  
  
• Once the sampling units are selected, alterations are not allowed. (This is one of the requirements of probability sampling.)

Step 3: Design of questionnaires

• Questions should be relevant to the survey objectives. They should also be arranged in proper order and their wording should be appropriate, specific and precise. If the questions involve expression of opinions, care should be taken to avoid questions leading respondents towards the direction of a certain answer or questions being loaded in favor of a particular response.
  
  
• If respondents reluctantly give answers to questions which they are unable to answer or do not hold any genuine opinions, such answers will only turn out to be invalid. Therefore, alternative answers such as "don't know" and "no opinion" should be included as options.
  
  
• Long questionnaires are undesirable.
  
  
• Questionnaires should be tested on some prospective respondents before they are finalized for use in surveys so as to ensure their effectiveness.

Step 4: Collection of raw data

• The mode of data collection includes personal interviewing, telephone interviewing and using self-administered questionnaires by mail. An appropriate mode should be selected by carefully considering respondents' willingness to cooperate, the degree of complexity of the subject of enquiry and other relevant factors. Very often, mixed modes of data collection can be used.
  
  
• Various arrangements related to fieldwork such as the proper allocation of workload amongst interviewers and the provision of adequate transport facilities, should be carefully planned to ensure smooth and efficient operation. Fieldwork procedures should be thoroughly tested before implementation.
  
  
• Interviewers should be carefully briefed on the concepts and definitions of terms used in the survey and properly trained in the procedures before they start working.
  
  
• Interviewers should ensure that respondents understand the questions. If in doubt, probing or giving further explanations should be done; but such should not be overdone so as not to exercise undue influence on respondents.
  
  
• Every effort should be made to achieve a high response rate, i.e. to minimize non-contacts with respondents or refusals to respond, because a low response rate may induce a serious bias in the survey results and is liable to mislead users of the results. Adequate publicity of the survey, proper introduction of the surveying organization, clear identification of interviewers and giving advance notice to sampled respondents should help. Arrangements should also be made to follow up non-contact respondents and to persuade uncooperative respondents to participate.
  
  
• If the non-response rate is not negligible, methods should be devised to assist in the interpretation of survey results (such as to obtain some observable characteristics of the non-respondents).
  
  
• The survey progress should be properly controlled. To ensure the quality of survey results, interviewers should be closely supervised and if possible, the contents of some of the completed questionnaires should be checked with the respondents.

Step 5: Compilation and analysis of statistics and dissemination of survey results

• Raw data should be carefully and thoroughly checked before compilation of statistics.
  
  
• Due deliberation should be made to decide on an appropriate sub-grouping scheme in analyzing survey results at some disaggregate levels, taking into account the precision of estimates derived from the sample.
  
  
• Appropriate statistical methodology should be adopted in compiling and analyzing data.
  
  
• A complete survey report should contain details about different aspects of the survey, in particular details on population coverage, sample design, sample size, sampling error, response rate and likely sources of non-sampling error. Where appropriate, specimen of the questionnaire should be attached.
  
  
• When the results of a survey are released to the mass media, sufficient details on the survey methodology should be provided in addition to survey findings, so that the mass media may report both; otherwise the general public will have no basis to assess the reliability of those findings.

Sources:
http://www.info.gov.hk/censtatd/eng/statliteracy/edu_booklet/sam_survey/sam_survey.htm
A Toolbook for Quality Improvement and Problem Solving, 1995, by David Straker

 

More Quality Tools