How to...
Use questionnaires effectively

The process of constructing a questionnaire

Stages

A questionnaire must go through several stages.

Establishing your objectives

You will find organising and writing the questionnaire much easier if you have its objectives in mind at all times, from the initial drafting of the questionnaire through to the revisions following pretesting. It is normal to proceed from a hypothesis (or hypotheses) when developing a questionnaire. If this has been proposed by a literature review, you may well have a framework which will help you suggest your questions and formulate their order, or you may be operating with an existing tool such as a student satisfaction questionnaire. It is also not unusual to combine a fully structured questionnaire technique with an exploratory study in which interviewees (possibly key informant) are asked a series of open-ended questions.

Examples

In "Is your TQM programme successful? A self-assessment tool for managers" (Mohammad Ahmadi and Marilyn M. Helms, The TQM Magazine, Vol. 7 No. 2), the authors researched the TQM literature and used the critical elements indicated by that literature to help develop their survey instrument.

Susan Aldridge and Jennifer Rowley, in "Measuring customer satisfaction in higher education" (Quality Assurance in Education, Vol. 6 No. 4), used a student satisfaction questionnaire as the basis for their survey.

The American Statistical Organization also suggests starting with your data collection goals – what information do you need and from whom – and drafting an outline of the final report, which pinpoints the information requirements. From this a data analysis plan can be formulated which may include a table or a flowchart which links everything together at a high level, and which thereby helps to formulate the questions needed to gather the data.

Administration method

See "Different methods of delivering questionnaires" section.

Questionnaire construction

See "The organization and presentation of questionnaires", "Constructing the questions", and "Question format and response elicitation" sections.

Pretesting

A very important part of the questionnaire construction process is its piloting, known as pretesting. This involves testing your research instrument in conditions as similar as possible to the research, not in order to report results but rather to check for glitches in wording of questions, lack of clarity of instructions, etc. – in fact, anything that could impede the instrument's ability to collect data in an economical and systematic fashion.

Pretests should be conducted systematically, with potential respondents and using the same method of administration. The temptation to hurry over them, using just a convenience sample, should be avoided.

It is also beneficial to pretest the questionnaire with specialists in question construction, who may be able to pick up potential difficulties which might not be revealed in a pretest with respondents.

If there are a variety of respondent types, all should be included in the pretest, and if the questionnaire is to be in several languages, it should be tested in each language.

Example

In "Linking manufacturing planning and control to the manufacturing environment" (W. Rocky Newman and V. Sridharan, Integrated Manufacturing Systems, Vol. 6 No. 4), the authors conducted a survey on the relationship between the manufacturing environment and the use of MPC systems. Their pretests were particularly thorough, and are described as follows:

"A two-stage pretest of the survey instrument included initially mailing it to colleagues in academia, consulting and industry, and soliciting comments to assess the instrument for its validity and consistency. A revised instrument was then mailed to 55 management contacts in a wide range of industries for completion. With about 40 per cent of these firms responding, several participants were contacted by telephone to clarify their responses further. Based on their combined feedback, a final version of the instrument was mailed to 1,500 manufacturing facilities in a mid-western state with more than 150 employees. The assumption underlying this choice was that larger facilities are more likely to have a formal infrastructure support system for manufacturing planning and control."

Different methods of delivering questionnaires

Some initial considerations 

Nicolaos Synodinos, in an article "The 'art' of questionnaire construction: some important considerations for manufacturing studies" (Integrated Manufacturing Systems , Vol. 14 No.3) describes several factors which need to be taken into account in deciding the most appropriate survey administration method:

While all surveys are self reporting, some are administered by interviews and others are self-administered.

Interview methods 

Here we are considering structured interviews, with a schedule of questions that has already been worked out in a pre-defined instrument, as opposed to semi-structured interviews, where the conversation can proceed in a less structured fashion. The main ways in which interviews are conducted are considered below, along with their main advantages and disadvantages.

The main ways in which interviews are conducted, along with their main advantages and disadvantages

Interview type	Advantages	Disadvantages
In person, as in "street mall intercept" interviews	Less burdensome to the respondent who does not have to write out responses Higher response rate Opportunity to observe respondents Good if response to visual materials required, or if questions are consequential as interviewer controls sequence	Interaction between interviewer and respondent may bias response Cost of the interviewer No good if respondents are dispersed over a wide geographical area Loss of anonymity
By telephone	Good for conseqential questions, but not for visual material Easy to supervise interviewer ideal for ongoing and recently completed events (hence its popularity for elections) Greater anonymity than personal interviews Less costly than personal, can be less costly than by mail	Negative perception of telephone interviews Can be difficult if there are a large number of alternatives to choose from Not suitable for longer questions

Self-administered

The general problem with all self-administered methods of surveys is the lack of control over response – once your questionnaire has been despatched to the respondent, there is a limited amount you can do to ensure its return, or indeed that the respondents have completed the questionnaire. Questions about to response rate, and how to improve it, are a big issue with surveys. The main methods are as follows:

Main response methods

Response method	Advantages	Disadvantages
By mail	Easy to administer Control over visual quality of the instrument, which can be made to look professional Generally not particularly costly Greater degree of anonymity	Inability to control whether or not the individual will return or complete the questionnaire, or fill in the questions in the right order Greater level of literacy required than for an interview Inappropriate for studies of rapidly changing opinions
By email	Even easier to administer and less costly than mail surveys People more likely to fill in than a mail questionnaire	Not easy to guarantee anonymity
By internet	Good if the population sample is highly specific, i.e. from an organisation Easier to show visual material than with an email questionnaire Not very costly	Requires IT literacy from sample Difficult for more complex questions

Increasing the likelihood of completion

There are a number of ways of increasing completion rates on self-administered surveys:

1. Provide an incentive – ideally, this should be monetary, but small gifts, charitable donations, promise of issuing the report of the results.
2. Make the survey easy to fill in – surveys that follow the KISS principle, i.e. fairly short, with simple questions, are more likely to be returned. Some writers say that as a rough rule of thumb, the questionnaire should not take more than 15 minutes to fill in.
3. Ensure that the questionnaire looks professional – make the instructions clear, use plenty of white space, etc.
4. Increase contact with the respondents – send out reminders, write to the respondents to notify them that you are going to contact them. (Increasing the number of contacts is especially beneficial for questionnaires aimed at institutions.)
5. State the purpose of the questionnaire – if it will be used for research purposes, what is the purpose of the research, etc.
6. Assure them that their answers will remain confidential.
7. Be polite and courteous – you are taking up their time, they are doing you a favour. So, liberal sprinklings of "please" and "thank you" will not go amiss! 

Example

"Mail survey response behavior" (S. Tamer Cavusgil and Lisa A. Elvey-Kirk, European Journal of Marketing, Vol. 32 No. 11/12) looks at some of the factors which influence mail survey response.

The organisation & presentation of questionnaires

The organisation of your questionnaire will greatly help you gain the information you want and will also help your respondent fill it in.

Sequencing & steering

Experienced researchers advocate writing out the individual questions on card indexes, on each of which should be:

Sequencing

• a number indicating the place in the questionnaire sequence 
• the wording of the instructions
• the wording of the question
• the wording of any alternative answers
• instructions for recording the data.

The issue of the sequence of the questions is very important. Researchers tend to agree on a number of rough guidelines:

• Start with an interesting question, but one that is non-threatening, and does not ask for tricky information.
• Have screening questions early on in the sequence, after the introduction, so that you can ensure that your respondents are actually part of the sample, and can exclude any that are not. Equally important is the use of such questions to organise your sample into sub-ground. This will enable you to analyse your responses into categories.

Example

In "Libraries and desktop storage options: results of a Web-based survey" (Arthur Hendricks and Jian Wang, Library Hi Tech, Vol. 20 No. 3), the authors start their questionnaire with a number of questions concerning types of libraries, position, etc. so that the survey could report on where people worked and what they did.

• Once the introductory and screening questions have been got out of the way, the main body of the questionnaire can deal with the main data-yielding questions.
• The sequence of the "main body" questions is important: make sure that it is clear and logical, and that the placement of questions will not affect subsequent responses in that the respondent will be unduly influenced by a response to a previous question, as the context within which a question is placed can influence the response. Clear sequencing is particularly important if the respondents have little formal education.
• Group similar questions together within sections, and go from the general to the more specific.
• Questions which relate to the respondent, and, in the case of an organisational questionnaire, to the organisation, are best left to the end as these are perceived to be sensitive.

Example

In "A longitudinal survey of robot usage in Australia" (Stuart C. Orr, Integrated Manufacturing Systems, Vol. 7 No. 5), the authors describe a survey which was conducted by interview and where the questions were grouped as follows:

• Were the expectations of management being met by robot technology, given the potential benefit it has to offer industry?
• What was the impact on the workforce of robot manufacturing technology?
• What were the strengths and weaknesses of the technology under local manufacturing conditions?
• What are the likely future impacts of the technology on the Australian manufacturing industry?

Instructions

Provide clear instructions for filling in the questionnaire, which should be clearly differentiated from the questions themselves.

The introduction to the questionnaire should state general background information, for example:

• the purpose of the survey
• who you are and the organisation you represent
• what you want the respondents to do.

If you organise the questionnaire according to sections, each section should have a brief introduction stating its purpose.

If particular questions require specific instructions, e.g. choosing a particular response, give those instructions with that question or group of questions.

If some questions branch, i.e. "if you..., go to question 12", state these instructions clearly and in a user-friendly manner. Also, make it clear when some questions do not apply to all respondents.

With mail questionnaires, it is also usual to send out a covering letter which should state:

• the purpose of the survey and who you are, and represent (if part of an organisation)
• how filling in the questionnaire will benefit the respondent
• how filling in the questionnaire will benefit research
• how in the case of questions which solicit opinion, there are no right or wrong answers
• how and why the respondent was selected
• how you will ensure the confidentiality of the respondent's material and their privacy
• acknowledgement of the respondent's time – thank them for completing the questionnaire
• what the deadline for return of the questionnaire is
• what are the instructions for filling in the questionnaire.

Finally, you should always supply a self-addressed envelope.

Presentation

The following points are important:

• It is very important that a questionnaire has a professional appearance, and that is it formatted in such a way as to make it easy to fill in.
• Make sure that instructions are distinguished stylistically from the questions.
• Include plenty of white space.
• Make sure that there is enough space for the answers.
• Make sure that branching instructions are clear: for example, "If YES, go to question 10", etc.

Constructing the questions

We come now to discuss the actual part of the questionnaire which elicits data from the respondent. According to N. Synodinos, in "The 'art' of questionnaire construction: some important considerations for manufacturing studies" (Integrated Manufacturing Systems , Vol. 14 No. 3), question construction is a "highly developed art form within the practice of scientific enquiry", and even small changes in wording can have a considerable effect on the resultant data.  

The anatomy of a questionnaire question

There are three parts to the wording of questions:

• The question stem, which sets the framework for the data, which may be a statement followed by a number of alternatives, as in the examples above, or a straightforward question, such as "What type of library do you work in?" or a statement, such as "since our firm began to implement a TQM programme".
• The alternatives, which are the range of possible answers from which the respondent selects – a range of library types, a number on a scale of values, etc.
• The responses , which are the actions which the respondent has to undertake in order to input to the questionnaire – circling a number on a scale of values, clicking a radio button, filling in a text field, or just ticking "yes", "no".

Some ground rules for writing questions

Here we are concerned with very basic general issues; we will deal with question formats and answers in the next section.

Some general guidelines:

• Each question should relate to one issue only. If you find the question becomes too complicated, then consider splitting it into two.

Example

Synodinos cites "An audit-based approach to the analysis, redesign and continuing assessment of a new product introduction system" (Gardiner, G.S. and Gregory, M.J., Integrated Manufacturing Systems , Vol. 7 No. 2):

• Relate your questions to your objectives at all times – ask yourself for each question, "How will it help me achieve my research objective?"
• Don't use any difficult or technical terms without an explanation, and avoid slang and jargon.
• Avoid questions that are complex, or which tax the respondent's memory, or which call for information which is not easily to hand.
• Make sure that you don't have any questions which indicate bias – for example 'Do you think that every school should have access to the internet in order that children have the best possible chance of growing up IT literate?'
• Avoid negatives, especially double negatives, in questions – frame them in a positive way!
• Remember the KISS principle! Questions should be short, clear, and unambiguous.
• Avoid asking for highly confidential information, as well as questions that may appear intrusive.
• Avoid asking people to predict the future.
• If the survey is based on a past survey, make sure that cultural and linguistic assumptions are still correct. For example, if repeating a study of telephone use that was conducted a number of years ago, remember that the frequency of telephone use has increased.
• When dealing with populations in different cultures, make sure that the question wording means the same thing in each context.

Examples

Joseph Janes, in "Survey construction" (Library Hi Tech, Vol. 17 No. 3), has an excellent list of points to remember in constructing questions, under the heading "Writing good questions", as does Nicolaos Synodinos, in "The 'art' of questionnaire construction: some important considerations for manufacturing studies" (Integrated Manufacturing Systems , Vol. 14 No. 3), in the section "Question wording".

Question format & response elicitation

There are two main types of question format:

• Open-ended, in which the format of the response is free, with the user phrasing their own replies.
• Closed-ended, in which the respondent selects one or more response from a set of possible answers.

Closed-ended questions are easier for the respondent to answer, and also to code and analyse, than open-ended questions which require content as opposed to statistical analysis. They should therefore be used sparingly in questionnaires, if at all.

For a discussion of these issues, see "Open versus closed questions – an open issue" by Gerald Vinten (Management Decision, Vol. 33 No. 4).

Open-ended questions

According to Constructing Effective Questionnaires (Peterson, R.A., 2000, Sage, Thousand Oaks, CA) there are a number of instances when open-ended questions may be appropriate:

• when it is not easy to foresee the answers
• when the responses may be influences by the choices
• when the variables measured dictate unaided recall
• where flexibility is called for by unanticipated events
• where initial responses necessitate more in-depth follow-up questions.

Open questions are often used when carrying out research on issues prior to developing a questionnaire, in order to find out more background information and develop hypotheses on which to base questions.

Closed-ended questions

These questions require some skill in writing, but are easier to code and analyse. The main types are as follows:

Multiple choice

The most straightforward format, this involves selecting from a range of options "Libraries and desktop storage options: results of a Web-based survey" (Arthur Hendricks and Jian Wang, Library Hi Tech, Vol. 20 No. 3):

Note the way in which the author has in both instances included an "Other (please specify)" field for options which are not in the categories provided.

Free choice

Here, the respondent can choose one or more option from a list, which should include all alternatives, and not be mutually exclusive.

Free choice and multiple choice collect nominal data, in other words, it is possible to assign a number to the respondent's choice.

Ranked

Here, the respondent is required to list alternatives according to an order, for example the desirable attributes of a holiday, ranked according to importance. The data collected are described as ordinal.

Rating

This type of answer rates items according to a scale (from "Is your TQM programme successful? A self-assessment tool for managers", by Mohammad Ahmadi and Marilyn M. Helms, The TQM Magazine, Vol. 7 No. 2):

It is one of the most commonly used methods in management research, and is particularly useful for measuring affective issues such as attitude.

Rating scales are particularly useful for dealing with affective measures – i.e. those that relate to a person's perceptions, beliefs, feelings, attitudes and values towards themselves, individuals or organisations. Their use in management research is common. Because they measure interval data, which is a higher order than nominal or ordinal data, it is possible to apply a wider range of statistical procedures.

There are a number of different ways of measuring affective issues, of which some of the most commonly used in questionnaires are:

• Numerical rating scales: respondent is asked to select a particular numbered response on a scale with either all points 'anchored' (i.e. strongly agree – strongly disagree) or just the beginning and end points.
• Likert rating scales: use a numbered scale as above but provide respondent with statements with which to agree/disagree.
• Semantic differential: use a numbered scale that is measured by bipolar adjectives.
• Thurstone scales: assign ratings to a series of statements which respondents are asked to check those which apply to them.
• Guttman scales: a similar approach to Thurstone, but here the checking of statements can be cross-checked to see whether or not attitudes are uni-dimensional.

Examples

An interesting example of the use of rating scales is the use of the Likert scale in research on library use and library anxiety as developed by Sharon L. Bostick and used in subsequent research by Qun G. Jiao, Anthony J. Onwuegbuzie in their research on the relationship between anxiety about library use and social interdependence, as published in a number of articles, for example "Dimensions of library anxiety and social interdependence: implications for library services" (Library Review, Vol. 51 No. 2):

"You are being asked to respond to statements concerning your feelings about college and university libraries. Please mark the number which most closely matches your feelings about the statement. The number ranges from:

1 = Strongly Disagree   2 = Disagree   3 = Undecided   4 = Agree   5 = Strongly Agree

I'm embarrassed that I don't know how to use the library
   1   2   3   4   5

A lot of the university is confusing to me
   1   2   3   4   5

The librarians are unapproachable
   1   2   3   4   5

The reference librarians are unhelpful
   1   2   3   4   5"

Numerical rating scales are used in "Libraries and desktop storage options: results of a Web-based survey".

The choice of question format will be determined by the type of data you seek to elicit, but also by the level of sophistication of the statistical analysis which you hope to perform – some formats elicit more exact data than others, as we shall explore in the next section.

Alternatives

Many of the points raised earlier about writing questions apply about writing answers:

• ensure that all items listed are strictly necessary to the question objectives
• ensure that all items are clearly worded, and mean the same thing to the respondent as they do to you
• avoid items which make use of jargon, technical terms or words which are outside the sample's knowledge range
• the respondent should not have to look outside the survey – in other words, try and recall forgotten events

In addition:

• response items should be exhaustive and mutually exclusive
• do not provide too many alternatives – 15 is considered a maximum
  by some
• decide what order to present the alternatives – this could be random, alphabetical, or with an order dictated by the research objective
• if including very specific questions (e.g. How many cigarettes do you smoke a day? 1-5, etc.) preface it by a screening question (Do you smoke? Yes/No)
• it may be appropriate in some instances to give range variables (0-5, 6-10), etc. in stead of precise measures, for example if asking a question such as 'How many emails do you receive in a day' when precise recall may be a problem
• when using the same question type (for example, multiple choice, Likert scale, etc.) use the same response format, as in the examples above.

The analysis and writing up of questionnaires

The goal with a questionnaire is not to produce a beautiful instrument but rather one that will yield data that will provide results. If you study articles published in Emerald journals you will note that relatively few report the questionnaire in detail, but give more space to its results. The latter have often been achieved by quite sophisticated statistical analysis which is a subject in its own right and beyond the scope of these pages. What we offer is a few preliminary observations on how to analyse and present.

Analysis

Your work in analysis should start when you are thinking about the objectives of your questionnaire: you will have decided what statistical tests to apply, and you will have constructed your questionnaire so that it is easy to code.

You will also have established a cut-off date for the return of your questionnaires, at which point you will start the analysis. The first task is to go through the questionnaires and establish which ones are usable – those which are unfinished, or which have more than a couple of botched responses, must be discarded (if only a couple of botched responses, these can be marked as "non response").

Open questions will need to be analysed by content analysis. For the majority of closed-ended questions, you need to begin to:

• record the answer to each question, using an Excel spreadsheet.
• decide how you are going to cluster the data – what subgroups of your sample are you going to highlight.
• decide how you are going to cross tabulate.
• look at trends – do these relate back to your hypotheses?

Statistical packages such as SPSS and Minitab should only be used if there are sufficient responses – if the sample is less than 100, then the analysis is much better done by hand; if the sample is between 100 and 200, then it depends upon how many cross tabulations there are; for samples of more than 200, it is advisable to use statistical packages.

Types of data

Researchers should always try and gather data at the highest possible level of sophistication. The main types of data are listed below in order of scale:

• Nominal data: this puts people into categories, i.e. gender, type of job, etc.
• Ordinal data: ordinal data allows for ranking, e.g. the degree to which people possess a characteristic (but without a specific interval between the data)
• Interval data: also allows for ranking, but the intervals between the scale are equal. 

Ratio data The most precise level of measurement, measures intervals but have a precise zero point (e.g. height, speed, time, etc.)

Presentation

In articles written for Emerald publications, authors rarely concentrate on the details of the questionnaire construction but may describe such features as are necessary to the design of the research, such as:

• the main categories of the questionnaire, as for example in "Measuring customer satisfaction in higher education" (Susan Aldridge and Jennifer Rowley, Quality Assurance in Education, Vol. 6 No. 4), see the section on questionnaire design.
• how the questionnaire was delivered (e.g. electronically, on paper, etc.).
• number of responses.

When reporting results, authors describe the main points in the body of the text, with relevant data listed after the conclusion in the form of tables, bar charts, etc. as relevant.