european organisation for the safety of air navigation · less costly to conduct than telephone...

EUROPEAN ORGANISATION FOR THE SAFETY OF AIR NAVIGATION

EUROCONTROL

EUROPEAN AIR TRAFFIC MANAGEMENT PROGRAMME

A Guide to Surveys and Questionnaires

AIM/AEP/S-LEV/0009

Edition : 0.2 Edition Date : 19 Apr 2002 Status : Draft Class : General Public

DOCUMENT IDENTIFICATION SHEET

DOCUMENT DESCRIPTION

Document Title A Guide to Surveys and Questionnaires

EWP DELIVERABLE REFERENCE NUMBER

PROGRAMME REFERENCE INDEX EDITION : 0.2

AIM/AEP/S-LEV/0009 EDITION DATE : 19 Apr 2002 Abstract

It presents a guide for survey and questionnaire preparation.

Keywords Survey Questionnaire

CONTACT PERSON : Paul Bosman TEL : 3333 UNIT : AIM

DOCUMENT STATUS AND TYPE

STATUS CATEGORY CLASSIFICATION Working Draft Executive Task General Public Draft Specialist Task EATMP Proposed Issue Lower Layer Task Restricted Released Issue INTERNAL REFERENCE NAME : AHEAD Electronic Filing System

A Guide to Surveys and Questionnaires AIM/AEP/S-LEV/0009

Edition: 0.2 Draft Page iii

DOCUMENT APPROVAL

The following table identifies all management authorities that have successively approved the present issue of this document.

AUTHORITY NAME AND SIGNATURE DATE

Author/Editor Ertan Ozkan 19 Apr 2002

Programme Manager

Conrad Cleasby 19 Apr 2002

Quality Assurance Paul Bosman 19 Apr 2002

AIM/AEP/S-LEV/0009 A Guide to Surveys and Questionnaires

Page iv Draft Edition: 0.2

DOCUMENT CHANGE RECORD

The following table records the complete history of the successive editions of the present document.

EDITION DATE REASON FOR CHANGE SECTIONS

PAGES AFFECTED

0.1 27 Nov 2001 Creation All

0.2 19 Apr 2001 A. Zarbov’s comments are handled


Edition: 0.2 Draft Page v


Page vi Draft Edition: 0.2

TABLE OF CONTENTS

1. INTRODUCTION ......................................................................................................1 1.1 Purpose and scope.................................................................................................1 1.2 References ..............................................................................................................1

2. ABOUT SURVEYS...................................................................................................2 2.1 What is a Survey?...................................................................................................2 2.2 When Should I Use a Survey? ...............................................................................2 2.3 What are the Common Survey Methods? .............................................................2

2.3.1 Interview Surveys........................................................................................3

2.3.2 Self-Reported Surveys ................................................................................3

2.3.3 Focus Groups .............................................................................................4

2.4 Basic Steps .............................................................................................................4 3. PLANNING...............................................................................................................5

3.1 What is the purpose of the survey? ......................................................................5 3.2 Who will be surveyed? ...........................................................................................5 3.3 Who will use the results?.......................................................................................5 3.4 What specific information is needed?...................................................................5 3.5 Who will be responsible for developing the survey?...........................................6 3.6 How will the survey be administered? ..................................................................6 3.7 What resources will be needed? ...........................................................................6 3.8 What survey items will be used?...........................................................................6 3.9 How will survey information be analysed and reported? ....................................6 3.10 How many people need to respond to the survey?..............................................7

4. QUESTIONNAIRE DESIGN .....................................................................................8 4.1 Creating Survey Items............................................................................................8 4.2 Selecting Response Formats.................................................................................8

4.2.1 Close Ended Questions ..............................................................................8

4.2.2 Comments and Open-ended Questions ....................................................10

4.2.3 Demographic Questions............................................................................11

4.3 Review Survey Items ............................................................................................11 4.4 Review Survey as a Whole...................................................................................13

5. PRE-TESTING .......................................................................................................15 6. ADMINISTERING...................................................................................................16 7. ANALYSIS .............................................................................................................17

7.1 Simple Measures ..................................................................................................17 7.2 Measures of Central Tendency............................................................................18 7.3 Measures of Variability.........................................................................................18 7.4 Rank Orderings.....................................................................................................19


Edition: 0.2 Draft Page vii

8. PRESENTATION....................................................................................................20 9. CONCLUSIONS .....................................................................................................21 APPENDIX A PLANNING A SURVEY ............................................................................22 APPENDIX B SIMPLE TIPS AND TRICKS FOR CONFIDENTIALITY............................24 APPENDIX C BUDGET CHECKLIST .............................................................................25


Edition: 0.2 Draft Page 1

1. INTRODUCTION

1.1 Purpose and scope

This guide provides an overview of developing and conducting a survey. It also covers tips for writing or modifying survey items to meet the needs of an AIS organisation.

This guide is intended to assist an AIS organisation during development and conduct of the customer satisfaction surveys within the scope of a quality management system.

This document is based mainly on the Survey Handbook [1]. However, some missing points are compiled from the references given in the following section. Throughout this document example questions and answers are given in colour blue and badly constructed questions and answers are given in colour red.

1.2 References

[1] Survey Handbook, Archester Houston, Ph.D., Organisational Systems Division-Total Quality Leadership Office-Office of the Under Secretary of the Navy.

[2] Guide to questionnaires and surveys, Frédéric D’Astous, http://members.tripod.com/frede_dast/conseil1_a.html, 23 May 2001.

[3] How to Plan a Survey, ASA Series: What is a Survey?, American Statistical Association, 1997.

[4] How to Collect Survey Data, ASA Series: What is a Survey?, American Statistical Association, 1997.

[5] Judging the Quality of a Survey, ASA Series: What is a Survey?, American Statistical Association, 1998.

[6] What are Focus Groups, ASA Series: What is a Survey?, American Statistical Association, 1997.

[7] What is a Margin of Error, ASA Series: What is a Survey?, American Statistical Association, 1998.

[8] Designing a questionnaire, ASA Series, American Statistical Association, 1999.

[9] More About Mail Surveys, ASA Series: What is a Survey?, American Statistical Association, 1997.

[10] More About Telephone Surveys, ASA Series: What is a Survey?, American Statistical Association, 1999.

[11] Surveys and Privacy, American Statistical Association, 1998.

[12] Questionnaire Design and Survey Sampling, Professor Hossein Arsham, 8th Edition, http://obelia.jde.aca.mmu.ac.uk/resdesgn/arsham/opre330Surveys.htm.

[13] Questionnaire Design: Asking questions with a purpose, E. Taylor-Powell, University of Wisconsin-Extension, May 1998.

[14] Sampling, E. Taylor-Powell, University of Wisconsin-Extension, May 1998.


Page 2 Draft Edition: 0.2

2. ABOUT SURVEYS

2.1 What is a Survey?

A survey is a systematic method of collecting information from a selected group of people by asking a series of questions. Surveys can be used to collect various types of information. They can collect information on people’s behaviour, job performance, knowledge, preferences, attitudes, beliefs, feelings, etc.

Surveys can be used for many purposes, including:

• Determining customer needs/assessing customer satisfaction;

• Identifying organisational strengths and weaknesses;

• Targeting areas needing improvement;

• Assessing the effectiveness of new or existing policies or programs.

Though it is easy to write a questionnaire, one needs a lot of skill and experience to write a good questionnaire:

• Where every question is clear;

• Where it can be answered accurately;

• That has usable results.

2.2 When Should I Use a Survey?

Although surveys are a popular method of collecting data, they must be used under the appropriate conditions. Consider using a survey:

• When it is faster, easier or less expensive to use than other methods;

• When the information does not already exist in some form.

2.3 What are the Common Survey Methods?

There are several ways to conduct surveys:

• Interview surveys

o Face-to-face interviews

o Telephone interviews

• Self-reported surveys

• Focus Groups

Some surveys combine various methods. For instance, a survey worker may use the telephone to "screen" or locate eligible respondents and then make appointments for an in-person interview.



2.3.1 Interview Surveys

Interview surveys -whether face-to-face or by telephone- offer distinct advantages over self-reported data collection. The presence of an interviewer can increase cooperation rates and make it possible for respondents to get immediate clarifications.

The main requirement for good interviewers is an ability to approach strangers in person or on the telephone and persuade them to participate in the survey. Once a respondent’s cooperation is acquired, the interviewers must maintain it while collecting the needed data. Furthermore data that must be obtained in exact accordance with instructions.

2.3.1.1 Face-to-Face interviews

Face-to-face interviews in a respondent's office are much more expensive than self-reported surveys or telephone interviews. They may be necessary, however, especially when complex information is to be collected.

Face-to-face interviews can be done in different ways. One format is to ask respondents to complete on-the-spot surveys as they receive products or services from the organisation. Only very short surveys are practical in this type of situation. The survey administrator usually stands by as the customer completes the survey and might give the customer some small token of appreciation for completing the survey. A second format is to ask respondents to come to a particular location to complete the survey. A third format is to visit respondents at their work sites and ask them to complete the survey.

2.3.1.2 Telephone Interviews

Telephone interviews are an efficient method of collecting data and are being increasingly used. They lend themselves particularly well to situations where timeliness is a factor and the length of the survey is limited.

They are more timely and less expensive than those done face-to-face. Interviewer effects can be better controlled in telephone surveying. Self-reported surveys are less costly to conduct than telephone surveys but generally take more calendar time.

2.3.2 Self-Reported Surveys

Self-reported surveys can be relatively low in cost. As with any other survey, problems exist in their use when insufficient attention is given to getting high levels of cooperation. Self-reported surveys can be most effective when directed at particular groups, such as subscribers to a specialised magazine or members of a professional association.

The following means (or their combinations) can be used for self-reported surveys:

• Mail

• E-mail

• Fax

• Internet



2.3.3 Focus Groups

Focus groups are not polls but in-depth, qualitative interviews with a small number of carefully selected people brought together to discuss a set of topics. The composition of a focus group is usually based on the homogeneity or similarity of the group members. Bringing people with common interests or experiences together makes it easier for them to carry on a productive discussion. Focus groups are conducted by moderators who are skilled in maintaining good group dynamic. Depending on the purpose of the focus group, the moderator may also be an expert in a given topic area. The moderator’s basic job is to keep the group focussed. Questions should be open-ended so that there are many possible replies. Short-answer questions (e.g., yes-no questions) and leading questions should be avoided. The ideal size for a focus group is generally between six and twelve people.

Main advantages of focus groups are: (1) A wide range of information can be gathered in a relatively short time span, (2) The moderator can explore related but unanticipated topics as they arise in the discussion, (3) Focus groups do not require complex sampling techniques.

On the other hand focus groups have some disadvantages: (1) The sample is neither randomly selected nor representative of a target population, so the results cannot be generalised or treated statistically, (2) The quality of the data is influenced by skills and motivation of the moderator, (3) A focus group analysis is truly qualitative. You use the actual words and behaviours of the participants to answer your questions, rather than counting response options.

2.4 Basic Steps

Basic steps of a survey planning are as follows:

1. Planning

2. Questionnaire Design

3. Pre-testing

4. Administering survey (i.e., execution of survey with the selected method)

5. Analysis of responses

6. Present results to decision makers and to respondents

These steps are explained in detail in the following chapters.



3. PLANNING

In order to plan a survey you have to answer the following ten basic questions:

Figure 1. Basic Planning Questions

3.1 What is the purpose of the survey?

Without a clear purpose it is likely that the survey effort will not be successful. This can result in wasted resources, useless data and disappointment on the part of those who initiated the survey and those who responded to it. Therefore, the objectives of a survey should be as specific, clear-cut and unambiguous as possible.

Be very careful about conducting a survey “just to see what’s going on.” Conducting a survey can draw attention to an organisation from the people who receive the survey and from those who learn that it is being conducted. This attention can lead to expectations that executive actions will soon follow.

3.2 Who will be surveyed?

Identify the types of people who can provide the information you are interested in. Do they belong to a single organisational level or do they come from a variety of levels? Do they work in the organisation or are they end-users of the organisation’s products or services?

3.3 Who will use the results?

Identify who will be using the results of the survey and what their information needs are. What types of decisions are they going to make based on the results? How should the information be sorted for them? Do they need detailed responses or is the “big picture” sufficient? How do they prefer to have information presented to them?

3.4 What specific information is needed?

To meet the purpose of the survey, identify the topics or issues of interest and the forms of information needed. For example, if you are interested in maintaining equipment at an athletic centre, then you might ask questions about how often people use the centre, what facilities they use and hours of use. You might also ask people to rate the quality of the swimming pool, treadmills, exercise machines,

1. What is the purpose of the survey?

2. Who will be surveyed?

3. Who will use the results?

4. What specific information is needed?

8. What survey items will be used?

6. How will the survey be administered?

5.Who will develop the survey?

7. What resources will be needed?

9. How will it be analysed and reported?

10. How many people need to respond to the survey?

Basic

Planning Questions



shower area, towel service, etc. If you are interested in meeting future needs, you might ask people to identify their anticipated physical fitness needs or interests.

Each of these topics could be the subject of one or more survey items.

3.5 Who will be responsible for developing the survey?

Surveys are often developed by small teams. When considering membership, include individuals who have some knowledge of the topics or issues of interest. Also try to include individuals who have experience planning tools or graphic methods. If possible, have a member with experience in survey development and statistical analysis.

3.6 How will the survey be administered?

Select the method that will provide sufficient information as efficiently as possible. Interviews can provide very detailed information, but they require special training to conduct effectively, they can be time-consuming and they might be difficult to analyse and interpret. Interviews can be useful when you need to collect detailed information from a relatively small group of people. Interviews can be used to explore issues and options to a greater extent than self-reported surveys. Self-reported surveys typically provide less detailed information but can be used to collect large amounts of information in a relatively short period of time.

3.7 What resources will be needed?

Identify the staff and other resources required for the survey. Depending on the type of survey being conducted, resources can include a project manager, trained interviewers, statisticians, data analysts, printing costs, mailing costs, data entry, and data processing. Keep in mind that the people who respond to the survey should be considered as a resource in terms of number, time invested and information provided.

3.8 What survey items will be used?

Based on the information requirements, determine if you can use an existing survey, modify an existing one or if you need to develop an entirely new survey.

3.9 How will survey information be analysed and reported?

Decide what methods will be used to summarise, interpret and communicate the survey results.

Data can be summarised with methods as simple as tallies, percentages or averages. Depending on the decisions that are going to be made based on the survey information; more complex analytic methods might be appropriate. These more complex methods could involve analysis of variance to compare the results from different groups such as managers and employees or regression analysis to check the relationship between two variables such as job satisfaction and intent to leave the organisation.

Decide how you will display the survey information. This decision should be influenced by the interests and preferences of the audience. Do they prefer simple graphs, statistical charts or text summaries? The presentation format of results to executives who are considering policy changes might be different from that offered to external customers. If you need to present the results to different groups, consider



tailoring the level of detail and data display methods to the different levels of interest.

3.10 How many people need to respond to the survey?

Due to limited resources, it might not be feasible to collect survey information from every member of a group you’re interested in. Sampling is a technique used to determine the size and types of group subsets needed to provide representative information about the larger group.

The sample size required for a survey partly depends on the statistical quality needed for survey findings; this, in turn, relates to how the results will be used.

Even so, there is no simple rule for sample size that can be used for all surveys. Much depends on the professional and financial resources available. Analysts, though, often find that a moderate sample size is sufficient statistically and operationally.

One way to simplify the situation is to think of respondents in terms of segmentation. Segmentation means to sort respondents into groups based on similar characteristics, such as relationship to the organisation (internal, end-user, supplier), rank or the types of products or services used. Sorting respondents into groups is a commonly used method to identify and distinguish the experiences and perceptions of distinct groups. Demographic items are used in surveys to identify respondents so that their data can be sorted and analysed as needed.



4. QUESTIONNAIRE DESIGN

4.1 Creating Survey Items

Write survey questions for each topic of interest; write down whatever comes to mind. At the beginning it is better to have too many items than too few. Do not focus on writing the perfect question. It is more important at the beginning to cover all of the topics than to be elegant in phrasing. Consider using brainstorming, tree diagrams and other idea-generation methods to create survey items.

Try to create items that are:

• Clearly written. Statements should be short and easy to read. Do not use jargon, technical terms, or acronyms that are unfamiliar to the people answering the survey.

• Concise. Get to the point as quickly as possible. Eliminate extraneous words or ideas, so the respondent can concentrate on the survey item without being distracted.

• Specific. Focus on one idea at a time. Each item should collect information on a single behaviour, attitude, opinion, event or subject.

• Explicit. Do not force people to guess about what is being asked. Be sure they understand what information you want by explicitly stating so. If necessary, highlight or underline what is needed by way of an answer.

4.2 Selecting Response Formats

Along with the statements and questions, you need to provide methods for people to give their answers. Typically, survey items are used to ask people how much they agree with some statement, how important something is, or how often something happens. Here are some common types of survey response formats used to collect information:

4.2.1 Close Ended Questions

Two-options. This is the simplest response format. Options may include No-Yes, Disagree-Agree, False-True, Oppose-Favour. Depending on the desired information it may be the appropriate response format. However, using a rating scale or ranking elicits more information [15].

Are you married?

Yes No

Rating Scales. Surveys often ask that products or services be rated according to some scale. Some survey items present statements and ask people to rate how much they agree or disagree with the statements. Some examples of rating scales are presented below:



My supervisor encourages subordinates to participate in important decisions.

1 2 3 4 5 6 Strongly Disagree

Disagree Slightly Disagree

Slightly Agree

Agree Strongly Agree

How satisfied are you with the service you received?

1 2 3 4 5 Very

Dissatisfied Dissatisfied Neutral Satisfied Very

Satisfied

Please rate the condition of the housing you were provided: 1 Bad, in need of total rehabilitation 2 Very Poor, in need of several major repairs 3 Poor, in need of at least one major repair 4 Fair, in need of minor repairs 5 Good Condition 6 Excellent Condition

When creating rating scales, be sure that the end points or anchors are equal and opposite in meaning. If that is not done, there is a risk of biasing the responses you get from the survey. For example, the following rating scale might provide answers you like, but not information you need:

1 2 3 4 5

Mostly Satisfied

Completely Satisfied

Very Satisfied

Extremely Satisfied

Raving Fan

Ranking Items. Another common response choice is to ask respondents to rank (order) a list of options in terms of some factor, such as importance. These data help to prioritise what is most important to respondents. Then efforts can be focused on those aspects most important to respondents.

Please rank the following five improvement objectives in terms of importance by marking a 1 next to the most important objective, a 2 next to the second most important objective, and so forth:

Having a quick success Increasing the amount of output Reducing the price charged to customers Reducing work backlog Reducing the number of defects

Selecting Options. This response format presents a list of statements or options and asks respondents to circle all that apply to them. This format is similar to the ranking question, but does not require survey respondents to put things in an order. This kind of question is easier to respond to than having to rank-order questions, for in many cases respondents cannot prioritise options when they feel that everything is of equal importance.



Please tick in the square next to each of the graphic or statistical methods that your team has used to analyse process performance data:

Pareto diagrams (Ranked Histograms) Histograms Scatter diagrams Run charts Control charts Regression analysis

Items in a Series. When several questions use the same response category, it is possible to present the responses in a table, rather than write separate questions for each.

How satisfied are you with the product you received? Very

Dissatisfied

Dissatisfied

Neutral

Satisfied Very

Satisfied A. NOTAM 1 2 3 4 5 B. Accuracy 1 2 3 4 5 C. AIP 1 2 3 4 5 D. AIP Amendment 1 2 3 4 5 E. AIP Supplement 1 2 3 4 5

4.2.2 Comments and Open-ended Questions

This question/response type allows respondents to provide additional comments or other information in response to general questions. These questions usually leave blank space where respondents can write whatever is important in their own words and format. Examples are listed below.

Please provide us suggestions on how we can improve?

Please specify any products or services you need that we do not currently provide?

Please list anything else you would like us to know?

This information can be very useful in understanding issues that have not been touched upon or to gather ideas and generate options for improvement. On the down side, responses to these types of questions can be difficult to interpret.



4.2.3 Demographic Questions

Demographic information is typically used to segment respondents into smaller groups based on specific characteristics such as rank, age or organisational level. This segmentation is important if one of the purposes of the survey is to determine if there are significant differences in responses between groups. The following are examples of demographic questions:

Which one of the following age categories do you currently fall in?

Under 20 20-29 30-39 40-49 50-59 Over 59

What is your organisational level?

Directorate Division Director Branch Head Section Head First-line Supervisor Non-Supervisory

4.3 Review Survey Items

After you have developed a set of potential survey items and response scales, review them to make sure that they are:

• Relevant to the purpose of the survey. Items that stray from the purpose will not provide the information needed. Be sure to have a specific reason for why an item is being asked in a survey. Always focus on the purpose of the survey and the type of information that will be needed to support that purpose. Carefully compare the items to your survey purpose to ensure that they address the issues that have been identified.

• Appropriate for the individuals being surveyed. Do not include items that people do not have the knowledge to answer. For example, store customers could probably answer questions about a store’s layout; but they would not be able to answer questions about a store’s compliance with fire regulations.

• Simple Wording. Adapt wording to the vocabulary and reading skills of your respondents. Are there any confusing words? Do any words have double meanings?

• Clear Wording. Avoid words or phrases that can be ambiguous or misinterpreted.



Instead of asking “Do you often use service ABC?” prefer “Do you use service ABC more than 2 times per week?”

Instead of asking “Is the service ABC used on a regular basis or on an occasional basis?” prefer “During the last year, did you use service ABC at least 4 times?”

Avoid adjectives like “Do you enjoy exciting tours?” or “Is this organisation energetic and active?”

• Capable of providing the appropriate type of results. Think ahead to how the information will be summarised. The summary of results should provide the types and level of information required by the survey users. Will the results be presented in simple bar graphs or subjected to advanced analysis? How much detail will be required to meet the information needs of those using the survey? For instance, if the survey users are interested in general impressions, there is little reason to present averages calculated to the seventh decimal point. On the other hand, if the survey users need to make precise distinctions among quality features, then just providing a list of verbatim comments is not going to be helpful either. Eliminate items that are inappropriate or redundant and clarify items that are unclear. Check items to ensure that they are not:

• Overlapping. Avoid presenting response choices that overlap. Overlapping choices can lead to confusion of the respondent and difficulty in interpreting information.

Circle the number that best represents the number of hours per week you spend on physical readiness training:

1. None at all

2. Less than one hour per week

3. One to two hours per week

4. Two to three hours per week

5. Three to four hours per week

6. Four or more hours per week

• Double-barrelled. Avoid having respondents address two different issues in the same item.

• Leading. Avoid giving clues that point to the desired answer or limiting the answers to those desired. Such questions influence people to respond in a way that does not accurately reflect their position. A question can be leading in several ways: (1) when it implies that the respondent should be engaged in a particular behaviour (2) when the response categories are unequal or loaded in one direction (3) when words with strong positive or negative emotional appeal are used, such as “freedom”, “equality”, “boss”, “bureaucratic”, etc. Here are some examples for leading questions



More pilots are using the service XXX than any other variety of integrated briefing. Do you use XXX?

Do you feel that you are being forced to pay too much for day-care services?

• Redundant. Avoid asking the same question more than once. Rewrite or delete inappropriate or repetitive items. Then review the remaining ones for clarity and to ensure that they can collect the information required. In addition to reviewing the items yourself, you might consider conducting a pre-test of the survey.

• Include all necessary information. In some cases, respondents may not know enough to adequately answer a question. For such cases provide necessary information for respondents.

• Avoid questions that are too demanding and time consuming.

4.4 Review Survey as a Whole

When you think you’ve finished the individual questions, step back and look at the questionnaire as a whole. Remember, the questionnaire is a total package and needs to be considered as such.

• It needs a strong introduction conveying to the respondent what the survey is about. Sometimes this information is explained in a cover letter attached to the survey.

• It should indicate why the questions are being asked.

• It should contain instructions on how, when and where respondents are to return the completed survey.

• It needs interesting and readily answerable questions at the beginning to gain respondent attention and build rapport. It should address important topics early, rather than late, in the questionnaire.

• The conclusion should be gentle and friendly, expressing gratitude for the respondent's time and effort.

• If necessary, it may contain short instructions on how to complete the survey usually follows the introduction. Sometimes an example is provided to show how to mark response boxes or fill out an answer sheet.

• If necessary, it may provide a statement to reassure people that their responses will remain anonymous.

• It should be printed in an easy-to-read typeface.

• A numbered response should mean the same thing throughout the questionnaire.

• It should not make respondents turn a page in the middle of a question or between question and answer.

• Questions are distinguishable from the instructions and the answer.



Furthermore the questions need to flow well from one to the next and designers should be aware that earlier questions provide information and context to the respondents that they may use in later answers. Often the answer to one question may influence the answer to a later question. For instance, suppose respondents are asked first

How do you feel about your job?

and later on

How do you feel about life in general?

Answers to the second question may be tempered by the first question. Because respondents have already reported their feelings about their job, including those feelings in the second answer may be redundant. On the other hand, if their job is very important to them (or salient for some other reason), then the answer to the first question may be used when constructing the second answer.



5. PRE-TESTING

Pre-testing is an important part of questionnaire design. Therefore, after you have a draft survey, you may conduct a test to further check and refine the survey. Pre-testing involves trying out a survey before wide-scale distribution to make sure it is easy to understand and provides appropriate data. Pre-testing is usually done with a small group of respondents who are representatives of the larger group. If you choose to conduct a pre-test, consider taking the following steps:

• Ask colleagues to review the questionnaire critically. Let co-workers read the questions to see if the wording and instructions are clear and if the questionnaire will accomplish the study objectives.

• Select people as similar to your respondents as possible to pre-test the questionnaire.

• Explain the process. Gather together the group who will be pre-testing the survey. Briefly explain that they are to evaluate the survey as they are taking it and identify questions that are unclear, topics that are missing and suggestions about improvements.

• Request written comments. Ask people to make notes on the survey where they have questions or comments. Some individuals will not feel comfortable discussing survey problems with the group, but will provide written feedback. Collect the completed surveys at the end of the meeting and review any written comments provided.

• Measure time requirements. Time how long it takes people to complete the survey. Use this information to determine if the survey is too long or if it could be expanded in length.

• Hold a group discussion. After everyone has completed the survey, review each section of the survey with the group to gather feedback. Record the information and study it after the testing session. Sometimes, comments that do not seem to be a good idea initially, can, upon further reflection, be very useful. Ask if important issues or satisfaction areas were left off the survey. Also ask if there are any other demographic questions that need to be added to the survey.

Use the information obtained through the test to modify the survey. Modifications could involve rewriting items to increase their clarity, eliminating items that do not provide relevant information or adding items to obtain more complete information on important issues. Once you have a final version of the survey and have identified the sample, it is time to administer the survey.



6. ADMINISTERING

This step is simply the conduct of survey with the method decided during your planning. It varies with respect to the survey method that has been chosen in the planning.

For self-reported surveys administering consists of the following steps:

• Develop advance notice materials for respondents.

• Develop follow-up postcards to remind respondents to return the survey.

• Prepare survey material.

• Send an advance notice about one week before you begin the survey.

• Assemble materials and send them for the first time.

• Track survey responses.

• Send follow-up postcards.

• Send a second complete set of materials to non-respondents.

For interview-based surveys administering consists of the following steps:

• Anticipate questions respondents may have.

• Develop interview logs.

• Establish guidelines for handling “no” answers, hesitant participants, etc.

• Develop advance notice materials for respondents.

• Assemble materials.

• Train interviewer, if necessary.

• Arrange facilities.

• Send an advance notice about one week before you begin the survey.

• Schedule interviews and collect data.



7. ANALYSIS

After surveys have been administered, you need to summarise, analyse and interpret the results. This requires sorting and consolidating individual responses to survey items so that they can be more easily displayed and understood. Although extensive statistical analysis can be used in order to analyse survey results, there are some common and easy mathematical techniques that can make your analysis more understandable. They are covered in four basic groups:

• Simple Measures (numerical counts and percentages).

• Measures of central tendency (mean, mode, median).

• Measures of variability (range, standard deviation, variance).

• Rank Ordering.

7.1 Simple Measures

These are simplest measures that you can use to analyse data like numerical counts and percentages.

Numerical Counts (Frequencies). Counts of frequencies tell us how many times something occurred or how many responses fit into a particular category. Numerical counts are typically presented as tables or bar graphs for ease of interpretation.

1

Very Dissatisfied

2

Dissatisfied

3

Neutral

4

Satisfied

5 Very

Satisfied Availability of technical support 3 14 3 30 50 Responsiveness of technical support 10 15 2 45 28 Time to complete technical support service 1 3 0 88 8 Courtesy shown by technical support provider 17 70 0 7 6

0102030405060708090

Very Dissatisfied Dissatisfied Neutral Satisfied Very Satisfied

Availability of technical supportResponsiveness of technical supportTime to complete technical support serviceCourtesy shown by technical support provider

Figure 2. Example of Frequency Distribution



Percentages. A commonly used statistic, the percentage expresses information as a proportion of a whole. They are also useful when we want to show a frequency distribution of grouped data. The frequency distribution is a classification of answers or values into categories arranged in order of size or magnitude. Percentages can be displayed using tables, bar graphs or pie charts.

1

Very Dissatisfied

2

Dissatisfied

3

Neutral

4

Satisfied

5 Very

Satisfied Availability of technical support 3 14 3 30 50

Very Dissatisfied

3%

Dissatisfied14%

Neutral3%

Satisfied30%

Very Satisfied50%

Figure 3. Example of Percentages with a Pie Chart

7.2 Measures of Central Tendency

Measures of central tendency are used to characterise what is typical for the group.

Mean. The mean (or average) is commonly used in reporting data. It is obtained by summing all answers or scores and dividing by the total number. It is also useful for summarising findings from rating scales. Even with the narrative scales, a number value can be assigned to each category and thus calculate mean.

Mode. The mode is the most commonly occurring answer or value.

Median. The median is the middle value. It is the midpoint where half of the cases fall below and half fall above the value. Generally it is used to divide a group of participants into upper and lower groups.

7.3 Measures of Variability

Measures of variability express the spread or variation in responses. The main reason for measuring variability is that the mean may mask important differences or be skewed by extreme values at either end of the distribution.



Range. The range is the simplest measure of variability. It compares the highest and lowest value to indicate the spread of responses or scores. It is often used in conjunction with the mean to show the range of values represented in the single mean score.

Standard Deviation. The standard deviation measures the degree to which individual values vary from the mean. It is the average distance the average score lies from the mean.

Variance. It is used instead of standard deviation. It is simply the square of the standard deviation.

7.4 Rank Orderings

The analysis techniques discussed so far involve calculating numbers (i.e., using the actual data to provide measures of results). Rankings, on the other hand, are not actual measurements. They are created measures to impose sequencing and ordering. Rankings indicate where a value stands in relation to other values or where the value stands in relation to the total.

Generally Pareto Charts are used to present ranked data. Pareto charts are particularly useful in helping to focus attention on major factors or issues. Pareto charts are bar graphs that present data in descending order of frequency or magnitude combined with a cumulative percentage line. Here is an example Pareto Chart from the AIS Quality and Service Level Management Survey ’01:

SLA Content n %Roles & responsibilities 6 100Goals&objectives 5 83Reporting&escalation 4 67Methods for adjusting 4 67Help desk 3 50Penalties 1 17Incentives 1 17

SLA Content

01234567

Roles &

resp

onsib

ilities

Goals&

objec

tives

Report

ing&es

calat

ion

Method

s for

adjus

ting

Help de

sk

Penalt

ies

Incen

tives

Figure 4. An Example of Pareto Chart (Ranked Histograms)



8. PRESENTATION

One of the most important actions after collecting and analysing survey data is communicating the results. You might need to tailor presentations of results for different groups in the organisation. First, report the results to managers in the organisation, particularly those in areas highlighted by the data.

Presentations of results to those working in the processes should focus on specific product or service requirements. People working in the process have intimate knowledge of that process and can offer ideas on how to improve it.

A summary of the survey results should be provided to respondents when possible, along with a discussion of planned efforts to address identified problems and concerns.



9. CONCLUSIONS

Surveys can be used to provide information to support improvement efforts. Surveys can be used to obtain information on external customer requirements, current satisfaction levels, ideas on how to improve and the effects of previous efforts. Based on this information, decisions can be made on where to focus the organisation’s future improvement efforts. Keep the following in mind when beginning a survey:

• Have a clear purpose for conducting a survey.

• Tailor your survey to the particular respondents and issues being assessed.

• Keep surveys short and simple.

• Communicate results to managers and respondents.

• Use results to guide future improvement efforts.



APPENDIX A PLANNING A SURVEY

SURVEY PLANNING

1. What are you going to survey? _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ 2. What is the purpose of the survey? _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ 3. Who will use the evaluation? How will they use it? Who/Users How will they use the information? How may others be involved in the survey? __________________ 4. What questions will the survey seek to answer? _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ 5. What information do you need to answer the questions? What I wish to know Indicators – How I will know it? 6. When is the survey needed? _______________________________________________________________________________ _______________________________________________________________________________ 7. What resources do you need? Time available to work on survey

Money

People (professional, participants, etc.)



8. How will the survey be administered? Method Self Reported Survey / Mail

Self Reported Survey / E-mail Self Reported Survey / Fax Self Reported Survey / Internet

Telephone Interviews Face-to-Face Interviews Focus Groups

Will a sample be used? Yes No If yes, describe the procedure you will use. ____________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ 9. How will the data be analysed? Data analysis methods. ____________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ Who is responsible? _______________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ 10. How will the information be interpreted and by whom? _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ 11. How will the survey result be communicated and shared? _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________



APPENDIX B SIMPLE TIPS AND TRICKS FOR CONFIDENTIALITY

The confidentiality of the data supplied by respondents is of prime concern to all reputable organisations. A simple policy for survey organisations to safeguard confidentiality is as follows:

• Using only number codes to link the respondent to a questionnaire and storing the name-to-code linkage information separately from the questionnaires.

• Refusing to give the names and addresses of survey respondents to anyone outside the survey organisation, including clients.

• Destroying questionnaires and identifying information about respondents after the responses have been entered into the computer.

• Omitting the names and addresses of survey respondents from computer files used for analysis.

• Presenting statistical tabulations by broad enough categories so that individual respondents cannot be singled out.



APPENDIX C BUDGET CHECKLIST

A checklist of budget factors, such as this partial one, may be useful in estimating total survey costs (whether in time or money).

• Staff time for planning the study and steering it through the various stages, including time spent with the sponsor in refining data needs.

• Sample selection costs, including central office staff labour and computing costs.

• For “area segments” samples, substantial field staff (interviewer) labour costs and travel expenses for listing sample units within the segments.

• Labour and material costs for pre-testing the questionnaire and field procedures; the pre-testing step may need to be done more than once and money and time should be set aside for this (especially when studying something new).

• Supervisory costs for interviewer hiring, training and monitoring.

• Interviewer labour costs and travel expenses (including meals and lodging, if out of town).

• Labour and expense costs of redoing a certain percentage of the interviews (as a quality assurance step) and for follow-up on non-respondents.

• Labour and material costs for getting the information from the questionnaire onto a computer file.

• Cost of spot-checking the quality of the process of computerizing the paper questionnaires.

• Cost of “cleaning” the final data - that is checking the computer files for inconsistent or impossible answers; this may also include the costs of “filling in” or imputing any missing information.

• Analyst costs for preparing tabulations and special analyses of the data; computer time for the various tabulations and analyses.

• Labour time and material costs for substantive analyses of the data and report preparation.

• Potentially important are incidental telephone charges, postage, reproduction and printing costs for all stages of the survey - from planning activities to the distribution of results.

End of Document

european organisation for the safety of air navigation · less costly to conduct than telephone...

Documents