ARIZONA IN THE GREEN ECONOMY

ARIZONA IN THE GREEN ECONOMY

Lessons Learned in Designing an Effective Business

SurveySession 4

April 26, 2011

CREDITS AND DISCLAIMER

2

This report was funded by a grant awarded by the U.S. Department of Labor’s Employment and Training Administration. The report does not necessarily reflect the official position of the U.S. Department of Labor. The Department of Labor makes no guarantees, warranties, or assurances of any kind, express or implied, with respect to such information, including any information on linked sites and including, but not limited to, accuracy of the information or its completeness, timeliness, usefulness, adequacy, continued availability, or ownership.  State of Arizona has the copyright to this report.  Internal use by an organization and/or personal use by an individual for non-commercial purposes is permissible.  All other uses require the prior authorization of the copyright owner.

SESSION OVERVIEW:PREPARING FOR A SURVEY Defining the survey purpose

Identifying survey recipients

Defining key concepts

Designing the survey instrument

Preparing the survey cover letter

DEFINING THE SURVEY PURPOSE

4

4 COMMON ANALYTIC RESEARCH METHODS

1. Archival Use of available

administrative records to analyze the past or forecast the future

Observations used from a time period before a research question is posed

2. Experiment/Quasi-experiment

“Pilot project” compared against the “real world”

Controls for a single variable Helps to factor out other

explanations for the occurrence of a few observations

3. Case study In-depth examination of a

particular story Review multiple influences Develop theories about possible

causal explanations

4. Survey Primary data collection Surveys typically focus

gathering a larger number of responses that can be analyzed using statistical methods

Interviews and focus groups that can be “de-constructed” and easily compared (often provide in-depth companion info for surveys)

5

DEFINING THE SURVEY PURPOSE:ANSWERING EXPLORATORY QUESTIONS

Gathering and reporting facts or opinions

Examples from the Arizona Green Survey: How many Arizona companies have “green”

economic activity? What proportion of Arizona’s employment is

dedicated to “green” economic activity? How rapidly are employers planning to add

“green” jobs? What are the wages provided to workers in these

“green” jobs?

DEFINING THE SURVEY PURPOSE:TESTING SPECIFIC HYPOTHESES

Assessing whether patterns emerge as expected

Examples from the Arizona Green Survey: In which industries does Arizona have a high

concentration of green activities? How much faster do companies with green jobs grow

relative to all other companies? How much more do jobs requiring green skills pay

than other jobs in the same occupation?

BIASES THAT MUST BE UNDERSTOOD

The selection of questions reflect the biases and perspectives of the researchers The questions being posed often include inherent

biases

Answers to survey questions reflect the perspective of respondent

In business surveys, multiple respondents may answer differently

The selected sample may not necessarily reflect the general population

Difficulty in getting “accurate” responses from larger and smaller businesses

IDENTIFYING SURVEY RECIPIENTSWho are the businesses that you are surveying?

9

DATABASES FOR BUSINESS LISTS Quarterly Census of

Employment and Wages Availability Suitability of contacts

Partners and stakeholders Availability Reliability

Proprietary vendors (e.g., InfoGroup; D&B) Cost Accuracy and

completeness

10

SELECTING A VALID SURVEY FRAME

Determining how to narrow the list Geography, industry, size, business age,

available contact info, etc.

Cleaning up the list Time, access to information, accuracy of

alternative data sources

Determining appropriate contacts within the company Related to questions being asked

11

GETTING THE RIGHT SAMPLE SIZE Identify sub-groups of survey recipients (“stratification”)

Identify potential biases within sub-groups and the need for “oversampling” some groups

Estimate expected response rates and overall sample size required

Establish quotas for sub-groups in the sample (balancing resources with desire for of statistical validity)

Adjust the sample size to reflect sub-group quotas as well as expected overall response rate

12

DEFINING KEY CONCEPTS13

WHAT ARE YOU TRYING TO MEASURE? Operationalizing key concepts

“Firm” vs. “establishment” “Jobs” “Employment” “Wages”

Identifying terms with “shared definitions”

Understanding the limits What can you expect respondent to ‘know’?

Selecting appropriate time periods The power of memory The limits of projecting the future (OR the problem with

“hypothetical questions”)14

EXAMPLE: DEFINING “GREEN” JOBS

Examples Of Green Activities

Making Goods or Providing Services

Improving Processes in Existing Goods or Services

Energy generation Producing renewable energy

Improving fuel efficiency

Energy efficiency in buildings, equipment, appliances, or vehicles

Building long-lasting batteries

Implementing smart grid technologies

Pollution reduction/removal, green-house gas reduction, recycling and reuse

Producing industrial scrubbers or generating nuclear energy

Managing energy proactively

Natural resources conservation

Organic farming Conserving soil, water, or wildlife

Environmental compliance, education and training, and public awareness

Training in regulatory compliance

15

AN EXAMPLE: “RENEWABLE ENERGY” What are key elements of the “renewable energy”

industry? Which segments do we focus on? How do we handle nuclear energy or other segements?

What are related industries? …activities? …occupations?

How do you handle suppliers that might not be unique to solar such as… Maintenance and repair? Accountants and attorneys? Educational institutions Utilities? Investors?

What occupations would want to include? 16

ANOTHER EXAMPLE:OPERATIONALIZING “INNOVATION” IN RENEWABLE ENERGY

Defining innovation Innovation—introducing something new—mostly

derives from technological advances that ultimately lead to productivity and prosperity gains.

Communities must provide an appropriate foundation to realize those gains, including: A strong educational system A solid physical infrastructure, A base of research and development activities, Access to financial resources, and other factors.

Science and engineering—in the educational system and in the workforce—are of particular importance.

17

Source: Adapted from Arizona Indicators, Morrison Institute for Public Policy

http://arizonaindicators.org/innovation

SELECTED WAYS TO MEASURE INNOVATION

Human capital University entrance

exam scores Graduate students

and postdoctorates Employed science,

engineering & health doctorate holders

Patents granted

Financial capital Research and

development Innovation grants Venture capital

High tech employment High tech industries High tech

occupations

18Source: Arizona Indicators, Morrison Institute for Public Policy

HOW COULD FURTHER RESEARCH HELP US BETTER UNDERSTAND “RENEWABLE ENERGY” RELATED TO…?

…Arizona’s educational system?

…the state’s physical infrastructure?

…research and development activities?

…access to financial resources?

…the workforce’s preparation for emerging jobs in this area?

19

DESIGNING A SURVEY INSTRUMENTCreating and aggregating information about a large number of respondents

20

5 ALTERNATIVE INTERVIEW METHODS

Interview type Advantages Disadvantages

Personal interview

•Stronger commitment to completing the interview.•Greater confidence in finding the right recipient•Longer interviews tolerated

•More costly•Greater potential for creating a non-representative sample

Telephone surveys

•Quicker turn-around •Wide availability of telephone numbers•Availability of CATI to address complex questions•Role skilled interviewers to elicit more complete answers

• Negative perception of surveyors as “telemarketers”•Challenges of getting past “gatekeepers” (e.g., administrative assistants, caller ID)

Mail surveys •Relative expense per response•Ability to use only names and address to reach respondent•Can use photos•Answer at respondent’s leisure•Viewed as less intrusive

•Survey takes longer•Response rates lower•Respondents are more likely to have “something to say” •Less ability to correct “misunderstanding” of questions

21

5 ALTERNATIVE INTERVIEW METHODS(CONTINUED)

Interview type Advantages Disadvantages

Email surveys •Inexpensive with quick turnaround•Simple•Allows attachments

•Cannot handle complex logic•Requires having email addresses•Perceptions about viruses and spam•Potential missed messages due to “junk” mail filters•Difficult to manage sample due to ease of forwarding•Loss of anonymity•Response bias/inability to generalize

Web surveys •High speed in getting responses•Low cost•Use of email to generate responses•Flexibility in creating complex logic•More responsive to ‘sensitive’ questions•Greater ability to use open-ended questions•Respondent inputs data

•Increase survey “quitting”•Restricting response to the “sample” •Even greater difficulty in generalizing results

22

TIPS IN SURVEY DESIGN

Keep it Short and Simple (KISS)

Keep questions simple and specific

Start with an interesting question What makes a question

interesting?

Ask “answerable” questions

Taking care with question skipping

Recognize that surveys “reveal a story” about the research Encourage a “logic flow”

for questions Understand how

question order might affect response 23

TIPS IN WRITING QUESTIONS

Allow don’t know, not applicable, or OTHER options Avoids “skipping” and early

termination

Use forced-choice questions whenever possible

Create appropriate answer categories How many categories?

Ensure that answer options are complete AND mutually exclusive

When to use open-ended questions Provide context for specific questions or

survey You get 1 or 2 shots at most

Recognize respondents’ desire to please Will affect responses Omit the middle alternative and

focus on “intensity” Provide option for “no opinion” or

“not applicable”

Watch for “double-barreled” or “negative” questions? About people or policies that might be

controversial Using words that unintentionally evoke

emotion24

THE “STORY” BEHIND THE SURVEYWHY DID WE CHOOSE THIS STRUCTURE?

25

KEY GOALS IN PRE-TESTING THE SURVEY INSTRUMENT

Variation

Meaning

Task difficulty

Respondent interest and attention

Flow and naturalness of the sections

The order of questions

Impact of skip patterns

Timing

Overall respondent interest and attention

Respondent reaction

Specific Questions Instrument as a Whole

26

PREPARING THE SURVEY COVER LETTER

27

TIPS FOR A GREAT COVER LETTER

Keep the letter short Personalize it to an

individual whom you want to respond

Tell the recipient how they could benefit directly (or pay them for their time)

Tell them how long it will take and BE HONEST

Give them options for responding

Provide a deadline within a week of receipt

Say something memorable (funny is good, but not offensive)

Use the “right” logo Careful about who signs

the letter

28

DEVELOPING YOUR OWN SURVEYAn Exercise29

RESOURCE IN DESIGNING AN INNOVATION SURVEY

30

WORKING GROUP EXERCISE: DESIGNING A BUSINESS SURVEY ON INNOVATION IN RENEWABLE ENERGY

1. Issues to explore: How many Arizona renewable

energy companies are

“innovative”?

What are the key characteristics of

“innovative” renewable energy

companies?

2. Hypotheses to test: “Innovative” companies were

much more likely to add jobs likely

to add jobs during the recession.

“Innovative” companies need

different kinds of skills and training

than do other renewable energy

companies.

What are the key talking

points in the letter?

Who should sign it and

why?

Design an Instrument– 2 survey design groups

Prepare a Cover Letter -- for each group

31

Who should get the survey?

What subgroups are there?

How should the survey be

adapted to the subgroups?

Determine Sample Frame-- for each group

THANK YOU