Regional Workshop on Science, Technology and Innovation Indicators for Gulf countries

Four Seasons Hotel, Doha, Qatar15 to 17 October 2012

The R&D Survey: governance, logistics and process Michael Kahn

Research and Innovation Associates, South Africa

R&D Survey: governance, logistics, process

• The R&D Survey is a project studying peer institutions of the innovation system

• Goal-directed project management

• Developing a project milestone plan

• Learning by doing

The R&D Survey as a Project

• A project is a process of change involving the application of resources over a fixed period of time.

• Has a clear beginning and deadline.

• Logical sequence, defined objectives, verifiable indicators, specified deliverables.

• Agreed budget and human resources.

• Mechanisms for learning and adjustment.

Adopting a project management methodology

• Method to plan, organize, lead and control projects.• Goal directed project management’ (GDPM)

methodology projects lead to development of a final product, the people and the organization involved.

• Successful projects have clarity of objectives, sound leadership and enjoy political support. They excel at communication among stakeholders and staff.

• Require clear statement of objectives and methods of verification, with associated timeline

• Break down the overall project into a clear set of milestones that hold the project together

An innovation System

UNIVERSITIES+

ACADEMIES?

BUSINESS

GOVT. DEPTS INSTITUTES &

MUSEUMS

S&T SERVICES

UTILITIES

PUBLIC

SECTOR

Regulatory, ethical & legal, environmental,

IPR

Cultural-

Political norms

Associations

EDUCATION

TRAINING

SKILLS DEV

STANDARDS

WORLD SYSTEM

S.O.E.sNGOs

INFORMAL

SECTOR

ECONO

MIC

SYSTEM

Project elements

• Objectives of the project

• Who is the target population?

• Major stakeholders

• Benefits of the project

• Approach for delivery

• Project Organization inc. budget

• Milestone Plan

• Constraints, risks and corrective actions

Governance (1)– Who is the champion?– Oversight Committee appointed.– Access and Confidentiality agreed.– Stakeholder workshop.– Oversight Committee approves fieldwork.– Oversight Committee approves results.– Media event for R&D Survey Results.– Evaluation and close out

Logistics (1)– Appoint Project manager.– Premises secured.– Staff appointed; initial training.– Database developer appointed.– Printer contracted.– Staff trained for fieldwork..

Process (1)• Scope and methodology.

– Which sectors are to be covered?– Method: census, sample or purposive?– Develop the survey registers– Who will complete the survey questionnaire?– Contact with organizations– Tracking incidents

A census of Government and HEIs, is to be conducted with the latter split into National and Foreign categories. A purposive survey will be carried out for the Business sector. This will allow for the development of a registry of Business sector R&D performers, split by National and Foreign categories. A purposive survey seeks out to identify R&D performers case-by-case making use of secondary data as well as referrals as needs be. A similar approach is followed with the private non-profit sub-sector (PNP). Given the important role of expatriate personnel in Qatar, personnel data will be captured regarding Qatari/foreign status.

The questionnaires will be administered by enumerators/analysts for completion by a designated person at each responding institution. Where possible a contact survey should be carried out. Each Survey questionnaire carries a unique identifier for each respondent. The questionnaires will be delivered to respondents by post, by e-mail, as well as through direct contact. It is hoped that respondents will complete the Survey within two weeks of receipt. The CEO of each responding institution is requested to sign off the completed questionnaire.

The Survey requires the collection of information from the various respondents that make up the Qatar innovation system. Building interest, trust, and commitment with these survey respondents is essential if their cooperation throughout the Survey life cycle is to be assured. Without a guarantee of confidentiality, respondents may be reluctant to divulge the required information.

13

A lesson from nature

• Covers basic, applied research, experimental development

• Related Scientific & Technological Activities excluded

• Clinical trials; software development …IN

• Recognizes increasing role of Services sector

• Much of Social Sciences & Humanities …OUT

What counts as R&D

Where is R&D performed?

• Universities• Public research organizations• Private sector• Stock exchange• Annual reports• Direct and indirect incentive registers• Chamber of Commerce• Industry associations • Regulators

• Research permits• Clinical trials• GM trials

• Referrals

R&D excludes

•S&T Education and Training (STET)•S&T Information and Documentation (STID)•Routine S&T Services (STS)• Testing• Mapping• Surveys• Database management & debugging• Software development• Policy-related research• Monitoring and evaluation

In practice however, especially as this is the first R&D Survey, the information that is being sought may not be readily available in compiled form. There may also be concerns at releasing information that is felt to contain ‘company secrets.’ Accordingly the enumerators must be prepared to provide assurances that only high-level information is needed. The contact point for Department-based Research Institutes (DBRIs) will be the senior civil servant such as the Director-General or Permanent Secretary. The case of public research institutes (PRIs) is somewhat different especially where these have been established as arm’s length statutory bodies. In this case the Chief Executive Officer or executive responsible for research management would be the contact point.

Interviews should be sought with the Chief Technology Officer or Chief Financial Officer of the largest local and foreign firms. It is always important carefully to identify the appropriate unit of measure. Ideally one would wish to capture data for each independent R&D performing entity. One test of independence might be to determine the income tax status of the various companies that make up a diversified holding company. Each subsidiary that is a registered tax-payer would in principle be of interest to the Survey. Where a firm indicates that it is performing R&D one should use the opportunity to enquire ‘who else in your industry sector is also doing R&D, and who should we contact to find out more?’

The first enumeration contact between the Survey and respondent is crucial: the old maxim applies:

“One can only make a first impression once”

First enumeration contact should follow an agreed script that states with clarity the purpose of the interaction and the authority under which the interaction takes place.

Salutation ……………. This is the Office of …………….my name is ------------------------- We are carrying out the official Survey of Research and DevelopmentCould you please let me speak to ------------------------ to discuss completion of the Survey

Who indeed is the target of the Survey?

We have contact

A typical conversation involving an enumerator and the R&D manager could go something as follows:

Salutation …………….• Thank you for your time. We are trying to estimate the

amount of R&D activity in your organization.• Does your firm carry out R&D?• Could you give me some idea of what R&D you are

working on?• I see….. and this work is done in-house?• How many staff are involved in this work?

Professionals; technicians?• Full time/part time?• How much do you think this costs at present?• Excellent. We would like you to complete a survey

questionnaire ….

The services sector is important everywhere, but tends to be neglected in the R&D Surveys of many countries. Services are a site of much innovation activity, and the sector may account for significant levels of R&D, up to 40% of BERD as it does in the United Kingdom. The growth of mobile telephony semi-monopolies, local banks, insurance and retail chains points to considerable domestic management capability as well as the ability to develop and integrate software solutions. Software engineering may involve the construction of unique algorithms that are part of the intangible value of a firm. The development of this intellectual property leads to commercial benefit, and is definable as R&D. It is strongly suggested that effort is made to engage with the leading banks, insurers, mobile telephony and ICT companies.

R&D in the private not-for-profit (PNP) sector presents measurement challenges similar to those found in the for-profit business sector. These difficulties are not only to do with identifying those PNPs that perform countable R&D, but also to do with their juristic status. PNPs may domestically owned, or foreign-owned. Their fields of activity are wide, ranging from health to environmental monitoring and protection. They may be channels of considerable funds for in-house as well as contract R&D. Foreign-owned PNPs present other issues of conformance for the R&D Survey. In some cases all professional staff are paid in the home country; data resulting from local R&D work may also be processed abroad, so that the local operation is essentially one restricted to data gathering. The contribution of such activities to ‘national’ R&D effort is thus questionable.

Governance (2)– Oversight Committee approves fieldwork.– Oversight Committee approves results.– Media event for R&D Survey Results.– Evaluation and close out

• Database user requirement specification• Pilot Survey completed• Questionnaires approved and printed• Registries complete• Databases debugged and commissioned

Process (2)

• Dissemination and collection– What is in-house R&D?– Head count of R&D staff and calculation of FTE– Capital expenditure– Other current expenditure– Sources of funds for R&D– Classification of R&D– The special case of foreign-registered PNPs

Process (3)

• Capture, verification and calculation• Results compiled• R&D Survey Report

Process (4)

Developing the milestone plan

Project plan

Gantt chart

There is an intimate connection between the enumerator responsible for a particular respondent, the completed questionnaire, and the eventual database record. The greater the care given to obtaining clean and verifiable data, the better will be the end product. All relevant fields should be populated by the respondent, checked, verified, and then signed off. Hardcopy questionnaires have considerable value if every incident is logged and signed off. This audit trail allows for error detection and correction.

Survey and Data Management

An additional benefit of incident logging is that this allows handover of work from one enumerator to another. Once the respondent has indicated that they have completed the questionnaire to their satisfaction, including for example declaring that ‘such and such data is unavailable,’ decisions must be made regarding what to do about missing data items. At this stage the primary data collection is over, and the questionnaire may be checked for accuracy and completeness. All calculations and totals must be verified. One should check that the full-time equivalent is smaller than the headcount. Compare value of R&D with secondary sources of annual income of the organization being measured.

Three ways to handle missing data. 1. Find alternative public source. E.g. company annual report and

search that for the relevant item. Whatever the outcome of that search, this should be logged as an incident.

2. Use historic data for the entity from a previous survey.3. Imputation techniques. The various incidents for each respondent when combined across the Survey give rise to the metadata needed to interpret the full meaning of the Survey. Once the questionnaires are completed and signed off by respondents, the data must be verified and cleaned and once all discrepancies are resolved, the final questionnaire may be signed off by the enumerator and the data captured to the database. After this all questionnaires should be placed in a secure storage area.

Logistics (2)– Archiving– Financial close out.

Differences between R&D and Innovation Surveys

• Census & purposive – all formal sectors• Some computation

• Sharp focus on nature of R&D

• Well-understood indicators• GERD;GDP is a good target indicator

• Random – formal business only

• Tabular information/ check boxes

• R&D ‘unproblematic,’ emphasis on relationships & knowledge flows

• Innovation attributes are relatively blunt

• Allows econometric modeling: academics love to play with the microdata

• Innovation is ‘sexy’

Learning from the Project

• Were the interests of the project sponsor communicated?

• Were the stakeholders committed ?

• Were the criteria for success defined and communicated?

• How accurate was the plan?

• Were staff competences adequate?

• What was the general response to the project?

• Was adequate external support forthcoming?

• Do you expect institutionalization?

• Who smiled?