valuing organizational vision in the development of performance measurement frameworks of...

CES 2012 Conference HalifaxTuesday, May 15, 2012 | 15:45‐17:15

Valuing organizational vision in the developmentof performance measurement frameworks

of science‐based organizationsLessons learned from two projects

2

Outline

Context and objective

Background on organizational vision and performance

Overview of two performance framework development projects: Different organization types, visions and projects (tailored)

Effect of organizational vision on the two projects: Approach Methods Results

Project update on framework implementation

Lessons learned For organizations

For external evaluators/consultants

Questions?

3

Context and objective

Research and Technology Organizations (RTOs) increasingly make use of performance measurement (PM) to monitor the achievement of strategic objectives

Approaches to develop performance measurement frameworks are directly aligned with an organization’s:

strategic vision/values

competitive environment

reporting requirements

Using two recent projects, this paper demonstrates how organizational values influenced:

Approach: performance measurement strategies

Methods: how the performance framework was developed

Results: design and implementation of plan for the proposed framework

4

R&D performance measurement

Common goals: Monitor the scientific performance and effectiveness of the

organization on a regular basis Monitor the efficiency and economy of R&D project management and

other supporting processes Inform operational and strategic decisions Feed the production of relevant performance reports Provide reliable information to support effective evaluationCommon challenges: Culture of performance measurement and management Organizational structure and management Internal capacity for planning / performance measurement / evaluation Data availability and quality Buy‐in of data provider and user (championship/ communication)

5

Organizational vision and performance

Many studies have examined the link between the shared vision, values, strategy, goals and practices within organizations (often referred to collectively as ‘organizational culture’) and organizational performance.

Given the difficulty of measuring organizational culture, the research has been inconclusive about the exact nature of this relationship. (Lim, 1995)

However, many researchers have asserted that organizational culture affects all aspects of organizational interactions (Henri, 2006) and that a strong culture is predictive of organizational performance. (Abu‐Jarad, Yusof, & Nikbin, 2010)

Very few studies, however, have looked specifically at how organizational culture influences performance measurement strategies.

6


Evidence suggests that companies that exhibit more ‘flexible’ or ‘adaptive’ cultures tend to:

have better long‐term performance overall (Abu‐Jarad, Yusof, & Nikbin, 2010; Lim, 1995)

use a greater number of performance measures and focus their performance measurement strategy on supporting decision‐making, justifying activities and establishing priorities. (Henri, 2006)

R&D organizations, in particular, have unique needs and challenges with respect to the capture of impacts

e.g., contributions may be impossible to measure, even in the medium‐long term.

7


In general, studies suggest that organizations: clarify and translate their vision and values through their PM strategy; align their PM system with their specific R&D profile (basic vs. applied,

short vs. long‐term); use multiple performance measures, including both financial and non‐

financial indicators, to determine short‐term results as well as long‐term processes, impacts and cause‐and‐effect functions;

use their culture as a basis for allocating resources, establishing milestones, and facilitating strategic review and feedback mechanisms.

(Abu‐Jarad, Yusof, & Nikbin, 2010; Jyoti, Banwet, & Deshmukt, 2006; Shahzad, Luqman, Khan, & Shabbir, 2012)

Not surprisingly, we observed the influence of organizational culture as we conducted two projects using two completely different performance measurement approaches that reflected each organization’s vision and values.

8

Overview of two performance framework development projects

Provincial RTO Project Federal RTO Project

Vision: Be a world leader in innovation aligned

with strategic issues of its main client

Approach: Focused +

Participatory

Approach:Exploratory +Comparative

Vision: Be among the top five leading RTOs in the

world

9


Provincial RTO Federal RTOVision Be a world leader in innovation

aligned with strategic issues of its main client

Be among the top five leading RTOs in the world

Organization type

Provincial RTO that mainly serves the scientific and technological needs of importance for a provincial crown owned company(main client)

Federal RTO that serves thescientific and technological needs of importance for the country

Strategic plan

• 5‐year plan• 4 guiding principles• 5 priorities/11 action items

New plan was in development at that time

10


Provincial RTO Federal RTOVision Be a world leader in

innovation aligned with strategic issues of its client


Overall project goal and approach

Goal:• Develop an internal

performance measurement framework to monitor the strategic plan

Expected results: • Actionable plan • Plan developed in

consultation with key staff for buy‐in

Goal:Develop a comparative framework strategy for assessing effectiveness and benchmarking performance against comparable international RTOsExpected results: • A conceptual framework strategy • Creation of linkages with the

international RTO community (for benchmarking)

11

Effect of organizational vision on project approach

Provincial RTO Federal RTO

Vision

Be a world leader in innovation aligned with strategic issues of its main client


Approa

ch

Focused • Focused on the strategic level• Focused on the identification of

limited/available number of indicators

Participatory• Close collaboration with senior

management (RTO Director General)

• Participatory and interactive approach with internal staff

• PM capacity building and buy‐in

Exploratory• Identify best practices for both operational

and strategic levels• Explore multiple indicators and

measurement approaches Comparative• Consult with other RTOs to compare/

benchmark • Examine how other RTOs are organized and

deliver R&D (client‐focused and impact driven)

• Engage a community of practice

12

Effect of organizational vision on project method


Vision Be a world leader in innovation aligned

with strategic issues of its main clientBe among the top five leading RTOs in the world

Metho

ds

Secondary data collection• Focused on provincial RTO’s internal

literature and document/data• Identification of data

availability/feasibilityPrimary data• 4 full‐day roundtable workshops with

different levels of managers: Directors Managers Lead scientists Integration/wrap‐up

• Presentation for validation

Secondary data collection• Extensive scan and review of

document/data on the federal RTO and on the comparable RTOs

• Extensive literature review Primary data• Consultations with some senior

managers from the Federal RTO• Consultation with representatives

(involved in PM and corporate strategy) from 7 comparable RTOs

• Presentation for validation

13

Effect of organizational vision on the project results


Vision

Be a world leader in innovation aligned with strategic issues of its main client


Results

• Report on the proposed/selected indicators and rationale for deployment

• Included alternative indicators for consideration

• Total of 25 indicators (1‐3 per action item)

• 25 page report• All levels of Provincial RTO

management were aware of the project results

• Profiles/analysis of performance measurement approaches and indicators used by other RTOs

• Characterization of operational and strategic indicators

• A strategy to assess the Federal RTO performance

• Step‐by‐step approach for engaging a community of practice (for benchmarking)

• More than 100 indicators explored• 60 page report and 31 pages of technical

material on international RTOs• Federal RTO performance and evaluation staff

were aware of the project results

14

Current status of project implementation …

15

Lessons learned for organizations

Using a focused and participatory approach:

Provides near‐to‐complete product for implementation Limited number of indicators

Data is (almost) readily available

Wider organizational awareness and buy‐in

Provides a clear set of available indicators, but many are proxy indicators at the strategic level

Indicators inform on the progress towards strategic outcomes, but not necessarily on how to improve performance

Potential tension between staff during the definition/selection of operational indicators to measure strategic performance

16

Lessons learned for organizations

Using an exploratory and comparative approach: Necessity to define and conceptualize a full framework, including

Internal measures (operational/corporate‐level indicators) External measures (strategic/impact indicators)

Need to distinguish indicators for internal performance measures and indicators for international benchmarking (can’t always get both)

Need to narrow down the number of potential indicators, especially for international comparison

Access/confidentiality/availability issues of comparable data for benchmark International representatives did not have the same conception of, or

experience with, internal and external measures Implementation requires further challenging steps (final set of indicators,

internal buy‐in, further engagement with RTOs, data sharing agreement, etc.) Organizational culture can impede/influence the selection of common

indicators and how performance is judged

17

Lessons learned for consultants

Using a focused and participatory approach: Data load and complexity more manageable End‐results/tools developed are more transferable to other

projects/RTOs Easy to access data (proximity, language and availability) Greater participation/input from a wider variety of internal

stakeholders Had to help operational managers to think in terms of strategic

indicators Had to mediate differences in performance values from different units No means to compare or calibrate selected indicators with other RTOs

18

Lessons learned for consultants

Using an exploratory and comparative approach: Learned and benefited from the experience of several organizations Further reflection and conceptualization of performance measurement Created links with the international community Challenges to select comparable RTOS and to reach the right

informants The “full comparative matrix syndrome”

Expectation of data in all cells Multiplier effect (50 indicators x 6 dimensions x 8 RTOs = 2,400)

Challenges in the definition and interpretation of collected indicators Language barrier with key PM documents/data sources Much more work compared to the other approach (for similar budget) Had to manage expectations given the ambitious nature of the

international and comparative dimensions

19

Thank you for your time and feedback

CONTACT INFO

Andréa Ventimiglia, BSc MJResearch Analyst | Science‐Metrix

514‐495‐6505 x124andrea.ventimiglia@science‐metrix.com

Frédéric Bertrand, MSc CEVice‐President, evaluation | Science‐Metrix

514‐495‐6505 x117frederic.bertrand@science‐metrix.com

ACKNOWLEDGEMENTSEmmanuel Trépanier MA (contributor)

Eric Archambault PhD (contributor)Julie Caruso MLIS (contributor)

Science‐Metrix 1335, Mont‐Royal E.Montreal, Quebec H2J 1Y6Telephone: 514‐495‐6505Fax: 514‐495‐6523E‐mail: info@science‐metrix.com

WEB SITEwww.science‐metrix.com

Questions?

20

References

Abu‐Jarad, I. Y., Yusof, N., & Nikbin, D. (2010, December). A review paper on organizational culture and organizational performance. International Journal of Business and Social Science, 1(3), 26‐46. Retrieved from http://www.ijbssnet.com/journals/Vol._1_No._3_December_2010/4.pdf

Henri, J. F. (2006). Organizational culture and performance measurement systems. Accounting, Organizations and Society, 31, 77–103. Retrieved from http://arafiki.edublogs.org/files/2011/05/Organizational‐culture‐and‐performance‐measurement‐systems‐1zth1xx.pdf

Jyoti, T., Banwet, D. K., & Deshmukt, S. G. (2006). Balanced scorecard for performance evaluation of R&D organizations: A conceptual model. Journal of Scientific and Industrial Research, 65, 879‐886. Retrieved from http://nopr.niscair.res.in/bitstream/123456789/4952/1/JSIR%2065(11)%20879‐886.pdf

Lim, B. (1995). Examining the organizational culture and organizational performance link. Leadership & Organization Development Journal, 16(5), 16‐21. Retrieved from http://www.emarketing.net.cn/upload/file/2008/06/25/211214400841549.pdf

Shahzad, F. Luqman, R. A., Khan, A. R., & Shabbir, L. (2012, January). Impact of organizational culture on organizational performance: An overview. Interdisciplinary Journal of Contemporary Research in Business, 3(9), 975‐985. Retrieved from http://www.journal‐archieves14.webs.com/975‐985.pdf

21

Theoretical model

J.‐F. Henri / Accounting, Organizations and Society 31 (2006) 77–103

Relationship between culture, use of PM systems and diversity of measurement

valuing organizational vision in the development of performance measurement frameworks of...

Documents