decision support: from ir, ie, and planning to...

1

Decision Support: From IR, IE, and Planning to Comprehensive Information Management

Detailed depiction of the evolution of a traditional Institutional Research office to that of a Decision Support office that accomplishes the following: • Economies and efficiencies – prevents duplication of

time and effort

• Connectivity – addresses how decision support professionals effectively interface with institutional decision makers

• Information and message synchronization – produces usable products to best communicate data and messages, and

• Integration – demonstrates best practices of data/information, planning, budgeting, and assessment working together

For presentation at the 2010 AIR Annual Forum, Chicago, IL Developed by the University of South Florida, Office of Decision Support:

Dr. Michael Moore, Associate Vice President Debbie Hayward, Director of Decision Support Systems Dr. Valeria Garcia, Assistant Director of Planning & Analysis Travis Thompson, Technology & Systems Analyst Jacqui Cash, Communications & Marketing Officer

2

Table of Contents UOverviewU....................................................................................................................................................... 3

UInformation ManagementU .............................................................................................................................. 4

UInformation Management ModelU ............................................................................................................... 5

UPhilosophyU ............................................................................................................................................. 5

UInformation StrategyU ............................................................................................................................. 6

UDeliveryU ................................................................................................................................................. 6

UProductsU ................................................................................................................................................ 7

UBudgetingU ..................................................................................................................................................... 11

UInstitutional Effectiveness and AssessmentU ................................................................................................. 12

UInstitutional EffectivenessU ........................................................................................................................ 12

UOutcomes AssessmentU ........................................................................................................................... 12

URegional AccreditationU ............................................................................................................................ 13

UPlanning and AnalysisU .................................................................................................................................. 14

UMessage SynchronizationU ............................................................................................................................ 17

3

Decision Support: From IR, IE, and Planning to Comprehensive Information Management

0BUOverview The evolution of institutional research (IR) from a behind-the-scenes source of institutional data into a comprehensive customer service model is the basis of the next generation of the profession. The demand for planning, focused budgeting, and accountability in higher education has increased in recent years, resulting in notable changes in the traditional functions of an IR office. In charting the course for the future of IR in higher education, the decision support structure is a model for the next 50 years. IR is a form of organizational intelligence that requires a level of institutional understanding and implementation of best practices. The IR office of the future is an active and centralized team supporting the decision making process of an organization as external and internal stakeholders place greater demand for the services and expertise of IR offices. A decision support office is characterized by its responsive and proactive nature, led by a team of knowledge workers that posses an enthusiasm for delivering just-in-time service. These attributes support the philosophy of an ODS office going beyond the traditional reporting and delivery structure by also including information management and data communications in support of the institution’s leadership and alignment with its strategic plan. A strategic plan is a long-term blueprint for action that publicly articulates the university’s core planning components: vision, mission, values and goals. When senior administrators are data driven with a laser focus on the institution’s strategic direction, a symbiotic relationship naturally develops between these decision makers and an office of decision support. To foster this relationship, a modern day decision support office must combine the elements of information management, budgeting, assessment and institutional effectiveness, planning and analysis, communication and cross functional tools to facilitate the decision making cycle. While clear division of labor and specialization is necessary, a strict chain of command or hierarchy is not necessary and can actually impede progress and creative thinking. A flat organizational structure encourages cross training and collaborative initiatives. An office management style in the context of mutually understood goals and direction facilitates self management and personal ownership in product development and project management. In turn, this management style fosters a sense of reward, contribution, and accountability which develops a sense of connectedness between ODS staff and the clients they serve. Below is an organizational chart for an office of decision support based on the current structure of the Office of Decision Support at the University of South Florida (USF), a major metropolitan research very-high university. Certain offices or reporting structures may or may not be applicable to the needs of different types of institutions.

4

Office of Decision Support Structure

Planning & Analysis • Analysis • Survey • Web Applications • Communications

4 professional staff

members

Institutional Effectiveness

• Assessment • Regional accreditation


members

Decision Support Systems

• Information

management (e.g., BI, DWH)

• Faculty reporting • State and federal

reporting


members

Central Space • Classroom scheduling • Utilization


members

Academic Budgets


members

This document is an exploration of the next generation of IR as an active and integral member of the decision making process in an organization. It explores the composition and products of an evolutionary decision support office that answers both external demands for data and supports the internal decision making processes. Our intent is to provide an example of a functioning prototype that transforms a traditional IR office model into a more effective decision support office that accomplishes the following:

• Addresses how decision support professionals effectively interface with institutional decision makers,

• Produces usable products to best communicate data and messages, and • Demonstrates best practices of data/information, planning, budgeting, and assessment

working together. By highlighting the functions and characteristics of the various teams within a decision support office, this functioning prototype can be modified to support the strategic direction of different types of organizations as well as various leadership styles. In this example, the services outlined include information management, budgeting, institutional effectiveness and assessment, planning and analysis, message synchronization, and central space.

1BUInformation Management Meeting data needs requires a central distributed information system at all levels: recognized as the authority source of institutional information, widely accessible, and coordinated from operational to executive levels to meet appropriate needs. Critical to this mission is data and metadata that is relevant, timely, high quality, actionable, and possesses cross-system consistency. Beyond extracting the data, an effective and modern information management team operates from a central repository (e.g. data warehouse) of integrated, consistent, and actionable information that is common across data structures; securely delivered through a uniform, centrally supported platform; and accessible with easy-to-use tool sets. By definition, a business intelligence (BI) model provides the structure to support such a unit, including enterprise data architecture and transformation processes.

5

6BInformation Management Model

A critical assessment of the institution’s data and how the data are to be used is the first step to building a functional model for information management. Understanding that information must have value to the user, be accessible, and is in a format that is easily understood is the cornerstone of information management. Certain characteristics impart quality of information, including: relevance, applicability to the strategic decision making process, ease of sharing across organizational units, timeliness, quality, and actionable. The way information is delivered also has a significant impact on the way it is used. An accessible, self-service delivery system ensures users can quickly and easily understand, identify, and apply the appropriate data for their needs.

10BPhilosophy A common barrier in higher education is a disregard, either intentional or unintentional, for constituents’ needs. User needs vary depending on the area of the institution they serve as well as the users’ target audiences. Most “off-the-shelf” products are typically not designed to serve the broad and specific needs of higher education. A potential solution is to design a custom “home-grown” system that supports the various needs and users’ abilities to access data, which positions decision support professionals as core information providers. As such, a mutually shared view of the institution’s core functions, as portrayed by the information, achieves the goal of developing heuristics for decision making and establishes a common vocabulary.

Information Management Model

6

11BInformation Strategy Building an integrated information management model begins with a comprehensive examination of how data can best support the needs of the institution to achieve its operational and strategic goals. A proactive approach to the support of decision making is essential for an effective model to be sustained. This approach is novel in that it contrasts the current practice of leaders reacting to data when solving problems rather than using data proactively to intercept potential problems before they develop. An integral part of developing an information strategy is establishing data administration within the institution. Who at the institution should take responsibility for data quality? A shared model approach might be appropriate. For example, data stewards would first establish business rules and element definitions. IT would then institutionalize those rules and definitions by developing the metadata repository (including descriptions and documentation). To ensure consistency of usage, some responsibility also falls on those analyzing, understanding, applying, and delivering the data. This is the point when the data become useful information for forward-thinking decision support, assessment, and post-performance accountability. Following these practices positions data quality management as a cornerstone of data administration. For data to be truly useful they must be made available and easily accessible to the user. Optimally, a central office provides data generated from a single source and ensures consistent, single interpretation of those data. This central office provides information that is accurate, consistent, and meets the documented business rules and definitions. Likewise a central office can provide a control and a guide to facilitate users’ questions and their alignment with the strategic plan of the institution. Whether or not a central office model is in place, an institution benefits from the implementation of an information strategy.

12BDelivery The implementation of an information management model should involve both the people who use the data for decision making as well as the teams who build and support information technology (IT). Collaboration with IT in the development and ongoing improvements of information management systems ensures projects are managed with a smooth cascade rather than rushing waterfall approach. For example, an IT department will undertake a major project, build the system and release it for immediate use by all – this is the waterfall. In contrast, a cascade approach is typically seen in offices such as decision support that are faced with short turnaround projects with a particular focus and deadline. A prototype is released to gather user feedback to make updates and modifications. This approach is responsive, encourages creativity, embraces an entrepreneurial spirit, and requires a team to tolerate risk taking and receive constructive criticism. Various delivery tools can be helpful in supporting the information management model. It is imperative that tools necessary for individuals to generate answers to their own questions are sanctioned and supported by the institution. Toolsets need to be leveraged across the institution. Enterprise reporting tools allow users to identify frequently used reports or customized reports (smart reporting) for specific needs. The development of an executive information system (EIS) essentially allows for intuitive drill downs to actionable data. Identified Key Performance Indicators (KPIs), which can be displayed on a balanced scorecard, are then emphasized. Regardless of how

7

comprehensive your delivery tools are there will always be a need for ad hoc reporting. Strategies to meet the demands of ad hoc requests vary based on the inherent culture of the institution and the relevance of such requests to the strategic goals. Frequently asked questions often become standard reports. Theoretically, better standard reporting mechanisms reduce ad hoc requests. OLAP (online analytical processing) data structures such as cubes coupled with front ends which users can manipulate are examples of ad hoc query solutions. There is utility in the exercise of developing an information matrix, a needs assessment that identifies who uses the information and how they use it. An information matrix allows an ODS office to contextualize data, making it appropriate for the various users by translating it into a language they can understand which gives value to the information.

Information Matrix

TRANSACTION MANAGERS EXECUTIVES PROGRAM

RESEARCH ORGANIZATION/

PUBLICS TASKS • Daily operational

work • Error check and

resolution

• Operational management

• Oversight • Monitor-exceptions,

alerts

• Tactical • strategic

• Detail to generalization

• Model • analytical

• General info • Relation building • Shared view

PRESENTATION FORM

• Trans screens • Trans reports • Direct access to

data store

• Trans access • Summary reports • Smart reports • Ad hoc reports

• Graphic/Table w/ drill down

• Internet • Documents

TIME • Current • Daily, weekly • Summary • Trend • model

• Term • trends

• Term • Trends • General

INFORMATION FORM

• Element • Case • Transaction • SQL

• Trans reports • Special reports • Alerts • dashboard

• Profiles w/ drill down

• KPI • dashboard

• Elements • DB access • mining

• Aggregate • Summary

13BProducts The way information is delivered also has a significant impact on the way it is used. An accessible, self-service delivery system ensures users can quickly and easily understand, identify, and apply the appropriate data for their needs. Providing users with too much data or open access to raw data can lead to unintentional misinterpretation of the data. Common mistakes include failure to examine the details of the data, improper use of metrics for data needs, and inconsistent time slices. Effective decision support professionals partner with and understand the needs of those who use the data for institution-wide decision making and provide the tools to make those decisions. As a customer service unit, decision support can provide data analysis and a broad array of management tools to make the information more approachable and usable for consumers. Tools include data reporting tools; compact planning systems; integrated planning, performance, and accountability systems; and dynamic executive information systems (EIS). The look and feel of the tools makes an impact on how people engage with the services. A clean, approachable design and presentation securely delivered through a uniform, centrally supported

8

platform with easy to use tool sets not only ensures users have access to data but also nurtures the relationship between decision support and the users. A key example of a data reporting tool which embodies the aforementioned characteristics is the USF InfoCenter, a web-based application that delivers various views of data in a just-in-time format from a single source of institutional truth. Screen shots on the following two pages illustrate the various components of the USF InfoCenter.

InfoCenter: EIS (Executive Information System)

This section of the USF InfoCenter is interchangeably referred to as “Executive Summary” or “EIS-Executive Information System” and houses a series of user-driven reports. Reports are designed by the users’ selection of data by domain (i.e., student, faculty, finance, cross-functional), campus, college, and department.

Section being used is also indicated in the upper right hand corner.

Reports are driven by selection of data by these categories.

Shaded area lets the user know what section they are using.

9

InfoCenter: InfoMart

The InfoMart section of the InfoCenter provides users with reports on several data elements pertinent to students at the university. Such reports include: Student Headcount, Student Course Load, Student Credit Hours, Student FTE, and Degrees Awarded. Drop down options are available to the user to specify the parameters being sought.

Additionally, with adequate human, technological, and fiscal resources, an office of decision support can develop customized tools to address data elements central to the strategic plan (student, faculty, and budget). Both cubes and data marts are viable options for offices facing a high demand of ad hoc requests not supported by existing reporting applications. These tools also provide users with relevant data through intuitive, user-friendly structures. The following page features a screen shot of a cube template accessed by the user in Microsoft Excel.

Metadata are consistent throughout most sections of the InfoCenter.

Parameters selected by user are visible.

Description of the report is provided for the user.

10

TITLE H

Another example of a “home-grown” developed reporting application is the USF Faculty Academic Information Reporting (FAIR). This tool is a faculty designed and developed application for collection of academic information by or about faculty and instructional staff activity (e.g., publications, presentations, awards, instructional activity). Below is an example of the vita module within FAIR.

Cube – Ad Hoc

FAIR – Faculty Academic Information Reporting

11

2BUBudgeting Decision support is more effective if permitted to take a central role in the planning and budget process. By facilitating budget effectiveness in the decision-making process, the office helps ensure resources from all funding sources are used efficiently and consistent with the institution’s strategic goals. The planning and budget process has recently become aligned with the strategic planning process, but much work must continue to connect the cyclical nature to other aspects of the institution’s measurement and evaluation processes.

Planning and Budget Integration

Expenditure Reporting

12

3BUInstitutional Effectiveness and Assessment The future of decision support necessitates closing the gap that commonly exists in the planning, evaluation, and budgeting cycle of an institution by fully integrating outcomes assessment and other qualitative data into the process. A gap commonly exists at most institutions, contributing to the continual challenge of meeting regional accreditation requirements and executing successful institutional effectiveness and assessment. By incorporating institutional effectiveness and assessment professionals in a decision support office, a mutually beneficial partnership between policy and data functions can help the leadership best meet the strategic goals of the institution.

7BInstitutional Effectiveness The function of institutional effectiveness is to engage in critical self-analysis, by providing the institution with the support necessary to hold up a mirror to itself in the context of planning, evaluation, and budgeting. Institutional effectiveness encompasses all academic and administrative processes that ultimately impact student success, a cornerstone of the institution’s strategic plan. Examples of institutional effectiveness data include, but are not limited to, the following:

• Tracking flow-through, retention, graduation rates, and job placement; • Policy analysis; • Data in support of enrollment management decision making; • Other indirect measures — NSSE and alumni, employer, and organizational climate and

satisfaction surveys; • Faculty allocation and credentialing; • Course evaluation (student assessment of instruction); and • Faculty workload analysis.

8BOutcomes Assessment Outcomes assessment is conducted at both the administrative and academic levels with a commitment to transparency, accountability, comparability, and benchmarking. Focus is on formative program assessment rather than assessment of students or faculty members. This results in an emphasis on “closing the loop” by improving programs based on what was learned from the assessment. In short, everything flows back to the impact on learning as examined through measurable objectives. Assessment is no longer just an exercise in response to accrediting bodies. Today, many state statutes and governing bodies require program assessment such as Florida’s academic learning compacts and program reviews. Strong partnerships between institutional effectiveness and outcomes assessment can position an institution for potential changes to the accreditation process in the United States. For example, there are some indications that accreditation could move to a process based on sectorization rather than geographic divisions. An institution with a strong assessment program would be better positioned to respond to such a change.

13

9BRegional Accreditation The function of monitoring regional accreditation policies and procedures as they are executed across the institution is another core component of the institutional effectiveness and assessment team. A nimble institutional effectiveness team monitors the Higher Education Reauthorization Act to quickly assess the implication of changes at the institutional level. The team also maintains extensive databases necessary to respond to requirements of regional accrediting bodies in an ongoing manner rather than only times of reaccreditation.

Xitracs™ – “Off the Shelf” Reporting Tool

14

4BUPlanning and Analysis As the senior leadership of an institution engages in its visionary planning, a decision support office operating in a centralized capacity can provide information and leadership to support decision-making about the effectiveness of the institution in achieving its mission. Whereas a traditional IR office supports the strategic plan, a planning and analysis team within a modern decision support office leads the offices’ direct contribution to the strategic planning process. The team is often called upon to provide not only data but analyses critical for informed decision making. Performance reports related to accountability measures consistent with the institution’s strategic plan assist in decision making. As a part of the planning, evaluation, and budgeting cycle, numerous tools provide opportunities to identify areas for investment and/or reallocation, a basis for accountability in evaluating performance, and establishing outcome expectations for accountability. The Compact Plans application is one example of a home-grown tool which connects the planning, evaluation, and budgeting cycles with the strategic plan blueprint. Compacts are short term (18-24 months), focused planning agreements negotiated between accountable officers (such as Deans) and their supervisors (such as the Provost). They are intended to align broad University goals with the priorities, investments, and actions of campuses and colleges, as well as academic and service units. Generally a Compact Plan consists of three parts: 1) unit background information which is supplied by the department and contains a brief overview of the unit including history, major accomplishments, and peer identification; 2) resource data which is pre-supplied to the department by Decision Support Systems with dollars and values for budgets, staffing, enrollment, degrees awarded, and research contracts and grants; 3) initiative information which is supplied by the department and consists of details on the unit’s top three to five initiatives.

Compact Plan Unit Resource Data

Second page of initiative entry in a Compact Plan

15

Among other uses, background information supplied by the department provides a reasonable base for performance comparison among other institutions through identification of specific universities which the unit considers as its peers. The pre-supplied resource data is used by the department as well as its supervisor as a common understanding of the unit’s situation in terms of both resources and past performance. When entering initiatives in the system, a department indentifies three main types of information: 1) using specific values and metrics, how the initiative will contribute to and align with the strategic plan; 2) how the initiative will be assessed and explanation of the criteria for success; and 3) how existing funds will be reallocated and explanation for request of new funds. By identifying specific practical, short-term, and intermediate steps toward meeting the university’s long range goals, every department (both academic and non-academic) can both contribute to strategic plan goals in identified ways and also communicate intentions to their supervisors. In addition to connecting the planning, evaluation, and budgeting cycles through Compact Plans, the planning and analysis team engages in various key strategic activities in support of the decision-making process at the university. In an ideal setting, the strategic plan is a living document and therefore requires an ongoing review and analysis of performance measures. Through the development and maintenance of a planning and performance matrix outlining measurable objectives, the team can communicate to both internal and external audiences the institution’s progress toward meeting its strategic objectives. Ways in which performance tracking are communicated include electronic and print versions of a matrix, dashboard, and a planning, performance analysis application. While not as detailed as the aforementioned tools, the team is also central in the annual development of a pocket fact booklet, as well as several data intensive publications used widely by key stakeholders. Beyond internal tracking and monitoring of institutional data is the process of benchmarking institutional metrics against those selected peer groups. Benchmarking can provide useful data for purposes of administrative activities such as annual college performance reviews and reports to governing bodies. Benchmarking data encompasses metrics such as national rankings, faculty productivity, research portfolio measures, and student success indicators. Further, the planning and analysis team engages in administering, monitoring, and reporting data of surveys driven both internally (e.g., NSSE, senior exit surveys) and externally (e.g., USNWR, Princeton Review). Included in the planning and analysis function exists traditional IR activities such as research studies and focus groups.

Planning, Performance and Accountability Matrix

16

The following are planning and accountability tools developed by the planning and analysis team at the University of South Florida.

Performance Dashboard

Planning and Performance Matrix documents institutional progress toward achieving strategic goals

17

5BUMessage Synchronization To face the challenge of effectively communicating the intricacies of institutions of higher education, there is a growing need to synchronize an institution’s data and its story. In a society that places increased emphasis on quantitative measures of quality, it is imperative an institution’s data is presented in a manner that is understandable, thus maintaining data integrity and consistent messaging of the values and achievements of the institution. Through the formation of a unique team of data and communications experts, the university is able to more effectively and consistently communicate institutional data to internal and external stakeholders (including senior administration, academic units, media, local and state governing bodies, community organizations, and business and industry partners) in a manner that embodies the vision and mission of the university as set forth in the strategic plan. Evidence of successful data and communication synchronization includes cross-university initiatives and outcomes that are utilized not only during the decision-making process but also when telling the story of the university. Effectively communicating data is the process by which institutional accountability transforms ideas into measurable results. Through the creation of publications, strategic planning documents, presentations, and relationship building, the team is a central source for vibrant institutional support in a data-driven society. Examples of communication products include a pocket fact booklet, peer comparison publications, strategic plan matrix, and a variety of data-driven presentations prepared for high level stakeholders.

Web-based tool to collect and archive national faculty and student awards

18

Pocket

Points of Pride Publication

Strategic Plan Excerpt Publication

BIG EAST Peer Comparision Publication

Pocket Fact Booklet

19

UConclusion

A necessary condition to the decision support model is the support of senior leadership in fostering a culture that values objective decision making. The institution committed to goal directed decision making will benefit from this model whose actions support the transparency of processes, are informed by empirical data, and provide for accountability. Transforming IR into an office of decision support serves to provide integrated information to support decision making at all levels throughout the institution. The evolution of information management moves from decentralized data operations to centralized data management, and finally to distributed data management which makes shared and mutually understood information the mission critical element that the institution desires. Information as a product of a decision support office must have these attributes: integrated and shared, timely, high quality, actionable, and relevant.