s519: evaluation of information systems analyzing data: value and importance ch6+7

S519: Evaluation of Information Systems

Analyzing data:

value and importance

Ch6+7

„Values“ in evaluation (DCh6)

Adding value to descriptive data to make our evaluation explicit

Our goal Using quantitative value to evaluate the quality or

value of the evaluand in a particular context. Build up our conclusions based on a level of

certainty What are values:

„good“, „valuable“, „worthwhile“

„Values“ in evaluation

Adding „values“ to descriptive data collected about Process, outcomes, costs, comparisons, exportabilities;

or Situated dimensions or components

Weighting all the strengthens or weaknesses of these values to draw overall conclusion about the evaluand. How Importance weighting Merit determination synthesis

methodology

Subjective

Before we go for methodology, we have to answer the question: Whether our data are subjective

Three types of subjectives (Scriven, 1991)

Subjective 1: Inappropriate application of personal or cultural preferences/biases: arbitrary, idiosyncratic, unreliable, highly personal (i.e., based

purely on personal preference, cultural biases, gender biases)

Subjective 2: assessment or interpretation by a person, rather than guidelines Using well-founded expert judgments Robust evidence

Subjective 3: about a person‘s inner life or experiences (e.g., headaches, fears, beliefs, emotions, stress) Usually not independently verifiable

Avoid Subjective

We provide our conclusion based on certainty in the relevant decision-making context

Keep the whole evaluation well documented and justified

All evaluations, especially high-stakes ones, should be meta-evaluated (i.e., evaluation itself should be evaluated)

Determining importance (D-ch7)

Importance determination is the process of assigning labels to dimensions or components to indicate their importance. Importance weighting

Prioritize improvements Identify whether identified strengths or weakness are serious

or minor Work out whether an evaluand with mixed results is doing

fairly well, quite poorly, or somewhere in between.

Determining importance (D-ch7)

Different evaluations Dimensional evaluation

Looking at multiple dimensions of merit that pertain to the evaluand as a whole rather than separately to its parts.

Component evaluation Looking at each of the evaluand‘s components (or parts) separately

and then synthesizing these findings to draw conclusion about the evaluand as a whole.

Each component can be evaluated on several dimensions that pertain to this component only rather than to the evaluand as a whole.

Holistic evaluation Looking evaluation as a whole without division into dimensions or

components

When to use what

Component analysis Evaluating policies, programs, or interventions

that have several quite distinct parts An international program consisting of projects

implemented in different locations (e.g. „WIC“ in IU) A government policy includes multiple policy

measurements (e.g. Juvenile delinquency) An organizational transformation includes several

distinct interventions (e.g. Career support)

When to use what

Dimensional evaluation Entities whose quality or value is experienced by

consumers on multiple dimensions that pertain to the evaluand as a whole Product evaluation (i.e. Car evaluation)

When to use what

Holistic evaluation Unusual in the evaluation of programs, policies

and other large complex evaluands. More common in personnel, product and service

evaluation (expertise-oriented evaluation) Judging the overall quality of a sample of writing Grading essays Classroom teaching Athletic performance cosmetics

Determining the importance of dimensions

Determining the importance of componentss

Determining importance

Weak performance on minor criteria (e.g. dimensions, components) may be no big deal,

But weak performance on important criteria can be very serious issues.

Determining importance: 6 strategies

1. having stakeholders or consumers „vote“ on importance Commonly used in both participatory and

nonparticipatory evaluations Collecting opinions from everybody Assumptions

Each person is well informed Stakeholder‘s belief what (s)he chooses is important Stakeholder‘s important should be treated equally

Pros and cons?


2. Drawing on the knowledge of selected stakeholders Using selected stakeholder input to guide the

assignment of importance weightings Collecting opinions from selected experts Setting up the Bars

A bar is a defined minimum level of criterion performance below which the evaluand is considered completely unacceptable, regardless of performance on other criteria.


2. Drawing on the knowledge of selected stakeholders Assumptions:

The stakeholders should be sufficiently well informed to provide valuable relevant information

The combination of stakeholder input will provide sufficient certainty about importance for the given decision-making context

Pros and cons?


3. Using evidence from the literature Literature review Evaluations of similar evaluations in similar contexts Research documenting the key drivers (or strongest predicators)

of success or failure with this type of evaluand. Assumptions

The volume and quality of the available research is sufficient to judge the importance

The context of other research is sufficiently similar to yours and therefore that the findings can be reasonably applied to your setting

Pros and cons?


4. Using specialist judgment When you have tight timeline, no time for

gathering stakeholders and looking for literature Identify one or two (or two or more) well-known

specialists in the domain Better be supplemented with other evidence

Pros and cons?


5. Using evidence from the needs and values assessments Determining the importance of criteria

(dimensions) Any frequently mentioned characteristics? Looking for poor-performing evaluators that cause

serious problem Looking for top-notch evaluators that have dramatic

impacts on success


5. Using evidence from the needs and values assessments Determining the importance of components

Severity of dysfunction addressed (primary consideration) Scarcity of alternatives: no other options for addressing the

need. Intent to use alternatives: if the evaluand component in question

did not exist. Rubrics to measure (Table7.4, 7.5 and 7.6, 7.7, 7.8 (combined))

Prons and cons?


6. Using program theory and evidence of causal linkages When criteria or components are linked to needs through a

complex logic chain. Such as „soft“ skills or attributes (e.g., inspirational leadership, self-

esteem, stress management, a kind of instrumental needs) More upstream variables (see Exhibit 7.5)

How to estimate the strengths of the links Interview Analyze your previous data ...

Strengths and weaknesses of the six strategies

Always think whether they are applicable Choose mulitple of them

Exercise

Table 7.10 (class dissusion) Form a group

Discuss which strategies you will choose to determine the importance for the “student services in the school health program” (see Table 7.8)

Discuss which strategies you will choose to determine the importance for your group project

s519: evaluation of information systems analyzing data: value and importance ch6+7

Documents