Stopping Rule Use During Stopping Rule Use During Information Search in Information Search in

Design TasksDesign Tasks

Glenn J. Browne

Texas Tech University

Mitzi G. Pitts

University of Memphis

OutlineOutline

Introduction: Aims of the StudyBackgroundSetting and MethodResultsConclusion

Introduction: Aims of the Introduction: Aims of the StudyStudy

Improving requirements determination for systems development

Understanding how analysts decide when to stop gathering information

Understanding role of experience in that decision

Background:Background:The Decision-Making ProcessThe Decision-Making Process

Simon’s Model– Intelligence– Design– Choice

Information SearchInformation Search

Why we might expect information search in different stages of the decision-making process to differ.

“Stopping Rules”

Information AcquisitionInformation Acquisition

Problems in acquisition– Underacquiring– Overacquiring

HeuristicsHeuristics

Defined – rules of thumb for taking actions in various situations

Examples– “80-20 Rule”– “Feed a cold, starve a fever”

Heuristics for Assessing Heuristics for Assessing LikelihoodLikelihood

Normative – Relative FrequencyDescriptive

– E.g., availability, representativeness, anchoring and adjustment

Heuristics for ChoiceHeuristics for Choice

Normative– E.g., expected value of information, expected

value of additional information, expected loss from terminating information acquisition.

Descriptive– E.g., Dominance, Conjunctive, Disjunctive,

“The Minimalist,” “Take the Best.”

Heuristics for Intelligence Heuristics for Intelligence Gathering and DesignGathering and Design

?

Heuristics for Intelligence Gathering Heuristics for Intelligence Gathering and Design:and Design:Some IdeasSome Ideas

Nickles, Curley, and Benson (1995)– Difference Threshold– Magnitude Threshold– Mental List– Representational Stability

Difference Threshold Stopping Rule

Magnitude Threshold Stopping Rule

Mental List Stopping Rule

Representational Stability Stopping RuleRepresentational Stability Stopping Rule

Impact of Analyst ExperienceImpact of Analyst Experience

On information gatheredOn stopping rules used

The ContextThe Context

Requirements gathering for information systems development

Application for grocery shopping on world wide web

54 practicing systems analysts in the Baltimore-Washington metro area

Participants asked to gather requirements until they felt they had enough information to draw diagrams representing requirements and proceed with system design.

Measuring Information Measuring Information RequirementsRequirements

Requirements TaxonomyTotal requirements (Quantity)BreadthDepth

HypothesesHypotheses

H1a: The use of some stopping rules will result in different quantities of requirements than the use of others.

H1b: The use of some stopping rules will result in different breadth of requirements than the use of others.

H1c: The use of some stopping rules will result in different depth of requirements than the use of others.

Hypotheses (cont.)Hypotheses (cont.) H2a: A greater number of experienced analysts will

use the mental list rule than will use the representational stability rule.

H2b: A greater number of experienced analysts will use the mental list rule than will use the difference threshold rule.

H2c: A greater number of experienced analysts will use the magnitude threshold rule than will use the representational stability rule.

H2d: A greater number of experienced analysts will use the magnitude threshold rule than will use the difference threshold rule.

Hypotheses (cont.)Hypotheses (cont.)

H3a: There will be no relationship between the experience of the analyst and the quantity of requirements elicited.

H3b: There will be no relationship between the experience of the analyst and the quality of requirements elicited.

Data AnalysisData Analysis

Verbal protocols and questionnairesCodingInterrater reliabilityStopping rule identification

ResultsResults

Stopping Rule Use– Difference Threshold – 22– Representational Stability – 13– Mental List - 10– Magnitude Threshold – 9

Results (cont.)Results (cont.)

Requirements Elicited by Stopping Rule– Quantity – F(3,50) = 2.72; p = .05– Breadth - F(3,50) = 1.72; p = .17

– Depth - 2(3) = 8.98; p = .03

Results (cont.)Results (cont.)

Impact of Experience on Stopping Rule Use– Mental List = 14.30 years– Magnitude Threshold = 14.06 years– Difference Threshold = 11.11 years– Representational Stability = 7.65 years

Results (cont.)Results (cont.)

Impact of Experience on Stopping Rule Use– Mental List rule users were more experienced than

users of the Representational Stability rule (t(21) = 2.27; p = .019), supporting Hypothesis 2a.

– Users of the Magnitude Threshold rule were also more experienced than users of the Representational Stability rule (t(20) = 2.00; p = .03), supporting Hypothesis 2c.

– Other two hypotheses were not supported.

Results (cont.)Results (cont.)

Impact of Experience on Requirements Elicited– Analysts’ years of experience were unrelated to

the total number of requirements elicited (Pearson’s r2 = .08; p = .59), supporting H3a.

– Breadth of requirements (r2 = .15; p = .27) and depth of requirements (r2 = .02; p = .91) were also unrelated to analysts’ years of experience, supporting H3b.

ConclusionConclusion

Identification of stopping rules during information search

Impacts of analyst experienceImpact on information systems

development process