empirical studies of design ideation: alignment of design experiments with lab experiments

1

Empirical Studies of Design Ideation: Empirical Studies of Design Ideation: Alignment of Design Experiments with Lab ExperimentsAlignment of Design Experiments with Lab Experiments

JamiJ. ShahNoe Vargas-Hernandez

Mechanical and Aerospace EngineeringArizona State University, Tempe, AZ

ASME 2003 International Conference on Design Theory and MethodologySeptember 5, 2003, Chicago Il

NSF Grant Number: DMI-0115447

Steve M. Smith David R. Gerkens

Department of PsychologyTexas A&M University, College Station, TX

Muqi WulanDepartment of Industrial and Manufacturing Systems Engineering

Beijing University of Aeronautics and Astronautics, Beijing, China

2

INTRODUCTIONINTRODUCTION

There are many methods for design synthesisHow useful are these? Which ones are better? There isn’t much empirical data on specific effectiveness of Idea Generation (IG) methods

MOTIVATION

For many years, Psychologists and Designologists have studied IG. Their experiments have different “ecological” validity (i.e. realism captured)

•Lab Experiments done by Psychologists

Low level – use few and simpler variables

•Design Experiments done by Engineers

High level – use more and complex variables

PSYCHOLOGY VS ENGINEERING

3


If our experiments at different levels have the same results, we can establish a connection and we can run more of the simpler Lab Experiments

More Experiments means more empirical data on specific IG methods

Our alignment approach is still WIP and is part of a bigger project

Our ultimate objective is to develop a theoretical model of design ideation

Such a model would help us better understand IG methods

SCOPE

At this time we do not consider human variables as experiment variables (e.g. experience, creativity of designers) Our focus is on Intuitive IG methods

OBJECTIVE

4


Figure 1. Classification of Idea Generation Methods (Shah et al., 2000)

INTUITIVE

LOGICAL

FORMAL IG

METHODS

DESIGN IDEA GENERATION METHODS

INVERSION

FORWARD STEPS

MORPHOLOGICAL ANALYSIS BRAINSTORMING K-J METHOD

CHECKLISTS RANDOM STIMULI P-M-I METHOD

METHOD 6-3-5 C-SKETCH (Collaborative Sketching) GALLERY METHOD

AFFINITY METHOD STORYBOARDING FISHBONE

SYNECTICS

GERMINAL

TRANSFORMATIONAL

PROGRESSIVE

ORGANIZATIONAL

ANALYTICAL

HYBRID

DESIGN CATALOGS OF PHYSICAL EFFECTS OF SOLUTIONS HISTORY

BASED TRIZ

SIT

5

PRELIMINARY RESEARCHPRELIMINARY RESEARCH

For several years we have been doing experimental studies to define the effectiveness of ideation

PAST EXPERIMENTS

We used two distinct approaches to conducting experiments:

•Direct Method – IG methods are studied as a whole

•Indirect Method – Ideation Components are studied

And two distinct approaches to assessing the effectiveness:

•Process – Assess the ideation process (e.g. protocol studies)

•Outcome – Assess the ideas produced (e.g. sketches)

6


IDEATION COMPONENTSIdeation Components are mechanisms that are believed to intrinsically promote IG or to help designers overcome mental blocks.

Examples of Blocks:•Being Judgmental•Emphasis on Quality•Lack of Motivation•Having a tight grip on problem specs.•Rigid Problem Representation•Design Fixation•Imposing Fictitious Constraints

Examples of Ideation Components:•Provocative Stimuli•Deferred Judgment•Flexible Representation•Frame of Reference Shifting•Incubation•Example Exposure

These are “common”: Known in Engineering Design Research and acknowledged by Cognitive Psychology

7


Table 1. Effectiveness Measures for Idea Generation Outcome.

MEASURE DESCRIPTIONQuantity How many ideas were generated n

Quality How close it comes to meeting the design specifications

Novelty How unusual and unexpected an idea is as compared to other ideas

Variety How well the design space was explored

OUTCOME EFFECTIVENESSFour measures were defined in our previous projects (Shah, Kulkarni, and Vargas-Hernandez, 2000).

k

n

kjk

m

jj pSfM

1

11

1

10x1jk

jkjkjk T

CTS

nbSfM kk

k

m

jj /

4

113

m

jjk

kjk

m

jj fnpSfM

1

2

112 *

8

RESEARCH APPROACHRESEARCH APPROACH

FUNDAMENTAL ISSUES

LAB EXPERIMENTS (Done A&M Psychologists)

•Focus on “atomic” cognitive processes•Little similarity between the condition of these experiments and design concept generation in the real world.

DESIGN EXPERIMENTS(Done by ASU Engineers)

•Simulate real world better•Incorporate more and complex variables•Require prohibitive number of experiments•Unable to explain the performance of methods under different conditions.

How can we compare results from experiments at different levels?

RESEARCH APPROACHAlignment: Agree on the Ideation Components to study and the Effectiveness Metrics for assessment

How can we align these two?

9


RESEARCH APPROACH

How can we compare results from experiments at different levels?

Alignment:Ideation Components Effectiveness Metrics Lab Experiments

Done byPsychologists

Design Experiments Done

By Engineers

Comparison

Results

Results

More Lab Exp.

Better Understanding of IG

Theoretical Model of

Design Ideation

Figure 2. Research Strategy

10


The number of components identified is more than a dozen. Because of limited resources and the prohibitive number of experiments required to study all possible interactions, only the most relevant were selected.

IDEATION COMPONENTS

COMPONENT DESCRIPTIONProvocative Stimuli Excite ideas by exposing the subject to a concept ideaSuspend Judgment Postpone reaching decisions or making conclusions of an ideaFlexible Representation Unconstraint the manner in which ideas are representedFrame of Reference Shifting

Change in the basic set of ideas on which other ideas are interpreted

Incubation Period of time that elapses generation of ideas for a problemExample Exposure Excite ideas by exposing the subject to a model idea

Table 2. Selected Components

11


Two levels were considered for each of the selected components. Although more levels could be defined, it is recommended to run experiments initially with few levels.

DESIGN OF EXPERIMENTS (DOE)

FACTORSRUN A B C

1 0 0 02 1 0 03 0 1 04 1 1 05 0 0 16 1 0 17 0 1 18 1 1 1

Full Factorial Experiments

A Frame of Reference Shifting

B Incubation

C Example Exposure

RUN LEVEL1 02 1

Table 3. Experiments in a 23 Full-Factorial Design

Table 4. Simple Comparative Experiments

All 6 Ideation Components were tested at ASU and TAMU simultaneously

Simple Comparative Experiments

•Provocative Stimuli

•Suspend Judgment

•Flexible Representation

12

Experiment VariablesSubjectsUndergraduate Engineering students. Comparable expertise/knowledge between subjects is assumed. Task One design problem was used for all experiments. The objective was to design a device for throwing a ping-pong ball the farthest distance. A list of allowed materials was given; this to improve the quality of sketches.Idea RecordingSubjects were asked to generate ideas individually using sketchesNuisance VariablesSimilar environmental settings procured for each run (classroom, noise, light, etc.)

DESIGN EXPERIMENTS DONE BY ASU ENGINEERS

RESULTSRESULTS

13

RESULTSRESULTSDESIGN EXPERIMENTS DONE BY ASU ENGINEERS

Figure 3. Sample Sketches from Design Experiments

Boat Wheels Attached

Catapult Cannon Hammer

Airplane

HIGH NOVELTY SET

LOW NOVELTY SET

14

RESULTSRESULTS

LAB EXPERIMENTS DONE BY TAMU PSYCHOLOGISTS

Experiment VariablesSubjectsUndergraduate Psychology students. Comparable expertise/knowledge between subjects is assumed. Task •Listing members of large taxonomic categories, sense impression categories, and ad-hoc categories.•Divergent thinking, unusual uses of common objects.Idea RecordingSubjects were asked to generate ideas individually using text for member listing tasks and sketches for divergent thinking tasksNuisance VariablesSimilar environmental settings procured for each run (classroom, noise, light, etc.)

15

RESULTSRESULTS

Figure 4. Sample Sketches from Lab Experiments

LAB EXPERIMENTS DONE BY TAMU PSYCHOLOGISTS

16

Table 5. Lab Experiments Done by TAMU Psychologists: Mean Ideation Effectiveness Scores

RESULTSRESULTS

CONDITION QUANTITY VARIETY QUALITY NOVELTYControl Group 1.51 1.20 1.47 0.00282Incubation Group 1.93 1.60 2.37 0.00373

Table 6. Design Experiments Done by ASU Engineers:Mean Ideation Effectiveness Scores

CONDITION QUANTITY VARIETY QUALITY NOVELTY

Control Group 4.86 2.81 6.15 4.71Incubation Group 5.11 6.24 7.31 6.76

IDEATION COMPONENT: INCUBATION

HIGH

HIGH

HIGH

HIGH

HIGH

HIGH

HIGH

HIGH

17

RESULTSRESULTSIDEATION COMPONENT: INCUBATION

Table 7. Two Sample t-test

CONDITION METRIC t0 P-value

LAB EXPERIMENTS BY TAMUPSYCHOLOGISTS

QUANTITY 1.31 0.19

QUALITY 1.85 0.07

NOVELTY 3.16 0.002

DESIGN EXPERIMENTS BY ASU ENGINEERS

QUANTITY 1.10 0.14QUALITY 5.86 0NOVELTY 12.38 0

18


RESULTSRESULTS

Table 8. Correlation Between Lab and Design Experiments

METRIC CONDITIONLEVEL

LAB DESIGNQUANTITY Control 1.51 4.86

Incubation 19.3 5.11Correlation 1.00

VARIETY Control 1.20 2.81Incubation 1.60 6.24

Correlation 1.00QUALITY Control 1.47 6.15

Incubation 2.37 7.31Correlation 1.00

NOVELTY Control 0.00282 4.71Incubation 0.00373 6.76

Correlation 1.00

19

CONCLUSIONSCONCLUSIONS

•Based on the results from (TAMU Psychologists) Lab and (ASU Engineers) Design Experiments, Incubation increases the effectiveness of ideas generated. Results correlate at both levels and show a satisfactory confidence level.

•Incubation’s positive impact on Design Ideation is substantiated by concrete Engineering evidence (from ASU Design Experiment results) and has a theoretical basis (from TAMU Lab Experiment results).


20

CONCLUSIONSCONCLUSIONS

Connection Proven

Experiments on other components have been completed at ASU and TAMU.

•More experiments needed to prove connection•The alignment procedure provides a framework for comparison between both levels. Results for Incubation exemplify how the alignment works.

OVERALL

Run more of the simpler Lab Experiments and less of the more complex Design Experiments

More empirical data

Better understanding of Ideation

Theoretical Model of Design Ideation.

21

FURTHER CONCLUSIONSFURTHER CONCLUSIONS

•According to our results we found Frame of Reference Shifting (FORS), Incubation (I), and Example Exposure (EE) to have similar main effects.

•Interaction effects weren't that clear, probably because some components are much alike (specially FORS and EE) and hence aren't independent.

•This generates a question: Maybe these Ideation Components have the same effect on ideation ?

•Two or more Ideation Components sharing the same effect could belong to the same higher level Ideation Principle

22

FUTURE WORKFUTURE WORKIDEATION PRINCIPLES

Table 9. Comparison of Ideation Principles with Cognitive ComponentsPRINCIPLE COMPONENTABSTRACTION Abstractions

IncubationAnalogies-Metaphors

JUDGMENT Suspend JudgmentBreak RulesImpose Constraints

RANDOM Frame of Reference ShiftingPERSISTANCE Provocative Stimuli

Example ExposureRandom Connections

COMBINATIONS CombinationsFLEXIBLE REPRESENTATION

Flexible Representation

23

FUTURE WORKFUTURE WORKIDEATION PRINCIPLES

•Refine Ideation Principles and its Implementations (Ideation Components)

•Run more exercises, collect and analyze more data to prove/disprove our theory about Ideation Principles

•We still continue experimenting on Ideation Components, but with a better understanding of Ideation Principles, experiments can be better targeted (e.g. less redundant) and more efficient.

24

REFERENCESREFERENCES1. Altshuller, G., 1984, Creativity as an Exact Science, Gordon and Breach, New York. 2. Dennehy, E.B., Bulow, P., Wong, F., Smith, S.M., and Aronoff, J.B. (April, 1992). A test of cognitive fixation

in brainstorming groups. Paper presented at the meeting of the Eastern Psych. Association, Boston, MA.

3. Dodds, R.A., and Smith, S.M., 1999, Fixation. In M.A. Runco & Pritzker (Eds.) Encyclopedia of Creativity, San Diego, CA: Academic Press.

4. Ericsson, K., and Simon, H., 1984- “Protocol Analysis - verbal reports as data”, MIT Press.5. Finke, R.A., Ward, T.B., and Smith, S.M., 1992, Creative Cognition: Theory, Research, and Applications,

Cambridge, MA: MIT Press. 6. Hale, C., “Analysis of the Engineer Design Process in an Industrial Context”, Grant Hill Pubs, Cambridge,

1987.7. Jansson, D. G., and Smith, S. M., 1991, “Design Fixation,” Design Studies, Vol. 12, pp. 3-11.8. Koestler, A., 1964, “The art of Creation”, Hutchinson and Co., London.9. Langley, P., and Jones, R., 1988, “Computational model of scientific insight,” in R. J. Sternberg, ed., The

nature of creativity – contemporary psychological perspectives, Cambridge University Press, NY. 10. McKoy, F., 2000, “Experimental Evaluation of Engineering Design Representations for Idea Generation”,

MS Thesis, Arizona State University, Tempe, AZ.11. Mckoy, F., Vargas-Hernandez, N., Summers, J. D., Shah, J., 2001, “Experimental Evaluation of

Engineering Design Representation on Effectiveness of Idea Generation”, Proceedings, ASME Design Theory and Methodology Conference, Pittsburgh, PA.

12. Koestler, A., 1964, “The art of Creation”, Hutchinson and Co., London.13. Martindale, C., 1995, “Creativity and Connectionism,” in S. M. Smith et al., eds., The creative cognition

approach, MIT Press, Cambridge, MA. 14. Schön, D., 1991,“Teaching and learning as a design transaction”, in Research in Design Thinking Delft

Press, 1991.

25

REFERENCESREFERENCES15. Schwartz, B.L. and Smith, S.M., 1997, The retrieval of related information influences tip-of-the-tongue

states. Journal of Memory & Language, 36, 68-86. 16. Shah, J., 1998, “Experimental Investigation of Collaborative Techniques for Progressive Idea

Generation,” Proceedings, ASME Design Theory and Methodology Conference, Atlanta, GA.17. Shah, J., Kulkarni, S., Vargas-Hernandez, N., 2000, “Guidelines for Experimental Evaluation of Idea

Generation Methods in Conceptual Design”, Journal of Mechanical Design, vol. 122, no. 4, pp. 377-384. 18. Shah, J. J., Vargas-Hernandez N., Summers, J. D., Kulkarni, S., 2001, “Evaluation of Collaborative

Sketching as an Idea Generation Technique for Engineering Design”, Journal of Creative Behavior, 35:3, pp.1-31.

19. Smith, D. K., Paradice, D. B., and Smith, S. M. (2000). Prepare your mind for creativity. Communications of the Association for Computing Machinery , 43, 110-116.

20. Smith, S. M., 1995, “Creative Cognition: Demystifying Creativity,” in C. N. Hedley et al., eds., Thinking and literacy – the mind at work, Lawrence Erlbaum Associates, Hillsdale, NJ.

21. Smith, S. M. and Blankenship, S. E., 1991, “Incubation and the persistence of fixation in problem solving”, American Journal of Psychology, 104, 61-87.

22. Smith, S. M., Carr, J. A., and Tindell, D. R., 1993, April, Fixation and incubation in word fragment completion. Paper presented at the meeting of the Midwestern Psychological Association, Chicago, IL.

23. Smith, S. M., and Vela, E., 1991. Incubated reminiscence effects. Memory & Cognition, 19 (2), 168-176. 24. Smith, S. M., Sifonis, C. M., and Tindell, D. R., 1998, Hints do not evoke solutions via passive spreading

activation. Proceedings of the Twentieth Annual Meeting of the Cognitive Science Society, Madison, WI. 25. Vargas-Hernandez, N. and Shah, J. J., 2002, Inventory of Creativity Exercises: 1995-2002, Tech. Report

ASU/DAL/IG/02-1, Arizona State University. 26. Wallas, G., 1926, “The Art of Thought”, Harcourt, New York.

empirical studies of design ideation: alignment of design experiments with lab experiments

Documents