CPSC 699

Scientific Evidence

Message

Computer Science research requires scientific evidence and well design research plan even in the presence of uncertainty

Lecture Outline

History of algorithms Problem solving Ways to deal with uncertainty:

assumptions, paradoxesScientific evidenceExperimental design

Algorithm Definition

Algorithm is a procedure that consists of a finite set of instructions which, given an input from some set of possible inputs, enable to obtain an output

Donald Knuth: “Computer Science is the Study of Algorithms”

Some simple examplesSuppose you and I have the same amount of

money. How much should I give you so that you have $10 more than I do?

A dealer bought a book for $7, sold for $8, bought for $9, sold for $10. How much profit did he make?

Solution provides an algorithm.

AssumptionsEach problem can be solved under a set of specific

assumptionsExamples:

Weight is given in kilogramsActivity takes place on Earth (Newton’s laws apply)When pouring the water into container, it does not

evaporate

“Which one is older?”Logical problem (Raymond Smullyan) A brother and a sister were asked who is older. The

brothers said “I am older”. The sister said “I am younger”. At least one of them lied. Who is older? Under what assumption the solution works?

Solution

If we assume they cannot be of the same age, then they both lied and the sister is older.

A pizza puzzle

How do you cut a pizza onto 8 pieces with 3 cuts. What is the minimum number of slices you can get with 3 cuts? Which assumptions did you use?

SolutionA piece is not necessarily should have a

cheese on it (be the top one).You can cut horizontally.All cuts are distinct.

Logical problemsR. Smullyan “The riddle of Scheherazade”1. Think of a yes/no question that will forceyou to tell the truth.

2. Think of a yes/no question that will forceyou to tell the truth.

SolutionsSolution 1. Will you answer “Yes” to this question?Solution 2. Will you answer “No” to this question?The first solution is obtained as a result of cognitive

process.The second solution can be obtained by analogy.

Problem solving

Applying specific rulesBy analogy (similar problem)Abstraction (generalization)Intuition (non-ordinary solution)

The biggest assumption“Math is perfect??”In 1931 Kurt Godel came up with a startling discovery

that mathematical truth can not be completely formalized. I.e. given the set of axioms and rules of inference there are some statements that cannot be proven true or false inside this system.

This is called: Incompleteness theorem

ParadoxesThis leads to paradoxes:“This sentence is false”“You have no reason to believe this sentence”Do you believe it?

How do you know?Many ways we come

to believe thingsOften, we get to

wrong conclusionsLook at

Gilovich, How we know what isn’t so, 1991.

Huff, How to lie with statistics, rev 1993.

Campbell and Stanley, Experimental and quasi-experimental designs for research, 1963.

Polya, How to Solve it, 1945

Cognitive People often see patterns in random data

“images in the clouds”we then mistakenly assume the pattern is

significantwe are also good at explaining patterns we

believe are there.

The hot handMany basketball fans and players believe in

the hot hand shooting success comes in streaks

No statistical evidenceConsider the sequence

OXXXOXXXOXXOOOXOOXXOOrandom62% of subject believe shows streaks

Illusion of validity

Anecdotal evidence –seen in TV ads“I followed the new x

diet and lost 25 lbs.”need more

informationlook at all four

quadrants

Lost Weight

anecdotal

evidence in

advertising

no lostweight

no diet diet

CognitiveSeeing what we expect to see

biased evaluationlike evidence/research that supports what we

already believeExamples

scrutinizing contrary evidence more closelygamblers’ rationalization of lossespolarizes people of opposing viewpoints (eg.,

capital punishment)

Social Seeing what we want to see

motivational determinants of beliefThe Lake Wobegon effect

“… the women are strong, the men are goodlooking, and all the children are above average.” (Garrison Keillor)

Examplesthe large majority of the general publicbelieves that they are

more intelligentmore fair-mindedless prejudicedmore skilled at driving than the average person

Social Believing what we are told

the biasing effects of second hand informationWhere there is smoke there is fire?

if you hear it often enough, it must be trueAlterations in recounted stories

it was this big

How to use statisticsHuff, “How to Lie with Statistics”, Norton,

New York, 1954.Who says so?How does he know?What is missing?Did somebody change the subject?Does it make sense?

examples (1)U of C 101

Higher Grades have been achieved by students who attended U of C 101 according to the research. Overall GPA of frosh who did not attend U of C 101: 2.26 Overall GPA of frosh who did attend U of C 101: 2.68

Can we know anything?

Yes, withExperimentation plus statisticsCampbell and Stanley show how to design

experiments that have validity

Experiment designOne-shot case study

X One-group pretest-posttest design

O X OPre-test-post-test control group design

R O X OR O O

ValidityInternal

does an experiment give a valid answer to the problem posed? How many times and in which settings?

Externalcan the results of an experiment transfer to

other contexts?Uncertainty

do the results have anything to do with conditions that occur in the real world?

3 research questionsEarly

what will happen if …?

what happens when …?

Middledoes this predict

what will happen?End

what causes this to happen

i.e., a causal explanation

Experiments and research questionsexperiments that do not have an underlying research

question areface value experimentsanything learned about a research question is

incidentalOne should try to have experiment address

underlying research question and critically compare results: under various settings, with variable size data sets, with different algorithms

Case studies

Where ideas come fromInternal Talk to other grads in your lab Get demonstrations of what they are doing Experiment with any software that's built by

others in your group so you get your hands dirty in the area (e.g., in the lab or downloadable from the net) that let you get your hands dirty in the area

External Read lots - conferences, journals, etc. Mark the articles you like so you can go back and

review it. Or copy the abstract and put it in a binder for later review.

Attend a key conference in your area if you can

Good area choiceHow do you decide what is a good area? Talk to your supervisor first. Some have specialized

areas of interests and focused projects in mind that they will want you to work on. Others may be willing to consider projects outside their direct interest.

It should be personally exciting and interesting to you. You will be working on this for a long time.

It should be rich in scope It should be topical i.e., something of relevance It should be related to your supervisor's experiences

Once you found an areaResearch the area to gain exposure to it. In particular,

collect all interesting material you find in a binder. photocopy /save good articlesmaintain a list of full references of these articles as a

bibliography better yet, create an annotated bibliography where

you briefly note what is interesting about them keep a folder or sketchbook or notebook where you

can collect ideas or snippets related to this area maintain a list of links to interesting web sites

Message

Computer Science research requires scientific evidence and well design research plan even in the presence of uncertainty

SourcesHistory of Mathematicshttp://www-groups.dcs.st-andrews.ac.uk/~history/Chronology/index.htmlFun Math http://www.cut-the-knot.com/content.html

R. Smullyan “The riddle of Scheherazade”Gilovich, How we know what isn’t so,

1991.Huff, How to lie with statistics, rev

1993.