ExperimentaldesignWHAT is an experimentwhat is a Good experiment

i identifyinsight stestwithoutconfounds

i what is an experimenti 1 Origins medical drugeffectiveness

agriculture cpesticideeffectivenessFisher 1926

theu psychology cauderstacedpeoplefocus HRI Cfestdesign decisions

Roboticscomparealgorithmstest insights1.2 Components

treatments responsesCcoudetious measuresa drug vs placebo a symptomprogressionrandom vs optimal ComsatCHOMPUSGSCHOMP cost

eapen.meaewutassignmeusubjects Csubj allocationpatients random cdrug placebousers eeveryusersees bothKapproblems everyproblem seesboth

1.3 AssignmentBetween subjects one conditionperuserno biases

within subjects all conditionseliminate inter user variation

caseswhere within not possibleIgivepatients both drugsnot realistic robotcollaboration1coordinate

onceyouSinislethetaskyou know it

Mixed

1.4 Operationalizationof variablesIndependent variables coudetious

whatyoumanipulate e.g drugtype motiontypedependentvariables measures

whatyou measure eg symptoms cougat

1.5 HypothesisIv x affects Dv y

motivateby a mechanism n analogousstudies

Better Iv xpositivelyaffects Du yBecause optimalmotion is morepredictable wehypothesize that

HI Optimizingmotion increases user cougutBecausegoal sets addflexibility wehypothesioethat

Ht Includinggoalsets in optimizationreduces the final trajectory eat

ayHypotheses extractkey insights

fy oIm9aeg HIagaeggpoguetagges

think aboutthe IV tscience notjustengineering

2 What is agoodexperiment2.1 Goodexperiments are contended

ucontrolled experimenter assigns experimentalunits to treatmentsas opposed to observational

e.g HowdoesestrogentreatmentaffecthealthObsstudy

outcomes

93676 women8yearsIIIIIffaeesh

estrogenposthoueffeatedwhatis wrong

ToCONFOUND health conciousnesscontrolled study estrogen negatively affects Hit

lg 2 lefthandednessObsstudy2000 peoplewhodied Lil dieyounger

2 peple U de dLH died gyearsyounger

LH deeyoungerwhatIswrong2TfCONFOUNDDo LH condemned youngerpeople

equal amounts butolderpeople areright handed2 2 Good experiments avoidconfoundsucougound a variable whoseeffectcannotbe

distinguishedfromthe effectofthe IVe.g health consciousnessartificialdecreaseofLHgenderexperience with robotstrajectoryexecution timealgorithm metaparamelersRegoptimizes step seaToolsfor avoiding coyoceads

e Racedomidation2525OE E E

EM50Goa

Note hapha2and randomisedegalternate ABABAB

an A in morningall 13 in evening 3 deffpopulation

similar populationga eachcondition

within subjects counter balancetheorderD P Neconditions 7 N orderings

I D l f Oude myP D u eatin square

stackthe cards againstyourselfraud pathlengthCHOMP squaredvelocity

DMipathleeyth

optimizethebaselinealgstamp stepsizeRRT improve RRT as well in all otherways

but parentselectionrewiringtestthekeg insight

2 2 Good experiments are reliablereliability Cow experimental error ClawvarianceToolsfor improving reliabilitywithin subjects exactsame user noeghihentbias

onBlocking9 blockgroupof homogenousexperimentalunits blocking arrangementgauetsinto blocks reducesknower but irrelevantsources of variation between coudetious

EI EI b P

EET E E EEVarCX y Varcx t Vancy 2 couchyminimize maximize

blockingSautercovariates secondary variable thateau affecttheDM

Multi item scalespredictable expected surprising

2.3 Good experiments have constructvalidityconstructvalidity the measures actually

measure whatyou wantIQ test intelligenceratingof predictability predictability2Toolsobjective subjective measurespredesxfueftuds.mg EIBL7eDdeoooemcgf.gt

Construct validity

reliable reliableinvalid valid

2 4 Good experiments have externalvalidityUexternal validity conclusionsTools

S

Samplefromthe targetpopulationEET EApplyyourinsightacrossproblems problemtypes even algorithms

goalsetsNO CHOMP STOMP

setupThgopt

YES GSTrajopt GSSTOMP

RepresentationWAYPOINTS CHOMP WP

Rkets CHOMARKHS I I2.5 Goodexperiments augactorial Tffactorial i conditions alleaubimatiousgall

levels of the IVsGS

ruts common pitfall adeaugusatouce

CHorapusTrajoPT 2udader treatment obstacles1st vs and soft hard

softsDF CHOMP MdaderCHOMP pointSDFhardigeonchohuffn Trajopt geomgeomgeomchamp

cu ou

CHOMP WP us CHOMP RUHS us Trajopt RkitsWhatig Rkits helps Atomp butmottnajopt

Factorials break down the changes and extractwhat actuallymatters

Recapetreatmeutfre.peexp cents assignment

Htt manipulated variablesdependentmeasuressubj allocation

controlled not confounded reliable gawdfactorial

coustueettextuual3Anestartingpoimt73l I lU 2 levels within subjectsMP Problem Level Cost

gtappqbeem

D CootI O l O 2I 1 82 O y 2 42 I I

test c If sample mean

I NTIsample sizesampleSD

N p value probabilityofobtaininga resultequal to a more extremethan uthasactuallyobserved whenthe mule hypothesis istrue

artificialthresholdjer significance E 05I larger more confidentNeargen more confident5 larger less confidentif Dv is binary1categorical chi squared XL

3.2 I lU L levels between subjectst Ie

VII t IIT3 3 Liv 2levelseach between subjects

THE 6 t tests Neumueilpee comparisonsproblem6

eachcomparison PCerror7 05PC 31 errors in 100Comparisons

1 Pc always correct

I 1 Calways care 7Cas indy I 95100 9941 Do

conservativecorrection Bayernoui a agame

better Tuckeycomparisons

Factorial designs OR IV with 72 levelsANOVA Analysisof Varianceprecursor to multiple comparisonsmain effects Nl NLinteraction effects we x i v2

a f ly wi I b f f ly w2 1

response 13a pcb 133ab tphadd user problem Jar withinsubjects

useful in roboticswhen multiplefactorIfactermultiplelevels

JMP MINI TAB