Learning to Order Objects using Haptic and Proprioceptive Exploratory Behaviors

Jivko Sinapov, Priyanka Khante, Maxwell Svetlik, and Peter Stone

Department of Computer ScienceUniversity of Texas at Austin, Austin TX 78712, USA{jsinapov,pkhante,maxwell, pstone}@cs.utexas.edu

2

Building-Wide Intelligence Project:http://www.cs.utexas.edu/~larg/bwi_web/

3

Building-Wide Intelligence Project:http://www.cs.utexas.edu/~larg/bwi_web/

4

Building-Wide Intelligence Project:http://www.cs.utexas.edu/~larg/bwi_web/

5

Motivation: Grounded Language Learning

Robot, fetch me the green empty bottle

6

Object Category Recognition in Robotics

Sridharan et al. 2008

Lai et al. 2011Rusu et al. 2009

Collet et al. 2009

7

Object Category Learning in Robotics

Thomason, J., Sinapov, J., Svetlik, M., Stone, P., and Mooney, R. (2016).Learning Multi-Modal Grounded Linguistic Semantics by Playing I, Spy

Robotics and Vision 3 Session

8

Now, when and where does this fail...

Consider the word, “weight” - how should it be grounded?

9

How do humans ground such words?

Sample Montessori toys designed to teach children about the ordinal properties of object weight, height, and size

10

Object Ordering in Psychology

11

Object Orderings in Human Environments

12

Problem Formulation

● Order Recognition: what property is a given series of objects ordered by?

“height”

“width”

13

Problem Formulation (2)

● Order Insertion: given an object series, insert a new object into the correct position

series test object

14

Three-Stage Approach

Stage 1: Object Exploration Stage 2: Unsupervised Order Discovery

. . . .

Stage 3: Semantic Grounding

weight width height

. . . .

16

Stage 1: Object Exploration

32 common household and office items

The objects vary along three ordinal properties:

1) Weight2) Width3) Height

17

Exploratory Behaviors

18

Video

19

Video

20

Video

21

Haptic and Proprioceptive Feature Extraction

Time

Join

t P

ositi

ons

(Pro

rioce

ptio

n)Jo

int

Effo

rts

(Hap

tics)

. . . . . .

22

Haptic and Proprioceptive Feature Extraction

Time

Join

t P

ositi

ons

(Pro

rioce

ptio

n)Jo

int

Effo

rts

(Hap

tics)

. . . . . .

23

Haptic and Proprioceptive Feature Extraction

Time

Join

t P

ositi

ons

(Pro

rioce

ptio

n)Jo

int

Effo

rts

(Hap

tics)

. . . . . .

24

Stage 2: Unsupervised Order Discovery

grasp

lift

hold

lower

drop

proprioception

Beh

avio

rsSensory Modalities

push

press

haptics

26

Unsupervised Order Discovery Example with Synthetic Data

Object order with highest likelihood using the method of [Kemp and Tennenbam, 2008]

Input Relational Count Matrix

27

Example Relational Count Matrix with the Press action and Haptic features

Similarity between objects i and j in the press-haptic sensorimotor context

28

Resulting Order (Press behavior and Haptic modality)

The number corresponds to the object's height in millimeters

29

Stage 2: Unsupervised Order Discovery

grasp

lift

hold

lower

drop

proprioception

Beh

avi

ors

Sensory Modalities

push

press

haptics

30

Stage 3: Order Grounding Stage

31

Order Grounding Example: “height”

Positive Examples: Negative Examples:

. . .

. . .

32

Object Order Representation

Training Example:

Object Orders Discovered During Stage 2

. . . .

33

Object Order Representation

Training Example:

. . . .

Object Orders Discovered During Stage 2

34

Object Order Representation

Training Example:

. . . .

Object Orders Discovered During Stage 2

35

Object Order Representation

Training Example:

x1

. . . .

Object Orders Discovered During Stage 2

36

Object Order Representation

Training Example:

x1

. . . .

Object Orders Discovered During Stage 2

37

Object Order Representation

. . . .

Training Example:

x1

x2

Object Orders Discovered During Stage 2

38

Object Order Representation

. . . .

Training Example:

x1

x2 . . . . x

n

Object Orders Discovered During Stage 2

39

Results: Order Recognition

40

Sample Learned Decision Trees

Hold Haptics

Lower Haptics

Lift Haptics

Press Proprioception

Press Haptics

Grasp Proprioception

weight width height

41

When does the robot make mistakes?

42

When does the robot make mistakes?

difficult easy

43

When does the robot make mistakes?

difficult easy

44

Object Order Insertion Results

45

Object Order Insertion Results

46

Object Order Insertion Results

47

Object Order Insertion Results

48

Conclusion

● A behavior-grounded framework for learning object ordering concepts

● The robot grounded three ordering concepts, “weight”, “height”, and “width”

● Future Work:– Active action selection

– Learn object ordering concepts in conjunction with object categories, pairwise object relations, etc.

– Learn from humans (for a preview, see our next talk at Robotics and Vision III)

49

Thank you!

Jivko Sinapov Maxwell Svetlik Peter Stone

http://www.cs.utexas.edu/~larg/bwi_web/

Priyanka Khante