yun jiang, changxi zheng, marcus lim, ashutosh...
TRANSCRIPT
![Page 1: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/1.jpg)
Learning to Place New Objects
Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena.
Cornell University.
![Page 2: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/2.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Placing Objects } Daily tasks:
} Setting a dinner table. } Arranging grocery in
a fridge. } Packing items in boxes. } Organizing closets. Grasping and placing objects.
} Placing is an important skill for a personal robot. } However, it has heretofore been little-studied.
![Page 3: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/3.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Related Work } Identify flat surfaces.
} Surface (green) vs clutter (red). } Place upright objects on flat areas
using tactile feedback.
Schuster et al., 2010
![Page 4: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/4.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Flat Surface Heuristic?
} It is not always as easy as finding flat surface
![Page 5: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/5.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Challenges } Different configurations in different
placing situations. } Plate: vertical in a dish-rack; slanted on
a support. } Glasses: upright on table, upside-down
on stemware holder,
} Stacking objects.
![Page 6: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/6.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Problem Specification } Input: Point cloud of an object (e.g., plate) and placing
environment (e.g., a dish rack).
} Output: a stable and preferred placement specified by 3D location and 3D orientation of the object. } Stability
} stay still after placing and stand small perturbation.
} Preference } E.g., plates and pens are flat on a table, but vertical in a dish-rack and a
pen-holder.
![Page 7: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/7.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Brute-force Approach } Test all orientations and locations randomly using a
physics-based simulator.
} Too many configurations to try. } Does not address the preferred placement issue.
Learning needed!
![Page 8: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/8.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Learning Approach
Supervised Learning. } Features X for placement ξ } Learning algorithm: X à y } Choose ξwith highest y.
![Page 9: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/9.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Features for Learning Algorithm } Supporting contacts
} Falling distance, #contacts, variance of contacts, etc.
} Caging features } Height of the neighborhood around the placement.
} Signatures of the object and placing environment } Histogram of #points in object/environment point
cloud. } Distance ratio of obj/env along different directions.
Total 120 features.
![Page 10: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/10.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Learning Algorithm } Support Vector Machines.
} Maximize the minimum margin.
} Parameters: ω, b. } Features X } Label y
![Page 11: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/11.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Learning Experiments. } Data
} 7 placing areas and 8 objects. } For each object-environment pair, 100 distinct 3D locations
with 18 different orientations. This gives 1800 different placements.
} Total 37655 collision-free placements.
} Metrics: } R0: rank of first correct placement. } Prec@5: precision of top 5 predictions.
![Page 12: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/12.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Learning Results } Results with different features.
} Same Environment Same Object (SESO).
} In New Environment New Object (NENO) setting. } R0 is 9.4. } Pre@5 is 0.54. } (This is unacceptable.)
![Page 13: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/13.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Learning Algorithm: Multiple Models } Intrinsic difference between different placing settings
} Independent models in conventional SVM } r models: wi
![Page 14: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/14.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Results: Independent Models. } New Environment New Object (NENO).
![Page 15: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/15.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Shared Sparsity SVM } Introduce sparsity (Jalali et al., NIPS 2010)
} Each model is composed of and
} is self-owned features reflecting individual difference } represents shared sparsity structure
} Shared sparsity SVM
iii BSw +=iw iS iB
iS
iB
where and
![Page 16: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/16.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Comparison of different algorithms. } New Environment New Object (NENO).
![Page 17: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/17.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Learning Experiments: Results.
Chance: 29.4 Flat surface: 18.6 Our method: 1.9
![Page 18: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/18.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Robotic Experiments
Panda (PersonAl Non-Deterministic Arm.)
Some test objects
Some test environments
![Page 19: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/19.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Results: Robotic Experiments.
![Page 20: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/20.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Challenges } Different configuration in different
placing situations. } Plate: vertical in a dish-rack; slanted on
a support. } Glasses: upright on table, upside-down
on stemware holder.
} Stacking multiple objects.
![Page 21: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/21.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Placing Multiple Objects (Future Work) } Multiple objects and several placing environments.
} Objects can choose different environments. } Objects can stack on another.
A B
C A
B
C
A
B C
![Page 22: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/22.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Multiple Object: Results. (Future Work)
![Page 23: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/23.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Robotic Experiments
![Page 24: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/24.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Questions? ! Download code and data at: !!!http://pr.cs.cornell.edu/placingobjects!!!!(Cornell Personal Robotics.)!
![Page 25: Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh …pr.cs.cornell.edu/papers/learning_to_place_objects...Yun Jiang, Changxi Zheng, Marcus Lim, Ashutosh Saxena. Cornell University. Learning](https://reader036.vdocuments.site/reader036/viewer/2022070807/5f05e7d17e708231d4154ed4/html5/thumbnails/25.jpg)
Learning to Place New Objects. Jiang, Zheng, Lim, Saxena.
Thank you