| |
|
| |
Grab a sundae & join Sundae Robotics, a private, invite-only Sunday series bringing together robotics researchers, founders, & builders working at the frontier of physical intelligence.
Sundae Robotics 12
Dexterous Manipulation, Tactile Perception & Robot Learning
Featured Talk: Beyond the Gripper - Teaching Robots Dexterity Through Touch, Vision, & Play
Beyond the Gripper: Teaching Robots Dexterity Through Touch, Vision, & Play
Keynote: Akash Sharma
Research Scientist, Amazon Frontier AI & Robotics CMU Robotics Institute PhD
Robots that impress in demos still mostly use parallel-jaw grippers, & true in-hand dexterity remains out of reach. The two standard routes each face fundamental limitations: demonstrations often force humans to act like robots, while simulation cannot faithfully model touch or real-world vision.
In this talk, Akash will present four ingredients that aim to close those gaps.
First, he will introduce PTLD, which learns tactile policies without ever simulating a tactile sensor by distilling a privileged simulation-trained teacher on real hardware.
He will then discuss INVAR, which learns visual invariance once from clean & randomized simulation pairs, allowing policies trained on clean simulation to transfer zero-shot.
Next, Akash will present CARD, which retargets human video, glove, & motion-capture demonstrations according to what the hand touches rather than simply how the hand moves.
Finally, he will discuss DSP, which turns minutes of human play into goal-conditioned dexterity through autonomous play, world models, & distillation.
Together, these works point toward a common recipe for scaling dexterous manipulation: let humans be humans, use a privileged teacher, scale with the robot's own experience, & distill what is learned into a deployable policy.
Akash Sharma is a Research Scientist at Amazon Frontier AI & Robotics (FAR), where he works with Jitendra Malik on robot hands & dexterous manipulation.
He received his PhD from the Carnegie Mellon University Robotics Institute, where he was advised by Prof. Michael Kaess. During his PhD, he was also a visiting researcher with the FAIR Robotics team at Meta, where he worked closely with Mustafa Mukadam & Francois Hogan.
Akash's research focuses on robot perception using varied sensing modalities, particularly vision & touch, to enable dexterous manipulation. He is also broadly interested in state estimation & reconstruction in dynamic environments.
His recent work explores how tactile sensing, simulation, human demonstrations, & large-scale robot experience can be combined to overcome limitations of conventional dexterous learning pipelines. His work on PTLD develops a sim-to-real approach for tactile dexterity that avoids tactile simulation by using privileged real-world sensing to distill information into deployable tactile policies.
Akash has also worked extensively on general-purpose tactile representations. Sparsh, Sparsh-skin, & Sparsh-X explore self-supervised representations across vision-based tactile sensors, distributed tactile skin, & multimodal touch, while DexterityGen investigates combining reinforcement-learned dexterous primitives with human guidance for complex manipulation.
His broader research spans multi-fingered manipulation, human-to-robot skill transfer, tactile perception, sim-to-real learning, object-centric manipulation, state estimation, & scalable data generation for robot learning.
Before his PhD, Akash received a Master's in Robotics from Carnegie Mellon University, where he worked on object-based SLAM for robot navigation. He received his bachelor's degree in Electronics & Communication Engineering from Sri Jayachamarajendra College of Engineering.
Topics
Moving beyond parallel-jaw grippers toward true in-hand dexterity
Vision & touch for multi-fingered robot manipulation
Why conventional human demonstrations struggle to scale to dexterous hands
Why tactile simulation remains a bottleneck for sim-to-real learning
PTLD & privileged tactile latent distillation
Learning tactile policies without simulating tactile sensors
Using privileged sensing on real hardware to supervise deployable policies
Sim-to-real transfer for tactile manipulation
INVAR & learning visual invariance for zero-shot transfer
Separating visual robustness learning from downstream policy learning
Training policies in clean simulation while remaining robust to real-world vision
CARD & contact-aware human-to-robot retargeting
Retargeting demonstrations based on interaction rather than pose matching
Learning from human video, gloves, & motion capture
Preserving task-relevant contacts across human & robot embodiments
DSP & learning dexterity from human play
Autonomous robot play as a scalable source of experience
World models for goal-conditioned dexterous manipulation
Distilling large-scale robot experience into deployable policies
Combining human priors, privileged teachers, & autonomous learning
Open Discussion + Q&A
Why has multi-fingered dexterity remained so much harder to scale than parallel-jaw manipulation?
Which aspects of touch are most important for in-hand manipulation: contact location, force, slip, pressure, or something else?
Can we obtain the benefits of tactile simulation without actually building an accurate tactile simulator?
What information should a privileged teacher have access to that the deployed robot should not?
How much real-world tactile data is needed to successfully distill privileged information?
Should visual domain randomization happen during every policy training run, or can invariance be learned once & reused?
What kinds of visual invariances transfer across different manipulation tasks?
Why is matching human & robot poses often the wrong objective for retargeting dexterous demonstrations?
Is preserving object contact more important than preserving human hand trajectories?
How can demonstrations from video, gloves, & motion capture be unified despite providing very different kinds of information?
What can humans demonstrate naturally that is difficult to express through robot teleoperation?
How far can a few minutes of human play be amplified through autonomous robot experience?
What role should world models play in scaling goal-conditioned dexterous behavior?
When is distillation preferable to deploying a larger teacher policy directly?
Could autonomous play eventually generate most of the data needed for dexterous robot learning?
What combination of vision, touch, demonstrations, simulation, & autonomous experience is most likely to unlock general-purpose robot dexterity?
|
|
|
|
|
|
|
|
