Animesh Garg

TRI's "LBM 1.0" paper appeared on arxiv last night! Large Behavior Models (LBMs) are foundation models for robots that map robot sensors (notably camera inputs) and natural language commands into robot actions. The robots are programmed just through demonstration; we can develop incredible new skills like the video below without writing a single line of new code. There is a lot of excitement in the field right now because of the incredible potential for this type of technology. Inevitably, there is also a lot of hype. One of our main goals for this paper was to put out a very careful and thorough study on the topic to help people understand the state of the technology, and to share a lot of details for how we're achieving it. The short version is: LBMs work! We see consistent and statistically significant improvements as we increase the amount of pretraining data. But doing the science is still hard; as a field we have more work to do to improve the statistical power of our experiments. Please check out our project website for the paper and more details: https://lnkd.in/eDn_sqGh. https://lnkd.in/epSksw5E

773

9 Comments

Our CoRL 2025 paper, GLOVER++, tackles a key question: How can we leverage human videos to boost robot manipulation performance and generalization? 🤖 We're excited to release HOVA-500K, a dataset of 500K annotated human object interaction images! It's perfect for training affordance reasoning models (e.g., "pour me water": given an image, the model outputs the mug handle's location). This enables zero-shot robot manipulation and enhances imitation learning precision. 🙌 Code & dataset are open-source! Check out GLOVER++! 👇 https://lnkd.in/gdUi7fQY #CoRL2025 #Robotics #AI

30

Excited to share our new work on constrained protein design with diffusion models 🚀 Diffusion models have transformed de novo protein design 🧬, but they often break when strict functional, stereochemical, or geometric constraints must hold 😬. This work introduces a constrained diffusion framework that embeds proximal feasibility updates directly into the generative process, enabling hard constraint satisfaction ✅ without sacrificing structural diversity 🎯 We report state-of-the-art results on motif scaffolding 🧩 and vacancy-constrained pocket design 🔒, and release a curated PDZ-domain scaffolding benchmark 📚 Work with Jacob Christopher, Austin Seamann, Jingyi Cui, and Sagar Khare. Paper: https://lnkd.in/e5AuiGea

164

1 Comment

A nice paper from Senthil Yogamani et.al (Jane Courtney Ganesh Sistu and me) "Let’s Go Bananas: Beyond Bounding Box Representations for Fisheye Camera-Based Object Detection in Autonomous Driving" Object detection is a mature problem in autonomous driving, with pedestrian detection being one of the first commercially deployed algorithms. It has been extensively studied in the literature. However, object detection is relatively less explored for fisheye cameras used for surround-view near-field sensing. The standard bounding-box representation fails in fisheye cameras due to heavy radial distortion, particularly in the periphery. In this paper, a generic object detection framework is implemented using the base YOLO (You Only Look Once) detector to systematically explore various object representations using the public WoodScape dataset. First, we implement basic representations, namely the standard bounding box, the oriented bounding box, and the ellipse. Secondly, we implement a generic polygon and propose a novel curvature-adaptive polygon, which obtains an improvement of 3 mAP (mean average precision) points. A polygon is expensive to annotate and complex to use in downstream tasks; thus, it is not practical to use it in real-world applications. However, we utilize it to demonstrate that the accuracy gap between the polygon and the bounding box representation is very high due to strong distortion in fisheye cameras. This motivates the design of a distortion-aware optimal representation of the bounding box for fisheye images, which tend to be banana-shaped near the periphery. We derive a novel representation called a curved box and improve it further by leveraging vanishing-point constraints. The proposed curved box representations outperform the bounding box by 3 mAP points and the oriented bounding box by 1.6 mAP points. In addition, the camera geometry tensor is formulated to provide adaptation to non-linear fisheye camera distortion characteristics and improves the performance further by 1.4 mAP points. DiCE Research Centre Ciarán Eising Brian Deegan Valentina Donzella Patrick Vandewalle Pepijn van de Ven Tejus Vignesh Vijayakumar Ankita Saini Ahmed Nofal ( 🇸🇦 🇮🇪 🇪🇬) Louis Kerofsky John O'Shea Adrian Nunez-Rocha Aileen Kelly-O'Sullivan David Guthrie Ankita Saini Aryan Singh Robin Jenkin Tim Brophy Alexis Lluis https://lnkd.in/e_Uwi9Zr

49

2 Comments