Publication

On the Dual-Use Dilemma in Physical Reasoning and Force

Nicolaus J Correll — Sat, 24 May 2025 16:55:34 +0000

On the Dual-Use Dilemma in Physical Reasoning and Force Nicolaus J Correll Sat, 05/24/2025 - 10:55

Humans learn how and when to apply forces in the world via a complex physiological and psychological learning process. Attempting to replicate this in vision-language models (VLMs) presents two challenges: VLMs can produce harmful behavior, which is particularly dangerous for VLM-controlled robots which interact with the world, but imposing behavioral safeguards can limit their functional and ethical extents. We conduct two case studies on safeguarding VLMs which generate forceful robotic motion, finding that safeguards reduce both harmful and helpful behavior involving contact-rich manipulation of human body parts. Then, we discuss the key implication of this result--that value alignment may impede desirable robot capabilities--for model evaluation and robot learning.

References

On the Dual-Use Dilemma in Physical Reasoning and Force

W Xie, E Rice, N Correll - arXiv preprint arXiv:2505.18792, 2025

Varying contextual semantics in the same scene can yield harm and help, often with a thin line separating them. We evaluate how VLMs under different prompt schemes which elicit physical reasoning for robot control navigate this line between harm and help for forceful, contact-rich tasks with potential for bodily danger.

Off

Traditional

White

Unfettered Forceful Skill Acquisition with Physical Reasoning and Coordinate Frame Labeling

Nicolaus J Correll — Wed, 14 May 2025 17:03:34 +0000

Unfettered Forceful Skill Acquisition with Physical Reasoning and Coordinate Frame Labeling Nicolaus J Correll Wed, 05/14/2025 - 11:03

A natural language query, together with head and wrist images both annotated with a coordinate frame at a VLM-generated grasp point (u, v) on the image, is provided to Gemini to estimate, using spatial and physical reasoning, an appropriate wrench and duration to execute the task. The wrench is then passed to a compliance controller and the resulting motion and visual data can be used for iterative task improvement.

Vision language models (VLMs) exhibit vast knowledge of the physical world, including intuition of physical and spatial properties, affordances, and motion. With fine-tuning, VLMs can also natively produce robot trajectories. We demonstrate that eliciting wrenches, not trajectories, allows VLMs to explicitly reason about forces and leads to zero-shot generalization in a series of manipulation tasks without pretraining. We achieve this by overlaying a consistent visual representation of relevant coordinate frames on robot-attached camera images to augment our query. First, we show how this addition enables a versatile motion control framework evaluated across four tasks (opening and closing a lid, pushing a cup or chair) spanning prismatic and rotational motion, an order of force and position magnitude, different camera perspectives, annotation schemes, and two robot platforms over 220 experiments, resulting in 51% success across the four tasks. Then, we demonstrate that the proposed framework enables VLMs to continually reason about interaction feedback to recover from task failure or incompletion, with and without human supervision. Finally, we observe that prompting schemes with visual annotation and embodied reasoning can bypass VLM safeguards. We characterize prompt component contribution to harmful behavior elicitation and discuss its implications for developing embodied reasoning. Our code, videos, and data are available at: https://scalingforce.github.io/.

References

Unfettered Forceful Skill Acquisition with Physical Reasoning and Coordinate Frame Labeling

W Xie, M Conway, Y Zhang, N Correll - arXiv preprint arXiv:2505.09731, 2025.

Off

Traditional

White

Towards forceful robotic foundation models: a literature survey

Nicolaus J Correll — Wed, 16 Apr 2025 17:12:35 +0000

Towards forceful robotic foundation models: a literature survey Nicolaus J Correll Wed, 04/16/2025 - 11:12

This article reviews contemporary methods for integrating force, including both proprioception and tactile sensing, in robot manipulation policy learning. We conduct a comparative analysis on various approaches for sensing force, data collection, behavior cloning, tactile representation learning, and low-level robot control. From our analysis, we articulate when and why forces are needed, and highlight opportunities to improve learning of contact-rich, generalist robot policies on the path toward highly capable touch-based robot foundation models. We generally find that while there are few tasks such as pouring, peg-in-hole insertion, and handling delicate objects, the performance of imitation learning models is not at a level of dynamics where force truly matters. Also, force and touch are abstract quantities that can be inferred through a wide range of modalities and are often measured and controlled implicitly. We hope that juxtaposing the different approaches currently in use will help the reader to gain a systemic understanding and help inspire the next generation of robot foundation models.

Common force or touch sensing methods on robot arms include joint torques, wrist force/torque (F/T) sensors, and end effector gripper sensors.

Off

Traditional

White

A Machine Learning Approach to Contact Localization in Variable Density Three-Dimensional Tactile Artificial Skin

Nicolaus J Correll — Wed, 04 Dec 2024 21:19:38 +0000

A Machine Learning Approach to Contact Localization in Variable Density Three-Dimensional Tactile Artificial Skin Nicolaus J Correll Wed, 12/04/2024 - 14:19

Estimating the location of contact is a primary function of artificial tactile sensing apparatuses that perceive the environment through touch. Existing contact
localization methods use flat geometry and uniform sensor distributions as a simplifying assumption, limiting their ability to be used on 3D surfaces with variable
density sensing arrays. This paper studies contact localization on an artificial skin
embedded with mutual capacitance tactile sensors, arranged non-uniformly in an
unknown distribution along a semi-conical 3D geometry. A fully connected neural
network is trained to localize the touching points on the embedded tactile sensors.
The studied online model achieves a localization error of 5.7 ± 3.0 mm. This
research contributes a versatile tool and robust solution for contact localization that
is ambiguous in shape and internal sensor distribution.

References

Murray, M., Zhang, Y., Kohlbrenner, C., Escobedo, C., Dunnington, T., Stevenson, N., Correll, N. and Roncone, A., A Machine Learning Approach to Contact Localization in Variable Density Three-Dimensional Tactile Artificial Skin. In 2nd NeurIPS Workshop on Touch Processing: From Data to Knowledge.

Contact localization model takes in a sensor image from any configuration of artificial tactile skin and determines the location of touch through a feedforward neural network.

Off

Traditional

White

Just Add Force for Delicate Robot Policies

Nicolaus J Correll — Fri, 18 Oct 2024 21:39:48 +0000

Just Add Force for Delicate Robot Policies Nicolaus J Correll Fri, 10/18/2024 - 15:39

Robot trajectories used for learning end-to-end robot policies typically contain end-effector and gripper position, workspace images, and language. Policies learned from such trajectories are unsuitable for delicate grasping, which require tightly coupled and precise gripper force and gripper position. We collect and make publically available 130 trajectories with force feedback of successful grasps on 30 unique objects. Our current-based method for sensing force, albeit noisy, is gripper-agnostic and requires no additional hardware. We train and evaluate two diffusion policies: one with (forceful) the collected force feedback and one without (position-only). We find that forceful policies are superior to position-only policies for delicate grasping and are able to generalize to unseen delicate objects, while reducing grasp policy latency by near 4x, relative to LLM-based methods. With our promising results on limited data, we hope to signal to others to consider investing in collecting force and other such tactile information in new datasets, enabling more robust, contact-rich manipulation in future robot foundation models.

Reference

W. Xie, S. Caldararu, N. Correll. Just Add Force for Delicate Robot Policies. https://arxiv.org/abs/2410.13124, 2024.

Off

Traditional

White

DeliGrasp: Inferring Object Mass, Friction, and Compliance with LLMs for Adaptive and Minimally Deforming Grasp Policies

Anonymous — Tue, 12 Mar 2024 06:00:00 +0000

DeliGrasp: Inferring Object Mass, Friction, and Compliance with LLMs for Adaptive and Minimally Deforming Grasp Policies Anonymous (not verified) Tue, 03/12/2024 - 00:00

Large language models (LLMs) can provide rich physical descriptions of most worldly objects, allowing robots to achieve more informed and capable grasping. We leverage LLMs’ common sense physical reasoning and code-writing abilities to infer an object’s physical characteristics—mass, friction coefficient, and spring constant —from a semantic description, and then translate those characteristics into an executable adaptive grasp policy. Using a current-controllable, two-finger gripper with a built-in depth camera, we demonstrate that LLM-generated, physically-grounded grasp policies outperform traditional grasp policies on a custom benchmark of 12 delicate and deformable items including food, produce, toys, and other everyday items, spanning two orders of magnitude in mass and required pick-up force. We also demonstrate how compliance feedback from DeliGrasp policies can aid in downstream tasks such as measuring produce ripeness. Our code and videos are available at: https://deligrasp.github.io.

References

Xie, W., Valentini, M., Lavering, J. and Correll, N., 2024. DeliGrasp: Inferring Object Properties with LLMs for Adaptive Grasp Policies. In 8th Annual Conference on Robot Learning.

Xie, W., Lavering, J. and Correll, N., 2024. DeliGrasp: Inferring Object Mass, Friction, and Compliance with LLMs for Adaptive and Minimally Deforming Grasp Policies. arXiv preprint arXiv:2403.07832.

Off

Traditional

White

A multifunctional soft robotic shape display with high-speed actuation, sensing, and control

Anonymous — Mon, 31 Jul 2023 06:00:00 +0000

A multifunctional soft robotic shape display with high-speed actuation, sensing, and control Anonymous (not verified) Mon, 07/31/2023 - 00:00

Shape displays that actively manipulate surface geometry are an expanding robotics domain with applications to haptics, manufacturing, aerodynamics, and more. However, existing displays often lack high-fidelity shape morphing, high-speed deformation, and embedded state sensing, limiting their potential uses. Here, we demonstrate a multifunctional soft-shape display driven by a 10 × 10 array of scalable cellular units which combine high-speed electrohydraulic soft actuation, magnetic-based sensing, and control circuitry. We report high-performance reversible shape morphing up to 50 Hz, sensing of surface deformations with 0.1 mm sensitivity, and external forces with 50 mN sensitivity in each cell, which we demonstrate across a multitude of applications including user interaction, image display, sensing of object mass, and dynamic manipulation of solids and liquids. This work showcases the rich multifunctionality and high-performance capabilities that arise from tightly-integrating large numbers of electrohydraulic actuators, soft sensors, and controllers at a previously undemonstrated scale in soft robotics.

References

B. K. Johnson, M. Naris, V. Sundaram, A. Volchko, K. Ly, S. K. Mitchell, E. Acome, N. Kellaris, C. Keplinger, N. Correll, J. S. Humbert & M. E. Rentschler. A multifunctional soft robotic shape display with high-speed actuation, sensing, and control. Nature Communications volume 14, Article number: 4516 (2023)

Off

Traditional

White

A versatile robotic hand with 3D perception, force sensing for autonomous manipulation

Anonymous — Mon, 10 Jul 2023 18:42:28 +0000

A versatile robotic hand with 3D perception, force sensing for autonomous manipulation Anonymous (not verified) Mon, 07/10/2023 - 12:42

We describe a force-controlled robotic gripper with built-in tactile and 3D perception. We also describe a complete autonomous manipulation pipeline consisting of object detection, segmentation, point cloud processing, force-controlled manipulation, and symbolic (re)-planning. The design emphasizes versatility in terms of applications, manufacturability, use of commercial off-the-shelf parts, and open-source software. We validate the design by characterizing force control (achieving up to 32N, controllable in steps of 0.08N), force measurement, and two manipulation demonstrations: assembly of the Siemens gear assembly problem, and a sensor-based stacking task requiring replanning. These demonstrate robust execution of long sequences of sensor-based manipulation tasks, which makes the resulting platform a solid foundation for researchers in task-and-motion planning, educators, and quick prototyping of household and warehouse automation tasks.

References

N. Correll, D. Kriegman, S. Otte and J. Watson. A versatile robotic hand with 3D perception, force sensing for autonomous manipulation. In Proceedings of Workshop on Perception and Manipulation Challenges for Warehouse Automation, Robotics: Science and Systems, Daegu, Korea.

Off

Traditional

White

Early failure prediction during robotic assembly using Transformers

Anonymous — Mon, 10 Jul 2023 06:00:00 +0000

Early failure prediction during robotic assembly using Transformers Anonymous (not verified) Mon, 07/10/2023 - 00:00

Peg-in-hole assembly of tightly fitting parts often requires multiple attempts. Parts need to be put together by performing a wiggling motion of undetermined length and can get stuck, requiring a restart. Recognizing unsuccessful insertion attempts early can help in reducing the makespan of the assembly. This can be achieved by analyzing time-series data from force and torque measurements. We describe a transformer neural network model that is three times faster, i.e. requiring much shorter time series, for predicting failure than a dilated fully convolutional neural network. Albeit the transformer provides predictions with higher confidence, it does so at reduced accuracy. Yet, being able to call unsuccessful attempts early, makespan can be reduced by almost 40% which we show using a dataset with force-torque data from 241 peg-in-hole assembly runs with known outcomes.

References

R. Montané-Güell, J. Watson and N. Correll, 2023. Early failure prediction during robotic assembly using Transformers. In Proceedings of Workshop on Robotics and AI: The Future of Industrial Assembly Tasks at Robotics: Science and Systems, Daegu, Korea.

Off

Traditional

White

Distributed Tactile Sensors for Palmar Surfaces of Prosthetic Hands

Anonymous — Fri, 19 May 2023 06:00:00 +0000

Distributed Tactile Sensors for Palmar Surfaces of Prosthetic Hands Anonymous (not verified) Fri, 05/19/2023 - 00:00

Sensory feedback provided by prosthetic hands shows promise in increasing functional abilities and promoting embodiment of the prosthetic device. However, sensory feedback is limited based on where sensors are placed on the prosthetic device and has mainly focused on sensorizing the fingertips. Here we describe distributed tactile sensors for the palmar surfaces of prosthetic hands. We believe a sensing system that can detect interactions across the palmar surfaces in addition to the fingertips will further improve the experience for the prosthetic user and may increase embodiment of the device as well. This work details the design of a compliant distributed sensor which consists of PiezoResistive and PiezoElectric layers to produce a robust force measurement of both static and dynamic loads. This assembled sensor system is easy to customize to cover different areas of the prosthetic hand, simple to scale up, and flexible to different fabrication form-factors. The experimental results detail a load estimation accuracy of 95.4% and sensor response time of less than 200ms. Cycle tests of each sensor shows a drifting of within 10% of sensing capability under load and 6.37% in a no-load longitudinal test. These validation experiments reinforce the ability of the DualPiezo structure to provide a valuable sensor design for the palmar surfaces of prosthetic hands.

References

Truong, H., Correll, N. and Segil, J., 2023, April. Distributed Tactile Sensors for Palmar Surfaces of Prosthetic Hands. In 2023 11th International IEEE/EMBS Conference on Neural Engineering (NER) (pp. 1-4). IEEE. [PDF]

Off

Traditional

White