AI-assisted learning platform helps robots reason beyond literal commands

Researchers from MIT and New York University have built FLEX, a feature-editing pipeline for robot motion planning that lets a robot infer broader preferences from conversational commands, MIT News reports. The study, led by MIT assistant professor Andreea Bobu with co-authors Nathan Dennler, Audrey Lee and NYU's Ilia Sucholutsky, is due to be presented this month at the Conference on Robot Learning.

FLEX runs in three stages that take about five seconds in total. "Add" places an object a person mentions on the robot's list of things that matter, which is how earlier methods worked. "Generalize," the new step, uses GPT-4o to look for patterns across those items and infer a wider rule. "Mask" then uses computer vision to remove objects no longer present, keeping the list from growing unwieldy.

The system has so far been shown only in simulation, with volunteers typing commands to a virtual robot arm serving coffee, moving a candle and carrying a bowling ball. Told not to spill on a laptop, the robot also steered clear of a nearby phone, reasoning that electronics should be avoided. Similar instructions led it to avoid flammable items and fragile objects.

Dennler says language models absorb a kind of common sense from internet text, such as the link between liquids and broken devices. Bobu argues that homes and priorities vary, and that picking up preferences from a sentence or two could make household and assistive robots far more appealing than ones needing every rule spelled out.