UMI-LINGO

Open access to a modular, multi-tool handheld gripper and a multimodal dataset for natural language-guided robot manipulation.

UMI-LINGO democratises robot learning by providing an open-source handheld gripper and a multimodal dataset from human demonstrations.

By integrating natural language instructions, visual inputs, and tactile/auditory sensing, the project enables cost-effective data collection and model training, accelerating automation adoption across sectors, particularly for SMEs.

Building upon the Universal Manipulation Interface (UMI), UMI-LINGO introduces a novel approach: task demonstrations are generated from natural-language descriptions. In this setup, one user describes a task based on before/after images, and another executes it from the spoken instruction. This method captures the diversity of human communication and execution strategies, enhancing the richness and generalisation capability of datasets used to train AI-powered robots.

UMI-LINGO is an initiative selected by the 3^rd Open Call “Creation of manipulation tasks dataset with vision and language instructions on tech exchange and collaborative projects on robotics and AI”, boosted by the euROBIN project.

In this sense, UMI-LINGO complements the euROBIN LLM-planner by offering automatic skills implementation, eliminating manual programming and enhancing workflow autonomy. Beyond euROBIN, this initiative benefits the entire EU industry by promoting dataset reusability, lowering R&D barriers, and enhancing collaboration.

To maximise accessibility, the project provides video tutorials on gripper assembly, data capture and model training, empowering SMEs and non-expert users to deploy robotic solutions independently and affordably.

UMI-LINGO is executed by Eurecat’s Robotics and Automation Unit, with mentoring from Inria, and it be validated across multiple domains: flexible manufacturing, domestic assistance, and outdoor robotics.