This project presents a multimodal human-robot interaction system integrating vision-language models (Florence 2), language interpretation (Llama 3.1), and speech recognition (Whisper) with fuzzy logic for precise control of a Dobot Magician robotic arm. Initial tests show 75% accuracy, highlighting its potential for more natural, efficient human-robot collaboration.
In this work, we developed a vision-based reactive adaptation method for contact-rich manipulation tasks, based on the compliant control and learning from demonstration. 
Tele-echography has emerged as a promising and effective solution, leveraging the expertise of sonographers and the autonomy of robots to perform ultrasound scanning for patients residing in remote areas, without the need for in-person visits by the sonographer. In this project, we develop a teleoperation system for robot-assisted sonography with two different interfaces, a haptic device-based interface and a low-cost 3D Mouse-based interface, which can achieve continuous and intuitive telemanipulation by a leader device with a small workspace..gif)
We propose a trust human-robot skill transfer framework, interpretive and reactive dynamic system (IRDS), by investigating the behaviour tree (BT) and dynamic movement primitives (DMPs) enhanced by the feedback from perceived information for dynamic and uncertain tasks and environments. Human sensorimotor control allows interactions with various environments and accomplishes complex manipulation tasks with uncertainty; however, it is still hard for robots to own this capability. In this work, we aim to transfer these reactive skills to robots, which enable robots to interact with humans and environments under varying uncertainty. 
Suction adhesion is widely used by natural creatures for gripping irregular objects (e.g., rocks), but their artificial counterparts show less adaptation in the same situation. In this work, we developed a contact-triggered suction cup with the ability to adapt to objects with complex and irregular shapes.
In this project, we developed a vision-based UAV autonomous landing solution on a moving car. This project designed a new landing pad and its recognition algorithm to achieve relative pose estimation with high sampling rate. Utilization of monocular gimbaled camera expand the field of view and improves the reliability of identification as well. All above investigates solutions for the challenging problem of autonomous landing on a moving vehicle of multi-rotors Unmanned Aerial Vehicles.
In this work, we proposed a novel robot-assisted ultrasound scanning framework, integrating deep multimodal imitation learning and model-based compliant control. We investigated the deep imitation learning model to fuse multimodal information, including the RGB image, force profile, ultrasound image, and proprioceptive information for robot-assisted ultrasound scanning artery. 
Sorting wood waste from other waste streams using robotic systems involves the use of advanced technology and automation to efficiently separate wood materials from mixed waste. This project studied vision-based solution for waste sorting by Baxter robot arm.
In this work, an impedance control-based framework for human-robot composite layup skill transfer was developed, and the human-in-the-loop mechanism was investigated to achieve human-robot skill transfer. Although there are some works on human-robot skill transfer, it is still difficult to transfer the manipulation skill to robots through teleoperation efficiently and intuitively. In this project, we developed an impedance-based control architecture of telemanipulation in task space for the human-robot skill transfer through teleoperation. 