SIGN-ACT – Inclusive Multimodal AI for Human–AI Collaboration

SIGN-ACT aims to study multimodal AI methods that support natural and inclusive interaction between people and AI-enabled systems. The main research direction is to treat sign language, gesture, visual information, and text as complementary interaction modalities rather than relying mainly on speech- or text-based interfaces. The project will investigate cross-modal representations linking sign video, pose, gestures, language, and task context, with attention to robustness across users and real-world conditions.

Possible research directions include sign-enabled human–AI interaction, adaptive multimodal interfaces, multimodal knowledge exchange, and AI support for training and task assistance. Potential application settings may include industrial training, knowledge capture, collaborative work, and AI-supported workplaces involving users with different communication needs.

We are interested in collaboration with partners in human–AI interaction, HCI and accessibility, industrial AI, manufacturing, human factors, sign language and Deaf studies, and trustworthy AI. We are also open to joining an existing Horizon Europe consortium where our expertise in computer vision, multimodal learning, and sign language AI can contribute to a broader human–AI collaboration project. The project is currently at the concept stage, and we are open to shaping its scope together with potential partners. Use cases, technical work packages, and validation settings will be refined based on complementary expertise and consortium interests.