(An Interactive Game-Based Approach to Multimodal Robot Learning)

Teaching robots about objects in their environment requires a multimodal correlation of images and linguistic descriptions to build complete feature and object models. These models can be created manually by collecting images and related keywords and presenting the pairings to robots, but doing so is tedious and unnatural. I Spy abstracts the problem of training robots to learn about the world around them by embedding the task in a vision- and dialogue-based game. Currently, two versions of this game are in development. One version is instantiated as an online game, and sources feature names, feature labels, and questions about those features from players as they try to guess which of 17 objects the system has in mind. In the other version, available in-person only, the scenario is reversed and a humanoid robot asks players questions to try to guess which objects the players have in mind. In this second version, the robot captures gameplay images to add additional visual information to the learned features in its knowledge base over time.

Awards

1st Prize - IRSS 2014

National Center for Scientific Research - Demokritos, 2014 The original research project, "Fun, Dynamic, Multimodal Robot Learning with I Spy and 20 Questions," won first prize out of 13 projects created by teams of students from around the world at the International Research-centered Summer School in Cognitive Systems and Interactive Robotics, Data and Content Analysis (IRSS 2014).

People's Choice Award - IRSS 2014

Center for Scientific Research - Demokritos, 2014 I Spy won the People's Choice Award at the International Research-centered Summer School in Cognitive Systems and Interactive Robotics, Data and Content Analysis (IRSS 2014), based on a vote by all participants in the summer school.

Related Publications

Natalie Parde, Adam Hair, Michalis Papakostas, Konstantinos Tsiakas, Maria Dagioglou, Vangelis Karkaletsis, and Rodney D. Nielsen. Grounding the Meaning of Words through Vision and Interactive Gameplay. In Proceedings of the 2015 International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina, July 25-31, 2015.

Natalie Parde, Michalis Papakostas, Konstantinos Tsiakas, Maria Dagioglou, Vangelis Karkaletsis, and Rodney D. Nielsen. (2015). I Spy: An Interactive Game-Based Approach to Multimodal Robot Learning. To appear in Proceedings of the AAAI-15 Workshop on Knowledge, Skill, and Behavior Transfer in Autonomous Robots, Austin, Texas, January 25, 2015.

Natalie Parde, Michalis Papakostas, Konstantinos Tsiakas, and Rodney D. Nielsen. (2015). "Is It Rectangular?" Using I Spy as an Interactive, Game-Based Approach to Multimodal Robot Learning. To appear in Proceedings of the AAAI-15 Conference on Artificial Intelligence, Austin, Texas, January 25-30, 2015.

Grace, Companionbot

Grace, a NAO robot, joined the HiLT Lab in the fall of 2013. She currently divides her time between learning knowledge via the I Spy project, and showing off her skills to new and future CS students. Since joining the lab, she's already traveled to Europe once and appeared in several UNT media projects. Grace has a twitter account (@GraceHilt), where you can follow the latest details of her activity in the HiLT lab.

Related Poster

Machine Learning and I Spy Accuracy

Rodney Nielsen & Adam Hair
Poster