Learning Situated Affordances in Robots by Playing Imitation Games

Research Areas: 

In this project we investigate how artificial systems can learn complex real-world tasks using a game situation. Situated affordances is an enlarged definition of affordances, a concept that was recently successfully adopted by robotic researchers to investigate the process of learning complex tasks in robots. Because learning is inextricably intertwined with the ability to memorize, we develop a cognitive architecture, inspired by biological and psychological findings, to study the interplay of perception, memory and the process of learning itself.

Methods and Research Questions: 

How humanoid robots can be enabled to learn complex real-world tasks is still an open research question. We investigate this question by developing a biologically inspired cognitive architecture that should learn a complex task by observation.

Recently the concept of affordances has become a popular paradigm in teaching robots and to investigate the process of learning. As affordances led to informative and interesting insights we subsequently enlarged the definition to situated affordances to follow this promising and successful path of research. Situated affordances describe a real-world relationship between action, objects, environments and effects or, one could say, a two stage cause and effect relationship, for example a pen that affords to write can only be used to write if there is paper that affords to be written on. Situated affordances can be used to learn affordances but in addition allow the learning of even more complex tasks that can not be learned using affordances alone.


To investigate how the learning of situated affordances takes place we develop a cognitive architecture that consists of several biologically inspired modules.


A perceptual module, a memory module and a reasoning module. By constructing the modules and the overall architecture we learn a lot about the corresponding problems.


In addition to the cognitive architecture we have developed a game that covers one exemplary situated affordances relationship. The game can be played by one person and the situated affordances relationship can be learned by observing the game.


In order to learn the situated affordance relationship, a demonstrator will play the game several times being observed by the cognitive system. During each demonstration the system shall recognize each entity of the game, extract relevant features to memorize it and shall also observes the outcome of the game. Then a coherent game descriptor, a situation signature will be created, that integratively describes each whole game situation. Having several different game situations memorized, the reasoning module will be applied to first learn which features are relevant and second which depend on each other, which would represent the learning of situated affordances. In the end the system should be capable to predict the appropriate action for a given game situation.


The anticipated goal of this research project is a cognitive architecture, implemented on the humanoid robot iCub, that is able to learn situated affordance relationships. The main constituents of this cognitive architecture are a biologically inspired memory module, a perceptual module as well as a reasoning module. The interplay of these three components should enable the overall system to learn a physical two stage cause and effect relationship by mere observation.


Therefore, a game was developed that covers a two stage real world cause and effect relationship, which is one form of the defined situated affordances concept. A demonstrator has to play several rounds of the game being observed by the cognitive architecture. During each demonstration the system shall extract important features and memorize them. If the system has memorized a sufficient number of training samples the reasoning module should learn which features are important and which are irrelevant to achieve the goal of the game. Finally the cognitive architecture should be capable to predict the appropriate action for a given game situation.