Eren Erdal Aksoy, Action Semantics for Robot Perception and Imitation

Robots require a generic action representation in order to recognize, learn, and imitate observed tasks without any human intervention. Defining such a representation is a chal- lenging problem due to the high inter-individual variability that emerges during the execu- tion of actions. Conventional methods approach this problem by either considering contin- uous trajectory profiles or employing predefined symbolic action knowledge. The main challenge, however, still remains in linking perceived continuous sensory signals to dis- crete symbolic object or action concepts.

In this talk, I will introduce a fundamentally new framework, so-called “Semantic Event Chain” (SEC), for creating a generic representation of visual sensory experiences while linking continuous signal streams (e.g. image sequences) to their symbolic descriptions (e.g. action primitives). The SEC concept is an implicit spatiotemporal formulation that encodes actions by coupling the observed effect with the exhibited roles of manipulated objects. I will explain how such a semantic action encoding can allow robots not only to ground high-level symbolic plans into the low-level sensory-motor data but also to have seamless data processing in a bottom-up and top-down manner. To highlight the scalabil- ity of SECs, I will finally introduce various applications on coupling language and vision, perceiving time, and memorizing episodic experiences.