Introduction to Cognitive Robotics
David Vernon
![]() A PR2 robot pours popcorn from a saucepan during a demonstration of cognitively-enabled robot manipulation using CRAM. Image courtesy of the Everyday Activity Science and Engineering (EASE) interdisciplinary research center at the University of Bremen, Germany.  
Course Description  |  Learning Objectives  |  Content  |  Lecture Notes  |  Course Textbook  |  Recommended Reading |  Software |  Resources  |  Acknowledgements Cognitive Robotics "The word cognition derives from the Latin verb cognosco, a composition of con (meaning related to) and gnosco (to know). Cognitive robotics, then, is the branch of robotics where knowledge plays a central role in supporting action selection, execution, and understanding. It focuses on designing and building robots that have the ability to learn from experience and from others, commit relevant knowledge and skills to memory, retrieve them as the context requires, and flexibly use this knowledge to select appropriate actions in the pursuit of their goals, while anticipating the outcome of those actions when doing so. Cognitive robots can use their knowledge to reason about their actions and the actions of those with whom they are interacting, and thereby modify their behavior to improve their overall long-term effectiveness. In short, cognitive robots are capable of flexible, context-sensitive action, knowing what they are doing and why they are doing it."
Sandini, G., Sciutti, A., and Vernon, D. (2021) "Cognitive Robotics", in Ang M., Khatib O., Siciliano B. (eds) Encyclopedia of Robotics. Springer, Berlin, Heidelberg.
Course Description
Since this course does not assume prior knowledge of robotics, it first covers the essentials of classical robotics and builds on these when covering the core topics in cognitive robotics.
After a general overview of the field, the course begins with the key elements of mobile robots, robot manipulators, and robot vision, using ROS (Robot Operating System), C/C++, and OpenCV. It then progresses to the main topics in artificial cognitive systems, including the different paradigms of cognitive science and cognitive architectures. These components form the foundation for the remainder of the course, involving a detailed study of the CRAM (Cognitive Robot Abstract Machine) cognitive architecture, building on ROS, and exploiting functional programming in Lisp to reason about and execute under-determined tasks in everyday activities.
The course covers both theory and practice, using robot simulators as well as low-cost robots and cameras for practical examples and exercises.
Learning Objectives
After completing this course, students should be able to:
Lecture Notes
Module 1: Overview of Cognitive Robotics
Module 2: The Robot Operating System (ROS)
Module 3: Mobile Robots
Module 4: Robot Manipulators
Module 5: Robot Vision
Module 6: Artificial Cognitive Systems
Module 7: Cognitive Architectures
Module 8: An Introduction to Functional Programming with Lisp
Module 9: The CRAM Plan Language
Module 10: Using Turtlesim with CRAM
Module 11: Cognition-enabled Robot Manipulation with CRAM
If you are an instructor and would like a copy of the complete set of PowerPoint slides, please contact me by email at david
At present, there is no textbook that covers all the material in this course. The recommended reading below provides partial coverage.
Recommended Reading
Cangelosi, A. and Asada, M. (Eds.), Cognitive Robotics, MIT Press, in press.
Corke, P. (2016). Robotics, Vision and Control, 2nd Edition, Springer.
O'Kane, J. M. (2018). A Gentle Introduction to ROS.
Paul, R. (1981). Robot Manipulators: Mathematics, Programming, and Control. MIT Press.
Szeliski, R. (2010). Computer Vision: Algorithms and Applications, Springer.
Vernon, D. (1991). Machine Vision: Automated Visual Inspection and Robot Vision, Prentice-Hall.
Vernon, D. Artificial Cognitive Systems, MIT Press, 2014.
Software Development Environment
Lecture 4 in Module 1 has detailed instructions for installing the software required for the various exercises in the course.
Resources
Additional material can be found on the Resources page of the IEEE Technical Committee for Cognitive Robotics website.
Acknowledgements
This course was developed over a four year period leading up to, during, and directly after the time I spent working at Carnegie Mellon University Africa in Rwanda. My thanks go to the students I taught there, several of whom have contributed directly or indirectly to the material. Their deep interest and searching questions made all the difference.
I wish to acknowledge the generous support I received for the preparation of aspects of this course from the IEEE Robotics and Automation Society under the program Creation of Educational Material in Robotics and Automation (CEMRA) 2020.
The material on CRAM (Cognitive Robot Abstract Machine) was derived from tutorials on the CRAM website. I am indebted to Michael Beetz and Gayane Kazhoyan for the time and effort they invested explaining CRAM and teaching me how to write CRAM Plan Language programs during my summer visits to the Institute for Artificial Intelligence, University of Bremen, and since joining Prof. Beetz's team in August 2020.
The module on mobile robots benefitted greatly from a course developed by Alessandro Saffiotti, Örebro University, Sweden, on Artificial Intelligence Techniques for Mobile Robots. I borrowed heavily from this material, while creating my own illustrations and diagrams.
The module on robot vision is a very short version of my course on applied computer vision which, in turn, drew inspiration from several sources, including courses given by Markus Vincze at Technische Universitat Wien, by Kenneth Dawson-Howe, Trinity College Dublin, and Francesca Odone, University of Genova. Many of the OpenCV examples are taken from Kenneth Dawson-Howe's book
A Practical Introduction to Computer Vision with OpenCV and the code samples.
All images and diagrams are either original or have their source credited. My apologies in advance for any unintended omissions. Technical drawings were produced in LaTeX using TikZ and the 3D Plot package.
David Vernon
|