An open course for the IEEE Robotics and Automation Society Technical Committee for Cognitive Robotics
Instructors: send an email to request the complete set of PowerPoint slides. Sample assignments, test questions, support notes, and code will be available soon.
A PR2 robot pours popcorn from a saucepan (left) and sets a table (right) during demonstrations of cognitively-enabled robot manipulation using CRAM.
Image courtesy of the Everyday Activity Science and Engineering (EASE) interdisciplinary research center at the University of Bremen, Germany.
Course Description | Learning Objectives | Content | Lecture Notes | Course Textbook | Recommended Reading | Software | Resources | Acknowledgements
Cognitive Robotics
"The word cognition derives from the Latin verb cognosco, a composition of con (meaning related to) and gnosco (to know). Cognitive robotics, then, is the branch of robotics where knowledge plays a central role in supporting action selection, execution, and understanding.
It focuses on designing and building robots that have the ability to learn from experience and from others, commit relevant knowledge and skills to memory, retrieve them as the context requires, and flexibly use this knowledge to select appropriate actions in the pursuit of their goals, while anticipating the outcome of those actions when doing so.
Cognitive robots can use their knowledge to reason about their actions and the actions of those with whom they are interacting, and thereby modify their behavior to improve their overall long-term effectiveness.
In short, cognitive robots are capable of flexible, context-sensitive action, knowing what they are doing and why they are doing it."
Sandini, G., Sciutti, A., and Vernon, D. (2021) "Cognitive Robotics", in Ang M., Khatib O., Siciliano B. (eds), Encyclopedia of Robotics . Springer, Berlin, Heidelberg.
Course Description
No prior knowledge of robotics is assumed and we first cover the essentials of classical robotics before exploring the core topics in cognitive robotics.
After a general overview of the field, the course begins with the key elements of mobile robots, robot manipulators, and robot vision, using ROS (Robot Operating System), C/C++, and OpenCV. It then progresses to the main topics in artificial cognitive systems, including the different paradigms of cognitive science and cognitive architectures. These components form the foundation for the remainder of the course, involving a detailed study of the CRAM (Cognitive Robot Abstract Machine) cognitive architecture, building on ROS, and exploiting functional programming in Lisp to reason about and execute under-determined tasks in everyday activities.
The course covers both theory and practice, using robot simulators as well as low-cost robots and cameras for practical examples and exercises.
Learning Objectives
After completing this course, students should be able to:
Lecture Notes
Module 1: Overview of Cognitive Robotics
Module 2: The Robot Operating System (ROS)
Module 3: Mobile Robots
Module 4: Robot Manipulators
Module 5: Robot Vision
Module 6: Artificial Cognitive Systems
Module 7: Cognitive Architectures
Module 8: An Introduction to Functional Programming with Lisp
Module 9: The CRAM Plan Language
Module 10: Using Turtlesim with CRAM
Module 11: Cognition-enabled Robot Manipulation with CRAM
Instructors: send an email to request the complete set of PowerPoint slides.
If you already have a copy, check the Version History to make sure you have the most recent version. If you don't, please send an email to request it.
At present, there is no textbook that covers all the material in this course. The recommended reading below provides partial coverage.
Recommended Reading
Cangelosi, A. and Asada, M. (2021), Eds. , Cognitive Robotics, MIT Press, in press.
Corke, P. (2017). Robotics, Vision and Control, 2nd Edition, Springer.
O'Kane, J. M. (2018). A Gentle Introduction to ROS.
Paul, R. (1981). Robot Manipulators: Mathematics, Programming, and Control. MIT Press.
Szeliski, R. (2010). Computer Vision: Algorithms and Applications, Springer.
Vernon, D. (1991). Machine Vision: Automated Visual Inspection and Robot Vision, Prentice-Hall.
Vernon, D. Artificial Cognitive Systems, MIT Press, 2014.
Software Development Environment
Lecture 4 in Module 1 has detailed instructions for installing the software required for the various exercises in the course.
The simulator for the Lynxmotion AL5D robot manipulator for Module 4 and the ROS example programs for Modules 2, 3, 4, and 5 are available on the course GitHub repository github.com/cognitive-robotics-course.
Resources
Additional material can be found on the Resources page of the IEEE Technical Committee for Cognitive Robotics website.
Acknowledgements
I wish to acknowledge with thanks the support I received from the IEEE Robotics and Automation Society under the program Creation of Educational Material in Robotics and Automation (CEMRA) 2020.
This course was developed over a four year period leading up to, during, and directly after the time I spent working at Carnegie Mellon University Africa in Rwanda. My thanks go to the students I taught there, several of whom have contributed directly or indirectly to the material. Their deep interest and searching questions made all the difference.
Special thanks go to Vinny Adjibe, Abrham Gebreselasie, Innocent Mukoki, Ribeus Munezero, and Timothy Odonga for their work developing material and tools for the course during their Summer 2020 CMU-Africa internships, and to Derrick Odonkor, who validated the material and developed the Pepper tutorial for CRAM during his internship at the Institute for Artificial Intelligence, University of Bremen, in the Spring of 2020. Derrick Odonkor's intership was mostly funded by the IEEE Robotics and Automation Society under the program Creation of Educational Material in Robotics and Automation (CEMRA) 2020.
My thanks go to Carnegie Mellon University Africa for its generous support in sponsoring the internships and teaching assistantships, and supplementing the funds provided by the IEEE Robotics and Automation Society to support Derek Odonkor's visit to the University of Bremen.
The module on mobile robots benefitted greatly from a course developed by Alessandro Saffiotti, Örebro University, Sweden, on Artificial Intelligence Techniques for Mobile Robots. I borrowed heavily from this material, whilst creating my own illustrations and diagrams.
The module on robot vision is a very short version of my course on applied computer vision which, in turn, drew inspiration from several sources, including courses given by Kenneth Dawson-Howeat Trinity College Dublin, Kris Kitani at Carnegie Mellon University, Francesca Odone at University of Genova, and Markus Vincze at Technische Universitat Wien. Many of the OpenCV examples are taken from Kenneth Dawson-Howe's book
A Practical Introduction to Computer Vision with OpenCV and the code samples.
The material on CRAM (Cognitive Robot Abstract Machine) was derived from tutorials on the CRAM website. I am indebted to Michael Beetz and Gayane Kazhoyan for the time and effort they invested explaining CRAM and teaching me how to write CRAM Plan Language programs during my summer visits to the Institute for Artificial Intelligence, University of Bremen, and since joining Prof. Beetz's team in August 2020. Thank you to the Institute for Artificial Intelligence, University of Bremen, for hosting the three-month visit by Derrick Odonkor.
All images and diagrams are either original or have their source credited. My apologies in advance for any unintended omissions. Technical drawings were produced in LaTeX using TikZ and the 3D Plot package.
David Vernon, Institute for Artificial Intelligence, University of Bremen, Germany.
Lecture 1. Component disciplines; the nature of cognition; definition of cognitive robotics.
Lecture 2. Operation of a cognitive robot; reasons for studying cognitive robotics.
Lecture 3. Industrial requirements; resources.
Lecture 4. Installation of software development environments for exercises.
Lecture 1. Introduction to ROS (Robot Operating System); the Turtlesim turtlebot simulator.
Lecture 2. Writing ROS software in C++: publishers, subscribers, services.
Lecture 1. Types of mobile robots; the challenge of robot navigation; relative position estimation using inertial sensors.
Lecture 2. Relative position estimation using odometry; kinematics of a two-wheel differential drive robot.
Lecture 3. Absolute position estimation.
Lecture 4. Closed-loop control and PID control; the go-to-position problem; divide-and-conquer controller.
Lecture 5. The go-to-position and go-to-pose problems; MIMO controller.
Lecture 6. Finding a shortest path in a map; breadth-first search algorithm; other search approaches.
Lecture 1. Robot programming; coordinate frames of reference and homogenous transformations.
Lecture 2. Object pose specification with homogenous transformations and vectors & quaternions.
Lecture 3. Robot programming by frame-based task specification.
Lecture 4. Pick-and-place example of task-level robot programming.
Lecture 5. Implementation of the pick-and-place example for a Lynxmotion AL5D robot arm using the Frame class in C++.
Lecture 6. Kinematics; Denavit-Hartenberg representation; kinematics and inverse kinematics of the LynxMotion AL5D arm.
Lecture 1. Computer vision; optics and sensors; image acquisition; image representation; image processing.
Lecture 2. Introduction to OpenCV.
Lecture 3. Segmentation; region-based approaches; feature-based thresholding; graph cuts.
Lecture 4. Segmentation; boundary-based approaches; edge detection.
Lecture 5. Image analysis; feature extraction.
Lecture 6. K-nearest neighbour, minimum distance, linear, maximum likelihood and Bayes classifiers.
Lecture 7. Perspective transformation; camera model; camera calibration.
Lecture 8. Inverse perspective transformation; stereo vision; epipolar geometry.
Lecture 1. The paradigms of cognitive science.
Lecture 2. Learning and development.
Lecture 3. Memory and prospection; internal simulation; the symbol grounding problem.
Lecture 4. Interaction, social cognition; theory of mind; instrumental helping; collaboration.
Lecture 1. Role and requirements; desirable characteristics; core cognitive abilities.
Lecture 2. Example cognitive architectures: Soar, ACT-R, CLARION, ICARUS, BBD, ISAC.
Lecture 3. The CRAM cognitive architecture: design principles; structure.
Lecture 4. The CRAM cognitive architecture: operation.
Lecture 1. Common Lisp - REPL, lists, structures, equality, conditionals, CONS, CAR, CDR, dotted and assoc-lists.
Lecture 2. Common Lisp - functions, I/O, recursion, iteration, lambda and mapping functions, CLOS, inference.
Lecture 3. Emacs.
Lecture 1. Fluents, concurrency, reasoning, exception handling.
Lecture 2. Designators, process modules.
Lecture 1. Pose specification in ROS; Transform Library (TF).
Lecture 2. Creating a CRAM package; controlling a turtle.
Lecture 3. Implementing plans to move a turtle.
Lecture 4. Using Prolog for reasoning.
Lecture 5. Creating motion designators for the TurtleSim.
Lecture 6. Creating process modules.
Lecture 7. Automatically choosing a process module for a motion.
Lecture 8. Using location designators with TurtleSim.
Lecture 9. Writing high-level plans for TurtleSim.
Lecture 10. Implementing failure handling for TurtleSim.
Lecture 1. Fetch-and-place CRAM plan for the PR2 robot; Bullet real-time physics simulation.
Lecture 2. Fetch-and-place CRAM plan for the PR2 robot; error handling and recovery looking in different places.
Lecture 3. Fetch-and-place CRAM plan for the PR2 robot; error handling and recovery using different arms.
March 2021.