Introduction to Cognitive Robotics

David Vernon
Institute for Artificial Intelligence
University of Bremen
Germany


Beta version
(Final version ~1/1/2021)
Work remaining: add material from Cognitive Robotics, A. Cangelosi and M. Asada, in press, add a simulator for the LynxMotion AL5D arm, add material on using LynxMotion AL5D arm and a Pepper robot with CRAM, links to support material and other courses, material targetted at the general public.


If you are an instructor and would like a copy of the PowerPoint files, please send a request to davidvernon.eu.
Suggestions for improvements are very welcome.

 


A PR2 robot sets a table during a demonstration of cognitively-enabled robot manipulation using CRAM.
Image courtesy of the EASE interdisciplinary research center at the University of Bremen, Germany.

 

Course Description  |  Learning Objectives  |  Content  |  Lecture Notes  |  Course Textbook  |  Recommended Reading |  Software |  Resources  |  Acknowledgements

Course Description

This course covers both the essentials of classical robotics and the core elements of cognitive robotics. After a general overview, it begins with mobile robots, robot manipulators, and robot vision before progressing to artificial cognitive systems and cognitive architectures. These components form the foundation for the remainder of the course, involving a detailed study of the CRAM (Cognitive Robot Abstract Machine) cognitive architecture, building on ROS (Robot Operating System), and exploiting functional, object-oriented, and logic programming to reason about and execute under-specified tasks in everyday activities.

The course covers both theory and practice, using robot simulators as well as low-cost robots and cameras for practical examples and exercises.

Support for the preparation of this course was provided by grant from the IEEE Robotics and Automation Society under the program Creation of Educational Material in Robotics and Automation (CEMRA) 2020.

Learning Objectives

After completing this course, students should be able to:

  1. Apply their knowledge of machine vision and robot kinematics to create computer programs that control mobile robots and robot arms, enabling the robots to recognize and manipulate objects and navigate their environments.

  2. Explain how a robot can be designed to exhibit cognitive goal-directed behaviour through the integration of computer models of vision, reasoning, learning, prospection, and social interaction.

  3. Use computer programs that realize limited instances of these faculties.

Course Content

Overview of Cognitive Robotics (Lectures 1 - 3)

  • Goals of the course.
  • Learning outcomes.
  • The emerging discipline of cognitive robotics.
  • Defining cognition.
  • The need for cognitive robots.
  • How cognitive robots operate.
  • Industrial requirements for cognitive robots.
  • Installation of software development environments for exercises and assignments.

Mobile Robots (Lectures 4 - 9)

  • Introduction to ROS (Robot Operating System).
  • Using the ROS Turtlebot simulator.
  • Writing ROS software.
  • Types of mobile robots.
  • The challenge of robot navigation
  • Relative inertial position estimation.
  • Kinematics of a two-wheel differential drive robot.
  • Relative odometry-based position estimation.
  • Absolute position estimation.
  • Closed-loop feedback control.
  • PID control.
  • Go-to-position and go-to-pose problems.
  • Divide and conquer controller.
  • MIMO controller.
  • Path tracking through waypoints
  • Finding a shortest path in a map.
  • Breadth-First Search (BFS).
  • Other search strategies.

Robot Manipulators (Lectures 10 - 13)

  • Robot programming.
  • Description of object pose with homogenous transformations.
  • Robot programming by frame-based task specification.
  • Convention for embedding T6 and end-effector frames.
  • Specifying orientation: roll, pitch, yaw; Euler angles.
  • Pick-and-place example of task-level robot programming.
  • Implementation for a Lynxmotion AL5D robot arm using Frame classes in C++.
  • Forward and inverse kinematics of a manipulator.
  • Denavit-Hartenberg reporesentation.
  • Kinematic specification of the LynxMotion AL5D arm.
  • Inverse kinematics of the LynxMotion AL5D arm.

Robot Vision (Lectures 14 - 20)

  • Computer vision.
  • Optics and sensors.
  • Image acquisition and image representation.
  • Image processing.
  • Introduction to OpenCV.
  • Segmentation.
  • Region-based approaches: feature-based thresholding, connected component analysis, graph cuts.
  • Boundary-based approaches: edge detection, boundary representations.
  • Image Analysis. Inspection, location, identification.
  • Feature extraction.
  • Classification: k-nearest neighbour, minimum distance, linear, maximum likelihood and Bayes classifiers.
  • Homogeneous coordinates and transformations.
  • Perspective transformation.
  • Camera model and inverse perspective transformation.
  • Camera calibration.

Artificial Cognitive Systems (Lectures 21 - 24)

  • Paradigms of cognitive science.
  • Learning and development.
  • Memory and prospection.
  • Knowledge and representation.
  • Social cognition.

Cognitive Architectures (Lectures 25 - 26)

  • Role and requirements.
  • desirable characteristics.
  • Core cognitive abilities.
  • Example cognitive architectures: Soar, Clarion, ISAC.
  • The Common Model of Cognition.

The CRAM Cognitive Architecture (Lectures 27 - 45)

  • Overview of the CRAM (Cognitive Robot Abstract Machine) cognitive architecture.
  • Programming paradigms.
  • Essentials of Common Lisp.
  • Emacs.
  • The CRAM plan language.
  • Description of object pose with quaternions in ROS.
  • CRAM and the ROS Turtlesim turtlebot simulator.
  • Creating a CRAM package.
  • Controlling a turtle.
  • Implementing simple plans to move a turtle.
  • Using Prolog for reasoning.
  • Creating motion designators for the TurtleSim.
  • Creating process modules.
  • Automatically choosing a process module for a motion.
  • Using location designators with TurtleSim.
  • Writing high-level plans for TurtleSim.
  • Implementing failure handling for TurtleSim.
  • Fetch-and-place CRAM plan for the PR2 robot.
  • Bullet Real-time Physics Simulation.
  • Error handling.
  • Recovery behaviors.

Lecture Notes

Lecture 1. Overview of cognitive robotics: component disciplines; defining cognition; the need for cognitive robots; how cognitive robots operate.
Lecture 2. Overview of cognitive robotics: industrial requirements.
Lecture 3. Overview of cognitive robotics: installation of software development environments for exercises and assignments.
Lecture 4. Mobile robots: introduction to ROS (Robot Operating System); the Turtlesim turtlebot simulator.
Lecture 5. Mobile robots: writing ROS software in C++.
Lecture 6. Mobile robots: types of mobile robots; the challenge of robot navigation; relative inertial position estimation.
Lecture 7. Mobile robots: relative odometry position estimation; kinematics of a two-wheel differential drive robot.
Lecture 8. Mobile robots: closed-loop control and PID control; the go-to-position and go-to-pose problems; divide-and-conquer and MIMO controllers.
Lecture 9. Mobile robots: finding a shortest path in a map; breadth-first search algorithm; other search approaches.
Lecture 10. Robot manipulators: robot programming; description of object pose with homogenous transformations.
Lecture 11. Robot manipulators: task specification; embedding T6 and end-effector frames; specifying orientation; roll, pitch, yaw; Euler angles.
Lecture 12. Robot manipulators: pick-and-place example; implementation for a Lynxmotion AL5D robot arm using Frame classes in C++.
Lecture 13. Robot manipulators: kinematics; Denavit-Hartenberg representation; kinematics and inverse kinematics of the LynxMotion AL5D arm.
Lecture 14. Robot vision: computer vision; optics and sensors; image acquisition; image representation; image processing; introduction to OpenCV.
Lecture 15. Robot vision: segmentation; region-based approaches; feature-based thresholding; graph cuts.
Lecture 16. Robot vision: segmentation; boundary-based approaches; edge detection.
Lecture 17. Robot vision: image analysis; feature extraction.
Lecture 18. Robot vision: k-nearest neighbour, minimum distance, linear, maximum likelihood and Bayes classifiers.
Lecture 19. Robot vision: perspective transformation; camera model; camera calibration.
Lecture 20. Robot vision: inverse perspective transformation; stereo vision; epipolar geometry.
Lecture 21. Artificial cognitive systems: the paradigms of cognitive science.
Lecture 22. Artificial cognitive systems: learning and development.
Lecture 23. Artificial cognitive systems: memory; prospection; internal simulation, the symbol grounding problem.
Lecture 24. Artificial cognitive systems: interaction, social cognition; theory of mind, instrumental helping, collaboration.
Lecture 25. Cognitive architectures: role and requirements; desirable characteristics; core cognitive abilities.
Lecture 26. Cognitive architectures: Example cognitive architectures: Soar, Clarion, ISAC; The Common Model of Cognition.
Lecture 27. The CRAM cognitive architecture: overview of CRAM (Cognitive Robot Abstract Machine).
Lecture 28. The CRAM cognitive architecture: Common Lisp - REPL, lists, structures, equality, conditionals, CONS, CAR, CDR, dotted and assoc-lists.
Lecture 29. The CRAM cognitive architecture: Common Lisp - functions, I/O, recursion, iteration, lambda and mapping functions, CLOS, inference.
Lecture 30. The CRAM cognitive architecture: Emacs.
Lecture 31. The CRAM cognitive architecture: CRAM plan language, fluents, concurrency, reasoning, exception handling.
Lecture 32. The CRAM cognitive architecture: CRAM plan language, designators, process modules.
Lecture 33. The CRAM cognitive architecture: Specifying pose in ROS, quaternions.
Lecture 34. The CRAM cognitive architecture: Turtlesim with CRAM; creating a CRAM package; controlling a turtle.
Lecture 35. The CRAM cognitive architecture: Turtlesim with CRAM; implementing plans to move a turtle.
Lecture 36. The CRAM cognitive architecture: using Prolog for reasoning.
Lecture 37. The CRAM cognitive architecture: creating motion designators for the TurtleSim.
Lecture 38. The CRAM cognitive architecture: creating process modules.
Lecture 39. The CRAM cognitive architecture: automatically choosing a process module for a motion.
Lecture 40. The CRAM cognitive architecture: using location designators with TurtleSim.
Lecture 41. The CRAM cognitive architecture: writing high-level plans for TurtleSim.
Lecture 42. The CRAM cognitive architecture: implementing failure handling for TurtleSim.
Lecture 43. The CRAM cognitive architecture: fetch-and-place CRAM plan for the PR2 robot; Bullet real-time physics simulation.
Lecture 44. The CRAM cognitive architecture: fetch-and-place CRAM plan for the PR2 robot; error handling and recovery looking in different places.
Lecture 45. The CRAM cognitive architecture: fetch-and-place CRAM plan for the PR2 robot; error handling and recovery using different arms.

Course Textbook

Future project.

Recommended Reading

Beetz, M., Mösenlechner, L., and Tenorth, M. (2010). "CRAM - A Cognitive Robot Abstract Machine for Everyday Manipulation in Human Environments", IEEE/RSJ International Conference on Intelligent Robots and Systems, 1012-1017.

Cangelosi, A. and Asada, M. (Eds.), Cognitive Robotics, MIT Press, in press.

Corke, P. (2016). Robotics, Vision and Control, 2nd Edition, Springer.

Paul, R. (1981). Robot Manipulators: Mathematics, Programming, and Control. MIT Press.

Szeliski, R. (2010). Computer Vision: Algorithms and Applications, Springer.

Vernon, D. (1991). Machine Vision: Automated Visual Inspection and Robot Vision, Prentice-Hall.

Vernon, D. Artificial Cognitive Systems, MIT Press, 2014.

Vernon, D. and Vincze, M. "Industrial Priorities for Cognitive Robotics", Proceedings of EUCognition 2016, Cognitive Robot Architectures, European Society for Cognitive Systems, Vienna, 8-9 December, 2016, R. Chrisley. V. C. Müller, Y. Sandamirskaya. M. Vincze (eds.), CEUR-WS Vol-1855, ISSN 1613-0073, pp. 6-9.

Software Development Environment

Lecture 3 has detailed instructions for installing the software required for the various exercises in the course.

Resources

Additional material can be found on the Resources page of the IEEE Technical Committee for Cognitive Robotics website.

Acknowledgements

The material on mobile robots is benefitted greatly from a course developed by Alessandro Saffiotti, Örebro University, Sweden, on Artificial Intelligence Techniques for Mobile Robots.

The material on CRAM (Cognitive Robot Abstract Machine) was derived from tutorials on the CRAM website. I am indebted to Michael Beetz and Gayane Kazhoyan for the time and effort they invested explaining CRAM and teaching me how to write CRAM Plan Language programs.

All images and diagrams are either original or have their source credited. Apologies in advance for any unintended omissions. Technical drawings were produced in LaTeX using TikZ and the 3D Plot package.

www.vernon.eu