Introduction to Cognitive Robotics

 Alpha version
(Beta version ~1/10/2020; Final version ~1/1/2021)

Work remaining: rewrite mobile robotics with new and improved figures, add material from Cognitive Robotics Handbook (A. Cangelosi and M. Asada, in press), add a simulator for the LynxMotion AL5D arm, add material on using a Pepper robot with CRAM, more figures, more software examples, links to support material and other courses, material targetted at the general public.


If you are an instructor and would like a copy of the PowerPoint files, please send a request to davidvernon.eu.
Suggestions for improvements are very welcome.

 


A PR2 robot cooks popcorn during a demonstration of cognitively-enabled robot manipulation using CRAM.
Image courtesy of the EASE interdisciplinary research center at the University of Bremen, Germany.

 

Course Description  |  Learning Objectives  |  Content  |  Lecture Notes  |  Course Textbook  |  Recommended Reading |  Software |  Resources

Course Description

This course covers both the essentials of classical robotics (mobile robots, robot arms for manipulation, and robot vision) and the principles of cognition (cognitive architectures, learning and development, prospection, memory, knowledge representation, internal simulation, and meta-cognition).

It brings these components together by working through some recent advances in robotics for everyday activities, and by including practical and detailed material based on the CRAM (Cognitive Robot Abstract Machine) cognitive architecture, incorporating the KnowRob knowledge base, building on ROS (Robot Operating System) and exploiting functional, object-oriented, and logic programming to reason about and execute under-specified tasks in everyday activities.

The course emphasizes both theory and practice and makes use of physical robots and robot simulators for visual sensing and actuation.

Support for the preparation of this course was provided by grant from the IEEE Robotics and Automation Society under the program Creation of Educational Material in Robotics and Automation (CEMRA) 2020.

Learning Objectives

After completing this course, students should be able to:

  1. Apply their knowledge of machine vision and robot kinematics to create computer programs that control mobile robots and robot arms, enabling the robots to recognize and manipulate objects and navigate their environments.

  2. Explain how a robot can be designed to exhibit cognitive goal-directed behaviour through the integration of computer models of vision, reasoning, learning, prospection, and social interaction.

  3. Use computer programs that realize limited instances of these faculties.

Course Content

Overview of Cognitive Robotics (Lectures 1 - 3)

  • Goals of the course.
  • Learning outcomes.
  • The emerging discipline of cognitive robotics.
  • Defining cognition.
  • The need for cognitive robots.
  • How cognitive robots operate.
  • Industrial requirements for cognitive robots.
  • Installation of software development environments for exercises and assignments.

Mobile Robotics (Lectures 4 - 9)

  • Types of mobile robots.
  • The challenge of robot navigation
  • Relative inertial position estimation
  • Relative odometry-based position estimation.
  • Kinematics of a two-wheel differential drive robot.
  • Absolute position estimation.
  • Closed-loop control.
  • Go-to-position and go-to-pose problems.
  • Divide and conquer controller.
  • MIMO controller.
  • Introduction to ROS (Robot Operating System).
  • Using the ROS Turtlebot simulator.
  • Writing ROS software.
  • Path planning.
  • The search problem in AI.
  • Path planning as a search problem.
  • Breadth-First Search (BFS): The Wavefront Algorithm.
  • Other search strategies.
  • Robot control architectures.

Robot Manipulators (Lectures 10 - 13)

  • Robot programming.
  • Description of object pose with homogenous transformations.
  • Robot programming by frame-based task specification.
  • Convention for embedding T6 and end-effector frames.
  • Specifying orientation: roll, pitch, yaw; Euler angles.
  • Pick-and-place example of task-level robot programming.
  • Implementation for a Lynxmotion AL5D robot arm using Frame classes in C++.
  • Forward and inverse kinematics of a manipulator.
  • Denavit-Hartenberg reporesentation.
  • Kinematic specification of the LynxMotion AL5D arm.
  • Inverse kinematics of the LynxMotion AL5D arm.

Robot Vision (Lectures 14 - 20)

  • Computer vision.
  • Optics and sensors.
  • Image acquisition and image representation.
  • Image processing.
  • Introduction to OpenCV.
  • Segmentation.
  • Region-based approaches: feature-based thresholding, connected component analysis, graph cuts.
  • Boundary-based approaches: edge detection, boundary representations.
  • Image Analysis. Inspection, location, identification.
  • Feature extraction.
  • Classification: k-nearest neighbour, minimum distance, linear, maximum likelihood and Bayes classifiers.
  • Homogeneous coordinates and transformations.
  • Perspective transformation.
  • Camera model and inverse perspective transformation.
  • Camera calibration.

Artificial Cognitive Systems (Lectures 21 - 24)

  • Paradigms of cognitive science.
  • Learning and development.
  • Memory and prospection.
  • Knowledge and representation.
  • Social cognition.

Cognitive Architectures (Lectures 25 - 26)

  • Role and requirements.
  • desirable characteristics.
  • Core cognitive abilities.
  • Example cognitive architectures: Soar, Clarion, ISAC.
  • The Common Model of Cognition.

The CRAM Cognitive Architecture (Lectures 27 - 45)

  • Overview of the CRAM (Cognitive Robot Abstract Machine) cognitive architecture.
  • Programming paradigms.
  • Essentials of Common Lisp.
  • Emacs.
  • The CRAM plan language.
  • Description of object pose with quaternions in ROS.
  • CRAM and the ROS Turtlesim turtlebot simulator.
  • Creating a CRAM package.
  • Controlling a turtle.
  • Implementing simple plans to move a turtle.
  • Using Prolog for reasoning.
  • Creating motion designators for the TurtleSim.
  • Creating process modules.
  • Automatically choosing a process module for a motion.
  • Using location designators with TurtleSim.
  • Writing high-level plans for TurtleSim.
  • Implementing failure handling for TurtleSim.
  • Fetch-and-place CRAM plan for the PR2 robot.
  • Bullet Real-time Physics Simulation.
  • Error handling.
  • Recovery behaviors.

Lecture Notes

Lecture 1. Overview of cognitive robotics: component disciplines; defining cognition; the need for cognitive robots; how cognitive robots operate.
Lecture 2. Overview of cognitive robotics: industrial requirements.
Lecture 3. Overview of cognitive robotics: installation of software development environments for exercises and assignments.
Lecture 4. Mobile Robotics: introduction to ROS (Robot Operating System); the Turtlesim turtlebot simulator.
Lecture 5. Mobile Robotics: writing ROS software in C++.
Lecture 6. Mobile Robotics: types of mobile robots; the challenge of robot navigation; relative inertial position estimation.
Lecture 7. Mobile Robotics: relative odometry position estimation; kinematics of a two-wheel differential drive robot.
Lecture 8. Mobile Robotics: closed-loop control; the go-to-position and go-to-pose problems; divide-and-conquer and MIMO controllers.
Lecture 9. Mobile robotics: path planning; BFS and the Wavefront algorithm; robot control architectures.
Lecture 10. Robot manipulators: robot programming; description of object pose with homogenous transformations.
Lecture 11. Robot manipulators: task specification; embedding T6 and end-effector frames; specifying orientation; roll, pitch, yaw; Euler angles.
Lecture 12. Robot manipulators: pick-and-place example; implementation for a Lynxmotion AL5D robot arm using Frame classes in C++.
Lecture 13. Robot manipulators: kinematics; Denavit-Hartenberg representation; kinematics and inverse kinematics of the LynxMotion AL5D arm.
Lecture 14. Robot vision: computer vision; optics and sensors; image acquisition; image representation; image processing; introduction to OpenCV.
Lecture 15. Robot vision: segmentation; region-based approaches; feature-based thresholding; graph cuts.
Lecture 16. Robot vision: segmentation; boundary-based approaches; edge detection.
Lecture 17. Robot vision: image analysis; feature extraction.
Lecture 18. Robot vision: k-nearest neighbour, minimum distance, linear, maximum likelihood and Bayes classifiers.
Lecture 19. Robot vision: perspective transformation; camera model; camera calibration.
Lecture 20. Robot vision: inverse perspective transformation; stereo vision; epipolar geometry.
Lecture 21. Artificial cognitive systems: the paradigms of cognitive science.
Lecture 22. Artificial cognitive systems: learning and development.
Lecture 23. Artificial cognitive systems: memory; prospection; internal simulation, the symbol grounding problem.
Lecture 24. Artificial cognitive systems: interaction, social cognition; theory of mind, instrumental helping, collaboration.
Lecture 25. Cognitive architectures: role and requirements; desirable characteristics; core cognitive abilities.
Lecture 26. Cognitive architectures: Example cognitive architectures: Soar, Clarion, ISAC; The Common Model of Cognition.
Lecture 27. The CRAM cognitive architecture: overview of CRAM (Cognitive Robot Abstract Machine.
Lecture 28. The CRAM cognitive architecture: Common Lisp - REPL, lists, structures, equality, conditionals, CONS, CAR, CDR, dotted and assoc-lists.
Lecture 29. The CRAM cognitive architecture: Common Lisp - functions, I/O, recursion, iteration, lambda and mapping functions, CLOS, inference.
Lecture 30. The CRAM cognitive architecture: Emacs.
Lecture 31. The CRAM cognitive architecture: CRAM plan language, fluents, concurrency, reasoning, exception handling.
Lecture 32. The CRAM cognitive architecture: CRAM plan language, designators, process modules.
Lecture 33. The CRAM cognitive architecture: Specifying pose in ROS, quaternions.
Lecture 34. The CRAM cognitive architecture: Turtlesim with CRAM; creating a CRAM package; controlling a turtle.
Lecture 35. The CRAM cognitive architecture: Turtlesim with CRAM; implementing plans to move a turtle.
Lecture 36. The CRAM cognitive architecture: using Prolog for reasoning.
Lecture 37. The CRAM cognitive architecture: creating motion designators for the TurtleSim.
Lecture 38. The CRAM cognitive architecture: creating process modules.
Lecture 39. The CRAM cognitive architecture: automatically choosing a process module for a motion.
Lecture 40. The CRAM cognitive architecture: using location designators with TurtleSim.
Lecture 41. The CRAM cognitive architecture: writing high-level plans for TurtleSim.
Lecture 42. The CRAM cognitive architecture: implementing failure handling for TurtleSim.
Lecture 43. The CRAM cognitive architecture: fetch-and-place CRAM plan for the PR2 robot; Bullet real-time physics simulation.
Lecture 44. The CRAM cognitive architecture: fetch-and-place CRAM plan for the PR2 robot; error handling and recovery looking in different places.
Lecture 45. The CRAM cognitive architecture: fetch-and-place CRAM plan for the PR2 robot; error handling and recovery using different arms.

Course Textbook

In preparation.

Recommended Reading

Beetz, M., Mösenlechner, L., and Tenorth, M. (2010). "CRAM - A Cognitive Robot Abstract Machine for Everyday Manipulation in Human Environments", IEEE/RSJ International Conference on Intelligent Robots and Systems, 1012-1017.

Cangelosi, A. and Asada, M. (Eds.), Cognitive Robotics Handbook, MIT Press, in press.

Corke, P. (2016). Robotics, Vision and Control, 2nd Edition, Springer.

Paul, R. (1981). Robot Manipulators: Mathematics, Programming, and Control. MIT Press.

Szeliski, R. (2010). Computer Vision: Algorithms and Applications, Springer.

Vernon, D. (1991). Machine Vision: Automated Visual Inspection and Robot Vision, Prentice-Hall.

Vernon, D. Artificial Cognitive Systems, MIT Press, 2014.

Vernon, D. and Vincze, M. "Industrial Priorities for Cognitive Robotics", Proceedings of EUCognition 2016, Cognitive Robot Architectures, European Society for Cognitive Systems, Vienna, 8-9 December, 2016, R. Chrisley. V. C. Müller, Y. Sandamirskaya. M. Vincze (eds.), CEUR-WS Vol-1855, ISSN 1613-0073, pp. 6-9.

Software Development Environment

Lecture 3 has detailed instructions for installing the software required for the various exercises in the course.

Resources

Additional material can be found on the Resources page of the IEEE Technical Committee for Cognitive Robotics website.

www.vernon.eu