|
|
| (115 intermediate revisions by the same user not shown) |
| Line 1: |
Line 1: |
| − | __NOTOC__
| + | The following pages provide support material for the course on [http://vernon.eu/cognitive_robotics/index.htm Cognitive Robotics]. |
| − | <span style="color:#AB0000">XX-YYY</span>
| + | |
| | | | |
| − | <span style="color:#AB0000">Course discipline: TBD</span>
| + | == General Resources == |
| | + | [[Cognitive Robotics Resources]] |
| | | | |
| − | <span style="color:#AB0000">Elective</span>
| + | == Software == |
| | + | <!-- [[CORO Software Development Environment]] <BR> --> |
| | + | [[Cognitive Robotics - Software Installation Guide | Software Installation Guide]] <BR> |
| | + | [https://github.com/cognitive-robotics-course GitHub Repository] |
| | | | |
| − | <span style="color:#AB0000">Units: 12 (could also be run as two seven-week independent 6 unit minis, running consecutively)</span>
| + | == Resources for Modules 9 - 12 on Implementing Robot Control with CRAM == |
| − | | + | Module 11: [[Zero Prerequisites Demo Tutorial: Simple Fetch and Place]] CRAM demo tutorial, including code to be copied to <code>pick-and-place.lisp</code><BR> |
| − | <span style="color:#AB0000">Lecture/Lab/Rep hours/week: 3 hours lecture/week, 3 hours lab/week</span>
| + | Module 12: [[Lynxmotion AL5D Pick and Place with CRAM]] using the Gazebo Lynxmotion AL5D simulator<BR> |
| − | | + | Legacy version of Module 11: [[Creating a CRAM Package for the Simple Mobile Manipulation Plan]] CRAM intermediate tutorial, including code to be copied to <code>simple-mobile-manipulation-plan.lisp</code> |
| − | <span style="color:#AB0000">Semester: Spring</span>
| + | |
| − | | + | |
| − | <span style="color:#AB0000">Pre-requisites: Programming skills</span>
| + | |
| − | | + | |
| − | ==<span style="color:#AB0000">Course description:</span> ==
| + | |
| − | Cognitive robotics is an emerging discipline that draws on robotics, artificial intelligence, and cognitive science. It often exploits models based on biological cognition.
| + | |
| − | | + | |
| − | There are at least two reasons why having a cognitive ability is useful in robotics:
| + | |
| − | | + | |
| − | # It allows the robot to work autonomously in challenging environments, adapting to changes and unforeseen situations, and anticipating outcomes when selecting the actions it will perform.
| + | |
| − | # It facilitates interaction with people. Humans have a strong preference for interaction with other cognitive agents so being able to exhibit a capacity for cognition encourages human robot interaction. Conversely, a cognitive ability provides the robot with the ability to infer the goals and intentions of the person it is interacting with and thereby allows it to do so in safe and helpful manner.
| + | |
| − | | + | |
| − | Cognitive robots achieve their goals by perceiving their environment, paying attention to the events that matter, planning what to do, anticipating the outcome of their own actions and the actions of other agents (people and other robots), and learning from the resultant interaction. They deal with the inherent uncertainty of natural environments by continually learning, reasoning, and sharing their knowledge.
| + | |
| − | | + | |
| − | A key feature of cognitive robotics is its focus on predictive capabilities to augment and compensate for perceptual capacities. Also, by being able to view the world from someone else’s perspective, a cognitive robot can anticipate that person’s intended actions and needs.
| + | |
| − | | + | |
| − | In cognitive robotics, the robot body can be more than just an instrument for physical manipulation or locomotion: it can also be a component of the cognitive process. In the particular case of humanoid robotics, the robot’s physical morphology, kinematics, and dynamics, as well as the environment in which it is operating, can help it to achieve its key characteristic of adaptive anticipatory interaction by mirroring the actions of the person it is interacting with.
| + | |
| − | | + | |
| − | This course provides introduces the key elements of cognitive robotics, touching on all of these issues. In doing so, it emphasizes both theory and practice and makes extensive use of physical robots, both mobile robots and manipulator arms, as well as different sensor technologies including RGB-D cameras.
| + | |
| − | | + | |
| − | ==<span style="color:#AB0000">Learning objectives:</span> == | + | |
| − | The primary goal of this course is provide students with an intensive treatment of a cross-section of the key elements of robotics, robot vision, AI, and cognitive science. Students will learn about the fundamentals of 2D and 3D visual sensing, focussing on the some essential techniques for mobile robots and robot arms. They will then learn about the kinematics and inverse kinematics of mobile robots, addressing locomotion, mapping, and path planning, as well as robot arm kinematics, manipulation, and programming. Based on these foundations, students will progress quickly to cover the topics that gives cognitive robotics its special focus, including reasoning, cognitive architectures, learning and development, memory, attention, prospection by internal simulation, and social interaction.
| + | |
| − | | + | |
| − | ==<span style="color:#AB0000">Outcomes:</span> ==
| + | |
| − | After completing this course, students should be able to:
| + | |
| − | * Apply their knowledge of machine vision and robot kinematics to create computer programs that control mobile robots and robot arms, enabling the robots to recognize and manipulate objects and navigate their environments.
| + | |
| − | * Explain how a robot can be designed to exhibit cognitive goal-directed behaviour through the integration of computer models of visual attention, reasoning, learning, prospection, and social interaction.
| + | |
| − | * Create computer programs that realize limited instances of each of these models.
| + | |
| − | | + | |
| − | ==<span style="color:#AB0000">Content details:</span> ==
| + | |
| − | (For a detailed plan of lectures and labs, see [[Cognitive Robotics Lectures and Labs]].
| + | |
| − | | + | |
| − | The course will cover the following topics:
| + | |
| − | * Cognitive robotics
| + | |
| − | * Robot vision
| + | |
| − | * Mobile robots
| + | |
| − | * Robot arms
| + | |
| − | * Constraint-based reasoning for robotics
| + | |
| − | * Cognitive architectures
| + | |
| − | * Learning and development
| + | |
| − | * Memory and Prospection
| + | |
| − | * Internal simulation
| + | |
| − | * Visual attention
| + | |
| − | * Social interaction
| + | |
| − | | + | |
| − | The detailed content for each of these topics follows.
| + | |
| − | | + | |
| − | <span style="color:#AB0000">Cognitive robotics</span><BR>
| + | |
| − | * Introduction to AI and cognition in robotics.
| + | |
| − | * Industrial requirements.
| + | |
| − | * Artificial cognitive systems.
| + | |
| − | * Cognitivist, emergent, and hybrid paradigms in cognitive science.
| + | |
| − | * Autonomy.
| + | |
| − | | + | |
| − | <span style="color:#AB0000">Robot vision </span><BR>
| + | |
| − | * Optics, sensors, and image formation.
| + | |
| − | * Image acquisition.
| + | |
| − | * Image filtering.
| + | |
| − | * Edge detection.
| + | |
| − | * Segmentation.
| + | |
| − | * Hough transform: line, circle, and generalized transform; extension to codeword features.
| + | |
| − | * Colour-based segmentation.
| + | |
| − | * Object recognition.
| + | |
| − | * Interest point operators.
| + | |
| − | * Gradient orientation histogram - SIFT descriptor.
| + | |
| − | * Colour histogram intersection.
| + | |
| − | * Haar features, boosting, face detection.
| + | |
| − | * Homogeneous coordinates and transformations.
| + | |
| − | * Perspective transformation.
| + | |
| − | * Camera model and inverse perspective transformation.
| + | |
| − | * Stereo vision.
| + | |
| − | * Epipolar geometry.
| + | |
| − | * Structured light & RGB-D cameras.
| + | |
| − | * Visual attention.
| + | |
| − | * Plane pop-out.
| + | |
| − | * RANSAC.
| + | |
| − | * Differential geometry.
| + | |
| − | * Surface normals and Gaussian sphere.
| + | |
| − | * Point clouds.
| + | |
| − | * 3D descriptors.
| + | |
| − | | + | |
| − | <span style="color:#AB0000">Mobile robots</span><BR> | + | |
| − | * Differential drive locomotion.
| + | |
| − | * Forward and inverse kinematics.
| + | |
| − | * Holonomic and non-holonomic constraints.
| + | |
| − | * Cozmo mobile robot.
| + | |
| − | * Map representation.
| + | |
| − | * Probabilistic map-based localization.
| + | |
| − | * Landmark-based localization.
| + | |
| − | * SLAM: simulataneous localization and mapping.
| + | |
| − | * Extended Kalman Filter (EKF) SLAM.
| + | |
| − | * Visual SLAM.
| + | |
| − | * Particle filter SLAM.
| + | |
| − | * Graph search path planning.
| + | |
| − | * Potential field path planning.
| + | |
| − | * Navigation. Obstacle avoidance.
| + | |
| − | * Object search.
| + | |
| − | | + | |
| − | <span style="color:#AB0000">Robot arms</span><BR>
| + | |
| − | * Homogeneous transformations.
| + | |
| − | * Frame-based pose specification.
| + | |
| − | * Denavit-Hartenberg specifications.
| + | |
| − | * Robot kinematics.
| + | |
| − | * Analytic inverse kinematics.
| + | |
| − | * Iterative approaches.
| + | |
| − | * Kinematic structure learning.
| + | |
| − | * Kinematics structure correspondences.
| + | |
| − | * Robot manipulation.
| + | |
| − | * Frame-based task specification.
| + | |
| − | * Vision-based pose estimation.
| + | |
| − | * Language-based programming.
| + | |
| − | * Programming by demonstration.
| + | |
| − | | + | |
| − | <span style="color:#AB0000">Constraint-based reasoning for robotics</span><BR>
| + | |
| − | * Constraint satisfaction problems (CSP).
| + | |
| − | * Meta-Constraints and Meta-CSP reasoning.
| + | |
| − | * Planning and navigation with multiple mobile robots.
| + | |
| − | | + | |
| − | <span style="color:#AB0000">Cognitive architectures</span><BR>
| + | |
| − | * Role and requirements.
| + | |
| − | * Cognitive architecture schemas.
| + | |
| − | * Example cognitive architectures including Soar, ACT-R, Clarion, LIDA, and ISAC.
| + | |
| − | * CRAM: Cognitive Robot Abstract Machine.
| + | |
| − | * CRAM Plan Language (CPL).
| + | |
| − | * KnowRob knowledge processing and reasoning.
| + | |
| − | | + | |
| − | <span style="color:#AB0000">Learning and development </span><BR>
| + | |
| − | * Supervised, unsupervised, and reinforcement learning.
| + | |
| − | * Hebbian learning.
| + | |
| − | * Predictive sequence learning (PSL).
| + | |
| − | * Cognitive development in humans and robots.
| + | |
| − | * Value systems for developmental and cognitive robots.
| + | |
| − | | + | |
| − | <span style="color:#AB0000">Memory and Prospection</span><BR>
| + | |
| − | *Declarative vs. procedural memory
| + | |
| − | *Semantic memory
| + | |
| − | *Episodic memory
| + | |
| − | | + | |
| − | <span style="color:#AB0000">Internal simulation </span><BR>
| + | |
| − | * Forward and inverse models.
| + | |
| − | * Internal simulation hypothesis.
| + | |
| − | * Internal simulation with PSL
| + | |
| − | * HAMMER cognitive architecture
| + | |
| − | | + | |
| − | <span style="color:#AB0000">Visual attention</span><BR>
| + | |
| − | * Visual attention.
| + | |
| − | * Spatial attention vs. selective attention
| + | |
| − | * Saliency functions.
| + | |
| − | * Selective Tuning.
| + | |
| − | * Overt attention.
| + | |
| − | * Inhibition of return.
| + | |
| − | * Habituation.
| + | |
| − | * Top-down attention
| + | |
| − | | + | |
| − | <span style="color:#AB0000">Social interaction</span><BR>
| + | |
| − | * Joint action.
| + | |
| − | * Joint attention.
| + | |
| − | * Shared intention.
| + | |
| − | * Shared goals.
| + | |
| − | * Perspective taking.
| + | |
| − | * Theory of mind.
| + | |
| − | * Action and intention recognition.
| + | |
| − | * Learning from demonstration.
| + | |
| − | * Humanoid robotics.
| + | |
| − | | + | |
| − | ==<span style="color:#AB0000">Faculty:</span> ==
| + | |
| − | David Vernon
| + | |
| − | | + | |
| − | ==<span style="color:#AB0000">Delivery:</span> ==
| + | |
| − | Face-to-face
| + | |
| − | | + | |
| − | ==<span style="color:#AB0000">Students assessment:</span> ==
| + | |
| − | (To be confirmed)
| + | |
| − | | + | |
| − | Lab assignments 50%
| + | |
| − | Mid-term exam 20%
| + | |
| − | Final exam 30%
| + | |
| − | | + | |
| − | ==<span style="color:#AB0000">Recommended reading:</span> ==
| + | |
| − | [https://www.researchgate.net/publication/222219268_A_survey_of_robot_learning_from_demonstration Argall, B. D., Chernova, S., Veloso,M., and Browning, B. (2009). "A survey of robot learning from demonstration". Robotics and Autonomous Systems, 57:469–483, 2009. ]
| + | |
| − | | + | |
| − | Beetz, M., Mösenlechner, L., and Tenorth, M. (2010). "CRAM - A Cognitive Robot Abstract Machine for Everyday Manipulation in Human Environments", IEEE/RSJ International Conference on Intelligent Robots and Systems, 1012-1017.
| + | |
| − | | + | |
| − | [http://calinon.ch/papers/Billard-handbookOfRobotics.pdf Billard, A., Calinon, S., Dillmann, R. and Schaal, S. (2008). "Robot programming by demonstration". In Springer Handbook of Robotics, pages 1371–1394.]
| + | |
| − | | + | |
| − | [http://exactdownload.com/download.php?filename=Agents%20and%20Artificial%20Intelligence%20(Joaquim%20Filipe,%20Ana%20Fred,%20Bernadette%20Sharp,%20(Ed.),%202011)%20-%20Conference%20Book.pdf Billing, E., Hellström, T., and Janlert, L-E. (2011). "Predictive Learning from Demonstration", in ICAART 2010, CCIS 129, Filipe, J., Fred, A., and Sharp, B. (Eds.), pp. 186-200. ]
| + | |
| − | | + | |
| − | [http://journal.frontiersin.org/article/10.3389/frobt.2016.00009/full Billing, E. Svensson, H., Lowe, R. and Ziemke, T. (2016). "Finding Your Way from the Bed to the Kitchen: Reenacting and Recombining Sensorimotor Episodes Learned from Human Demonstration", Frontiers in Robotics and AI, Vol. 3. ]
| + | |
| − | | + | |
| − | [http://ilab.usc.edu/publications/doc/Borji_Itti13pami.pdf Borji, A. and Itti, L. (2013). "State-of-the-Art in Visual Attention Modeling", IEEE Transactions on Pattern Analysis and Machine intelligence, Vol. 35, No. 1, pp. 185-207. ]
| + | |
| − | | + | |
| − | Cangelosi, A. and Schlesinger, M. (2015). Developmental Robotics: From Babies to Robots. Cambridge, MA: MIT Press.
| + | |
| − | | + | |
| − | [http://iccm-conference.org/2013-proceedings/papers/0048/paper0048.pdf Chella, A., Kurup, U., Laird, J., Trafton, G., Vinokurov, J., Chandrasekaran, B. (2013). "The Challenge of Robotics for Cognitive Architectures", Proc. 12th International Conference on Cognitive Modelling.] | + | |
| − | | + | |
| − | Dechter, R. (2003). Constraint Programming, Morgan Kaufman.
| + | |
| − | | + | |
| − | Demiris, Y. and Khadhouri, B. (2006). "Hierarchical attentive multiple models for execution and recognition (HAMMER). Robotics and Autonomous Systems, 54:361–369.
| + | |
| − | | + | |
| − | [http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA323194 Harmon, M. and Harmon, S. (1997). Reinforcement Learning: A Tutorial] | + | |
| − | | + | |
| − | Kragic, D. and Vincze, M. (2010). "Vision for Robotics", Foundation and Trends in Robotics, Vol 1, No 1, pp 1–78.
| + | |
| − | | + | |
| − | [https://www.researchgate.net/publication/220233671_Developmental_robotics_A_survey Lungarella, M., Metta, G., Pfeifer, R. and Sandini, G. (2003). "Developmental Robotics: A Survey", Connection Science, 17, pp. 151-190. ]
| + | |
| − | | + | |
| − | [https://pdfs.semanticscholar.org/7a29/f09f4b5a71496f658dffb5570e2dcbbb1f39.pdf Mansouri, M. and Pecora, F. "More knowledge on the table: Planning with space, time and resources for robots", Proc. IEEE Int. Conf. on Robotics and Automation (ICRA). ]
| + | |
| − | | + | |
| − | Merrick, K (2016). "Value systems for developmental cognitive robotics: a survey", Cognitive Systems Research, in press.]
| + | |
| − | | + | |
| − | Paul, R. (1981). Robot Manipulators: Mathematics, Programming, and Control. MIT Press.
| + | |
| − | | + | |
| − | Russell, S. and Norvig. P. (2014). Artificial Intelligence: A Modern Approach, Pearson Education.
| + | |
| − | | + | |
| − | [https://spiral.imperial.ac.uk:8443/handle/10044/1/12540 Sarabia, M., Ros, R. & Demiris, Y. (2011) Towards and open-source social middleware for humanoid robots in Proceedings of the IEEE-RAS International Conference on Humanoid Robots, pp.670-675.]
| + | |
| − | | + | |
| − | [https://hrilab.tufts.edu/publications/scheutzetal13acs.pdf Scheutz, M., Harris, J., Schermerhorn, P. (2013). Systematic Integration of Cognitive and Robotic Architectures, Advances in Cognitive Systems, Vol. 2, pp. 277-296.]
| + | |
| − | | + | |
| − | [http://www.sts.rpi.edu/~rsun/sun.expert01.pdf Sun, R. and C. L. Giles (2001). "Sequence Learning: From Recognition and Prediction to Sequential Decision Making", IEEE Intelligent Systems and Their Applications, Vol. 16, No. 4, pp. 67-70.]
| + | |
| − | | + | |
| − | [http://szeliski.org/Book/drafts/SzeliskiBook_20100903_draft.pdf Szeliski, R. (2010). Computer Vision: Algorithms and Applications, Springer. ]
| + | |
| − | | + | |
| − | [http://homepages.inf.ed.ac.uk/rbf/BOOKS/VERNON/vernon.htm Vernon, D. (1991). ''Machine Vision: Automated Visual Inspection and Robot Vision'', Prentice-Hall, 1991. ]
| + | |
| − | | + | |
| − | [http://vernon.eu/publications/14_Vernon_Artificial_Cognitive_Systems.pdf Vernon, D. (2014). Artificial Cognitive Systems - A Primer, MIT Press, 2014. ]
| + | |
| − | | + | |
| − | [http://vernon.eu/publications/16_Vernon_vonHofsten_Fadiga_BICA.pdf Vernon, D., von Hofsten, C. and Fadiga, L. (2016). "Desiderata for Developmental Cognitive Architectures", Biologically Inspired Cognitive Architectures, in press. ]
| + | |
| − | | + | |
| − | ==<span style="color:#AB0000">Documentation:</span> ==
| + | |
| − | [http://cozmosdk.anki.com/docs/index.html Cozmo SDK API ]
| + | |
| − | | + | |
| − | [http://mowl-power.cs.man.ac.uk/protegeowltutorial/resources/ProtegeOWLTutorialP4_v1_3.pdf Manchester OWL Tutorial ]
| + | |
| − | | + | |
| − | [http://docs.opencv.org/3.0-beta/doc/py_tutorials/py_tutorials.html OpenCV Python Tutorial ]
| + | |
| − | | + | |
| − | [http://pointclouds.org/documentation/ Point Cloud Tutorial ]
| + | |
| − | | + | |
| − | [http://protegewiki.stanford.edu/wiki/Protege4Pizzas10Minutes Protégé Tutorial ]
| + | |
| − | | + | |
| − | [https://docs.python.org/3.5/tutorial/index.html Python Tutorial ]
| + | |
| − | | + | |
| − | [http://wiki.ros.org/ROS/Tutorials ROS Tutorial ]
| + | |
| − | | + | |
| − | [http://miguelsdc.bitbucket.org/HAMMER/index.html HAMMER Tutorial ]
| + | |
| − | | + | |
| − | [https://www.youtube.com/watch?v=iu_Y7mxDjgA&feature=youtu.be OpenEase ]
| + | |
| − | | + | |
| − | ==<span style="color:#AB0000">Acknowledgments:</span> ==
| + | |
| − | The syllabus for this course drew inspiration from several sources. These include the following.
| + | |
| − | | + | |
| − | * Course 15-494/694 Cognitive Robotics given by Dave Touretzky at Carnegie Mellon University
| + | |
| − | * Course VO 4.0 376.054 Machine Vision and Cognitive Robotics given by Markus Vincze, Michael Zillich, and Daniel Wolf at the Vienna University of Technology
| + | |
| − | * Course IT921F Artificial Cognitive Systems given by David Vernon at the University of Skövde
| + | |
| − | * Tutorial on 3D semantic perception at the Third Örebro University Winter School on Artificial Intelligence and Robotics given by Joachhim Hertzberg and Thomas Wiemann, Osnabrück University, and Martin Günther, DFKI Robotics Innovation Center, Germany
| + | |
| − | * Tutorial on Constraint-based Reasoning at the Third Örebro University Winter School on Artificial Intelligence and Robotics given by Federico Pecora and Masoumeh Mansouri, University of Örebro, Sweden
| + | |
