5.3 Learning a Task Plan
5.4 Learning Task Objectives
5.5 Learning Task Features
5.6 Learning Frame of Reference
5.7 Learning Object Affordances
5.8 Techniques for Handling Suboptimal Demonstrations
5.9 Discussion and Open Challenges
6.1 Batch vs. Incremental Learning
6.2 Reinforcement Learning Based Methods
6.3 Corrective Refinement from the Teacher
6.4 Active Learning
6.4.1 Label Queries
6.4.2 Demonstration Queries
6.4.3 Feature Queries
6.5 Summary
7 Designing and Evaluating an LfD Study
7.1 Experimental Design
7.2 Evaluating the Algorithmic Performance
7.3 Evaluating the Interaction
7.3.1 Subjective Measures
7.3.2 Objective Measures
7.4 Experimental Controls
7.5 Experimental Protocol
7.6 Data Analysis
7.6.1 Choosing the Right Statistical Tool
7.6.2 Drawing Conclusions
7.7 Additional Resources
8 Future Challenges and Opportunities
8.1 Real Users, Real Tasks
8.2 HRI Considerations
8.3 Advancing Learning through Benchmarking and Integration
8.4 Opportunities
8.5 Additional Resources
CHAPTER 1
Introduction
Machine Learning techniques have had great success in many robotics applications. Today’s robots are able to study the depths of the Earth’s oceans, carry equipment while following soldiers through mountainous terrain, and explore the peaks and valleys of Mars. Robots build (and will soon drive) our cars, gather the items for shopping orders in busy warehouses, and keep hospital shelves stocked with supplies. Robots can vacuum your floor, mow your lawn, and clean your pool. Yet robots, or more specifically the algorithms that control them, are still unable to handle many of the complexities of the real world. Today, and for the foreseeable future, it is not possible to go to a store and bring home a robot that will clean your house, cook your breakfast, and do your laundry. These everyday tasks, while seemingly simple, contain many variations and complexities that pose insurmountable challenges for today’s machine learning algorithms.
What separates impossible domains from challenging-yet-achievable ones for today’s autonomous technologies is the degree of structure and consistency within the problem domain. Vacuuming robots require a flat floor to operate and not much else. Since the vast majority of house floors meet this requirement, the deployment of robotic vacuum cleaners has been highly successful. The new owner of such a robot simply needs to press the Clean button and in most cases the robot performs its function as expected (until it gets stuck on that sock you left on the floor, but that’s why it’s not a room cleaning robot).
Now consider the scenario of bringing a new house cleaning robot home, putting it in the kitchen, and pressing a similar Clean button for the first time. Some of the tasks such a robot might be expected to do is to load the dishwasher with all the dirty dishes, toss waste into the trash, and return clean items to their correct locations. The level of complexity of these tasks is higher not only in terms of perception and manipulation capabilities, but also in the required degree of adaptation to the new environment. Each house is unique, with custom layouts, preferred object locations, and rules (e.g., “never put the knife with the red handle in the dishwasher or it will rust”). Just as a human house guest arriving to a home for the first time, the robot needs to adapt to the customs of a particular household. This means that a single Clean button is no longer sufficient for such a system, instead the platform, and its underlying algorithms, must support the ability for the user to customize the robot’s policy of behavior.
Robot Learning from Demonstration (LfD) explores techniques for learning a task policy from examples provided by a human teacher. The field of LfD has grown into an extensive body of literature over the past 30+ years, with a wide variety of approaches for encoding human demonstrations and modeling skills and tasks. In this book we provide an introduction to the field with a focus on the unique technical challenges associated with designing robots that learn from human instruction. The book is written for AI researchers who are interested in developing Learning from Demonstration methods, or who would like to learn more about different modes of interaction or evaluation methods for such systems.
1.1 MACHINE LEARNING FOR END-USERS
The above household robot scenario describes one possible application area for LfD techniques. More generally, this scenario is motivated by the challenge of enabling a novice user, a non-programmer, to customize existing robot behaviors or develop new ones through intuitive teaching methods. The motivation for tackling this challenge centers on the belief that it is impossible to pre-program all the necessary knowledge into a robot operating in a diverse, dynamic and unstructured environment. Instead, end-users must have the ability to customize the functionality of these robotic systems. Since it is impractical to assume that every end-user will have programming experience, natural and intuitive methods of interactions must be developed to enable non-roboticists to effectively use such systems.
LfD techniques build upon many standard Machine Learning methods that have had great success in a wide range of applications. However, learning from a human teacher poses additional challenges, such as limited human patience and inconsistent user input. Traditional Machine Learning techniques have not been designed for learning from ordinary human teachers in a real-time interaction, resulting in a need for new, or modified, methods. Figuring out at which level of the algorithm
