3.1.1 Single vs. Multiple Policies
3.1.2 Linear vs. Monotonically Increasing Scalarization Functions
3.1.3 Deterministic vs. Stochastic Policies
3.2 Solution Concepts
3.2.1 Case #1: Linear Scalarization and a Single Policy
3.2.2 Case #2: Linear Scalarization and Multiple Policies
3.2.3 Case #3: Monotonically Increasing Scalarization and a Single Deterministic Policy
3.2.4 Case #4: Monotonically Increasing Scalarization and a Single Stochastic Policy
3.2.5 Case #5: Monotonically Increasing Scalarization and Multiple Deterministic Policies
3.2.6 Case #6: Monotonically Increasing Scalarization and Multiple Stochastic Policies
3.3 Implications for MO-CoGs
3.4 Approximate Solution Concepts
3.5 Beyond the Taxonomy
4.1 Inner Loop Approach
4.1.1 A Simple MO-CoG
4.1.2 Finding a PCS
4.1.3 Finding a CCS
4.1.4 Design Considerations
4.2 Inner Loop Planning for MO-CoGs
4.2.1 Variable Elimination
4.2.2 Transforming the MO-CoG
4.2.3 Multi-Objective Variable Elimination
4.2.4 Comparing PMOVE and CMOVE
4.3 Inner Loop Planning for MOMDPs
4.3.1 Value Iteration
4.3.2 Multi-Objective Value Iteration
4.3.3 Pareto vs. Convex Value Iteration
5.1 Outer Loop Approach
5.2 Scalarized Value Functions
5.2.1 The Relationship with POMDPs
5.3 Optimistic Linear Support
5.4 Analysis
5.5 Approximate Single-Objective Solvers
5.6 Value Reuse
5.7 Comparing an Inner and Outer Loop Method
5.7.1 Theoretical Comparison
5.7.2 Empirical Comparison
5.8 Outer Loop Methods for PCS Planning
6.1 Offline MORL
6.2 Online MORL
7.1 Energy
7.2 Health
7.3 Infrastructure and Transportation
8.1 Conclusions
8.2 Future Work
8.2.1 Scalarization of Expectation vs. Expectation of Scalarization
8.2.2 Other Decision Problems
8.2.3 Users in the Loop
Preface
Many real-world decision problems have multiple, possibly conflicting, objectives. For example, an autonomous vehicle typically wants to minimize both travel time and fuel costs, while maximizing safety; when seeking medical treatment, we want to maximize the probability of being cured, but minimize the severity of the side-effects, etcetera.
Although interest in multi-objective decision making has grown in recent years, the majority of decision-theoretic research still assumes only a single objective. In this book, we argue that multi-objective methods are underrepresented and present three scenarios to justify the need for explicitly multi-objective approaches. Key to these scenarios is that, although the utility the user derives from a policy—which is what we ultimately aim to optimize—is scalar, it is sometimes impossible, undesirable, or infeasible to formulate the problem as single-objective at the moment when the policies need to be planned or learned. We also present the case for a utility-based view of multi-objective decision making, i.e., that the appropriate multi-objective solution concept should be derived from what we know about the user’s utility function.
This book is based on our research activities over the years. In particular, the survey we wrote together with Peter Vamplew and Richard Dazeley [Roijers et al., 2013a] forms the basis of how we organize concepts in multi-objective decision making. Furthermore, we use insights from our work on multi-objective planning over the years, particularly in the context of the PhD research of the first author [Roijers, 2016]. Another important source for writing this book were the lectures we gave on the topic at the University of Amsterdam, and the tutorials we did at the IJCAI-2015 and ICAPS-2016 conferences, as well as the EASSS-2016 summer school.
Aim and Readership This book aims to provide a structured introduction to the field of multi-objective decision making, and to make the differences with single-objective decision theory clear. We hope that, after reading this book, the reader will be equipped to conduct research in multi-objective decision-theory or apply multi-objective methods in practice.
We expect our readers to have a basic understanding
