AAAI-2017: Tutorial on Risk-averse Decision Making and Control

Presenters:

Location: Continental 1-3, Ballroom Level

Schedule

Slides

Detailed Description

Traditional decision methods in artificial intelligence focus on maximizing the expected return (or minimizing the expected cost). This is appropriate when the decision-makers are risk-neutral. Yet, many decision-makers are risk-sensitive and are willing to give up some of the expected reward in order to protect against large losses. The desire to avoid risk when making decision was recognized early on, but developing appropriate models to capture risk has been challenging. Useful models of risk-aversion must be easy to understand and interpret for decision-makers; but they also must be general, flexible, and more importantly, they must produce tractable optimization problems.

The classical approach to modeling risk aversion is to use expected utilities, but they are difficult to specify and significantly complicate optimization methods. This tutorial focuses on the new approach to risk-aversion which is based convex measures of risk. Convex measures of risk replace the expectation operator by a more general operator which puts more weight on negative outcomes. Perhaps the most well-known risk measure is CVaR, which at level alpha computes the expectation of the lowest alpha-quantile of returns.

There has been tremendous progress in developing the theory and practice of risk measures since their introduction in the late 1990s. Researchers and practitioners have proposed and used many other risk measures besides CVaR and many stochastic optimization methods now work with convex risk measures. Due to the ease of modeling and optimization with them, convex risk measures have become the standard method for capturing risk sensitivity in operations research. Robust optimization, a related concept for modeling risk-aversion and avoidance, has flourished similarly.

In recent years, there has been growing interest in developing risk averse decision-making methods in artificial intelligence and machine learning. Risk-aversion is required to make machine learning relevant in many practical settings since solutions from risk neutral methods are often too risky in mission-critical problems. Convex risk measures and robust optimization are now being used in methods that range from classification, through multi-armed bandits, to reinforcement learning. While the general concept of risk measures is relatively simple, their true power can only be realized through deeper understanding. For example, integrating risk aversion with sequential decision-making requires overcoming a full set of challenges concerning time consistency. Our tutorial will shed light on these issues and provide numerous pointers for further research.

Goals and Target Audience

This tutorial will introduce the tools and methodology of convex risk measures and robust optimization, developed in operations research and stochastic finance, to the machine learning community. The goal is to make these often complex results accessible and provide a starting point for people interested in exploring this research direction in greater detail. We will introduce basic concepts of risk measures and robust optimization, describe connections and advantages w.r.t. the existing methods, and describe how risk aversion can be used in sequential decision problems.

This tutorial should be of interest to researchers in any area that involves decision-making or control. This in particular includes the reinforcement learning and online learning communities, in which the application of risk aversion presents the most pitfalls. Risk aversion can also be important in classification and regression problems as several recent publications attest to. We plan to introduce the general risk-modeling framework and assume just knowledge of measure theoretical concepts, linear algebra, and basic optimization.

References

Coherent risk measures

Sequential decision making

Other machine learning