Decision Making and Reward, Computational Perspectives

Reinforcement Learning

Reinforcement learning (RL) has been a powerful force in machine learning, psychology, and neuroscience since the 1980s. It blends a computational approach to decision making with the behaviorist or classical-economic assumption that agents act to maximize, or at least improve, earnings. RL theory developed from the theoretical foundations established in control theory by Richard Bellman in the 1950s and 1960s, but specifically exploited the recursive structure of equations for predicting future reward as a function of an action policy (specifying which action to take in every state of the environment) applied to a discrete state-space representation of the world. (A discrete state-space is a representation consisting of a list of all the unique states in which an agent could find itself; in contrast, a continuous set, such as the set of real numbers used in calculus, is uncountable and could never be exhaustively written down, even in an infinitely long list.) Using the discrete state-space approach, Richard Sutton and Andrew Barto showed that unsupervised, online learning by trial and error was an effective method for creating artificial agents without programming in all possible relevant knowledge—indeed, without the programmer even having this knowledge. Despite this success, RL approaches to decision making are often hindered by their frequent reliance on a compound-serial representation of [Page 219]time: an extremely memory-intensive representation in which a binary state variable is assigned to every relevant subinterval of a time period and linked in a chain (a binary state variable is a memory slot for a 1 or a 0, indicating whether the world is in the corresponding environmental state).

...