11. Reinforcement Learning

Transcription

1 Artificial Intelligence 11. Reinforcement Learning prof. dr. sc. Bojana Dalbelo Bašić doc. dr. sc. Jan Šnajder University of Zagreb Faculty of Electrical Engineering and Computing (FER) Academic Year 2015/2016 Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

2 Markov Decision Processes and Non-Deterministic Search Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

3 The Model of the World Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

4 GridWorld A maze-like problem Noisy movement Actions don t always end up as planned Immediate reward ( Living reward) Each time-step the agent lives, he receives a reward. The goal of the game is to maximize the sum of rewards - Utility Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

5 Deterministic vs Stochastic Movement Deterministic Stochastic Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

6 Markov Decision Processes Andrey Markov ( ) Markov in MDP s means that an action outcome depends only on the current state: P (S t+1 = s S t = s t, A t = a t ) The successor function cares only about the current state, and not where you came from Therefore, we want to model the optimal action choice (Policy) for all given states π : S A A policy π gives an action for each state An optimal policy maximizes expected utility Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

7 Formalized Markov Decision Processes A Markov Decision Process (MDP) is a Non-deterministic search problem MDP Formalization A set of states s S A set of actions a A A transition function T (s, a, s ) The probability that taking an action a in state s takes you to state s Also known as the model of the world the agent lives in A reward function R(s, a, s ) A start state Maybe, a terminal state(s) In the scope of the course, only the transition function T is a probability distribution, while generally even the reward function could be a probability distribution itself Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

8 Gridworld demo Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

9 Optimal Policies and Immediate Rewards (Punishments?) r(s,, s ) = 0.01 r(s,, s ) = 0.03 r(s,, s ) = 0.4 r(s,, s ) = 2.0 How would the policy look like for positive immediate rewards? Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

10 MDP Search Trees Imagine that instead of playing a game against an opponent that wants to minimize your reward, your opponent takes random actions Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

11 Sequence Preference Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

12 Sequence Preference Two main, seemingly straightforward issues More or less 1, 2, 2 vs 1, 2, 3 Now or later 0, 0, 1 vs 1, 0, 0 It s logical and simple to prefer the larger sum of rewards How to adapt the algorithm to prefer equal amounts sooner instead of later? Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

13 Discounting Exponential decay of rewards Each step we take, the rewards are multiplied by a factor of γ Discounted utility = t=0 γt r t The utility for an action a in the state s that ends up in state s is the sum of the immediate reward and the discounted future reward V (s) = R(s, a, s ) + γv (s ) Finding the same utility faster is usually better (otherwise, the reward function is not defined properly) = ( = 0.107) Gamma values are usually from [0.8, 1] Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

14 Infinite utilities Some MDP s can go on forever with infinite rewards Solutions 1 Discounted utility = t=0 γt r t Rmax (1 γ) Smaller gamma = shorter focus 2 Limited number of steps (terminate learning after T steps) 3 Absorbing state - a terminal state s A which will always be reached (imagine that every state has a small probability to transition into s A ) Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

15 Markov Decision Processes Revisited MDP = (S, s 0 S, A, T (s, a, s ), R(s, a, s ), γ) A set of states s S An initial state s 0 Maybe, a terminal state(s) A set of actions a A A transition function T (s, a, s ) The probability that taking an action a in state s takes you to state s Also known as the model of the world the agent lives in A reward function R(s, a, s ) Discounting factor γ Terms Policy = which action to take in each state Utility = sum of discounted rewards Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

16 Solving Markov Decision Processes Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

17 Optimal Quantities Optimal quantities are the values we want at the end of our process Optimal policy π (s) = optimal action to take from state s Value (utility) of a state V (s) = expected utility when starting in s and acting optimally Value (utility) of a Q-state Q (s, a) = expected utility when having commited to an action a from a state s and acting optimally afterwards Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

18 Values of states Computing the expected maximum value of a state Expectation E[X] = x X P (x) x The utility is the probability weighted sum of rewards of possible transitions when committed to an action a Or, in more simple terms, the utility is the expected future reward The optimal action provides maximum future utility Two recursive relations ( Bellman equations ) The value of a state is the maximum Q-value of all the possible actions V (s) = max a Q (s, a) The Q-value of a state is the expectation of the immediate reward for a transition (s, a, s ) and the discounted future reward Q (s, a) = s T (s, a, s )[R(s, a, s ) + γv (s )] Bellman form - substituting Q(s, a) V (s) = max a s T (s, a, s )[R(s, a, s ) + γv (s )] Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

19 Value Iteration Initialize the vector of values to zero V 0 (s) = 0 Instead of 0, we can use any constant - the system is still going to converge to stationary values in the limit Iterate over the Bellman form Vk+1 (s) max a s T (s, a, s )[R(s, a, s ) + γv k (s )] Repeat until convergence Complexity of each step = O(S 2 A) Policy (optimal action) will converge before the values values (Demo) Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

20 Demo: value iteration No randomized movement, no living penalty, gamma = 0.9 python gridworld.py -n 0 -a value -v -i 20 Randomized movement p = 0.2, living penalty = -0.01, gamma = 0.9 python gridworld.py -r a value -v -i 20 Randomized movement p = 0.2, living penalty = -0.01, gamma = 1 python gridworld.py -r a value -v -i 20 -d 1 Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

21 Policy Iteration Value iteration is slow - it considers every possible action in every possible step (complexity S 2 A) The maximum value (policy) rarely changes! Policy iteration 1 Fix a non-optimal policy π 2 Policy evaluation: calculate the utilities (values) for the fixed (non-optimal) policy until convergence 3 Policy improvement: update the policy by using the resulting (non-optimal) utilities 4 Repeat until convergence Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

22 Policy Iteration Policy Iteration 1 Fix π 0 2 V π k+1 (s) s T (s, π i(s), s )[R(s, π i (s), s ) + γv π k (s )] 3 π i+1 (s) = arg max a s T (s, a, s )[R(s, a, s ) + γv π k (s )] 4 Repeat until convergence In value iteration, we calculate the optimal values without an explicit policy and then compute the policy by finding the max value in the successor states. In policy iteration, we use a fixed non-optimal policy which we then improve until convergence. Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

23 Double Bandit Problem Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

24 Double Bandit Problem First bandit (blue): With probability p = 1, win one dollar Second bandit (red): With probability p w = 0.75, win two dollars, with probability p l = 0.25 win zero dollars Even without explicit calculation, we have a good idea which strategy we should use Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

25 Game on! $2 $2 $0 $2 $2 $0 $2 $2 $2 $2 Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

26 Offline Planning When all the values are known, solving MDP s is easy V (blue) = x X P (x) x = 1 1 V (red) = x X P (x) x = = 1.5 What happens when we don t know the values? Same problem, but the transition function T is now unknown for both bandits Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

27 Game on again!! $2 $2 $0 $0 $0 $0 $0 $0 $0 $2 Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

28 Reinforcement Learning Intro What happened? We changed our policy after sampling long enough The underlying problem is still the same (MDP), however the transition function is unknown - forcing us to estimate the transition probabilities Keywords Exploration: Trying unknown actions to gather information Exploitation: Use what you know to be optimal Sampling: Trying things out with varying degrees of success and estimating the underlying probability distribution of the MDP Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

29 Reinforcement Learning Ivan Petrovič Pavlov Conditioned reflexes - based on repeated sampling, the dogs learned a reaction to hearing a bell Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

30 Reinforcement Learning Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

31 Reinforcement Learning Take actions based on current knowledge of the environment Receive feedback in the form of rewards Learn based on received rewards and update behavior (policy) in order to maximize the expected reward (utility) We start with zero knowledge and sample our way to the optimal policy Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

32 Example videos: learning to walk, game playing, crawler Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

33 Reinforcement Learning Assume an underlying Markov Decision Process A set of states s S A set of actions a A A transition function T (s, a, s ) A reward function R(s, a, s ) Goal: find optimal policy π (s) Difficulty: Don t know T or R Possible approaches 1 Estimate the transition and reward functions and then find the optimal model 2 Directly estimate the optimal model without explicitly calculating the transition and reward function Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

34 Offline Planning vs. Online Learning Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

35 Model-Based Learning Learn an approximate model based on experience Assume the learned model is correct and solve for its values 1 Learn empirical MDP model Count outcomes s for each (s, a) Normalize values to estimate T (s, a, s ) Discover each R(s, a, s ) when sampling 2 Solve the learned MDP Use value iteration Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

36 Example: expected age Estimation E[A] = a P (a) a = 1 N N i=0 a i Model-Based learning Estimate the prior ˆP (a) = count(a i) N Calculate the expectation E[A] a ˆP (a) a Model-Free learning Learn the estimation as you go E[A] 1 N N i=0 a i Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

37 Passive Reinforcement Learning Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

38 Passive Reinforcement Learning Evaluate a given, non-optimal policy π(s) Both the transition and reward function are unknown Goal: Learn state values The learner follows instructions given from an oracle and has no choice regarding which action to take This is not offline planning - we cannot deduce the state values without explicitely trying out sequences of moves (playing) Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

39 Direct evaluation Goal: Compute the value for each state under fixed non-optimal policy π Solution: Average sampled values Act according to π On each visit to a state, write down what the sum of discounted rewards turned out to be in the end, starting from that state Average the samples Similar to model-free learning Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

40 Example of direct evaluation γ = 1 Sample 1 B, east, E, -1 E, east, D, -1 D, exit, x, 10 Sample 2 C, north, E, -1 E, east, D, -1 D, exit, x, 10 Sample 3 B, east, E, -1 E, east, D, -1 D, exit, x, 10 Sample 4 C, north, E, -1 E, east, A, -1 D, exit, x, -10 Observed transitions T(B, e, E) = 1.0 T(E, e, D) = 0.75 T(E, e, A) = 0.25 Observed rewards R(B, e, E) = -1 R(D, exit, x) = Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

41 Example of direct evaluation γ = 1 Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

42 The Good and the Bad Pros of direct evaluation Easy to understand Requires no knowledge of T and R Eventually computes the correct values just by sampling Cons of direct evaluation Information about state connections is unused (T) Each state is learned independently Learning takes a long time Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

43 Policy Evaluation Value iteration: V0 π(s) = 0 Vk+1 π (s) max a s T (s, a, s )[R(s, a, s ) + γvk π(s )] We don t have T and R! How to average without knowing the weights? Sampling! Vk+1 π (s) 1 n n i=1 [R(s, π(s), s 1 ) + γv k π(s 1 )] We need samples for each outcome from state s - hard to do in practice (some states are hard to reach) Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

44 Temporal Difference Learning Keep track of the past and adapt to the present Still using a fixed, non-optimal policy! Update V (s) every time we get a new experience (s, a, s, r) The outcomes s that are more likely based on the unknown probability distribution T will contribute to the updates more often Temporal difference learning Sample from V(s): sample = (s, a, s, r) Update V(s): V π (s) = V π (s) + α[(r(s, π(s), s ) + γv π (s ) V π (s)] The update factor is the difference between the observed value and the current estimate scaled by the learning rate α difference = [R(s, π(s), s ) + γv π (s )] V π (s) If the observed value is higher than what we estimated, we increase the estimation, otherwise we decrease (or keep it the same) This type of update is called the running average - we move the values toward the value of whichever successor occurs Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

45 Caveats of Temporal Difference Value Learning Temporal difference value learning allows us to approximate the value of each state However, we wanted to learn the policy - and all we do is estimate the values based on a given non-optimal policy π(s) = arg maxa Q(s, a) Q(s, a) = s T (s, a, s )[R(s, a, s ) + γv (s )] We learned all the values, so in theory we could find the policy - but we didn t keep track of the transition function T, and therefore cannot find the arguments to the max Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

46 Active Reinforcement Learning Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

47 Active Reinforcement Learning The differences Now, you select which action to take based on prior experience The transitions T(s,a,s ) and rewards R(s,a,s ) are still unknown The goal: Learn the optimal policy and values Q-value Iteration Remember the recursive definition of the Bellman updates V (s) = max a Q (s, a) Q (s, a) = s T (s, a, s )[R(s, a, s ) + γv (s )] Instead of learning the value function, learn all of its arguments - if we have all the Q-values (expected utilities when commited to an action in a state), calculating the policy is a simple max over the actions for a state Q 0(s, a) = 0 Q k+1 (s, a) s T (s, a, s )[R(s, a, s ) + γ max a Q k (s, a )] The policy π is the action that for a given state gives the maximal Q-value Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

48 Q-Learning Sample-based Q-value iteration Qk+1 (s, a) s T (s, a, s )[R(s, a, s ) + γ max a Q k (s, a )] Learn Q(s,a) values on-the-fly Sample (s,a,s,r) Incorporate it into the running average Q(s, a) Q(s, a) + (α)[r(s, a, s ) + γ max a Q(s, a ) Q(s, a)] Q-learning converges to an optimal policy even if we make suboptimal decisions on the way Off-policy learning Caveats! It takes a long time to converge Eventually, you have to reduce the learning rate to converge, however you have to do it at the right time However, as the number of iterations approaches infinity, as long as you explore enough, the algorithm will learn the optimal policy! Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

49 Exploration vs Exploitation Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

50 Epsilon-Greedy Forcing exploration through random actions When choosing an action With probability ɛ choose a random action With probability 1 ɛ act according to current optimal policy We still need to converge to a fixed policy 1 Decay ɛ over time (similar to the discounting factor) 2 Smarter exploration functions Explore each state proportionally to its estimated (Q-)value Sample enough times to be sure that a state is bad and not just due to the movement noise Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

51 Approximate Q-Learning Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

52 Generalizing So far, we keep track of every single possible state We have no notion of similarities between states What happens when we scale the problem? Ideally, we want to be able to generalize from a small number of observed experiences For this, we need to have a notion of similarity between states Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

53 Similar states Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

54 Feature space embeddings Describe a state using a vector of features Features are mappings from the full problem space to a lower dimensional space (boolean values, enumerations, categories) which captures the important properties of the problem space Features Distance to the closest ghost Distance to closest food dot Number of ghosts Minimal distance to ghost Distance to ghost when moving in direction d Is Pacman in a tunnel We can also describe a q-state (state, action pair) with features, such as This action moves us further away from the ghosts Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

55 Linear Value Functions By using feature based representation, we can describe the value of each state (or q-state) by the linear combination of its weights Features are lower-dimensional mappings of the current state f(s) = [f 1 (s),..., f n (s)] We assign a weight to each feature to measure its importance W = [wn,..., wn ] The value of a (Q-)state is the dot product of the weights and features of that state V (s) = w 1 f 1 (s) + + w n f n (s) = W f(s) Q(s, a) = w 1 f 1 (s, a) + + w n f n (s, a) = W f(s) We get: Reduced dimensionality Faster learning Loss of information! States may share features but have different values Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56

56 Approximate Q-Learning Q-state decomposition Q(s, a) = w 1 f 1 (s, a) + + w n f n (s, a) Q-learning with linear Q-functions: difference = [R + γ maxa Q(s, a )] Q(s, a) wi w i + α(difference)f i (s, a) Adjust weights instead of Q-values When something unexpected happens (our current policy leads us to a bad outcome), find the responsible features (the ones that contributed the most to the policy) and update them Dalbelo Bašić, Šnajder (UNIZG FER) AI Reinforcement Learning Academic Year 2015/ / 56