Reinforcement Learning for Mobile Robots with Continuous States

Reinforcement Learning for Mobile Robots with Continuous States Yizheng Cai Department of Computer Science University of British Columbia Vancouver, V6T 1Z4 Email:yizhengc@cs.ubc.ca Homepage: www.cs.ubc.ca/~yizhengc Abstract It is very tedious to program a mobile robot in the real world. Also, it is difficult for the programmed mobile robot to get adapted to a new environment. Reinforcement learning provides a proper mechanism for the mobile robot to learn how to accomplish a task in any given environment with very little work for programming. However, traditional reinforcement learning methods only assume discrete state and action space, which is not applicable for mobile robots in the real world. This project simulated a mobile robot with continuous states and discrete actions using a safe approximation of the value function to learn the optimal policy. Experiment result shows that learning can be very successful and efficient with bootstrapped information provided by human controls. 1 Introduction The control task for a mobile robot is very tedious and time consuming because it is very difficult to translate high-level knowledge of how to accomplish a task into low-level control for the mobile robot to understand. This control task becomes more difficult if the environment of the task is complex. Also, this approach is not adaptive to the changes of the environment. The best way to solve this problem is to find a proper mechanism for the robot to learn how to accomplish the task itself. Reinforcement learning is just a proper way for the robot to learn the environment and accomplish the task. However, the traditional reinforcement learning methods always assume discrete state and action space, which is not applicable for mobile robot with real value states. In 1994, Boyan and Moore [1] introduced a method to safely approximate the value function for reinforcement learning, which makes it possible to apply reinforcement learning to the continuous state space. Later, Smart and Kaelbling [2, 3] adopted the safe value function approximation method in their work and used Q-learning to solve the control problem of mobile robots in continuous space. Another important contribution of their work is that they introduced an efficient and practical learning-system construction methodology that can augment the reinforcement learning process by information that is easy for human expert to provide. The experiment results in their work demonstrate that, with the bootstrapped

information provided by human controls, the learning process can be very efficient and the learning results can be extremely good. The motivation of this project is to take a closer look at reinforcement learning applied in the mobile robot control task and do some experiments to verify the efficiency of reinforcement learning in such tasks. The main task of this project is to simulate a mobile robot with continuous states and discrete actions using the same approach as Smart and Kaelbling did in their work so that experiments can be done to verify those hypotheses in their work. One of the important hypotheses is the effectiveness of save value function approximation which is the basis for the application of reinforcement learning in continuous state space. Another important hypothesis is that learning with bootstrapped information provided by human experts can significantly shorten the learning time and produce very good learning results for the robot to accomplish the task. The first part of this report will describe the basic idea of reinforcement learning and Q-learning. The second part will describe how the Q-learning method is implemented for the control of mobile robot. In the third part, some experiment results will be presented with discussion. The last part will be the discussion and future work of this project. 2 Reinforcement Learning Reinforcement learning had attracted much attention in the community of machine learning and artificial intelligence in the past several decades. It provides a very good way to teach a robot with some rewards or punishments so that the robot can learn how to accomplish a task without hard coding the low-level control strategies. The traditional reinforcement learning assumes that the world can be represented by a set of discrete states, S, and the agent has a finite set of actions, A, to take. The interaction between the agent and the world is represented by the reward, R. Here, the time is also discretized into time steps. For any agent, the state at time t is represented as s t, and the action to take is a t. After taking the action, a t, the agent will go to a new state, represented by s t+1, and will also be given a reward r t+1, which is an immediate reward to evaluate how good the action taken is. Thus, the agent can have cumulative experience represented by a sequence of tuples (s t, a t, r t+1, s t+1 ). For each action, it can affect both the immediate reward and the next state the agent might be in, which might affect the delayed reward. So, the ultimate goal of reinforcement learning is to find an optimal policy of behaviors that perform best in the environment; or an optimal value function that best maps the states (or the state and action) to a measure of long-term value of being in a state. The method used in this project, which is called Q-learning, is to find an optimal function that maps the given state to the long-term value of being in that state. 2.1 Q-learning Q-learning, introduced by Watkins and Dayan [4], is a method that is typically used to solve the RL problems. One of the big advantages of Q-Learning is that it is a model-free algorithm which does not require any prior knowledge about the MDP (Markov Decision Process) model. The optimal value function for Q-learning is defined as follows: Q * ( s, a ) = E [ R ( s, a ) + γ max a ' Q * ( s ', a ' )]

Here, the optimal value function is the expected value, over the next state, of the reward for taking action a in state s, ending up with s and perform optimal action from then on. γ is known as the discount factor which is a measure of how much weight should be given to the future reward. With the definition of Q-function, it is easy to define the function for the optimal policy: * π ( s) = arg maxq( s, a) During the Q-learning process, the agent performs trial actions around the environment to get a sequence of experience tuples and store the mapping from state and action to value into a table. As the learning process goes on, the state-action pair will be visited multiple times and the corresponding value in the table will then be updated according to the following form: Q( st, at ) Q( st, at ) + α ( rt + 1 + γ maxa' Q( st + 1, a') Q( st, at )) where α is known as the learning rate. It has been proved by Watkins and Dayan [4] that the Q-function mapping will finally converge after infinite visits of those state-action pairs. 2.2 Safe Value Function Approximation The table mapping approach described in the previous section was the traditional method of Q-learning. But, in the real world, most mobile robots will have real value states and actions which are not possible to be very properly discretized. An alternative approach is to use some function approximation techniques to approximate the Q-function. One of the major problems of this approach is the error of prediction. During the process of Q-learning, it is required to predict the Q-value given some state-action pair. According to the updating function in the previous section, the error in predicting both the current Q-value and the maximum Q-value of the next state will easily accumulate when the process goes on, and finally dominate the approximation. 2.2.1 Reducing Approximation Error In order to reduce the approximation error, Boyan and Moore [1] introduced a method to safely approximate the value function. They claimed that most approximation methods suffer from a problem called hidden extrapolation which means that the approximation extrapolate data that are not within the train data space. In order to safely approximate the Q-function, they suggested to only use function approximators to interpolate training data instead of extrapolate from it. So, they tried to create a convex hull around the training data and only allow for prediction queries within it. To compromise the computation complexity and the safety, the structure adopted is what called independent variable hull (IVH). For a training data matrix, X, where the rows of X correspond to the training data points, the hat matrix can be computed in the following way: V = X ( X ' X ) 1 X ' For any query vector, it lies within the hull only if it fulfills the following criteria: x'( X ' X ) 1 a x maxυ where υii are the diagonal elements of V. The next section will describe the detail of how to use this IVH for prediction. Another important technique to reduce the prediction error is to use the Locally Weighted Regression [5]. Instead of using all the training data to approximate the i ii

function, LWR only uses those nearby neighbors that are close to the query points. Each of these neighbor will be associated a certain weight according to their distance to the query point. Any kernel function can be used to compute the weight. The one used in this project is a Gaussian function: ω = e 2 ( x q) h where, q is the query point, x is neighbor of that query. 2.2.2 The HEDGER Algorithm HEDGER algorithm [2, 3] is a function approximation algorithm using linear regression. It is based on both safe value function approximation [1] and Locally Weighted Regression [5]. The following is the pseudocode of the HEDGER algorithm for predicting the Q-value used in the simulation of this project. It is based on the one given by [6] with some minor modification for implementation purposes. 2 Algorithm 1 HEDGER prediction Input: Set of training examples, S, with tuples of the form (s, a, q, r) Query state, s Query action, a LWR minimum set size, k LWR distance threshold, D LWR bandwidth, h Output: Predicted Q-value, q s,a x ( s, a) K training points with distance to x smaller than D if the total number of training points is less than k q s,a = q default else Construct an IVH, H, with training points in K if K K is singular q s,a = q default else if x is outside of H then q s,a = q default else q s,a = LWR prediction using x, K, and h if q s,a > q max or q s,a < q min then q s,a = q default Return q s,a The minimum size of LWR set is easy to decide. It is just the number of the parameters that are to be decided for linear regression. For the distance threshold for LWR, it is better that the threshold is large enough to include as many training points as possible. But, as the size of K grows, the computation would be prohibitively expensive. Also, points that are far from the query might have very small weight due to the bandwidth so that their contribution can safely be ignored and the computation spent on them at early stages is wasted. So, it is better to define a threshold, κ, as the minimum weight

for points within K so that they will not be ignored. Thus, the distance threshold can be computed in the following way, if the kernel function is Gaussian: D = 2 h logκ where, the value of the bandwidth, h, is empirically determined in this project because Smart claimed in his thesis [6] that the improvement on h does not bring significant benefit for the overall performance. In addition, the predicted Q-value should be within the boundary of the possible minimum and maximum value of Q-value here. Because the problem in this project is set to be infinite discounted Q-learning, the possible maximum and minimum number of the Q-value can be computed as follows: Q max = t = 0 γ r t max rmax = 1 γ The minimum value of Q-value can be computed the same way. The boundary for the reward can be easily updated by the new reward received in each stage so that the Q-value can always be safely within the actual boundary. In addition, the predefined default value for Q-value should also be defined within the boundary. The HEDGER algorithm for training is a modification to the traditional Q-learning. The pseudocode is presented in Algorithm 2 with some changes of the one in [6]. Algorithm 2 HEDGER training Input: Set of training examples, S, with tuples of the form (s, a, q, r) Initial state, s t Action, a t Next state, s t+1 Reward, r t+1 Learning rate, α Discount factor, γ LWR minimum set size, k LWR distance threshold, D Output: New set of training set, S Update q max and q min based on r t+1 and γ x s t, a ) qt +1 ( t maximum predicted Q-value at state st+1 based on S, k predicted Q-value for query of x, based on S, k K set of training points used for prediction of qt κ set of weights of corresponding training points q new α ( rt + 1 + γqt+ 1 qt ) + qt S' SU ( x, qnew) for each point in K q i ακ ( qnew qi ) + qi Return S qt In this project, due to the limit of time, it assumes that the robot has discrete action so that the maximum predicted Q-value at state s t+1 can be obtained by compare the Q-values from all possible actions.

3 Reinforcement Learning in Mobile Robot Control As is seen from the previous sections, the Q-learning method requires the agent to perform randomly at the beginning of learning to have sufficient experience. For environment with very sparse rewards, the learning process might spend quite a lot of time to explore the world without any useful information. One of the possible and reasonable ways to solve the problem is to use supplied control for the robot and lead it to those more interesting points as soon as possible, so that these information can bootstrap the learning process. 3.1 Two Phase Learning Process Smart and Kaelbling introduced the two phase learning process [2, 3] which uses some predefined policy or the direct control of human experts in the first phase to collect sufficient experience. Meanwhile, the RL learning system just learns the experience passively so that the information can be used to for the value-function approximation. In the second phase, the reinforcement learning system takes the control of the robot and learns the experience while the supplied control no long have effect on the robot. Here, the RL system is not trying to learn the trajectory but only uses the experience for the value-function approximation. The two phase learning process can be demonstrated in the following figure [2]: Environment Environment R O Supplied Control A R O Supplied Control A Policy Policy Learning System Learning System (a) Phase 1 (b) Phase 2 Figure 1. The graph demonstration of the two learning phases 3.2 Corridor-Following Task The task simulated in this project is the Corridor-Following task which is similar to the experiment done with the real robot by Smart and Kaelbling [2, 3]. The task is depicted by figure 2. Position in corridor Reward Area Distance to the end Figure 2. The corridor following task

In the corridor-following task, the state space has three dimensions, the distance to the end of the corridor, the distance to the left wall of the corridor as a fraction of the total width of the corridor and the angle to the target, shown in figure 3. Left wall d Target φ ψ w θ Figure 3. Relation between three dimensions of the state space According to figure 3, the relation between the three dimensions can be represented by the following formula: θ = tan 1 d w π + ϕ 2 All the three dimensions of the state are of real values. In the simulation, the scenario is of sparse reward, which means that the robot will be given reward of 10 when it reaches the reward area, and 0 elsewhere. The reason to use sparse reward is that, intuitively, if the learning works for the sparse reward situation, it would probably work for dense reward situations because everything is the same, but the robot will more easily get to the interesting point and update the approximation quickly so that the learning process will be faster. It is also a reason why it is necessary to adopt the two phase learning procedure to speed up the learning process. Because of the sparse reward, it takes longer time for the robot to get to those spots with rewards by taking random actions. One of the main purposes of this project is just to see how the learning strategy works for the sparse reward scenario. 3.3 Implementation Issues For simulation purpose, the corridor is modeled by a 300 by 800 pixel region and the robot is denoted by a circle with a line from the center to indicate the direction the robot is facing. The action is rotation from the original direction of robot. The goal of the task is to reach the end of the corridor. Due to the limit of time for this project, the task is simplified to some extent. The translation speed of the robot, υ t, is constant everywhere in the corridor. The action of the robot is the counterclockwise angle from the direction of the robot at current state. The angle takes the value from 30 degree to 360 degree with interval of 30 degree. Here, using 360 degree instead of 0 is in order to avoid getting a singular matrix K in Algorithm 1. Also, for the same reason, the task uses the angle to the target as one of the dimension in the state space so that its value will rarely be zero. As is seen in the formula in section 3.2, even if the actions are discrete and speed is constant, the position and direction of the robot is definitely real value. In order to simulate the continuous states, all the elements of the state vector are stored as real values. Only when the robot is plotted on the screen, the position of the robot will take the closest integer of that real value position in the state vector. Thus, even if it is simulated on the computer with discrete pixels on display, the simulation still take all the state entries as real value so that the simulation is very close to the real situation. For the parameters in the simulation, the learning rate, α, is set to 0.2 and the discount factor, γ, is 0.99, which are all adopted directly from Smart s work. The LWR

bandwidth is empirically set to 0.06. For the minimum size, k, of LWR set, it should be at least the same as the size of query vector (s, a), which has four entries. So, k, is set to 5 in the simulation. The supplied control in the first phase is hard coded to force the robot always moving forward, but the exact action is random. In addition, for a real robot, it is better not to hit any obstacle to avoid break the sensors or other components, the robot should have some mechanism to avoid hitting into the wall. So, in the simulation of this project, the robot will approximate whether it will bump into the wall according to the state information from the sensor and the current action generated by the action module. If the action leads to a bump, the action module will just generate a random action. In the simulation, at the second phase, the action module controlled by the RL learning system will not always follow the greedy policy which always takes the best action. For the exploration purpose, it will have some probability to perform random actions. Also, in the evaluation, the action module will also produce noisy action so that the simulation looks more real. 4 Results In order to evaluate the learning results, after every 5 training rounds, the training process will be evaluated. In all the training rounds, the robot starts at the same position but with different random directions. The following figure depicts the steps to goal of the robot during evaluation. Phase One Training Runs Phase Two Training Runs Figure 4. The steps to the goal in evaluation of the two phase training According to figure 4, with several training runs in the first phase, the number of steps to goal decreases rapidly. Although there are some peaks, the overall number of steps to goal is decreasing. After 30 rounds of first phase training, the system switches to the control of the RL system. The phase change leads to a peak in the plot because it needs to get adaptive to those new data brought by RL control and exploration. Finally, with sufficient exploration of the world, the number of steps to the goal converges and is close to the optimal value. Different from the result in Smart s work, there is a latency of the peak when the phase changes. There are several reasons that might cause this problem. At the early stage of

the second phase learning, there are not enough new data points, brought by the RL module, to affect the Q-value prediction. The prediction highly depends on the previous data points from the supplied control. Also, because of the exploration strategy, the action module controlled by RL system will take the random actions with some probability, which leads the robot to the area that it is not familiar. Thus, the robot have to perform randomly and update the Q-value in those regions until it arrive the regions that it is more familiar with. So, those newly created data, which are still not stable, might cause the robot to perform poorly. Those unexpected peaks both in the two phases of training are probably caused by the noise added to the action module at the evaluation stage. Because, before the dataset cover most regions in the corridor, the random action would more likely to lead the robot to a strange regions and the robot need to take many random actions to go back to the familiar region. Finally, when there are sufficient training data which explore most regions from the starting position to the goal, one random action will not make the robot jump out of the safe region. So, the total number of steps to the goals will finally converge. In this simulation, because the k-d tree is not implemented to speed up the algorithm by shortening the time to find the nearest neighbor in the experience, it takes more than one day to simulation the two phase training with more than 80 training runs. Figure 4 only reflects one whole simulation and no more simulation are done due to the limit of time. Figure 5 shows two images that plot the path of the robot to achieve the corridor-following task. The first image is plotted at the early stage of the first phase training. It well reflects the pattern of Q-value propagation. Because it is a sparse reward situation, the propagation of the Q-value start from the reward area and back propagate slowly. Those steps in the early stages are more like random action. But in the end, the robot moves quickly to the end of corridor. The second image is at the late stage of second phase training, it is seen that the Q-value is well distributed so that almost all the actions are forward to the target rather than perform randomly. Figure 5. The path of the robot in the early stages of the whole training procedure

In conclusion, from the simulation, the safe value approximation works well for RL in continuous state space. The robot does converge very close to an optimal policy after reasonable runs of training. Also, the two phase training procedure does make the number of steps quickly decrease to reasonable value and speed up the whole training procedure significantly, especially in environments with very sparse reward. This simulation really confirms the effectiveness of using the human aided control as a practical method for the control task of mobile robots. 5 Discussion and Future work The results from this simulation well support the hypotheses by Smart and Kaelbling, especially the idea of using two phase learning. It is really a promising approach that can use simple and high level method to handle those annoying problems of low level control and information mapping. Anyhow, there are still two major problems remain to be solved. Firstly, the task in this simulation is really simple. Simulations with more complicated tasks are needed to further prove the effectiveness of the approach. Secondly, the action space in this simulation is discrete, which significantly simplify the problem. But in real world, most actions have real values and are difficult to be discretized. So, some fancy methods are required to find the maximum of the Q-value of the next state, which is just to find the extremum of the approximated Q-function around a specified region. It will probably make the policy more difficult to converge to optimal. Brent s method is the one used by Smart and Kaelbling and proved to be effective. Some other Root-Finding methods should be tried as well. Acknowledgments Thanks to David Poole for useful suggestions and information for the literature. References [1] Boyan J.A. and Moore A.W.. Generalization in reinforcement learning: Safely approximating the value function. In Advances in Neural Information Processing Systems: Proceedings of the 1994 Conference (pp. 369-376). Cambridge, MA: MIT Press, 1995 [2] William D. Smart and Leslie Pack Kaelbling. Effective Reinforcement Learning for Mobile Robots. In International Conference on Robotics and Automation, 2002 [3] William D. Smart and Leslie Pack Kaelbling. Practical Reinforcement Learning in Continuous Spaces. In Proceedings of the Seventeenth International Conference on Machine Learning, 2000. [4] Christopher J. C. H. Watkins and Peter Dayan, Q-learning. In Machine Learning, vol. 8, pp. 279-292, 1992 [5] Atkeson C.G., Moore A.W. and Schaal, S.. Locally weighted learning. In AI Review, vol. 11, pp. 11-73. 1997 [6] William D. Smart. Making Reinforcement Learning Work on Real Robots. PhD Thesis, Department of Computer Science, Brown University, 2002