2 Description of Progress and Implementation

Transcription

1 Genetic Algorithms and Reinforcement Learning: Societies and Species: A midterm report, by Andrew Albert and Marc Lanctot 1 Overview This midterm report serves as both an indicator of progress and project plan for the remaining of the time remaining to implement the course project. A reminder: our project is based on Evolutionary Reinforcement Learning (ERL) [3]. 2 Description of Progress and Implementation 2.1 Implementation Our implementation is based on David Ackley and Michael Littman s ERL. Each agent in the world has two neural networks: an action network and an evaluation network. The neural networks are implemented as feed-forward neural networks (see figure below) which have a set of input neurons, a number of hidden layers each containing hidden units, and a set of output neurons. An example of a feed-forward neural network taken from [5]. This network has 4 input units, 1 hidden layer (with 3 hidden units), and 2 output units. The learning algorithm used is Michael Littman s Complementary Reinforcement Back- Propagation (CRBP) [1]. The structure of the current implementation is given in the UML diagrams below:

2 Below, we given a rough estimate for the percentage of completion for each class in the above UML diagram. 100% completion represents the final product. Note that the classes for the genetic algorithms have not yet been created.

3 Class Name % Impl. Completed % Impl. Tested Neuron Perceptron SigmoidNeuron NeuralNetwork FeedForwardNN BackProp CRBP 90 0 GridCell Grid Agent GeneticCode 5 0 Simulator 10 0 The evaluation networks are obtained directly from the individual s genetic code. The genetic code in our application is augmented to include bits which describe its physical makeup and determine to which species (agent society) it belongs. The genetic code describes the agent s evaluation function as well as its species. Note: the Beholder image was taken from [6]. 2.2 Progress As a measure of progress, we verify that the basic back-propagation learning algorithm as well as the underlying data structures worked as they are intended to. We implemented the basic back-propagation algorithm found in [7], and tested it in on the AND function. That is, f(x, Y ) = X Y, where X and Y are boolean inputs. A simple experiment was designed to collect the speed of convergence for 2 example feed-forward networks. The first network has 2 inputs, 3 hidden units, and 1 output. The second network has one fewer hidden unit: 2 inputs, 2 hidden units, and 1 output.

4 The figure below shows the sum of the error for all 4 possible combinations of inputs after every epoch. The results of back-propagation of the first network is always something similar to the figure on the left (it is an example). Most of the time, backprop on the second network produces something similar, but sometimes it gets stuck in a local minimum. One such case of this is shown in the right-hand side of the figure below. 1.8 BP Error 0.8 BP Error error error epochs The errors of a neural network after being trained using back-propagation. The left figure shows convergence to the global minimum. The right figure shows convergence to a local minimum. epochs These results are informative; they clearly show the need for us to carefully select the number of hidden units and structure of the neural networks in our final application. 3 Current Plan The current plan, as of Oct. 20th, is the following. Note that there are 2 major phases: the implementation/testing phase, and the results-collecting phase. Date Milestone Implementor Fri, Oct. 20th Genetic Code implemented Andrew Sat, Oct. 21st Simple Env. (like ALife) built Marc Sun, Oct. 22nd CRBP Fully Implemented Marc Sun, Oct. 22nd Agent Societies Prototype Andrew Wed, Oct. 25th CRBP Tested on Simple Env Marc Thu, Oct. 26th Agent Societies Tested Andrew Fri, Oct. 27th Full Implementation of ERL, Tested on Simple Env. Marc Sun, Oct. 29th Agent Societies and ERL Integrated Marc+Andrew Oct. 29th - Nov 4th Collect Results of Basic Impl. Marc+Andrew Nov. 4th - Nov. 20th Impl. extensions and collect results Marc+Andrew Nov. 13th - Nov. 20th Write final report and prepare presentation Marc+Andrew

5 4 Integration of Ideas from Related Work There are several interesting concepts that can be explored, once the basic implementation has been built. Here are the ideas that we would like to explore, listed in order of decreasing priority. We will implement each idea into our project, after a working base implementation is reached, time permitting. 4.1 Lamarckian vs. Baldwinian Evolution In the original ERL paper, the authors describe the Baldwin Effect [2] and show how it is observed in their analysis data of ERL applied to an artificial world they developed called AL. The Baldwin Effect is best described as an eventual discovering of desirable genetic traits through the process of survival and evolution. The Lamarckian mechanism, on the other hand, describes a way to pass on the learning that an individual has incurred during a generation to its successors. These two different types of evolution have been compared in the literature; different results and opinions have been presented for both approaches. In particular, the same authors that wrote the paper on ERL and the Baldwin Effect also wrote a paper on a Lamarckian approach. We plan to implement both approaches and compare the results. 4.2 Reinforcement Learning Techniques Since the first introduction of ERL, a significant amount of research has been done in the field of reinforcement learning. Re-applying techniques that have been applied in the RL literature to this evolutionary case could be interesting. There are too many such techniques to list here; one of particular interest is function approximation. If the evaluation function was treated as a function approximator, it can be improved over time and hence produce more meaningful rewards for the agents. 4.3 Cascade Correlation Cascade correlation is a technique for self-modification of neural network topology given input training data [4]. The technique is an improvement on regular back-propagation neural network learning which automatically determines and fixes its layout from experience. The algorithm detects features by measuring correlation between values of the outputs of the neurons and the residual error. Neurons are added to minimize the correlation of features. As a result, the speed back-propagation can be greatly improved. 5 Issues Pending Here we list the problems that we are likely to face, based on what we have already done. These are not major problems that will prevent us from succeeding, but they are certainly things we need to resolve.

6 5.1 Number of Hidden Units Preliminary experiments (see below) show that the back propagation algorithm is sensitive to the number of hidden units in the middle layer of a feed-forward neural network. We must find out what is the best number to use so that we can circumvent such problems. A general rule mentioned in [7] is to use log N hidden units, where N is the number of inputs, but no justification is given for this formula. 5.2 Phenotypic Plasticity This term is used in most of the related papers that we have read. In all cases, it seems to be a way to prevent certain genes from changing in future generations. The problem is that we re uncertain as to how this is done, specifically. We must find out more about this topic, and in particular, how to apply it in ERL. 5.3 Use of Genetic Algorithms The authors of ERL outline typically used operation in genetic algorithms, such as crossover and mutation. One point of ambiguity at this point in the project is precisely how to apply these notions. For example, what are the probabilities used for mutation and how is crossover applied? What are the conditions for two individuals to reproduce? Is asexual reproduction needed? If so, why? These questions can be answered by reviewing the proper text or consulting the proper community. References [1] D. H. Ackley and M. S. Littman. Generalization and scaling in reinforcement learning. In D. S. Touretzky, editor, Advances in Neural Information Processing Systems, volume 2, pages , Denver 1989, Morgan Kaufmann, San Mateo. citeseer.ist.psu.edu/ackley90generalization.html. [2] J. M. Baldwin. A new factor in evolution. Americal Naturalist, 30: , [3] M. L. Littman D. H. Ackley. Interactions between learning and evolution. Artificial Life II, pages , [4] Scott E. Fahlman and Christian Lebiere. The cascade-correlation learning architecture. Technical Report CMU-CS , Carnegie Mellon University, [5] William H. Gemmill and Vladimir M. Krasnopolsky. The Use of SSM/I Data in Operational Marine Analysis. [6] Wizards of the Coast. Beholder.

7 [7] Joelle Pineau. Neural network lecture notes. McGill University: COMP424, jpineau/comp424/lectures/19neuralnets.pdf.