Evolution of Time in Neural Networks: Present to Past to Future What is Time? CSCE 644 (Based on Forum for AI talk 211) Yoonsuck Choe Department of Computer Science and Engineering Texas A&M University * Joint work with Ji Ryang Chung and Jaerock Kwon No clear understanding (or consensus) tensed vs. tenseless psychological vs. thermodynamic vs. relativistic time and change, their relation 1 2 What is Time? Why Time? Common (psychological) concepts of time: Past Present Future Recollection Prediction Past Present Future A key to understanding brain function may lie in understanding time, as it relates to brain function. The brain generates (psychological) time! 3 4
Time and Memory Without memory, there can be no concept of time: No concept of the past Thus, no concept of the future Time, in the Context of Neural Networks Feedforward neural networks: Have no memory of past input. Only an eternal present. Recurrent neural networks: Have memory of past input. e.g., Elman (1991) 5 6 Feedforward Networks Recurrent Networks Output1 Output2 Output3 Output1 Output2 Output3 Input1 Input2 Input3 Input1 Input2 Input3 Input sequence 1 Input sequence 1 Output2 Output1 Output3 Output2 Output1 Output3 111 111 111 111 111 111 111 11 Input2 Input1 Input3 Input2 Input1 Input3 Input sequence 2 7 Input sequence 2 8
Time, in the Context of Neural Networks Feedforward nets: Reactive Living in the eternal present No past, no future, no time Recurrent nets: Contemplative Memories of the past Research Questions Recollection Prediction Past Present Future [Q1] how did recollection (memory) evolve? - From feedforward to recurrent architecture [Q2] how did prediction evolve? - Emergence of prediction in recurrent architecture Dynamic Note: The brain is a recurrent net e.g., Elman (1991) 9 1 Recollection in Feedforward Networks? Part I: Recollection Is it possible for a feedforward network to show memory capacity? What would be a minimal augmentation? Largely based on Chung et al. (29) Idea: allow material interaction, dropping and detecting of external markers. 11 12
Memory Task: Catch the Balls Three Networks A 5 distance sensors B C Evolve three different networks: B1 B2 Feedforward speed = 1 θ speed = 2 D E Recurrent agent Dropper/Detector (with Feedforward net) cf. Beer (2); Ward and Ward (26) Agent with range sensors move left/right. Must catch both falling balls. Memory needed when ball goes out of view. 13 14 Feedforward Network Recurrent Network O(t) H(t) I(t) v ji w kj O 1 O 2 H 1 H 2 H 3 z 1 I 1 I 2 I 3 I 4 I 5 u jl u jl u jl H 1 H 2 H 3 H 1 H 2 H 3... H 1 H 2 H 3 H(t 1) λ H(t 2) λ λ H(t N mem ) Stardard feedforward network. Standard recurrent network (Elman 1991). 15 16
Feedforward Net + Dropper/Detector Results: Feedforward O 1 O 2 O 3 Feed-Forward Network I 1 H 1 H 2 H 3 I 2 I 3 I 4 I 5 I 6 I 7 if O 3 > θ, DropMarker = True (1) else, DropMarker = False (2) (1) (2) Catch Performance (%) 1 8 6 4 2 Fast Left Ball Fast Right Ball On average, only chance-level performance (5%). Feedforward network plus: Extra output to drop markers. Always move to the fast ball. Randomly pick fast or slow ball and approach it. Extra sensors to detect the markers. 17 18 Results: Recurrent vs. Dropper Behavior (Short Sensors) 6 Catch Performance (%) 1 8 6 4 2 Recurrent Network Fast Left Ball Fast Right Ball Catch Performance (%) 1 8 6 4 2 Dropper Network Fast Left Ball Fast Right Ball position 4 2-2 -4-6 No difference in performance between dropper/detector net (right) and recurrent network Left 1 Left 2 Left 3 Trial (time) Right 1 Recurrent (Short Memory, SM) Dropper (Short Sensor, SS) Right 2 Right 3 (left). Slight overshoot and drop the marker. 19 Subsequent move repelled away from the marker. 2
Behavior (Long Sensors) Part I Summary position 6 4 2-2 -4 Reactive, feedforward networks can exhibit memory-like behavior, when coupled with minimal material interaction. Adding sensors and effectors could have been easier than adjusting the neural architecture. -6 Left 1 Left 2 Left 3 Trial (time) Right 1 Recurrent (Long Memory, LM) Dropper (Long Sensor, LS) Slight overshoot and drop the marker. Right 2 Subsequent move repelled away from the marker. Right 3 21 Transition from external olfactory mechanism to internal memory mechanism? Similar results obtained in 2D foraging task (Chung and Choe 29). 22 Emergence of Prediction in RNN? C z Part II: Prediction A Output Hidden x y z... Input feedback x B Activation x y z y time Largely based on Kwon and Choe (28) Can prediction emerge in internal state dynamics? Idea: Test if (1) internal state dynamics is predictable in evolved recurrent nets, and (2) if that 23 correlates with performance. 24
Task: 2D Pole Balancing x High ISP y θy Low ISP θx Example Internal State Trajectories Anderson (1989) Standard 2D pole balancing problem. Typical examples of high (top) and low (bottom) ISP. Keep pole upright, within square bounding region. High ISP=predictable, Low ISP=unpredictable. Evolve recurrent neural network controllers. Note: Both meet the same performance criterion! 25 Measuring Predictability 26 Experiment: High vs. Low ISP All Controllers High perform. Controllers te al sta intern is s ly a n a High ISP evolutionary selection process inter n analy al state sis Train a simple feedforward network to predict the Low ISP internal state trajectories. 1. Train networks to achieve same performance mark. Measure prediction error made by the network. High vs. low internal state predictability (ISP) 2. Analyze internal state predictability (ISP). 3. Select top (High ISP) and bottom (Low ISP) five, and 27 compare their performance in a harder task. 28
Generation Number Number of Steps Prediction Success Rate (%) Results: Internal State Predictability Comparison High ISP and Low ISP (ISP) 1 9 Comparison of High and Low Predictability 8 7 6 5 4 3 High Low 2 1 1 2 3 4 5 6 7 8 9 1 Test Case Number Trained 13 pole balancing agents. Chose top 1 highest ISP agents and bottom 1 lowest ISP. high ISPs: µ = 95.61% and σ = 5.55%. low ISPs: µ = 31.74% and σ = 1.79%. A comparison of the average predictability from two groups: high ISP and low ISP. The predictive success rate of the top 1 and the 29 bottom 1 agents. 3 Results: Learning Time Performance and Int. State Dyn. Performance and Internal State Dynamics 16 Learning Time 6 5 14 4 12 1 8 6 4 2 High Low 3 2 1 1 2 3 4 5 6 7 8 9 1 High Low 1 2 3 4 5 6 7 8 9 1 Test Case Number Test Case Number No significant difference in learning time Made the initial conditions in the 2D pole balancing task harsher. Performance of high- and low-isp groups compared. High-ISP group outperforms the low-isp group in the changed environment. 31 32
Prediction Success Rate (%) Behavioral Predictability Behavioral Predictability 9 8 7 6 5 High 4 Low 3 2 1 x pos y pos x angle y angle Examples of cart x and y position from high ISP.2.1.5 -.6 -.4 -.2.2.4.6.8 1 1.2 -.2.1.2.3.4.5.6 -.5 -.4 -.1 -.6 -.15 -.8 -.2-1 -.25-1.2 -.3-1.4 -.35-1.6 -.4 Success of high-isp group may simply be due to simpler behavioral trajectory..7.6.5.4.3.2.1 -.6 -.5 -.4 -.3 -.2 -.1.1 -.1 1.6 1.4 1.2 1.8.6.4.2.2.4.6.8 1 1.2 However, predictability in behavioral predictability is no different between high- and low-isp groups. Behavioral trajectories of x and y positions show complex trajectories. 33 34 Examples of cart x and y position from low ISP.6.6.4.5.2.4.3 -.25 -.2 -.15 -.1 -.5.5.1.15.2.25 -.2.2 -.4 Part II Summary Simulations show potential evolutionary advantage of predictive internal dynamics..1 -.35 -.3 -.25 -.2 -.15 -.1 -.5.5 -.1.4.2 -.2 -.15 -.1 -.5.5.1 -.2 -.4 -.6 -.8-1 -1.2 1.5 -.2.2.4.6.8 1 -.5 Predictive internal dynamics could be a precondition for full-blown predictive capability. -.6 -.8-1 -1.2-1 -1.5-2 Behavioral trajectories of x and y positions show complex trajectories. 35 36
Discussion No memory Olfactory system? Memory (External) Hippocampus? Memory (Internal) Predictive dynamics Wrap-Up Present Past Future From external memory to internalized memory (cf. Rocha 1996). Analogous to olfactory vs. hippocampal function? Pheronomes (external marker) vs. neuromodulators (internal marker)? 37 38 Discussion (cont d) Future Work No memory Olfactory system? Memory (External) Hippocampus? Memory (Internal) Predictive dynamics Implications on the evolution of internal properties invisible to the process evolution. Consciousness Self (subject of consciousness) Subject of action Authorship (property of action) 1% predictable (property of authorship, objectively investigatable) 39 Present Past Future Actual evolution from dropper/detector net to recurrent net. Actual evolution of predictor that can utilize the predictable dynamics. 4
Conclusion Other Projects From reactive to contemplative to predictive. Brain connectomics project Recollection: External material interaction can be a low-cost intermediate step toward recurrent Delay, delay compensation, and prediction architecture. etc. Prediction: Predictable internal state dynamics in recurrent neural nets can have an evolutionary edge, thus prediction can and will evolve. Time is essential for neural networks, and neural networks gives us time. 41 Knife-Edge Scanning Microscope 42 Delay Comp.: Flash-Lag Effect Line scan Camera M Light source icr os co pe ob j ec tiv e Diamond knife Specimen FLE Actual Perceived Nijhawan (1994) Various other FLEs exist (orientation, luminance, etc.). Choe et al. (28); Mayerich et al. (28) Connectomics for the whole mouse brain. Delay compensation methods at the synaptic level (Lim 1µm3 resolution, 2TB of data per brain. and Choe 25, 26, 28). 43 44
References Anderson, C. W. (1989). Learning to control an inverted pendulum using neural networks. IEEE Control Systems Magazine, 9:31 37. Beer, R. D. (2). Dynamical approaches to cognitive science. Trends in Cognitive Sciences, 4:91 99. Choe, Y., Abbott, L. C., Han, D., Huang, P.-S., Keyser, J., Kwon, J., Mayerich, D., Melek, Z., and McCormick, B. H. (28). Knife-edge scanning microscopy: High-throughput imaging and analysis of massive volumes of biological microstructures. In Rao, A. R., and Cecchi, G., editors, High-Throughput Image Reconstruction and Analysis: Intelligent Microscopy Applications, 11 37. Boston, MA: Artech House. Chung, J. R., and Choe, Y. (29). Emergence of memory-like behavior in reactive agents using external markers. In Proceedings of the 21st International Conference on Tools with Artificial Intelligence, 29. ICTAI 9, 44 48. Chung, J. R., Kwon, J., and Choe, Y. (29). Evolution of recollection and prediction in neural networks. In Proceedings of the International Joint Conference on Neural Networks, 571 577. Piscataway, NJ: IEEE Press. Elman, J. L. (1991). Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 7:195 225. Lim, H., and Choe, Y. (25). Facilitatory neural activity compensating for neural delays as a potential cause of the flashlag effect. In Proceedings of the International Joint Conference on Neural Networks, 268 273. Piscataway, NJ: IEEE Press. Lim, H., and Choe, Y. (26). Delay compensation through facilitating synapses and STDP: A neural basis for orientation flash-lag effect. In Proceedings of the International Joint Conference on Neural Networks, 8385 8392. Piscataway, NJ: IEEE Press. Lim, H., and Choe, Y. (28). Extrapolative delay compensation through facilitating synapses and its relation to the flash-lag effect. IEEE Transactions on Neural Networks, 19:1678 1688. Mayerich, D., Abbott, L. C., and McCormick, B. H. (28). Knife-edge scanning microscopy for imaging and reconstruction of three-dimensional anatomical structures of the mouse brain. Journal of Microscopy, 231:134 143. Nijhawan, R. (1994). Motion extrapolation in catching. Nature, 37:256 257. Rocha, L. M. (1996). Eigenbehavior and symbols. Systems Research, 13:371 384. Ward, R., and Ward, R. (26). 26 special issue: Cognitive conflict without explicit conflict monitoring in a dynamical agent. Neural Networks, 19(9):143 1436. Kwon, J., and Choe, Y. (28). Internal state predictability as an evolutionary precursor of self-awareness and agency. In Proceedings of the Seventh International Conference on Development and Learning, 19 114. IEEE. 44-1 44-2