6. Learning and Adaptation

Transcription

1 Autonomous Systems Tutorial: Part II 6. Learning and Adaptation David J. Atkinson, Ph.D Senior Research Scientist

2 Outline Review: Types of Knowledge Why Learn and Adapt? Desired Capabilities Bootstrapping a Mind Key Theoretical Questions

3 Types of Knowledge Declarative: Statements of fact (beliefs) Procedural: How to perform tasks (skills) Semantic: Relations of objects, situations (conceptual) Episodic: Entities and events encountered (cases) Meta-Knowledge: The agent's own capabilities (self)

4 Why Learn and Adapt? Why should an autonomous system learn and adapt? It is highly unlikely it knows everything it needs Some of what it knows may be irrelevant What it knows likely contains errors It's problem-solving skills are sub-optimal Even if that was not the case: It will be tasked and used in unanticipated ways We require that agent performance improves Uncertainty dominates The world is constantly changing Predicting the actions of other agents is difficult A great deal of relevant information may be hidden or not readily observable Realistically, confidence in information is never absolute

5 Practical Matters Knowledge engineering has proven to be difficult, error-prone and incomplete for domains of any significant complexity Procedural knowledge is especially difficult to encode Problem-solving skills can benefit immensely by optimization as a result of learning Knowledge and skills can rapidly become obsolete unless continually assessed and improved Autonomous systems interacting with human users must be able to adapt to new contexts and at extended time-scales, in a variety of environments that cannot be foreseen at during design and development.

6 Desired Capabilities To learn, a system must make effective judgments about: Similarity Representativeness Randomness Coincidences as clues to hidden causes Causal strength and evidential support Essential for: Diagnostic and conditional reasoning (causal knowledge) Predictions about events (episodic knowledge) Correctly identifying new instances of objects, actors, and situations (semantic knowledge)

7 Bootstrapping a Mind To go beyond the data requires other sources of data and processes that make up the difference Something more abstract must generate and delimit potential hypotheses or meaningful generalization would be impossible (computationally intractable) Psychologists and Linguists: Constraints Artificial Intelligence: Inductive Bias Statisticians: Priors The key question is what data, information, or process is needed to bootstrap learning knowledge

8 Three Key Theoretical Questions 1) How does abstract knowledge guide learning and inference from sparse data? Question of Constraints and Inductive Bias 2) What forms does abstract knowledge take, across different domains and tasks? Question of Representation 3) How is abstract knowledge itself acquired? Question of Cognitive Development From (Tenenbaum, et. al., 2011)

9 Schools of Thought Associative Learning (connectionism) Simple, unstructured forms of knowledge Statistical learning; correlations Assumes knowledge is induced with trivial mechanisms Learning is about adjust weights, strengths, parameters Example: Artificial Neural Networks (ANNs) Conceptual Learning (semanticism) Symbolic, richly structured knowledge Logical, heuristic, other non-statistical methods Assumes (some degree) of abstract knowledge is innate Learning is about discovery of rich symbolic structures Example: Explanationbased learning

10 Acquisition of Abstract Knowledge Discovering a structure (form) for the properties and data about objects enables new inferences Clusters; nameable categories, tree-like hierarchies Associative learning algorithms assume a single fixed structure (e.g., clusters) Cannot learn other forms Conceptual learning algorithms start with some knowledge of multiple structures, then adapt data to the one(s) that fit the best Capable of learning new forms

11 Importance of Representation Representations The type of structured symbolic form(s) used has a strong influence on ease of encoding of concepts and inference Imposes constraints on induction (generalization) Compact representations reflect real-world granularity and make reliable induction easier and computationally efficient Neural Network vs. Belief Networks Distributed representation Network variables have only one degree of activation Once trained, inference can be executed in linear time Localized representation Network nodes may have many active dimensions (properties, range of values, probabilities) General inference is NP-Hard (computationally complex) Associative Conceptual

12 Graph Representation Every Every form form of of abstract abstract knowledge knowledge can can be be represented as as a graph. graph. The The principles principles of of the the form form are are equivalent equivalent to to a grammar grammar for for growing growing graphs graphs of of that that form form learning learning grammars grammars == == learning learning new new forms forms Very useful! We have rigorous mathematical tools for analyzing graphs and grammars to make formal proofs Different machine learning algorithms work may work better with graphs or with grammars Now we know we can (theoretically) transform one into the other

13 Cognitive Models Cognitive model = structured symbolic forms and the processes that operate on them Important model properties enable machine learning: Generative: Supports hypotheses about hidden variables Abstract: Represents not only specific situations but classes over which generalization is possible

14 Inference in Learning Deduction Knowledge-intensive Explain and analyze an example instance Apply generalized concepts to infer facts about new instances Induction Data-intensive; requires many examples Generate a general description of a concept Abduction The art of good guessing - making reasonable hypotheses Identify an explanation of the sufficient conditions for describing a concept (there may be many explanations) Motivates simple, efficient explanations (e.g., Occam's razor)

15 Putting it all together Associative and Conceptual learning algorithms have each proven to be useful for different classes of problems Historically, these have been separate developments with different communities of interest Difficult conceptually to unite them in theory or practice Recently, Bayesian learning methods have shown a bridge between the two schools of thought: Hierarchical Bayesian Models combine richly structured, expressive knowledge representations with powerful statistical (probabilistic) inference engines The best of both conceptual and associative learning!

16 Hierarchical Bayesian Models Key insight: multiple levels of hypothesis networks arranged hierarchically can be used to address the origins of the hypothesis spaces and priors (probabilities) Hypothesis spaces of hypothesis spaces! Each layer generates a probability distribution on variables at the level below; higher levels pool variables from below Advantage: Hypotheses and priors can be learned at longer time scales while still constraining lower level learning (thus avoiding computational intractability)

17 Hierarchical Bayes Models (HBM) Can discover the basis of similarity in a problem domain: Can infer the correct (and best) forms of structure (grammars) for many domains Can learn abstract causal knowledge and specific causal relations at a level below Fast (polynomial), from relatively little data HMBs have been effectively applied to a wide range of analysis and learning problems in multiple domains

18 Other Dimensions of Machine Learning Conceptual vs. Associative Blends such as Hierarchical Bayes Models Orthogonal dimensions Supervised vs. Unsupervised Off-line vs. On-line (learning while doing; active) Many possible hybrid techniques are possible... This is very much the frontier of research!

19 Supervised Learning The learner is provided with labeled training data, examples such as (instance, class) An instance is a vector of features A learning system may be given many sets of training data The learning algorithm infers a function from the training data: called a Classifier (assumes discrete data) A Classifier is valid if it produces the correct out given a new instance of an unknown class Inductive reasoning Key challenges: What are the important features of an instance? Bias vs. variance (flexibility vs. consistency) Amount of training data vs. complexity of classifier Dimensionality of features (supervisor should reduce #)

20 Unsupervised Learning Conventional algorithms for unsupervised learning assume a single fixed form of structure is to be discovered Hierarchical clustering, principal components analysis, multidimensional scaling, clique detection Cannot learn multiple forms of structure, or discover new forms in novel data Examples: Genetic / Evolutionary Algorithms Neural networks (Self-organizing map; Adaptive resonance theory) Statistical methods (clustering; density estimation)

21 Off-line vs. On-line An off-line (passive) learner simply watches the world going by, and tries to learn the utility of being in various states An on-line (active) learner must also act using the learned information, and can use its problem generator to suggest explorations of unknown portions of the environment May be more typical of autonomous systems, although both are useful

22 Learning by Doing Includes learning while planning and learning while executing a plan (active, on-line learning) Motivation: The knowledge in domain theory is not usually effective/efficient ab initio An agent must learn how to use knowledge Learning in Planning: Opportunities Search Efficiency: Learn control knowledge to guide a planner though the search space Domain Specification: Learn the preconditions and effects of the planning actions Quality: Learning control knowledge to create higher quality plans Widely used methods Explanation-based learning Reinforcement learning

23 Explanation-based Learning A deductive learning method Purpose is not to learn more about target concept To re-express target concept in a more operational manner Control learning leads to greater efficiency Domain theory contains the information Not usually effective; rarely complete Examples focus on the relevant operational knowledge: Characterize only examples that actually occur Very useful in learning how to plan

24 Explanation-based Learning Inputs: Target concept definition Training example Domain theory Operationality criterion Output: Generalization of the training example that is: Sufficient to describe the target concept, and Satisfies the operationality criterion (adapted from Veloso and Simmons, 2010

25 SAFE-TO-STACK Example

26 SAFE-TO-STACK Example

27 SAFE-TO-STACK Example

28 Generating Operational Knowledge

29 Reinforcement Learning Concerned with maximizing reward by appropriate action The agent receives some evaluation of its action (such as a hefty bill for rear-ending the car in front) but is not told the correct action A general problem studied by many disciplines Typically formulated as a Markov Decision Process (MDP) or Partially Observable Markov Decision Process (POMDP) Focus is on-line performance, a balance of exploratory learning with using that knowledge to accomplish tasks Agent chooses actions, gets reward, then adapts selection function Especially useful when the only way to get information is by interacting with the environment (no training data) Many uses in robot control Adaptation

30 Ubiquitous Learning (Forbus, 2009) People learn continually in all sorts of situations Computers (typically) learn only when directed Consumes most or all resources Incompatible with highly interactive systems Ubiquitous learning aims to learn constantly: Compute-intensive learning tasks are off-loaded to background processing on dedicated notes Learning is focused via explicit learning goals constructed on the fly, prioritized, scheduled, reasoned about Not just learning about domain knowledge Learning how knowledge is communicated Learning about agent's own expertise and understanding This will be essential for long-lived autonomous agents

31 Learning vs. Adaptation Learning: Adds possibilities Conceptual, abstract Relations Generalization Adaptation: Constrains possibilities Concrete Parameters Refinement, tuning Both are important to intelligently react to change and to improve performance

32 Ultimate Challenge of Learning Formalizing the content of all intuitive theories requires Turing-complete compositional representations; not yet invented Probabilistic first-order logic Probabilistic programming languages But we can usually do good enough Consequence: => formally proving correctness of (most) learning systems is still a major stretch

33 Challenges for Humans Introspection, Learning and Bootstrapped Ontologies Autonomous learning systems will develop their own ways of clustering phenomena What they've been exposed to Their successes and failures They will use this information to optimize themselves Internal problem-solving capabilities States and Processes No one else will be able to understand this intuitively No one else has the identical history of experience! Subsequent effects of using those personal concepts may exacerbate the complexity and idiosyncratic character of the autonomous agent's internal processing

34 Challenges for Humans For many if not most machine learning algorithms, it is hard to see where human input can make an impact...possibly: Selection of training examples Ordering presentation Providing criticism, reward The products of associative learning are hard to explain because they are distributed and have little structure Statistical, reinforcement, fuzzy, genetic algorithms Undoing what has been learned is very hard 2 nd Order logics let us retract beliefs theoretically Long-lived learning No systems have learned over extended periods

35 ARMAR-III learns about objects and what actions can be applied to them by touching and manipulating, with human guidance, hints and demos. M. Cakmak, Georgia Tech Simon learns concepts from a human teacher through demonstration, asking questions and active learning Autonomous Agents that Learn Dexter performs cooperative assembly task with human and learns how to: Shichao Ou, Univ. Mass - generate effective expressive behavior - build robust, scalable knowledge model of humans - recognize human behavior and infer human intention Embodied cognition Tamim Asfour, Karlsruhe Inst. Tech

36 Thank you! Questions?