Machine Learning and Expert Systems By Alexander Bailey Introduction Why use machine learning? Strategies Usefulness and success Collaborative Systems 1
Why Use Machine Learning? Knowledge acquisition bottleneck ML helps in the knowledge elicitation process. Problems with KE Difficult to find experts Experts disagree Inconsistent knowledge Paradox of expertise There must be an expert! 2
What to do? Machine learning techniques can be used to attempt to tackle these problems. Shows lots of promise. Caveats: Knowledge is dynamic. Must be updated as experience is gained. There are limits to how ML can be applied! 3
Ideas Machine Learning is important for the long-term success of advanced AI systems. Hybrid systems should be built. 4
Primary Application of ML The primary application of machine learning in expert systems is to attempt to solve the knowledge acquisition bottleneck. Idea: Build knowledge base from a set of examples. Agreeable format, humans and machines can read. Many algorithms exist to do such a thing. ID3 and successors, GA. 5
Focus: ID3 An inductive inference algorithm Given a set of data, will try and build a decision tree which can be converted into rules. A top-down approach The idea is to select the next node that yields the highest information gain recurse. Terminate when growing tree makes no statistically significant difference in solution quality. ID3 is fast, simple, and produces small trees. 6
Focus: ID3 Problems Not very robust when noise exists in the data Guided locally by information gain. Prone to finding local optimums Equivalent to hill-climbing Hard to obtain different rules with near-by accuracy. There are strategies that overcome this. 7
Focus: Genetic Algorithms Genetic Algorithms can be used to build decision trees, or even rules expressed in predicate or first order logic. Genetic operators can be performed on decision trees. Subtree crossover. Fitness Function is the error. Speculation: Suffers from many of the same problems as GP. Decision tree bloat. 8
EFOPREL EFOPREL system by Vladamir Estivill-Castro. Inductive supervised learning system. Uses GAs to build rules in first order predicate logic. Knowledge is explicit for human understanding. 9
EFOPREL BNF of logic rules used by EFOPREL <Logic_Rule> <Consequent> <Antecedent> <Bool_Expr> ::= IF <Antecedent> Then <Consequent> ::= <Class_Label> ::= <Boolean_Expression> ::= <Predicate> <Bool_Expr> AND <Bool_Expr> <Bool_Expr> OR <Bool_Expr> NOT <Bool_Expr> Can also operate on non-discrete values Very_*High_Ai 10
EFOPREL Rule Encoding example: Fisher - Iris dataset If normal petal width(x) OR high petal width(x) OR normal petal length(x) Then versicolor 11
EFOPREL Rule Simplification is employed to reduce the size of the trees generated. Performs very well in comparison with ID3 and similar algorithms. Problems: Much slower than single-pass algorithms such as ID3. Because it is a GA it may not always produce the same tree with the same data. 12
What to do? Idea: Learning and evolution are not quite the same thing. GA is an evolutionary system while ID3 is a learning system. Evolution and learning can work together. [Hinton and Nowlan, 1987] Learning eases the pressure on evolution. 13
Focus: Hybrid Hybrid System for Pattern Classification [Bala and DeJong, 1995] 14
Focus: Hyrbid Performs very well! 15
ML's Dirty Little Secret How do ML techniques compare to traditional KE approaches? Recall the intro slide, there are limits to what ML can do for Knowledge acquisition. Paper by D Grzymala-Brusse and J Grzymala- Brusse. Any machine learning approach that uses learning by example produces only a subset of all potential rules. An exact covering by example is difficult and may be impossible. 16
ML's Dirty Little Secret There are few comparative studies! Grzymala-Brusse study is contrived, but illustrates the point. 17
ML's Dirty Little Secret Comparison 18
Collaborative Systems EVOPROL and EFOPREL Have a collaborative aspect! Idea: Allow human interaction with the evolutionary process. Rules are in a human-readable form. Human expert can add, modify or create partial rules as the GA runs. Better results than non-collaborative systems! 19
Conclusion Machine Learning can greatly speed up the process of knowledge acquisition. Evolutionary methods provide a powerful approach. Machine Learning can not, at present state, replace the knowledge engineer. SkyNet is a long way off! 20
Citations Bala J., Huang J., Vafaie H., Hybrid Learning Using Genetic Algorithms and Decision Trees For Pattern Classification, IJCAI Conference, Montreal, August, 1995. Grzymala-Brusse D., Grzymala-Brusse J., On The Usefulness Of Machine Learning Approach To Knowledge Acquisition, Computational Intelligence 11, 1995. Vladimir Estivill-Castro, Integration of Machine Learning and Knowledge Acquisition with a Genetic Algorithm, 1997. Vladimir Estivill-Castro, "Collaborative Knowledge Acquisition with a Genetic Algorithm," Tools with Artificial Intelligence, IEEE International Conference on, pp. 0270, 9th International Conference on Tools with Artificial Intelligence (ICTAI '97), 1997. Fogel, D., Hanson, J. C., Kick, R., Malki, H. A., Sigwart, C., Stinson, M., and Turban, E. 1993. The impact of machine learning on expert systems. In Proceedings of the 1993 ACM Conference on Computer Science (Indianapolis, Indiana, United States, February 16-18, 1993). CSC '93. ACM, New York, NY, 522-527. DOI= http://doi.acm.org/10.1145/170791.171158 21