Instructor: Anthony Kuh POST 205E / 484 Holmes EE645 Machine Learning Fall 2009 Dept. of Electrical Engineering University of Hawaii Phone: 956-7527, 956-4214 Email: kuh@hawaii.edu
Preliminaries Class Meeting Time: MW 9:10-10:25(389 Holmes) after this week Website: Laulima Office Hours: MW 10:30-12 or by appointment Probability: EE342 or equivalent Random variables, Bayes analysis, Gaussian RVs and Gaussian processes Linear Algebra: vector and matrix operations Programming: Matlab or C experience
Objectives and Grading Topics: Machine learning, pattern recognition, signal processing, neural networks, applications Objectives: obtain basic understanding and knowledge of fundamental concepts of machine learning, learn about current research in area, conduct project on topic of current research Grading: Homework: 30% Exam:30% Final project: 40% (oral presentation and written report)
Motivation Develop paradigms for learning that mimic features of natural learning for applications in engineering and science Processing data: CPUs and storage device technology have improved dramatically, algorithm development to process data has not increased nearly as rapidly Multidisciplinary area requiring tools from EE, CS, Statistics, Physics, Math, Biology
Overview of Course Material Linear algorithms for classification and regression Linear Threshold Unit (Perceptron Learning Algorithm) Optimum margin classifiers Linear Unit LMS Algorithm Least Squares Algorithm
Overview Continued Kernel Methods Optimization methods Kernels Support Vector Machines Least Squares kernel algorithms On-line algorithms Other learning algorithms Generative classifier: Naive Bayes Discriminative classifier: Logistic regression Multilayer networks: Backpropagation
Overview Continued Learning Theory Tools Bayesian decision theory Learning and generalization Structural risk minimization Dimensionality and generalization bounds Graphical Models Bayesian Networks Conditional independence Inference
Overview Continued Other Topics Mixture Models and EM Ensemble Learning and boosting Unsupervised Learning Component Analysis: PCA, Kernel PCA, ICA Competitive Learning Self Organizing Feature Maps Vector quantization
Overview Continued Reinforcement learning: Markov decision processes and dynamic programming TD learning, Q learning
Historical Notes 1940s: Hebb, The organization of behavior, McCulloch-Pitts model, Von Neumann 1950s-1960s: Rosenblatt, Minsky-Papert, Perceptrons, artificial intelligence, Widrow 1970s-1980s: Pioneers (Grossberg, Amari, Kohonen), Hopfield, PDP Group 1990s-2000s: Multidisciplinary area (machine learning, statistics, physics, biology), mathematical rigor (learning theory, kernel methods, reinforcement learning,bayesian learning, unsupervised learning)
Applications Character recognition Text classification Biomedical classification: disease diagnosis Bioinformatics: gene sequencing and protein classification Time series prediction Communication applications
References C. Bishop. Pattern Recognition and Machine Learning. Springer, 2006. T. Hastie, R. Tibshirani, and J. Friedman. Elements of Statistical Learning, Springer, 2003. J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, 2004. S. Haykin. Neural Networks and Learning Machines. 3 rd Ed. Prentice Hall, Englewood Cliffs,NJ, 2008. R. Duda, P. Hart, and D. Stork. Pattern Classification. 2 nd Ed. Wiley, 2000. B. Scholkopf and A. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization and Beyond. MIT Press, Cambridge, MA, 2002. N. Cristianini and J. Shawe-Taylor. An Introduction to Support Vector Machines: and other Kernel Based Learning Methods. Cambridge University Press, Cambridge, UK, 2000. Websites: IEEE CIS, INNS, Neural Computation, NIPS, IJCNN, kernel machines, machine learning