ECE 5258 - Pattern Recognition Syllabus Fall 2014 Dr. Georgios C. Anagnostopoulos August 11, 2014 (ver. 1.0) 1. Contents 2 Course Description 2 2.1 Objectives & Outcomes................................... 2 2.2 Target Audience & Prerequisites.............................. 2 3 Instructor Information 3 4 Meeting Times 3 5 Course Resources 3 6 Performance Assessment & Grading Policy 4 7 Tentative Course Schedule 4 8 Other Important Dates 6 9 Lecture Contents 6 10 Course Conduct 7 10.1 Lectures............................................ 7 10.2 Homework........................................... 7 10.3 Mini-Projects.......................................... 7 10.4 Individual Course Project.................................. 8 10.5 Oral Exam........................................... 9 10.6 Online Course Management................................. 9 10.7 Student-Instructor Interaction................................ 9 10.8 Make-Up Policy........................................ 9 10.9 Academic Dishonesty Policy................................. 9 11 Disclaimer 10 1
2. Course Description Pattern recognition is a major branch of machine learning that deals with the study of the broad subject of machine-based recognition. Intelligent systems that are able to recognize fingerprints, our voice commands, recognize text, detect human faces in a photo, intrusions to computer networks or detect signs of extraterrestrial life from cosmic radiation have been around now for a while. Moreover, year by year their sophistication and the breadth of their applicability is rapidly increasing. The basic drive behind this discipline is the effort to design and implement machines that can compete with humans in the ability to recognize objects and occurrences of important events. Even further, we are often in need to recognize objects and events, whose analysis of associated characteristics and measurements are beyond human capabilities. All these reasons have led to making pattern recognition a booming, interdisciplinary direction of research that is already playing in important role in our everyday life (see, for example, Apple s intelligent personal assistant, Siri). 2.1 Objectives & Outcomes ECE5258 is an introductory course to pattern recognition and detection. The student attending this course will be first introduced to the theoretical fundamentals of recognition. Next, a variety of recognition models will be examined, ranging from simple to the more sophisticated. The class is project-centered so that it can also offer the necessary opportunities to students to explore a variety of the subject matters aspects. Even more, the student will engage in a major project to acquire hands-on experience and further develop the necessary analytical skills and experiences. 2.2 Target Audience & Prerequisites The course is intended for graduate students or experienced seniors (with instructor permission) of engineering or science majors that already have sufficient background in Calculus III, Linear Algebra and Probability & Statistics at the undergraduate level. Furthermore, ECE5258 is a core course for graduate students wishing to pursue a specialization of Machine Intelligence in Computer Engineering. Apart from the aforementioned background, a basic competency (not mere familiarity) in computer programming (in some high level language) is required. Such competency is essential to gain practical understanding of the course material via its projects. In fact, a background in using and programming in an integrated computational and visualization platform, such as MATLAB TM by MathWorks (commercial), Mathematica TM by Wolfram Research (commercial), GNU Octave by John W. Eaton (a free MATLAB clone) and R by the R Development Core Team (free) a library-rich high level language like Python can be quite advantageous for the purposes of this course, especially, the first kind. Former attendees of the course have been M.S. and doctoral students of engineering (Electrical, Computer, Systems and Mechanical Engineering) and sciences (Computer Sciences, Operations Research, Applied Mathematics and Physics). 2
3. Instructor Information Dr. Georgios C. Anagnostopoulos Associate Professor, Electrical & Computer Engineering Room 345, Olin Engineering Building Florida Institute of Technology 150 West University Boulevard Melbourne, Florida 32901-6975, USA. +1.321.674.7125 (office) +1.321.674.8192 (fax) georgio AT fit DOT edu http://my.fit.edu/ georgio 4. Meeting Times Lectures Tuesdays & Thursdays, 17:00 18:15 (1h 15min) Aug 19 th (first lecture) - Dec 2 nd (last lecture) Room 230, Crawford Tower, Melbourne Main Campus Office Hours My office hours will be always kept current and posted outside my office and on my website. In order to accommodate and serve you best, please make an appointment before meeting with me in my office. 5. Course Resources Material will be drawn primarily from the two textbooks shown below. Additional material may be supplied during the course of the lectures in the form of lecture notes. 1. Pattern Recognition & Machine Learning, C.M. Bishop, Springer Science+Business Media, LLC, New York, NY, 2006. 2. Pattern Classification, R.O. Duda, P.E. Hart, D.G. Stork, 2 nd Ed., Wiley & Sons, Inc., New York, NY, 2001. Additional valuable references are 3. Pattern Recognition, S. Theodoridis & K. Koutroumbas, 4 th Ed., Academic Press, 2008. 4. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, T. Hastie, R. Tibshirani & J. Friedman, 2 nd Ed., Springer, 2009. 3
Finally, here are MS Windows installers for useful free software GNU Octave: http://mxeoctave.osuv.de/ R: http://cran.cnr.berkeley.edu/ Python (latest 2.x & 3.x versions): https://www.python.org/download/ 6. Performance Assessment & Grading Policy This course considers the following student performance assessment instruments: (i) homework, (ii) Mini-Projects (MPs), (iii) an Individual Course Project (ICP) and (iv) an oral exam. The score weighting of these instruments is displayed below. Instrument Weighting Homework 10% Mini-Project I 15% Mini-Project II 15% Mini-Project III 15% Individual Course Project 25% Oral Exam 20% The overall course score to letter grade conversion that will be used for this course is the standard one and is depicted in the table below. As a reminder, for graduate students, a letter grade of D or lower is not a passing grade. Overall Score Letter Grade 90 100 A 80 89 B 70 79 C 60 69 D 0 59 F 7. Tentative Course Schedule TUESDAY Aug 19th 1 Linear Algebra Review 26th 3 Elements of Unconstrained Optimization Sep 2nd 5 Fundamentals of PR (cont.) THURSDAY 21st 2 Linear Algebra Review (cont.) 28th 4 Fundamentals of PR 4th 6 k-nearest Neighbor Classification 4
TUESDAY 9th 7 Parzen Windows Classifier 16th 9 Quadratic & Linear Discriminant Analysis (cont.) 23rd 11 Fundamentals of Kernel Methods 30th 13 Multinomial Regression ICP topics announced 7th 15 Support Vector Machines (cont.) 14th Fall Break (No lecture) 21st 18 k-means & Mean Shift Clustering 28th 20 (Kernel) Principal Component Analysis Nov 4th 22 Graph-based Dimensionality Reduction 11th Veteran s Day (No lecture) 18th 25 Generalization Bounds 25th 27 Recent Developments in PR Dec 2nd 28 ICP presentations ICP materials due THURSDAY 11th 8 Quadratic & Linear Discriminant Analysis MP-I announced 18th 10 Elements of Constrained Optimization 25th 12 Basis Function Classifiers MP-I due Oct 2nd 14 Support Vector Machines MP-II announced 9th 16 Mixture Models & Expectation Maximization ICP topic selection due 16th 17 Mixture Models & Expectation Maximization (cont.) MP-II due 23rd 19 Spectral Clustering MP-III announced 30th 21 Locally Linear Embeddings 6th 23 Ensembles of Classifiers MP-III due 13th 24 Ensembles of Classifiers (cont.) 20th 26 Generalization Bounds (cont.) 27th Thanksgiving Day (No lecture) 4th 29 Study Day (No lecture) 5
8. Other Important Dates Below is a list of additional important dates pertaining to the course. October 24 th 2014 Last day to withdraw with a letter grade of W. December 4 th & 5 th 2014 Oral Exams December 15 th 2014 Grades reported to the Registrars Office 9. Lecture Contents A brief content description of each lecture is provided next. Linear Algebra Review Vector (sub)spaces, basis, matrices in the context of linear transformations, vector & matrix norms, special matrices, orthogonal projections, eigenvalue decomposition, singular value decomposition, matrix definiteness. Elements of Unconstrained Optimization Multi-variate calculus, vector & matrix derivatives, necessary & sufficient conditions for extrema, saddle points, convexity, Lipschitz continuity, gradient descent, subgradient method. Fundamentals of PR mixture models, 0/1 loss function, Bayes decision criterion, Neyman-Pearson decision criterion, Min-Max decision criterion, outlier detection problem, performance assessment & comparison of classification models. k-nearest Neighbor Classification k-nn classification rule, proof of asymptotic generalization bounds, acceleration of classification rule via trees. Parzen Windows Classifier Probabilistic generative models, kernel density estimation, asymptotic properties, PWC. Quadratic & Linear Discriminant Analysis Multi-Variate Gaussian distribution, maximum likelihood estimation, labeled mixture of Gaussians, regularization of QDA, LDA & QDA decision boundaries. Elements of Constrained Optimization Feasible region, Karush-Kuhn-Tucker conditions, convex problems, proximal methods. Fundamentals of Kernel Methods Hilbert Spaces of functions, Reproducing Kernel Hilbert Spaces, positive definite kernels, properties & construction, kernel trick. Basis Function Classifiers Probabilistic Discriminative Models, basis functions, radial basis function classifiers, kernelization and training of basis function classifiers. Multinomial Regression Framework, invariances and training of MR models. 6
Support Vector Machines Maximum-margin formulation, primal & dual problem (derivation), Multi-Kernel Learning. Mixture Models & Expectation Maximization Gaussian mixture models, basics of EM philosophy, EM for maximum likelihood of mixture parameters, semi-supervised and transductive QDA/LDA. k-means & Mean Shift Clustering The Majorization/Minimization framework, K-Means loss function & algorithm, MSC algorithm derivation and properties. Spectral Clustering Similarity graphs, graph Laplacian, types of spectral clustering approaches. (Kernel) Principal Component Analysis PCA motivation & framework, Gram matrix based PCA, Kernel PCA. Locally Linear Embeddings Manifold learning, LLE framework and algorithm. Graph-based Dimensionality Reduction Motivation and properties of Diffusion Maps and related methods. Ensembles of Classifiers Linear combination schemes of classifiers, ADABOOST formulation, properties and algorithm. Generalization Bounds Introduction to VC dimension and Rademacher complexity measures along with bounds for different hypotheses spaces. 10. Course Conduct 10.1 Lectures Lectures will be delivered in a single 1h 15m segment, twice a week according to schedule. Lecture notes will be made accessible prior to class. In absence of lecture notes, students are encouraged to take their own notes. 10.2 Homework Homework assignments will be occasional and will primarily focus on theoretical aspects of past material discussed in class. Students are much encouraged to interact with each other on such assignments. However, each student needs to submit their own, individual work by the designated deadline. Note that no late homework submissions will be allowed. 10.3 Mini-Projects The announcement and due dates for each Mini-Project (MP) are indicated in the tentative class schedule depicted further below. Course participants will have about 9 days to complete their MP work. Late submissions will be penalized linearly by 1 point every 14.4 minutes, i.e. 100 points in 7
24 hours; submission of your MP beyond 24 hours will not be possible and, in case of no submission, you will be assigned a score of 0 for that particular MP. Each MP will consist of several parts, such as implementations of algorithms, performance of experiments and interpretation of results, questions on theory that may require analytical work (mathematical derivations, critical thinking questions, etc.) and, possibly, some literature review. Overall, the role of the MPs is to motivate you to gain significant intuition and understanding about new concepts and techniques discussed during past lectures. In the process you will get exposed to some typical instances of PR in scientific and/or engineering problems. The MP work you will submit has to reflect your personal effort and understanding; collaboration with other individuals is not allowed. On the other hand, you are strongly encouraged to interact with your instructor before submitting your MP-related work. However, do not ask your instructor to review any code/implementations. Also, you are strongly encouraged to start working on your MP as early as possible; do not overestimate your capabilities or underestimate the level of effort you need to invest in the MPs. 10.4 Individual Course Project Each student will select an Individual Class Project (ICP) to work over the period of about 8 weeks. The choice will be made among a small set of potential topics previously identified by your instructor. Student-originated suggestions may also be entertained as an exception. For example, doctoral students may be interested in a topic due to their particular research focus. Such suggestions are welcomed, as long as they are reviewed and approved by your instructor before any project work commences. Overall, the role of the ICP is to provide the student the opportunity to independently study material that is beyond the scope of the lectures and apply her/his knowledge/skills learnt in the course to an application domain, potentially, of her/his interest. Work on ICPs typically will consist of surveying the literature and implementing particular PRrelated algorithms, describing a specific application domain, gathering and processing relevant data to be used in modeling, explaining the experimental setting in use, reporting on experimental results and, finally, draw pertinent conclusions. Each ICP will have as deliverables (i) a project report in a predetermined format of a conference paper, (ii) presentation slides and (iii) a small demonstration. Deadline(s) to submit ICP deliverables will be announced in time and have to be strictly observed; no late submissions will be accepted. Presentations of projects will be held during the last week of classes. Each student will be presenting for 20 minutes in front of the class. If you would like to invite someone to attend your presentation, please feel free to do so, as long as your guest will not be disruptive to the presentation process. Finally, more specific details on the ICPs, such as topics, expectations and assessment approach, will be provided in due time. 8
10.5 Oral Exam At the end of the term, during the Study Days before the final exam period, we will hold oral examinations. Each student will be examined on an individual basis in the instructor s office for 20-25 minutes. The examination is rather informal and will consist of theory questions related to lecture material taught in class, as well as material you have turned in for the MPs or ICPs. At the end of each lecture and upon request of the class, you will be provided with questions that would be considered typical of an oral exam on the lecture matter under discussion. 10.6 Online Course Management FITs online course management system, Canvas (http://fit.instructure.edu), will be utilized to manage a couple of important course components. First of all, Canvas will be used as a repository of recourse materials, such as lecture notes. Furthermore, all deliverables connected to homework, MPs and ICPs must be submitted via Canvas; submissions via email will not be taken into account. If you are a new FIT student, please ask Tech Support to provide you with login credentials for Canvas. 10.7 Student-Instructor Interaction Students are strongly encouraged to interact with their instructor on a one-on-one basis for clarifications, questions, guidance, etc. as well as regarding their work on their deliverables. 10.8 Make-Up Policy No opportunity will be granted for make-up homework, MPs and ICPs, unless provable, legitimate reasons are offered. By legitimate reasons, one means circumstances that were clearly beyond the control of the student and that prohibited him/her from completing the particular assignment. By provable, it is meant that the student can produce proof/documentation that supports the claimed circumstances. Therefore, it is the students responsibility to turn in complete assignments on time. Moreover, please note that, due to logistical reasons, as well as reasons of fairness, there will be no additional opportunity (like extra credit assignments, etc.) to improve upon the attained total score for the course. In addition, a letter grade of I ( incomplete ) is reserved only for those individuals that qualify under Policy 5219 of FIT s Academic Standards. 10.9 Academic Dishonesty Policy As students pursuing graduate studies are held at higher overall academic standards, academic dishonesty, on part of a course participant, will not be tolerated. Appropriate disciplinary action will definitely be pursued starting with failing the course. If necessary, please consult Policy 2490 of FIT s Student Handbook to inform yourself of what constitutes academic dishonesty. Furthermore, in the aforementioned policy, please ensure that you are cognizant of what constitutes plagiarism. With regards to ECE5258, pay specific attention to the following: If you happen to use code, implementations and/or libraries that (i) are very specific to your MP or ICP, (ii) you have not authored/developed them and (iii) you turn them in as part of your deliverables implying that these are products of your own efforts, this will be 9
considered an act of plagiarism, unless (a) you obtain your instructor s prior approval to do so, (b) you clearly disclose in your deliverable(s) that the pertinent components are not your work and (c) you provide references (e.g. URLs) to these components. If you author/compile a report for a MP or ICP and include verbatim parts of other unattributed / unacknowledged sources, this constitutes a plagiarism incident, unless such text is appropriately quoted and/or the parts in question are clearly attributed to their original sources. 11. Disclaimer Every effort will be made to follow this syllabus as close as possible throughout the course s duration. However, it is up to the instructors discretion to introduce changes, whenever deemed necessary. In this case, course participants will be provided with notice as early as circumstances allow for. 10