Python. for Data Science. by Luca Massaron and John Paul Mueller

Transcription

1

2

3 Python for Data Science by Luca Massaron and John Paul Mueller

4 Python for Data Science For Dummies Published by: John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ , Copyright 2015 by John Wiley & Sons, Inc., Hoboken, New Jersey Media and software compilation copyright 2015 by John Wiley & Sons, Inc. All rights reserved. Published simultaneously in Canada No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without the prior written permission of the Publisher. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) , fax (201) , or online at Trademarks: Wiley, For Dummies, the Dummies Man logo, Dummies.com, Making Everything Easier, and related trade dress are trademarks or registered trademarks of John Wiley & Sons, Inc. and may not be used without written permission. Python is a registered trademark of Python Software Foundation Corporation. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc. is not associated with any product or vendor mentioned in this book. LIMIT OF LIABILITY/DISCLAIMER OF WARRANTY : THE PUBLISHER AND THE AUTHOR MAKE NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE ACCURACY OR COMPLETENESS OF THE CONTENTS OF THIS WORK AND SPECIFICALLY DISCLAIM ALL WARRANTIES, INCLUDING WITHOUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE. NO WARRANTY MAY BE CREATED OR EXTENDED BY SALES OR PROMOTIONAL MATERIALS. THE ADVICE AND STRATEGIES CONTAINED HEREIN MAY NOT BE SUITABLE FOR EVERY SITUATION. THIS WORK IS SOLD WITH THE UNDERSTANDING THAT THE PUBLISHER IS NOT ENGAGED IN RENDERING LEGAL, ACCOUNTING, OR OTHER PROFESSIONAL SERVICES. IF PROFESSIONAL ASSISTANCE IS REQUIRED, THE SERVICES OF A COMPETENT PROFESSIONAL PERSON SHOULD BE SOUGHT. NEITHER THE PUBLISHER NOR THE AUTHOR SHALL BE LIABLE FOR DAMAGES ARISING HEREFROM. THE FACT THAT AN ORGANIZATION OR WEBSITE IS REFERRED TO IN THIS WORK AS A CITATION AND/OR A POTENTIAL SOURCE OF FURTHER INFORMATION DOES NOT MEAN THAT THE AUTHOR OR THE PUBLISHER ENDORSES THE INFORMATION THE ORGANIZATION OR WEBSITE MAY PROVIDE OR RECOMMENDATIONS IT MAY MAKE. FURTHER, READERS SHOULD BE AWARE THAT INTERNET WEBSITES LISTED IN THIS WORK MAY HAVE CHANGED OR DISAPPEARED BETWEEN WHEN THIS WORK WAS WRITTEN AND WHEN IT IS READ. For general information on our other products and services, please contact our Customer Care Department within the U.S. at , outside the U.S. at , or fax For technical support, please visit Wiley publishes in a variety of print and electronic formats and by print on demand. Some material included with standard print versions of this book may not be included in e books or in print on demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at For more information about Wiley products, visit Library of Congress Control Number: ISBN: ISBN (ebk); ISBN epdf (ebk) Manufactured in the United States of America

5 Table of Contents Introduction... 1 About This Book...1 Foolish Assumptions...2 Icons Used in This Book...3 Beyond the Book...4 Where to Go from Here...5 Part I: Getting Started with Python for Data Science... 7 Chapter 1: Discovering the Match between Data Science and Python Defining the Sexiest Job of the 21st Century...11 Considering the emergence of data science...11 Outlining the core competencies of a data scientist...12 Linking data science and big data...13 Understanding the role of programming...13 Creating the Data Science Pipeline...14 Preparing the data...14 Performing exploratory data analysis...15 Learning from data...15 Visualizing...15 Obtaining insights and data products...15 Understanding Python s Role in Data Science...16 Considering the shifting profile of data scientists...16 Working with a multipurpose, simple, and efficient language...17 Learning to Use Python Fast...18 Loading data...18 Training a model...18 Viewing a result...20 Chapter 2: Introducing Python s Capabilities and Wonders Why Python?...22 Grasping Python s core philosophy...23 Discovering present and future development goals...23 Working with Python...24 Getting a taste of the language...24 Understanding the need for indentation...25 Working at the command line or in the IDE...25

6 iv Python for Data Science For Dummies Performing Rapid Prototyping and Experimentation...29 Considering Speed of Execution...30 Visualizing Power...32 Using the Python Ecosystem for Data Science...33 Accessing scientific tools using SciPy...33 Performing fundamental scientific computing using NumPy...34 Performing data analysis using pandas...34 Implementing machine learning using Scikit learn...35 Plotting the data using matplotlib...35 Parsing HTML documents using Beautiful Soup...35 Chapter 3: Setting Up Python for Data Science Considering the Off the Shelf Cross Platform Scientific Distributions...38 Getting Continuum Analytics Anaconda...39 Getting Enthought Canopy Express...40 Getting pythonxy...40 Getting WinPython...41 Installing Anaconda on Windows...41 Installing Anaconda on Linux...45 Installing Anaconda on Mac OS X...46 Downloading the Datasets and Example Code...47 Using IPython Notebook...47 Defining the code repository...48 Understanding the datasets used in this book...54 Chapter 4: Reviewing Basic Python Working with Numbers and Logic...59 Performing variable assignments...60 Doing arithmetic...61 Comparing data using Boolean expressions...62 Creating and Using Strings...65 Interacting with Dates...66 Creating and Using Functions...68 Creating reusable functions...68 Calling functions in a variety of ways...70 Using Conditional and Loop Statements...73 Making decisions using the if statement...73 Choosing between multiple options using nested decisions...74 Performing repetitive tasks using for...75 Using the while statement...76 Storing Data Using Sets, Lists, and Tuples...77 Performing operations on sets...77 Working with lists...78 Creating and using Tuples...80 Defining Useful Iterators...81 Indexing Data Using Dictionaries...82

7 Table of Contents v Part II: Getting Your Hands Dirty with Data Chapter 5: Working with Real Data Uploading, Streaming, and Sampling Data...86 Uploading small amounts of data into memory...87 Streaming large amounts of data into memory...88 Sampling data...89 Accessing Data in Structured Flat File Form...90 Reading from a text file...91 Reading CSV delimited format...92 Reading Excel and other Microsoft Office files...94 Sending Data in Unstructured File Form...95 Managing Data from Relational Databases...98 Interacting with Data from NoSQL Databases Accessing Data from the Web Chapter 6: Conditioning Your Data Juggling between NumPy and pandas Knowing when to use NumPy Knowing when to use pandas Validating Your Data Figuring out what s in your data Removing duplicates Creating a data map and data plan Manipulating Categorical Variables Creating categorical variables Renaming levels Combining levels Dealing with Dates in Your Data Formatting date and time values Using the right time transformation Dealing with Missing Data Finding the missing data Encoding missingness Imputing missing data Slicing and Dicing: Filtering and Selecting Data Slicing rows Slicing columns Dicing Concatenating and Transforming Adding new cases and variables Removing data Sorting and shuffling Aggregating Data at Any Level...128

8 vi Python for Data Science For Dummies Chapter 7: Shaping Data Working with HTML Pages Parsing XML and HTML Using XPath for data extraction Working with Raw Text Dealing with Unicode Stemming and removing stop words Introducing regular expressions Using the Bag of Words Model and Beyond Understanding the bag of words model Working with n grams Implementing TF IDF transformations Working with Graph Data Understanding the adjacency matrix Using NetworkX basics Chapter 8: Putting What You Know in Action Contextualizing Problems and Data Evaluating a data science problem Researching solutions Formulating a hypothesis Preparing your data Considering the Art of Feature Creation Defining feature creation Combining variables Understanding binning and discretization Using indicator variables Transforming distributions Performing Operations on Arrays Using vectorization Performing simple arithmetic on vectors and matrices Performing matrix vector multiplication Performing matrix multiplication Part III: Visualizing the Invisible Chapter 9: Getting a Crash Course in MatPlotLib Starting with a Graph Defining the plot Drawing multiple lines and plots Saving your work Setting the Axis, Ticks, Grids Getting the axes...167

9 Table of Contents vii Formatting the axes Adding grids Defining the Line Appearance Working with line styles Using colors Adding markers Using Labels, Annotations, and Legends Adding labels Annotating the chart Creating a legend Chapter 10: Visualizing the Data Choosing the Right Graph Showing parts of a whole with pie charts Creating comparisons with bar charts Showing distributions using histograms Depicting groups using box plots Seeing data patterns using scatterplots Creating Advanced Scatterplots Depicting groups Showing correlations Plotting Time Series Representing time on axes Plotting trends over time Plotting Geographical Data Visualizing Graphs Developing undirected graphs Developing directed graphs Chapter 11: Understanding the Tools Using the IPython Console Interacting with screen text Changing the window appearance Getting Python help Getting IPython help Using magic functions Discovering objects Using IPython Notebook Working with styles Restarting the kernel Restoring a checkpoint Performing Multimedia and Graphic Integration Embedding plots and other images Loading examples from online sites Obtaining online graphics and multimedia...212

10 viii Python for Data Science For Dummies Part IV: Wrangling Data Chapter 12: Stretching Python s Capabilities Playing with Scikit learn Understanding classes in Scikit learn Defining applications for data science Performing the Hashing Trick Using hash functions Demonstrating the hashing trick Working with deterministic selection Considering Timing and Performance Benchmarking with timeit Working with the memory profiler Running in Parallel Performing multicore parallelism Demonstrating multiprocessing Chapter 13: Exploring Data Analysis The EDA Approach Defining Descriptive Statistics for Numeric Data Measuring central tendency Measuring variance and range Working with percentiles Defining measures of normality Counting for Categorical Data Understanding frequencies Creating contingency tables Creating Applied Visualization for EDA Inspecting boxplots Performing t tests after boxplots Observing parallel coordinates Graphing distributions Plotting scatterplots Understanding Correlation Using covariance and correlation Using nonparametric correlation Considering chi square for tables Modifying Data Distributions Using the normal distribution Creating a Z score standardization Transforming other notable distributions...254

11 Table of Contents ix Chapter 14: Reducing Dimensionality Understanding SVD Looking for dimensionality reduction Using SVD to measure the invisible Performing Factor and Principal Component Analysis Considering the psychometric model Looking for hidden factors Using components, not factors Achieving dimensionality reduction Understanding Some Applications Recognizing faces with PCA Extracting Topics with NMF Recommending movies Chapter 15: Clustering Clustering with K means Understanding centroid based algorithms Creating an example with image data Looking for optimal solutions Clustering big data Performing Hierarchical Clustering Moving Beyond the Round-Shaped Clusters: DBScan Chapter 16: Detecting Outliers in Data Considering Detection of Outliers Finding more things that can go wrong Understanding anomalies and novel data Examining a Simple Univariate Method Leveraging on the Gaussian distribution Making assumptions and checking out Developing a Multivariate Approach Using principal component analysis Using cluster analysis Automating outliers detection with SVM Part V: Learning from Data Chapter 17: Exploring Four Simple and Effective Algorithms Guessing the Number: Linear Regression Defining the family of linear models Using more variables Understanding limitations and problems...307

12 x Python for Data Science For Dummies Moving to Logistic Regression Applying logistic regression Considering when classes are more Making Things as Simple as Naïve Bayes Finding out that Naïve Bayes isn t so naïve Predicting text classifications Learning Lazily with Nearest Neighbors Predicting after observing neighbors Choosing your k parameter wisely Chapter 18: Performing Cross Validation, Selection, and Optimization Pondering the Problem of Fitting a Model Understanding bias and variance Defining a strategy for picking models Dividing between training and test sets Cross Validating Using cross validation on k folds Sampling stratifications for complex data Selecting Variables Like a Pro Selecting by univariate measures Using a greedy search Pumping Up Your Hyperparameters Implementing a grid search Trying a randomized search Chapter 19: Increasing Complexity with Linear and Nonlinear Tricks Using Nonlinear Transformations Doing variable transformations Creating interactions between variables Regularizing Linear Models Relying on Ridge regression (L2) Using the Lasso (L1) Leveraging regularization Combining L1 & L2: Elasticnet Fighting with Big Data Chunk by Chunk Determining when there is too much data Implementing Stochastic Gradient Descent Understanding Support Vector Machines Relying on a computational method Fixing many new parameters Classifying with SVC Going nonlinear is easy Performing regression with SVR Creating a stochastic solution with SVM...368

13 Table of Contents xi Chapter 20: Understanding the Power of the Many Starting with a Plain Decision Tree Understanding a decision tree Creating classification and regression trees Making Machine Learning Accessible Working with a Random Forest classifier Working with a Random Forest regressor Optimizing a Random Forest Boosting Predictions Knowing that many weak predictors win Creating a gradient boosting classifier Creating a gradient boosting regressor Using GBM hyper parameters Part VI: The Part of Tens Chapter 21: Ten Essential Data Science Resource Collections Gaining Insights with Data Science Weekly Obtaining a Resource List at U Climb Higher Getting a Good Start with KDnuggets Accessing the Huge List of Resources on Data Science Central Obtaining the Facts of Open Source Data Science from Masters Locating Free Learning Resources with Quora Receiving Help with Advanced Topics at Conductrics Learning New Tricks from the Aspirational Data Scientist Finding Data Intelligence and Analytics Resources at AnalyticBridge Zeroing In on Developer Resources with Jonathan Bower Chapter 22: Ten Data Challenges You Should Take Meeting the Data Science London + Scikit learn Challenge Predicting Survival on the Titanic Finding a Kaggle Competition that Suits Your Needs Honing Your Overfit Strategies Trudging Through the MovieLens Dataset Getting Rid of Spam s Working with Handwritten Information Working with Pictures Analyzing Amazon.com Reviews Interacting with a Huge Graph Index

14

15 Introduction You rely on data science absolutely every day to perform an amazing array of tasks or to obtain services from someone else. In fact, you ve probably used data science in ways that you never expected. For example, when you used your favorite search engine this morning to look for something, it made suggestions on alternative search terms. Those terms are supplied by data science. When you went to the doctor last week and discovered the lump you found wasn t cancer, it s likely the doctor made his prognosis with the help of data science. In fact, you might work with data science every day and not even know it. Python for Data Science For Dummies not only gets you started using data science to perform a wealth of practical tasks but also helps you realize just how many places data science is used. By knowing how to answer data science problems and where to employ data science, you gain a significant advantage over everyone else, increasing your chances at promotion or that new job you really want. About This Book The main purpose of Python for Data Science For Dummies is to take the scare factor out of data science by showing you that data science is not only really interesting but also quite doable using Python. You might assume that you need to be a computer science genius to perform the complex tasks normally associated with data science, but that s far from the truth. Python comes with a host of useful libraries that do all the heavy lifting for you in the background. You don t even realize how much is going on, and you don t need to care. All you really need to know is that you want to perform specific tasks and that Python makes these tasks quite accessible. Part of the emphasis of this book is on using the right tools. You start with Anaconda, a product that includes IPython and IPython Notebook two tools that take the sting out of working with Python. You experiment with IPython in a fully interactive environment. The code you place in IPython Notebook is presentation quality, and you can mix a number of presentation elements right there in your document. It s not really like using a development environment at all. You also discover some interesting techniques in this book. For example, you can create plots of all your data science experiments using MatPlotLib, for which this book provides you with all the details. This book also spends