Apache Spark Machine Learning Blueprints

Apache Spark Machine Learning Blueprints

by Alex Liu
Released May 2016
Publisher(s): Packt Publishing
ISBN: 9781785880391

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Develop a range of cutting-edge machine learning projects with Apache Spark using this actionable guide

About This Book

  • Customize Apache Spark and R to fit your analytical needs in customer research, fraud detection, risk analytics, and recommendation engine development
  • Develop a set of practical Machine Learning applications that can be implemented in real-life projects
  • A comprehensive, project-based guide to improve and refine your predictive models for practical implementation

Who This Book Is For

If you are a data scientist, a data analyst, or an R and SPSS user with a good understanding of machine learning concepts, algorithms, and techniques, then this is the book for you. Some basic understanding of Spark and its core elements and application is required.

What You Will Learn

  • Set up Apache Spark for machine learning and discover its impressive processing power
  • Combine Spark and R to unlock detailed business insights essential for decision making
  • Build machine learning systems with Spark that can detect fraud and analyze financial risks
  • Build predictive models focusing on customer scoring and service ranking
  • Build a recommendation systems using SPSS on Apache Spark
  • Tackle parallel computing and find out how it can support your machine learning projects
  • Turn open data and communication data into actionable insights by making use of various forms of machine learning

In Detail

There's a reason why Apache Spark has become one of the most popular tools in Machine Learning ? its ability to handle huge datasets at an impressive speed means you can be much more responsive to the data at your disposal. This book shows you Spark at its very best, demonstrating how to connect it with R and unlock maximum value not only from the tool but also from your data.

Packed with a range of project "blueprints" that demonstrate some of the most interesting challenges that Spark can help you tackle, you'll find out how to use Spark notebooks and access, clean, and join different datasets before putting your knowledge into practice with some real-world projects, in which you will see how Spark Machine Learning can help you with everything from fraud detection to analyzing customer attrition. You'll also find out how to build a recommendation engine using Spark's parallel computing powers.

Style and approach

This book offers a step-by-step approach to setting up Apache Spark, and use other analytical tools with it to process Big Data and build machine learning projects.The initial chapters focus more on the theory aspect of machine learning with Spark, while each of the later chapters focuses on building standalone projects using Spark.

Table of contents

  1. Apache Spark Machine Learning Blueprints
    1. Table of Contents
    2. Apache Spark Machine Learning Blueprints
    3. Credits
    4. About the Author
    5. About the Reviewer
    6. www.PacktPub.com
      1. eBooks, discount offers, and more
        1. Why subscribe?
    7. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
        1. Downloading the color images of this book
        2. Errata
        3. Piracy
        4. Questions
    8. 1. Spark for Machine Learning
      1. Spark overview and Spark advantages
        1. Spark overview
        2. Spark advantages
      2. Spark computing for machine learning
      3. Machine learning algorithms
      4. MLlib
        1. Other ML libraries
      5. Spark RDD and dataframes
        1. Spark RDD
        2. Spark dataframes
        3. Dataframes API for R
        4. ML frameworks, RM4Es and Spark computing
        5. ML frameworks
        6. RM4Es
        7. The Spark computing framework
      6. ML workflows and Spark pipelines
        1. ML as a step-by-step workflow
      7. ML workflow examples
      8. Spark notebooks
        1. Notebook approach for ML
          1. Step 1: Getting the software ready
          2. Step 2: Installing the Knitr package
          3. Step 3: Creating a simple report
        2. Spark notebooks
      9. Summary
    9. 2. Data Preparation for Spark ML
      1. Accessing and loading datasets
        1. Accessing publicly available datasets
        2. Loading datasets into Spark
        3. Exploring and visualizing datasets
      2. Data cleaning
        1. Dealing with data incompleteness
        2. Data cleaning in Spark
        3. Data cleaning made easy
      3. Identity matching
        1. Identity issues
        2. Identity matching on Spark
        3. Entity resolution
          1. Short string comparison
          2. Long string comparison
          3. Record deduplication
        4. Identity matching made better
          1. Crowdsourced deduplication
          2. Configuring the crowd
          3. Using the crowd
      4. Dataset reorganizing
        1. Dataset reorganizing tasks
        2. Dataset reorganizing with Spark SQL
        3. Dataset reorganizing with R on Spark
      5. Dataset joining
        1. Dataset joining and its tool – the Spark SQL
        2. Dataset joining in Spark
        3. Dataset joining with the R data table package
      6. Feature extraction
        1. Feature development challenges
        2. Feature development with Spark MLlib
        3. Feature development with R
      7. Repeatability and automation
        1. Dataset preprocessing workflows
        2. Spark pipelines for dataset preprocessing
        3. Dataset preprocessing automation
      8. Summary
    10. 3. A Holistic View on Spark
      1. Spark for a holistic view
        1. The use case
        2. Fast and easy computing
      2. Methods for a holistic view
        1. Regression modeling
        2. The SEM approach
        3. Decision trees
      3. Feature preparation
        1. PCA
        2. Grouping by category to use subject knowledge
        3. Feature selection
      4. Model estimation
        1. MLlib implementation
        2. The R notebooks' implementation
      5. Model evaluation
        1. Quick evaluations
        2. RMSE
        3. ROC curves
      6. Results explanation
        1. Impact assessments
      7. Deployment
        1. Dashboard
        2. Rules
      8. Summary
    11. 4. Fraud Detection on Spark
      1. Spark for fraud detection
        1. The use case
        2. Distributed computing
      2. Methods for fraud detection
        1. Random forest
        2. Decision trees
      3. Feature preparation
        1. Feature extraction from LogFile
        2. Data merging
      4. Model estimation
        1. MLlib implementation
        2. R notebooks implementation
      5. Model evaluation
        1. A quick evaluation
        2. Confusion matrix and false positive ratios
      6. Results explanation
        1. Big influencers and their impacts
      7. Deploying fraud detection
        1. Rules
        2. Scoring
      8. Summary
    12. 5. Risk Scoring on Spark
      1. Spark for risk scoring
        1. The use case
        2. Apache Spark notebooks
      2. Methods of risk scoring
        1. Logistic regression
          1. Preparing coding in R
        2. Random forest and decision trees
          1. Preparing coding
      3. Data and feature preparation
        1. OpenRefine
      4. Model estimation
        1. The DataScientistWorkbench for R notebooks
        2. R notebooks implementation
      5. Model evaluation
        1. Confusion matrix
        2. ROC
        3. Kolmogorov-Smirnov
      6. Results explanation
        1. Big influencers and their impacts
      7. Deployment
        1. Scoring
      8. Summary
    13. 6. Churn Prediction on Spark
      1. Spark for churn prediction
        1. The use case
        2. Spark computing
      2. Methods for churn prediction
        1. Regression models
        2. Decision trees and Random forest
      3. Feature preparation
        1. Feature extraction
        2. Feature selection
      4. Model estimation
        1. Spark implementation with MLlib
      5. Model evaluation
      6. Results explanation
        1. Calculating the impact of interventions
      7. Deployment
        1. Scoring
        2. Intervention recommendations
      8. Summary
    14. 7. Recommendations on Spark
      1. Apache Spark for a recommendation engine
        1. The use case
        2. SPSS on Spark
      2. Methods for recommendation
        1. Collaborative filtering
        2. Preparing coding
      3. Data treatment with SPSS
        1. Missing data nodes on SPSS modeler
      4. Model estimation
        1. SPSS on Spark – the SPSS Analytics server
      5. Model evaluation
      6. Recommendation deployment
      7. Summary
    15. 8. Learning Analytics on Spark
      1. Spark for attrition prediction
        1. The use case
        2. Spark computing
      2. Methods of attrition prediction
        1. Regression models
          1. About regression
          2. Preparing for coding
        2. Decision trees
          1. Preparing for coding
      3. Feature preparation
        1. Feature development
        2. Feature selection
          1. Principal components analysis
            1. Subject knowledge aid
          2. ML feature selection
      4. Model estimation
        1. Spark implementation with the Zeppelin notebook
      5. Model evaluation
        1. A quick evaluation
        2. The confusion matrix and error ratios
      6. Results explanation
        1. Calculating the impact of interventions
        2. Calculating the impact of main causes
      7. Deployment
        1. Rules
        2. Scoring
      8. Summary
    16. 9. City Analytics on Spark
      1. Spark for service forecasting
        1. The use case
        2. Spark computing
        3. Methods of service forecasting
        4. Regression models
          1. About regression
          2. Preparing for coding
        5. Time series modeling
          1. About time series
          2. Preparing for coding
      2. Data and feature preparation
        1. Data merging
        2. Feature selection
      3. Model estimation
        1. Spark implementation with the Zeppelin notebook
        2. Spark implementation with the R notebook
      4. Model evaluation
        1. RMSE calculation with MLlib
        2. RMSE calculation with R
      5. Explanations of the results
        1. Biggest influencers
        2. Visualizing trends
          1. The rules of sending out alerts
          2. Scores to rank city zones
      6. Summary
    17. 10. Learning Telco Data on Spark
      1. Spark for using Telco Data
        1. The use case
        2. Spark computing
      2. Methods for learning from Telco Data
        1. Descriptive statistics and visualization
        2. Linear and logistic regression models
        3. Decision tree and random forest
      3. Data and feature development
        1. Data reorganizing
        2. Feature development and selection
      4. Model estimation
        1. SPSS on Spark – SPSS Analytics Server
      5. Model evaluation
        1. RMSE calculations with MLlib
        2. RMSE calculations with R
        3. Confusion matrix and error ratios with MLlib and R
      6. Results explanation
        1. Descriptive statistics and visualizations
        2. Biggest influencers
        3. Special insights
        4. Visualizing trends
      7. Model deployment
        1. Rules to send out alerts
        2. Scores subscribers for churn and for Call Center calls
        3. Scores subscribers for purchase propensity
      8. Summary
    18. 11. Modeling Open Data on Spark
      1. Spark for learning from open data
        1. The use case
        2. Spark computing
        3. Methods for scoring and ranking
        4. Cluster analysis
        5. Principal component analysis
        6. Regression models
        7. Score resembling
      2. Data and feature preparation
        1. Data cleaning
        2. Data merging
        3. Feature development
        4. Feature selection
      3. Model estimation
        1. SPSS on Spark – SPSS Analytics Server
        2. Model evaluation
        3. RMSE calculations with MLlib
        4. RMSE calculations with R
      4. Results explanation
        1. Comparing ranks
        2. Biggest influencers
      5. Deployment
        1. Rules for sending out alerts
        2. Scores for ranking school districts
      6. Summary
    19. Index

Product information

  • Title: Apache Spark Machine Learning Blueprints
  • Author(s): Alex Liu
  • Release date: May 2016
  • Publisher(s): Packt Publishing
  • ISBN: 9781785880391