Books & Videos

Learning Spark

Lightning-Fast Big Data Analysis

By Matei Zaharia, Holden Karau, Andy Konwinski, Patrick Wendell

Publisher: O'Reilly Media

Release Date: February 2015

Pages: 276

Read on Safari with a 10-day trial

Start your free trial now Buy on Amazon

Where’s the cart? Now you can get everything on Safari. To purchase books, visit Amazon or your favorite retailer. Questions? See our FAQ or contact customer service:

1-800-889-8969 / 707-827-7019
support@oreilly.com

Download Example Code

View/Submit Errata

Data in all domains is getting bigger. How can you work with it efficiently? Recently updated for Spark 1.3, this book introduces Apache Spark, the open source cluster computing system that makes data analytics fast to write and fast to run. With Spark, you can tackle big datasets quickly through simple APIs in Python, Java, and Scala. This edition includes new information on Spark SQL, Spark Streaming, setup, and Maven coordinates.

Written by the developers of Spark, this book will have data scientists and engineers up and running in no time. You’ll learn how to express parallel jobs with just a few lines of code, and cover applications from simple batch jobs to stream processing and machine learning.

Quickly dive into Spark capabilities such as distributed datasets, in-memory caching, and the interactive shell
Leverage Spark’s powerful built-in libraries, including Spark SQL, Spark Streaming, and MLlib
Use one programming paradigm instead of mixing and matching tools like Hive, Hadoop, Mahout, and Storm
Learn how to deploy interactive, batch, and streaming applications
Connect to data sources including HDFS, Hive, JSON, and S3
Master advanced topics like data partitioning and shared variables

Chapter 1 Introduction to Data Analysis with Spark
1. What Is Apache Spark?
2. A Unified Stack
3. Who Uses Spark, and for What?
4. A Brief History of Spark
5. Spark Versions and Releases
6. Storage Layers for Spark
Chapter 2 Downloading Spark and Getting Started
1. Downloading Spark
2. Introduction to Spark’s Python and Scala Shells
3. Introduction to Core Spark Concepts
4. Standalone Applications
5. Conclusion
Chapter 3 Programming with RDDs
1. RDD Basics
2. Creating RDDs
3. RDD Operations
4. Passing Functions to Spark
5. Common Transformations and Actions
6. Persistence (Caching)
7. Conclusion
Chapter 4 Working with Key/Value Pairs
1. Motivation
2. Creating Pair RDDs
3. Transformations on Pair RDDs
4. Actions Available on Pair RDDs
5. Data Partitioning (Advanced)
6. Conclusion
Chapter 5 Loading and Saving Your Data
1. Motivation
2. File Formats
3. Filesystems
4. Structured Data with Spark SQL
5. Databases
6. Conclusion
Chapter 6 Advanced Spark Programming
1. Introduction
2. Accumulators
3. Broadcast Variables
4. Working on a Per-Partition Basis
5. Piping to External Programs
6. Numeric RDD Operations
7. Conclusion
Chapter 7 Running on a Cluster
1. Introduction
2. Spark Runtime Architecture
3. Deploying Applications with spark-submit
4. Packaging Your Code and Dependencies
5. Scheduling Within and Between Spark Applications
6. Cluster Managers
7. Which Cluster Manager to Use?
8. Conclusion
Chapter 8 Tuning and Debugging Spark
1. Configuring Spark with SparkConf
2. Components of Execution: Jobs, Tasks, and Stages
3. Finding Information
4. Key Performance Considerations
5. Conclusion
Chapter 9 Spark SQL
1. Linking with Spark SQL
2. Using Spark SQL in Applications
3. Loading and Saving Data
4. JDBC/ODBC Server
5. User-Defined Functions
6. Spark SQL Performance
7. Conclusion
Chapter 10 Spark Streaming
1. A Simple Example
2. Architecture and Abstraction
3. Transformations
4. Output Operations
5. Input Sources
6. 24/7 Operation
7. Streaming UI
8. Performance Considerations
9. Conclusion
Chapter 11 Machine Learning with MLlib
1. Overview
2. System Requirements
3. Machine Learning Basics
4. Data Types
5. Algorithms
6. Tips and Performance Considerations
7. Pipeline API
8. Conclusion

Books & Videos

Learning Spark

Lightning-Fast Big Data Analysis

Read on Safari with a 10-day trial

Table of Contents

Chapter 1 Introduction to Data Analysis with Spark

What Is Apache Spark?

A Unified Stack

Who Uses Spark, and for What?

A Brief History of Spark

Spark Versions and Releases

Storage Layers for Spark

Chapter 2 Downloading Spark and Getting Started

Downloading Spark

Introduction to Spark’s Python and Scala Shells

Introduction to Core Spark Concepts

Standalone Applications

Conclusion

Chapter 3 Programming with RDDs

RDD Basics

Creating RDDs

RDD Operations

Passing Functions to Spark

Common Transformations and Actions

Persistence (Caching)

Conclusion

Chapter 4 Working with Key/Value Pairs

Motivation

Creating Pair RDDs

Transformations on Pair RDDs

Actions Available on Pair RDDs

Data Partitioning (Advanced)

Conclusion

Chapter 5 Loading and Saving Your Data

Motivation

File Formats

Filesystems

Structured Data with Spark SQL

Databases

Conclusion

Chapter 6 Advanced Spark Programming

Introduction

Accumulators

Broadcast Variables

Working on a Per-Partition Basis

Piping to External Programs

Numeric RDD Operations

Conclusion

Chapter 7 Running on a Cluster

Introduction

Spark Runtime Architecture

Deploying Applications with spark-submit

Packaging Your Code and Dependencies

Scheduling Within and Between Spark Applications

Cluster Managers

Which Cluster Manager to Use?

Conclusion

Chapter 8 Tuning and Debugging Spark

Configuring Spark with SparkConf

Components of Execution: Jobs, Tasks, and Stages

Finding Information

Key Performance Considerations

Conclusion

Chapter 9 Spark SQL

Linking with Spark SQL

Using Spark SQL in Applications

Loading and Saving Data

JDBC/ODBC Server

User-Defined Functions

Spark SQL Performance

Conclusion

Chapter 10 Spark Streaming

A Simple Example

Architecture and Abstraction

Transformations

Output Operations

Input Sources

24/7 Operation

Streaming UI

Performance Considerations