Apache Spark is amazing when everything clicks. But if you haven’t seen the performance improvements you expected, or still don’t feel confident enough to use Spark in production, this practical book is for you. Authors Holden Karau and Rachel Warren demonstrate performance optimizations to help your Spark queries run faster and handle larger data sizes, while using fewer resources.
Ideal for software engineers, data engineers, developers, and system administrators working with large-scale data applications, this book describes techniques that can reduce data infrastructure costs and developer hours. Not only will you gain a more comprehensive understanding of Spark, you’ll also learn how to make it sing.
With this book, you’ll explore:
How Spark SQL’s new interfaces improve performance over SQL’s RDD data structure
The choice between data joins in Core Spark and Spark SQL
Techniques for getting the most out of standard RDD transformations
How to work around performance issues in Spark’s key/value pair paradigm
Writing high-performance Spark code without Scala or the JVM
How to test for functionality and performance when applying suggested improvements
Using Spark MLlib and Spark ML machine learning libraries
Spark’s Streaming components and external community packages
Chapter 1Introduction to High Performance Spark
What Is Spark and Why Performance Matters
What You Can Expect to Get from This Book
Chapter 2How Spark Works
How Spark Fits into the Big Data Ecosystem
Spark Model of Parallel Computing: RDDs
Spark Job Scheduling
The Anatomy of a Spark Job
Chapter 3DataFrames, Datasets, and Spark SQL
Getting Started with the SparkSession (or HiveContext or SQLContext)
Spark SQL Dependencies
Basics of Schemas
Data Representation in DataFrames and Datasets
Data Loading and Saving Functions
Extending with User-Defined Functions and Aggregate Functions (UDFs, UDAFs)
Debugging Spark SQL Queries
Chapter 4Joins (SQL and Core)
Core Spark Joins
Spark SQL Joins
Chapter 5Effective Transformations
Narrow Versus Wide Transformations
What Type of RDD Does Your Transformation Return?
Minimizing Object Creation
Iterator-to-Iterator Transformations with mapPartitions
Reducing Setup Overhead
Chapter 6Working with Key/Value Data
The Goldilocks Example
Actions on Key/Value Pairs
What’s So Dangerous About the groupByKey Function
Choosing an Aggregation Operation
Multiple RDD Operations
Partitioners and Key/Value Data
Dictionary of OrderedRDDOperations
Secondary Sort and repartitionAndSortWithinPartitions
Straggler Detection and Unbalanced Data
Chapter 7Going Beyond Scala
Beyond Scala within the JVM
Beyond Scala, and Beyond the JVM
Calling Other Languages from Spark
Chapter 8Testing and Validation
Getting Test Data
Property Checking with ScalaCheck
Chapter 9Spark MLlib and ML
Choosing Between Spark MLlib and Spark ML
Working with MLlib
Working with Spark ML
General Serving Considerations
Chapter 10Spark Components and Packages
Stream Processing with Spark
Using Community Packages and Libraries
Appendix Tuning, Debugging, and Other Things Developers Like to Pretend Don’t Exist
Spark Tuning and Cluster Sizing
Basic Spark Core Settings: How Many Resources to Allocate to the Spark Application?
Holden Karau is transgender Canadian, and an active open source contributor. When not in San Francisco working as a software development engineer at IBM's Spark Technology Center, Holden talks internationally on Apache Spark and holds office hours at coffee shops at home and abroad. She is a Spark committer with frequent contributions, specializing in PySpark and Machine Learning. Prior to IBM she worked on a variety of distributed, search, and classification problems at Alpine, Databricks, Google, Foursquare, and Amazon. She graduated from the University of Waterloo with a Bachelor of Mathematics in Computer Science. Outside of software she enjoys playing with fire, welding, scooters, poutine, and dancing.
Rachel Warren is a data scientist and software engineer at Alpine Data Labs, where she uses Spark to address real world data processing challenges. She has experience working as an analyst both in industry and academia. She graduated with a degree in Computer Science from Wesleyan University in Connecticut.
The animal on the cover of High Performance Spark is a fire-tailed sunbird (Aethopyga ignicauda) native to Southeast Asia and the Indian subcontinent. Sunbirds are very distant relatives of the hummingbirds of the Americas and the honeyeaters of Australia.
As their name suggests, these birds (especially males) have very vivid coloring. Their tails and the back of their necks are red, their wings are green, their bellies are yellow and orange, and their heads are an iridescent blue. Male fire-tailed sunbirds are slightly larger than females, at an average of 15 centimeters long. Both members of a mating pair participate in feeding their chicks.
The fire-tailed sunbird's preferred habitat is conifer forest, where it consumes a diet of insects and nectar. The downward curve of its beak and a tubular tongue help it reach into flowers for food.
Many of the animals on O'Reilly covers are endangered; all of them are important to the world. To learn more about how you can help, go to animals.oreilly.com.
The cover image is from Wood's Illustrated Natural History.
The best book on writing production-ready Spark code
from Manchester, UK
About Me Developer
Easy to understand
Comments about oreilly High Performance Spark:
There are quite a few good books on getting started with Spark, launching the interactive shell, running a few queries, and so on, but this book is fairly unique in showing you the ways to get the best of the Spark programming APIs.
The chapter on "Joins" covering RDD, DataFrame, and Dataset APIs will save you hours if not days of research alone.
Bottom Line Yes, I would recommend this to a friend