For many organizations, Hadoop is the first step for dealing with massive amounts of data. The next step? Processing and analyzing datasets with the Apache Pig scripting platform. With Pig, you can batch-process data without having to create a full-fledged application, making it easy to experiment with new datasets.
Updated with use cases and programming examples, this second edition is the ideal learning tool for new and experienced users alike. You’ll find comprehensive coverage on key features such as the Pig Latin scripting language and the Grunt shell. When you need to analyze terabytes of data, this book shows you how to do it efficiently with Pig.
Delve into Pig’s data model, including scalar and complex data types
Write Pig Latin scripts to sort, group, join, project, and filter your data
Use Grunt to work with the Hadoop Distributed File System (HDFS)
Build complex data processing pipelines with Pig’s macros and modularity features
Embed Pig Latin in Python for iterative processing and other advanced tasks
Use Pig with Apache Tez to build high-performance batch and interactive data processing applications
Create your own load and store functions to handle data formats and storage mechanisms
Chapter 1What Is Pig?
Pig Latin, a Parallel Data Flow Language
Pig on Hadoop
What Is Pig Useful For?
The Pig Philosophy
Chapter 2Installing and Running Pig
Downloading and Installing Pig
Chapter 3Pig’s Data Model
Chapter 4Introduction to Pig Latin
Input and Output
Chapter 5Advanced Pig Latin
Advanced Relational Operations
Integrating Pig with Executables and Native Jobs
split and Nonlinear Data Flows
Pig Latin Preprocessor
Chapter 6Developing and Testing Pig Latin Scripts
Testing Your Scripts with PigUnit
Chapter 7Making Pig Fly
Writing Your Scripts to Perform Well
Writing Your UDFs to Perform
Tuning Pig and Hadoop for Your Job
Using Compression in Intermediate Results
Data Layout Optimization
The JAR Cache
Processing Small Jobs Locally
Schema Tuple Optimization
Dealing with Failures
Chapter 8Embedding Pig
Embedding Pig Latin in Scripting Languages
Using the Pig Java APIs
Chapter 9Writing Evaluation and Filter Functions
Writing an Evaluation Function in Java
The Algebraic Interface
The Accumulator Interface
Writing Filter Functions
Writing Evaluation Functions in Scripting Languages
Chapter 10Writing Load and Store Functions
Shipping JARs Automatically
Handling Bad Records
Chapter 11Pig on Tez
What Is Tez?
Running Pig on Tez
Potential Differences When Running on Tez
Pig on Tez Internals
Chapter 12Pig and Other Members of the Hadoop Community
Pig and Hive
Chapter 13Use Cases and Programming Examples
intersect and except
Pig at Yahoo!
Pig at Particle News
Appendix Built-in User Defined Functions and PiggyBank
Alan is co-founder of Hortonworks and an original member of the engineering team that took Pig from a Yahoo! Labs research project to a successful Apache open source project. Alan also designed HCatalog and guided its adoption as an Apache Incubator project. Alan has a BS in Mathematics from Oregon State University and a MA in Theology from Fuller Theological Seminary. He is also the author of Programming Pig, a book from O’Reilly Press. Follow Alan on Twitter: @alanfgates.
Daniel is an Apache Pig PMC member/committer involved with Pig for 6 years at Yahoo and now at Hortonworks. He has a PhD in Computer Science from University of Central Florida, with a specialization in distributed computing, data mining and computer security.
The animal on the cover of Programming Pig is a domestic pig (Sus scrofa domesticusor Sus domesticus). While the larger pig family is naturally distributed in Africa, Asia,and Europe, domesticated pigs can now be found in nearly every part of the worldthat people inhabit. In fact, some pigs have been specifically bred to best equip themfor various climates; for example, heavily coated varieties have been bred in colderclimates. People have brought pigs with them almost wherever they go, for good reason:in addition to their primary use as a source of food, humans have been using theskin, bones, and hair of pigs to make various tools and implements for millennia.
Domestic pigs are directly descended from wild boars, and evidence suggests thatthere have been three distinct domestication events; the first took place in the TigrisRiver Basin as early as 13,000 BC, the second in China, and the third in Europe,though the last likely occurred after Europeans were introduced to domestic pigsfrom the Middle East. Despite the long history, however, taxonomists do not agree asto the proper classification for the domestic pig. Some believe that domestic pigsremain simply a subspecies of the larger pig group including the wild boar (Susscrofa), while others insist that they belong to a species all their own. In either case,there are several hundred breeds of domestic pig, each with its own particular characteristics.
Perhaps because of their long history and prominent role in human society, and theirtendency toward social behavior, domestic pigs have appeared in film, literature, andother cultural media with regularity. Examples include “The Three Little Pigs,” MissPiggy, and Porky the Pig. Additionally, domestic pigs have recently been recognizedfor their intelligence and their ability to be trained (similar to dogs), and have consequentlybegun to be treated as pets.