Although you don’t need a large computing infrastructure to process massive amounts of data with Apache Hadoop, it can still be difficult to get started. This practical guide shows you how to quickly launch data analysis projects in the cloud by using Amazon Elastic MapReduce (EMR), the hosted Hadoop framework in Amazon Web Services (AWS).
Authors Kevin Schmidt and Christopher Phillips demonstrate best practices for using EMR and various AWS and Apache technologies by walking you through the construction of a sample MapReduce log analysis application. Using code samples and example configurations, you’ll learn how to assemble the building blocks necessary to solve your biggest data analysis problems.
Get an overview of the AWS and Apache software tools used in large-scale data analysis
Go through the process of executing a Job Flow with a simple log analyzer
Discover useful MapReduce patterns for filtering and analyzing data sets
Use Apache Hive and Pig instead of Java to build a MapReduce Job Flow
Learn the basics for using Amazon EMR to run machine learning algorithms
Develop a project cost model for using Amazon EMR and other AWS tools
Chapter 1 Introduction to Amazon Elastic MapReduce
Amazon Web Services Used in This Book
Amazon Elastic MapReduce
Amazon EMR and the Hadoop Ecosystem
Amazon Elastic MapReduce Versus Traditional Hadoop Installs
Application Building Blocks
Chapter 2 Data Collection and Data Analysis with AWS
Log Analysis Application
Log Messages as a Data Set for Analytics
Simulating Syslog Data
Developing a MapReduce Application
Custom JAR MapReduce Job
Running an Amazon EMR Cluster
Viewing Our Results
Debugging a Job Flow
Our Application and Real-World Uses
Chapter 3 Data Filtering Design Patterns and Scheduling Work
Extending the Application Example
Understanding Web Server Logs
Finding Errors in the Web Logs Using Data Filtering
Building Summary Counts in Data Sets
Job Flow Scheduling
Scheduling with AWS Data Pipeline
Chapter 4 Data Analysis with Hive and Pig in Amazon EMR
Amazon Job Flow Technologies
What Is Pig?
Utilizing Pig in Amazon EMR
What Is Hive?
Utilizing Hive in Amazon EMR
Our Application with Hive and Pig
Chapter 5 Machine Learning Using EMR
A Quick Tour of Machine Learning
Python and EMR
Chapter 6 Planning AWS Projects and Managing Costs
Developing a Project Cost Model
Optimizing AWS Resources to Reduce Project Costs
Amazon Tools for Estimating Your Project Costs
Appendix Amazon Web Services Resources and Tools
Amazon AWS Online Resources
Amazon AWS Cost Estimation Tools
AWS Best Practices and Architecture
Amazon EMR Distributions
Appendix Cloud Computing, Amazon Web Services, and Their Impacts
Kevin J. Schmidt is a senior manager at Dell SecureWorks, Inc., anindustry leading MSSP, which is part of Dell. He is responsible for the design and development of a major part of the company’s SIEM platform. This includes data acquisition, correlation, and analysis of log data. Prior to SecureWorks, Kevin worked for Reflex Security, where he worked on an IPS engine and anti-virus software. And prior to this, he was a lead developer and architect at GuardedNet, Inc., which built one of the industry’s first SIEM platforms.
He is also a commissioned officer in the United States Navy Reserve (USNR). He has over 19 years of experience in software development and design, 11 of which have been in the network security space. He holds a Bachelor of Science in Computer Science.
Kevin has spent time designing cloud services components at Dell, including virtualized components to run in Dell’s own vCloud. These components are used to protect customers who use Dell’s cloud infrastructure. Additionally, he has been working with Hadoop, machine learning, and other technology in the cloud.
Kevin is co-author of Essential SNMP, second edition (O’Reilly and Associates, ISBN: 978-0-596-00840-6) and also Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management (Syngress, ISBN: 978-1-597-49635-3).
Christopher Phillips is a manager and senior software developer at Dell SecureWorks, Inc, an industry leading MSSP, which is part of Dell. He is responsible for the design and development of the company’s Threat Intelligence service platform. He also has responsibility for a team involved in integrating log and event information from many third-party providers that allow customers to have all of their core security information delivered to and analyzed by the Dell SecureWorks systems and security professionals.
Prior to Dell SecureWorks, Chris worked for McKesson and Allscripts, where he worked with clients on HIPAA compliance, security, and healthcare systems integration. He has over 18 years of experience in software development and design. He holds a Bachelor of Science in Computer Science and an MBA.
Chris has spent time designing and developing virtualization and cloud Infrastructure as a Service strategies at Dell to help our security services scale globally Additionally, he has been working with Hadoop, Pig scripting languages, and Amazon Elastic Map Reduce to develop strategies to gain insights and analyze Big Data issues in the cloud.
Chris is co-author of Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management (Syngress, ISBN: 978-1-597-49635-3).
The animal on the cover of Programming Elastic MapReduce is the eastern kingsnake(lampropeltis getula getula). The eastern kingsnake is a subspecies of the common kingsnake(Lampropeltis getula) that mostly lives in the Eastern United States. Commonkingsnakes can be found in swamps, streams, grasslands, and deserts across the UnitedStates and Mexico.Adult common kingsnakes, depending on the subspecies, are 20 to 78 inches in lengthand weigh between .62 and 5 pounds. They can be black, blue-black, or dark browncolored with two to four dozen white rings around their body. Even though they eatlizards, rodents, and birds, they also frequently eat other snakes. They eat other snakesby biting the mouth of their prey and stoping it from being able to counterattack. Commonkingsnakes are also immune to the venom of other snakes. They have no venomthemselves, however, and are considered harmless to humans.There are eight subspecies of common kingsnake, which may be the reason why thisspecies is known by several dozen names. Depending on the location, the commonkingsnake is called the Carolina Kingsnake, North American king snake, oakleaf rattler,thunder snake, black moccasin, thunderbolt, wamper, master snake, and pine snake.The cover image is from Wood’s Animate Creation. The cover fonts are URW Typewriterand Guardian Sans. The text font is Adobe Minion Pro; the heading font is AdobeMyriad Condensed; and the code font is Dalton Maag’s Ubuntu Mono