Apache Flume: Distributed Log Collection for Hadoop

Book description

Stream data to Hadoop using Apache Flume

  • Integrate Flume with your data sources
  • Transcode your data en-route in Flume
  • Route and separate your data using regular expression matching
  • Configure failover paths and load-balancing to remove single points of failure
  • Utilize Gzip Compression for files written to HDFS

In Detail

Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. Its main goal is to deliver data from applications to Apache Hadoop's HDFS. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with many failover and recovery mechanisms.

Apache Flume: Distributed Log Collection for Hadoop covers problems with HDFS and streaming data/logs, and how Flume can resolve these problems. This book explains the generalized architecture of Flume, which includes moving data to/from databases, NO-SQL-ish data stores, as well as optimizing performance. This book includes real-world scenarios on Flume implementation.

Apache Flume: Distributed Log Collection for Hadoop starts with an architectural overview of Flume and then discusses each component in detail. It guides you through the complete installation process and compilation of Flume.

It will give you a heads-up on how to use channels and channel selectors. For each architectural component (Sources, Channels, Sinks, Channel Processors, Sink Groups, and so on) the various implementations will be covered in detail along with configuration options. You can use it to customize Flume to your specific needs. There are pointers given on writing custom implementations as well that would help you learn and implement them.

By the end, you should be able to construct a series of Flume agents to transport your streaming data and logs from your systems into Hadoop in near real time.

Table of contents

  1. Apache Flume: Distributed Log Collection for Hadoop
    1. Table of Contents
    2. Apache Flume: Distributed Log Collection for Hadoop
    3. Credits
    4. About the Author
    5. About the Reviewers
    6. www.PacktPub.com
      1. Support files, eBooks, discount offers and more
        1. Why Subscribe?
        2. Free Access for Packt account holders
    7. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
        1. Errata
        2. Piracy
        3. Questions
    8. 1. Overview and Architecture
      1. Flume 0.9
      2. Flume 1.X (Flume-NG)
      3. The problem with HDFS and streaming data/logs
      4. Sources, channels, and sinks
      5. Flume events
        1. Interceptors, channel selectors, and sink processors
        2. Tiered data collection (multiple flows and/or agents)
      6. Summary
    9. 2. Flume Quick Start
      1. Downloading Flume
        1. Flume in Hadoop distributions
      2. Flume configuration file overview
      3. Starting up with "Hello World"
      4. Summary
    10. 3. Channels
      1. Memory channel
      2. File channel
      3. Summary
    11. 4. Sinks and Sink Processors
      1. HDFS sink
        1. Path and filename
        2. File rotation
      2. Compression codecs
      3. Event serializers
        1. Text output
        2. Text with headers
        3. Apache Avro
        4. File type
          1. Sequence file
          2. Data stream
          3. Compressed stream
        5. Timeouts and workers
      4. Sink groups
        1. Load balancing
        2. Failover
      5. Summary
    12. 5. Sources and Channel Selectors
      1. The problem with using tail
      2. The exec source
      3. The spooling directory source
      4. Syslog sources
        1. The syslog UDP source
        2. The syslog TCP source
        3. The multiport syslog TCP source
      5. Channel selectors
        1. Replicating
        2. Multiplexing
      6. Summary
    13. 6. Interceptors, ETL, and Routing
      1. Interceptors
        1. Timestamp
        2. Host
        3. Static
        4. Regular expression filtering
        5. Regular expression extractor
        6. Custom interceptors
      2. Tiering data flows
        1. Avro Source/Sink
        2. Command-line Avro
        3. Log4J Appender
        4. The Load Balancing Log4J Appender
      3. Routing
      4. Summary
    14. 7. Monitoring Flume
      1. Monitoring the agent process
        1. Monit
        2. Nagios
      2. Monitoring performance metrics
        1. Ganglia
        2. The internal HTTP server
        3. Custom monitoring hooks
      3. Summary
    15. 8. There Is No Spoon – The Realities of Real-time Distributed Data Collection
      1. Transport time versus log time
      2. Time zones are evil
      3. Capacity planning
      4. Considerations for multiple data centers
      5. Compliance and data expiry
      6. Summary
    16. Index

Product information

  • Title: Apache Flume: Distributed Log Collection for Hadoop
  • Author(s): Steve Hoffman
  • Release date: July 2013
  • Publisher(s): Packt Publishing
  • ISBN: 9781782167914