Mastering Ceph

Mastering Ceph

by Nick Fisk
Released May 2017
Publisher(s): Packt Publishing
ISBN: 9781785888786

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Deep dive into the unified, distributed storage system in order to provide excellent performance

About This Book

  • Leverage Ceph's advanced features such as erasure coding, tiering, and Bluestore
  • Solve large-scale problems with Ceph as a tool by understanding its strengths and weaknesses to develop the best solutions
  • A practical guide that covers engaging use cases to help you use advanced features of Ceph effectively

Who This Book Is For

If you are a developer and an administrator who has deployed a Ceph cluster before and are curious about some of the most advanced features in order to improve performance then this book is for you

What You Will Learn

  • Know when and how to use some of Ceph's advanced new features
  • Set up a test cluster with Ansible and some virtual machines using VirtualBox and Vagrant
  • Develop novel solutions to massive problems with librados and shared object classes.
  • Choose intelligent parameters for an erasure coded pool and set it up.
  • Configure the Bluestore settings and see how they interact with different hardware configurations.
  • Keep Ceph running through thick and thin with tuning, monitoring and disaster recovery advice.

In Detail

Mastering Ceph covers all that you need to know to use Ceph effectively. Starting with design goals and planning steps that should be undertaken to ensure successful deployments, you will be guided through to setting up and deploying the Ceph cluster, with the help of orchestration tools. Key areas of Ceph including Bluestore, Erasure coding and cache tiering will be covered with help of examples. Development of applications which use Librados and Distributed computations with shared object classes are also covered. A section on tuning will take you through the process of optimisizing both Ceph and its supporting infrastructure. Finally, you will learn to troubleshoot issues and handle various scenarios where Ceph is likely not to recover on its own.

By the end of the book, you will be able to successfully deploy and operate a resilient high performance Ceph cluster.

Style and Approach

A practical guide which has each chapter explaining the concept, sharing tips and tricks and a use case to implement the most powerful features of Ceph

Table of contents

  1. Preface
    1. What this book covers
    2. What you need for this book
    3. Who this book is for
    4. Conventions
    5. Reader feedback
    6. Customer support
      1. Downloading the example code
      2. Downloading the color images of this book
      3. Errata
      4. Piracy
      5. Questions
  2. Planning for Ceph
    1. What is Ceph?
    2. How Ceph works?
    3. Ceph use cases
      1. Replacing your storage array with Ceph
      2. Performance
      3. Reliability
      4. The use of commodity hardware
    4. Specific use cases
      1. OpenStack- or KVM-based virtualization
      2. Large bulk block storage
        1. Object storage
        2. Object storage with custom application
        3. Distributed filesystem - web farm
        4. Distributed filesystem -SMB file server replacement
    5. Infrastructure design
      1. SSDs
        1. Consumer
        2. Prosumer
        3. Enterprise SSDs
          1. Enterprise -read intensive
          2. Enterprise - general usage
          3. Enterprise -write intensive
      2. Memory
      3. CPU
      4. Disks
      5. Networking
      6. 10G networking requirement
    6. Network design
      1. OSD node sizes
        1. Failure domains
      2. Price
      3. Power supplies
    7. How to plan a successful Ceph implementation
      1. Understanding your requirements and how it relates to Ceph
      2. Defining goals so that you can gauge if the project is a success
      3. Choosing your hardware
      4. Training yourself and your team to use Ceph
      5. Running PoC to determine if Ceph has met the requirements
      6. Following best practices to deploy your cluster
      7. Defininga change management process
      8. Creating a backup and recovery plan
    8. Summary
  3. Deploying Ceph
    1. Preparing your environment with Vagrant and VirtualBox
      1. System requirements
      2. Obtaining and installing VirtualBox
      3. Setting up Vagrant
      4. The ceph-deploy tool
    2. Orchestration
    3. Ansible
      1. Installing Ansible
      2. Creating your inventoryfile
      3. Variables
      4. Testing
    4. A very simple playbook
    5. Adding the Ceph Ansible modules
      1. Deploying a test cluster with Ansible
    6. Change and configuration management
    7. Summary
  4. BlueStore
    1. What is BlueStore?
    2. Why was it needed?
      1. Ceph's requirements
        1. Filestore limitations
      2. Why is BlueStore the solution?
    3. How BlueStore works
      1. RocksDB
      2. Deferred writes
      3. BlueFS
    4. How to use BlueStore
      1. Upgrading an OSD in your test cluster
    5. Summary
  5. Erasure Coding for Better Storage Efficiency
    1. What is erasurecoding?
      1. K+M
    2. How does erasure coding work in Ceph?
    3. Algorithms and profiles
      1. Jerasure
      2. ISA
      3. LRC
      4. SHEC
    4. Where can I use erasure coding?
    5. Creating an erasure-coded pool
      1. Overwrites on erasure code pools with Kraken
      2. Demonstration
      3. Troubleshooting the 2147483647 error
        1. Reproducing the problem
    6. Summary
  6. Developing with Librados
    1. What is librados?
    2. How to use librados?
    3. Example librados application
      1. Example of the librados application with atomic operations
      2. Example of the librados application that uses watchers and notifiers
    4. Summary
  7. Distributed Computation with Ceph RADOS Classes
    1. Example applications and the benefits of using RADOS classes
    2. Writing a simple RADOS class in Lua
    3. Writing a RADOS class that simulates distributed computing
      1. Preparing the build environment
      2. RADOS class
      3. Client librados applications
        1. Calculating MD5 on the client
        2. Calculating MD5 on the OSD via RADOS class
      4. Testing
    4. RADOS class caveats
    5. Summary
  8. Monitoring Ceph
    1. Why it is important to monitor Ceph
    2. What should be monitored
      1. Ceph health
      2. Operating system and hardware
      3. Smart stats
      4. Network
      5. Performance counters
    3. PG states -the good, the bad, and the ugly
      1. The good
        1. The active state
        2. The clean state
        3. Scrubbing and deep scrubbing
      2. The bad
        1. The inconsistent state
        2. The backfilling, backfill_wait, recovering, recovery_wait states
        3. The degraded state
        4. Remapped
      3. The ugly
        1. The incomplete state
        2. The down state
        3. The backfill_toofull state
    4. Monitoring Ceph with collectd
      1. Graphite
      2. Grafana
      3. collectd
      4. Deploying collectd with Ansible
      5. Sample Graphite queries for Ceph
        1. Number of Up and In OSDs
        2. Showing most deviant OSD usage
        3. Total number of IOPs across all OSDs
        4. Total MBps across all OSDs
        5. Cluster capacity and usage
        6. Average latency
      6. Custom Ceph collectd plugins
    5. Summary
  9. Tiering with Ceph
    1. Tiering versus caching
      1. How Cephs tiering functionality works
    2. What is a bloom filter
    3. Tiering modes
      1. Writeback
      2. Forward
        1. Read-forward
      3. Proxy
        1. Read-proxy
    4. Uses cases
    5. Creating tiers in Ceph
    6. Tuning tiering
      1. Flushing and eviction
        1. Promotions
    7. Promotion throttling
      1. Monitoring parameters
      2. Tiering with erasure-coded pools
      3. Alternative caching mechanisms
    8. Summary
  10. Tuning Ceph
    1. Latency
    2. Benchmarking
      1. Benchmarking tools
        1. Fio
        2. Sysbench
        3. Ping
        4. iPerf
      2. Network benchmarking
      3. Disk benchmarking
      4. RADOS benchmarking
      5. RBD benchmarking
    3. Recommended tunings
      1. CPU
      2. Filestore
        1. VFS cache pressure
        2. WBThrottle and/or nr_requests
        3. Filestore queue throttling
          1. filestore_queue_low_threshhold
          2. filestore_queue_high_threshhold
          3. filestore_expected_throughput_ops
          4. filestore_queue_high_delay_multiple
          5. filestore_queue_max_delay_multiple
        4. PG Splitting
      3. Scrubbing
      4. OP priorities
      5. The Network
      6. General system tuning
      7. Kernel RBD
        1. Queue Depth
        2. ReadAhead
      8. PG distributions
    4. Summary
  11. Troubleshooting
    1. Repairing inconsistent objects
    2. Full OSDs
    3. Ceph logging
    4. Slow performance
      1. Causes
        1. Increased client workload
        2. Down OSDs
        3. Recovery and backfilling
        4. Scrubbing
        5. Snaptrimming
        6. Hardware or driver issues
      2. Monitoring
        1. iostat
        2. htop
        3. atop
      3. Diagnostics
    5. Extremely slow performance or no IO
      1. Flapping OSDs
      2. Jumbo frames
      3. Failing disks
      4. Slow OSDs
    6. Investigating PGs in a down state
    7. Large monitor databases
    8. Summary
  12. Disaster Recovery
    1. What is a disaster?
    2. Avoiding data loss
    3. What can cause an outage or data loss?
    4. RBD mirroring
      1. The journal
      2. The rbd-mirror daemon
      3. Configuring RBD mirroring
      4. Performing RBD failover
    5. RBD recovery
    6. Lost objects and inactive PGs
    7. Recovering from a complete monitor failure
    8. Using the Cephs object store tool
    9. Investigating asserts
      1. Example assert
    10. Summary

Product information

  • Title: Mastering Ceph
  • Author(s): Nick Fisk
  • Release date: May 2017
  • Publisher(s): Packt Publishing
  • ISBN: 9781785888786