Mastering SaltStack - Second Edition

Mastering SaltStack - Second Edition

by Joseph Hall
Released November 2016
Publisher(s): Packt Publishing
ISBN: 9781786467393

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Take charge of SaltStack to automate and configure your enterprise-grade environments

About This Book

  • Automate tasks effectively and take charge of your infrastructure
  • Effectively scale Salt to manage thousands of machines and tackle everyday problems
  • Explore Salt's inner workings and advance your knowledge of it

Who This Book Is For

This book is ideal for IT professionals and ops engineers who already manage groups of servers, but would like to expand their knowledge and gain expertise with SaltStack. This book explains the advanced features and concepts of Salt. A basic knowledge of Salt is required in order to get to grips with advanced Salt features.

What You Will Learn

  • Automate tasks effectively, so that your infrastructure can run itself
  • Start building more complex concepts
  • Master user-level internals
  • Build scaling strategies
  • Explore monitoring strategies
  • Learn how to troubleshoot Salt and its subcomponents
  • Explore best practices for Salt

In Detail

SaltStack is a powerful configuration management and automation suite designed to manage servers and tens of thousands of nodes. This book showcases Salt as a very powerful automation framework. We will review the fundamental concepts to get you in the right frame of mind, and then explore Salt in much greater depth. You will explore Salt SSH as a powerful tool and take Salt Cloud to the next level. Next, you'll master using Salt services with ease in your infrastructure. You will discover methods and strategies to scale your infrastructure properly. You will also learn how to use Salt as a powerful monitoring tool.

By the end of this book, you will have learned troubleshooting tips and best practices to make the entire process of using Salt pain-free and easy.

Style and approach

This book follows a step-by-step conversational tone. Topics are covered in detail through examples and a user-friendly approach.

Table of contents

  1. Mastering SaltStack Second Edition
    1. Mastering SaltStack Second Edition
    2. Credits
    3. Foreword
    4. About the Author
    5. About the Reviewer
    6. www.PacktPub.com
      1. Why subscribe?
    7. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
        1. Errata
        2. Piracy
        3. Questions
    8. 1. Essentials Revisited
      1. Executing commands remotely
        1. Master and minions
        2. Targeting minions
          1. Glob
          2. Perl Compatible Regular Expression (PCRE)
          3. List
          4. Subnet
          5. Grain
          6. Grain PCRE
          7. Pillar
          8. Compound
          9. Nodegroup
        3. Using module functions
          1. test.ping
          2. test.echo
          3. test.sleep
          4. test.version
          5. pkg.install
          6. pkg.remove
          7. file.replace
          8. Other file functions
          9. Various user and group functions
          10. sys.doc
      2. SLS file trees
        1. SLS files
        2. Tying things together with top files
        3. Organizing the SLS directories
      3. Using states for configuration management
        1. Using include blocks
        2. Ordering with requisites
          1. require
          2. watch
          3. onchanges
          4. onfail
          5. use
        3. prereq
        4. Inverting requisites
        5. Extending SLS files
      4. The basics of grains, pillars, and templates
        1. Using grains for minion-specific data
        2. Centralizing variables with pillars
        3. Managing files dynamically with templates
        4. A quick Jinja primer
      5. Summary
    9. 2. Diving into Salt Internals
      1. Understanding the Salt configuration
        1. Following the configuration tree
          1. Looking inside /etc/salt/
          2. Managing Salt keys
          3. Exploring the SLS directories
        2. Examining the Salt cache
          1. The master job cache
          2. The master-side minion cache
          3. The external file server cache
          4. The minion-side proc/ directory
          5. External modules
      2. The renderer
        1. Rendering SLS files
        2. Render pipes
        3. Serving templated files
      3. Understanding the loader
        1. Dynamic modules
        2. Execution modules
        3. Cloud modules
      4. Plunging into the state compiler
        1. Imperative versus declarative
        2. Requisites
        3. High and low states
        4. High states
        5. Low states
        6. Enforcing statefulness
          1. name
          2. result
          3. changes
          4. comment
      5. Summary
    10. 3. Managing States
      1. Handling multiple states
        1. Including other SLS files
      2. Spanning multiple environments
        1. Using the base environment
        2. Breaking out environments
      3. Understanding master_tops
      4. Using Salt Stack Formulas on GitHub
        1. Examining the standard layout
          1. formula name
          2. pillar.example
          3. README
          4. LICENSE
          5. FORMULA
        2. Cloning formulas
      5. Using the Salt Package Manager
        1. Thinking of SPM as a package manager
        2. Thinking of SPM as a repository manager
          1. Configuring SPM repositories
          2. Downloading repository metadata
        3. Creating SPM repositories
          1. Set aside a directory to hold packages
          2. Share that directory
          3. Populate the directory
          4. Create the repository metadata
        4. Building packages
          1. version
          2. release
          3. summary
          4. description
          5. name
          6. os
          7. os_family
          8. dependencies
          9. optional
          10. recommended
          11. minimum_version
          12. top_level_dir
          13. license
        5. Where to use SPM
        6. SPM configuration
          1. spm_conf_file
          2. formula_path
          3. pillar_path
          4. reactor_path
          5. spm_logfile
          6. spm_default_include
          7. spm_repos_config
          8. spm_cache_dir
          9. spm_build_dir
          10. spm_build_exclude
          11. spm_db
      6. Summary
    11. 4. Exploring Salt SSH
      1. Grappling with SSH
        1. Remote shells
      2. Using rosters
        1. The flat roster
          1. host
          2. port
          3. user
          4. passwd
          5. sudo
          6. priv
          7. timeout
          8. thin_dir
        2. Other built-in rosters
          1. scan
          2. cache
          3. cloud
          4. ansible
        3. Building dynamic rosters
        4. Using Salt SSH
          1. Using a Saltfile
        5. Salt versus Salt SSH
          1. Architecture
          2. Performance
      3. Understanding the salt-thin agent
        1. Building the thin package
          1. Including extra modules
        2. Deploying the thin package
        3. Executing the thin package
          1. The Salt SSH shim
          2. Preparing for Salt states
          3. Running Salt
          4. Salt's running data
      4. Using the raw SSH mode
        1. Caching SSH connections
      5. Summary
    12. 5. Managing Tasks Asynchronously
      1. Looking at the event system
        1. Reviewing the basics
        2. The structure of event data
        3. Watching event data
          1. Installing the event listener
          2. Using the event listener
          3. Firing custom data
        4. Namespacing events
          1. Namespacing guidelines
        5. Some common events
          1. salt/auth
          2. salt/key
          3. salt/minion/minion_id/start
          4. salt/job/job_id/new
          5. salt/job/job_id/ret/minion_id
          6. salt/presence/present
          7. salt/presence/change
        6. Common cloud events
          1. salt/cloud/vm_name/creating
          2. salt/cloud/vm_name/requesting
          3. salt/cloud/vm_name/querying
          4. salt/cloud/vm_name/waiting_for_ssh
          5. salt/cloud/vm_name/deploying
          6. salt/cloud/vm_name/created
          7. salt/cloud/vm_name/destroying
          8. salt/cloud/vm_name/destroyed
        7. Salt API events
          1. salt/netapi/url_path
      2. Building reactors
        1. Configuring reactors
        2. Writing reactors
          1. Calling execution modules
          2. Calling runner modules
          3. Calling wheel modules
        3. Writing more complex reactors
          1. Sending out alerts
          2. Using webhooks
          3. Reactors calling reactors
      3. Using Thorium
        1. A word on engines
        2. Looking at Thorium basics
        3. Enabling Thorium
        4. Setting up the Thorium directory tree
        5. Writing Thorium SLS files
        6. Using requisites
        7. Using the register
        8. Looking forward
      4. Using the queue system
        1. Learning how queues work
          1. Adding to the queue
          2. Listing queues
          3. Listing items in a queue
          4. Processing queue items
          5. Deleting items from a queue
        2. Using queues with the reactor
          1. Spreading out State runs
          2. Dividing tasks among minions
      5. Summary
    13. 6. Taking Advantage of Salt Information Systems
      1. Understanding pillar_roots
        1. Templating pillar_roots
        2. Calling out to other modules
      2. Using Salt's external pillars
        1. Configuring the etcd pillar
        2. Using git_pillar
        3. Using the mysql pillar
        4. Some final thoughts on external pillars
          1. Using multiple external pillars
          2. Caching pillar data
      3. Understanding SDB
        1. Securely storing passwords
        2. Staying simple
        3. Using SDB URIs
        4. Configuring SDB
        5. Performing SDB lookups
          1. Getting data
          2. Setting data
          3. Deleting data
      4. Comparing pillars and SDB
        1. Where the data is generated
        2. Differences in drivers
        3. Salt Cloud and others
      5. Summary
    14. 7. Taking Salt Cloud to the Next Level
      1. Examining the Salt Cloud configuration
        1. Global configurations
        2. Provider and profile configuration
          1. Providers
          2. Profiles
        3. Extending configuration blocks
      2. Using SDB with Salt Cloud
        1. Using SDB with OpenStack
        2. Using SDB with AWS/EC2
        3. Using SDB with other cloud providers
      3. Building custom deploy scripts
        1. Understanding the Salt Bootstrap script
          1. Installing from prebuilt packages
          2. Installing from Git
        2. Looking back at legacy deploy scripts
        3. Writing your own deploy scripts
        4. Passing arguments to scripts
        5. Using file maps
      4. Taking a look at cloud maps
      5. Using reactors with Salt Cloud
        1. Setting up the event tags
      6. Working with autoscale reactors
        1. Cloud cache
        2. Using cloud cache events
          1. Setting up a schedule
          2. Catching cloud cache events
      7. Summary
    15. 8. Using Salt with REST
      1. Looking at Salt's HTTP library
        1. Why a Salt-specific library?
        2. Using the http.query function
          1. GET versus POST
          2. Decoding return data
        3. Using the http.query state
          1. Using http.query with reactors
      2. Understanding the Salt API
        1. What is the Salt API?
        2. Setting up the Salt API
          1. CherryPy
          2. Tornado
          3. WSGI
        3. Creating SSL certificates
        4. Configuring authentication
          1. The external authentication module
        5. Taking your first steps with the Salt API
          1. Issuing one-off commands
      3. Working with webhooks
        1. Reacting with Thorium
        2. Security considerations
        3. More complex authentication
      4. Summary
    16. 9. Understanding the RAET and TCP Transports
      1. Comparing RAET and ZeroMQ
        1. Starting with HTTP
        2. SSH - the old favorite
        3. Using ZeroMQ
          1. ZeroMQ and security
        4. The need for RAET
      2. Flow-based programming
        1. The pieces of the puzzle
          1. Black boxes
          2. Shared storage
          3. Concurrent scheduling
      3. Driving with RAET
        1. Configuring RAET
        2. The RAET architecture
          1. The basics
          2. The RAET scheduler
          3. Estates and yards
      4. Looking at asynchronous programming
        1. Cooks in a restaurant
      5. Examining the TCP transport
      6. Using the TCP transport
      7. Summary
    17. 10. Strategies for Scaling
      1. All about syndication
        1. Different folks, different strokes
        2. No need for micromanaging
        3. Configuring syndication
      2. High availability with multiple masters
        1. Built-in high-availability configuration
        2. Old-school high availability
          1. The round-robin DNS
          2. IP-based load balancing
        3. Synchronizing files
          1. Base configuration files
          2. Synchronizing the nonexternal files
          3. Using rsync
          4. Using the event reactor
      3. Incorporating external data sources
        1. The external job cache
          1. Using returners on the minions
          2. Using the master job cache
        2. External filesystems
          1. GitFS
            1. base
            2. root
            3. mountpoint
            4. user and password
            5. insecure_auth
            6. pubkey, privkey, and passphrase
          2. Other source-control backends
          3. SVNFS
            1. root and mountpoint
            2. trunk
            3. branches
            4. tags
          4. HGFS
          5. S3FS
            1. One environment per bucket
            2. Multiple environments per bucket
          6. AzureFS
        3. External pillars
          1. cmd_yaml/cmd_json
          2. git
          3. redis
          4. mysql
      4. Using the master API
        1. The Salt keys
        2. Configuration
        3. The file and pillar roots
        4. Using the wheel reactor
          1. Using wheel with Thorium
      5. Testing the load in the infrastructure
        1. Using the minionswarm.py script
          1. Swarm internals
      6. Summary
    18. 11. Monitoring with Salt
      1. Monitoring basics
        1. Establishing a baseline
        2. Reading the system vitals in Salt
          1. status.loadavg
          2. status.cpustats
          3. status.meminfo
          4. status.vmstats
          5. disk.usage and status.diskusage
          6. status.w
          7. status.all_status and status.custom
      2. Monitoring with returners
        1. Deciding on a returner
      3. Using monitoring states
        1. Defining a monitoring state
        2. Monitoring with web calls
      4. Working with beacons
        1. Monitoring file changes
        2. Beacon intervals
      5. Setting up alerts
        1. Alerting in state files
        2. Alerting from beacons
          1. Watching file changes
          2. Monitoring bad logins
      6. Using aggregate data with Thorium
      7. Summary
    19. 12. Exploring Best Practices
      1. Future-proofing your infrastructure
      2. Setting up your directories
        1. Standard directory locations
        2. .sls versus init.sls
        3. Shallow versus deep
        4. Subdividing further
      3. The SLS efficiency
        1. Includes and extends
          1. Using includes
          2. Using extends
        2. Using templates to simplify SLS files
          1. Working with loops
          2. Decisions, decisions
        3. Using the built-in states
      4. Naming conventions
        1. Generic names
        2. Explicit names
      5. Templates and variables
        1. Nested variables
        2. Referring to variables in templates
      6. Summary
    20. 13. Troubleshooting Problems
      1. What the...?
        1. Addressing the problem source
        2. Where is the trouble?
          1. Master-to-minion communication
          2. Network and CPU congestion
          3. Checking minion load
          4. Querying the Salt job data
        3. Using debug and trace modes
          1. info
          2. warn
          3. error
          4. debug/trace
          5. Running services in debug mode
      2. Using salt-call locally
      3. Working with YAML
        1. YAML basics
          1. dict
          2. list
        2. YAML idiosyncrasies
          1. Spacing
          2. Numbers
          3. Booleans
          4. List items
        3. Troubleshooting YAML
      4. Asking the community for help
        1. The salt-users mailing list
          1. Asking questions
        2. The Salt issue tracker
          1. Researching before posting
          2. Formatting your issues
          3. Requesting features
        3. #salt on IRC
        4. Final community thoughts
      5. Summary

Product information

  • Title: Mastering SaltStack - Second Edition
  • Author(s): Joseph Hall
  • Release date: November 2016
  • Publisher(s): Packt Publishing
  • ISBN: 9781786467393