Hands-On Continuous Integration and Delivery

Book description

Understand various tools and practices for building a continuous integration and delivery pipeline effectively

Key Features

  • Get up and running with the patterns of continuous integration
  • Learn Jenkins UI for developing plugins and build an effective Jenkins pipeline
  • Automate CI/CD with command-line tools and scripts

Book Description

Hands-On Continuous Integration and Delivery starts with the fundamentals of continuous integration (CI) and continuous delivery (CD) and where it fits in the DevOps ecosystem. You will explore the importance of stakeholder collaboration as part of CI/CD.

As you make your way through the chapters, you will get to grips with Jenkins UI, and learn to install Jenkins on different platforms, add plugins, and write freestyle scripts. Next, you will gain hands-on experience of developing plugins with Jenkins UI, building the Jenkins 2.0 pipeline, and performing Docker integration. In the concluding chapters, you will install Travis CI and Circle CI and carry out scripting, logging, and debugging, helping you to acquire a broad knowledge of CI/CD with Travis CI and CircleCI.

By the end of this book, you will have a detailed understanding of best practices for CI/CD systems and be able to implement them with confidence.

What you will learn

  • Install Jenkins on multiple operating systems
  • Work with Jenkins freestyle scripts, pipeline syntax, and methodology
  • Explore Travis CI build life cycle events and multiple build languages
  • Master the Travis CI CLI (command-line interface) and automate tasks with the CLI
  • Use CircleCI CLI jobs and work with pipelines
  • Automate tasks using CircleCI CLI and learn to debug and troubleshoot
  • Learn open source tooling such as Git and GitHub
  • Install Docker and learn concepts in shell scripting

Who this book is for

Hands-On Continuous Integration and Delivery is for system administrators, DevOps engineers, and build and release engineers who want to understand the concept of CI and gain hands-on experience working with prominent tools in the CI ecosystem. Basic knowledge of software delivery is an added advantage.

Table of contents

  1. Title Page
  2. Copyright and Credits
    1. Hands-On Continuous Integration and Delivery
  3. Packt Upsell
    1. Why subscribe?
    2. PacktPub.com
  4. Contributors
    1. About the author
    2. About the reviewers
    3. Packt is searching for authors like you
  5. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
      1. Download the example code files
      2. Conventions used
    4. Get in touch
      1. Reviews
  6. CI/CD with Automated Testing
    1. Business scenario
      1. Manual processes – a hypothetical scenario
      2. Employee frustration
      3. Introducing automation
      4. Developer productivity
      5. Breaking down communication barriers
      6. Creating an environment of collaboration
    2. Summary
    3. Questions
    4. Further reading
  7. Basics of Continuous Integration
    1. Technical requirements
    2. What is CI?
      1. What is a software build anyway?
      2. CI process steps in a nutshell
      3. The value of CI
      4. Mitigating risks by utilizing CI
        1. But it works on my machine!
        2. Database synchronization
        3. A missing deployment automation phase
        4. Late discovery of defects
        5. Test coverage not known
          1. Code coverage tools
        6. Lack of project visibility
      5. Software builds at source code check-in
        1. What is a software build again?
        2. Scripting tool
        3. Performing single command builds
        4. Building your software in a nutshell
        5. Separating build scripts from your IDE
        6. Software assets should be centralized
        7. Creating a consistent directory structure
        8. Software builds should fail fast
        9. Building for any environment
      6. Small build and large build breakdown
      7. CI build practices
        1. Private build
        2. Usage of CI server
        3. Manual integration builds
        4. Running fast builds
    3. Summary
    4. Questions
    5. Further reading
  8. Basics of Continuous Delivery
    1. Technical requirements
    2. Problems delivering software
      1. What do we mean by delivering software?
      2. Common release anti-patterns
        1. Deploying software manually
          1. Deployment automation to the rescue
        2. Manual configuration management
          1. Configuration management automation
        3. How a production environment differs from other environments
          1. The production environment should be the same as the staging environment
      3. How to conduct a software release
        1. Frequent releases
        2. Automated releases
      4. The benefits of automation in delivering software
        1. Team empowerment
        2. Error reduction
        3. Stress reduction
    3. Configuration management
      1. What does configuration management mean anyway?
      2. Version control
        1. Example properties file
        2. Version control management tools
        3. Version control practices
        4. Conducting software check-ins often
        5. Writing descriptive and meaningful commit messages
      3. Dependency management
        1. Example Node.js dependency file and Gopkg.toml dependency file
        2. Managing software components
      4. Software configuration management
        1. Configurability and flexibility concepts
        2. Specific types of configuration
        3. Configuration management across applications
      5. Environment management
        1. Manual environment setup
        2. Important configuration information for environments
        3. Containerized environments
    4. Deployment pipeline
      1. What is a deployment pipeline?
      2. Deployment pipeline practices
        1. Building binaries once
        2. Deployment should be done the same way in every environment
        3. Commit stage – first step of the deployment pipeline
      3. Test gates
        1. Acceptance tests build stage
        2. Manual testing
        3. Nonfunctional testing
      4. Release preparation
        1. Automating release processes
        2. Conducting rollbacks
    5. Deployment scripting
      1. Overview of build tools
      2. Deployment scripting concepts
        1. Writing a script for each stage in the deployment pipeline
        2. Every environment should use the same scripts
        3. The deployment process should not change on each run
      3. Deployment scripting best practices
        1. Testing only known good foundations
        2. Testing the environment configuration
        3. Using relative paths
        4. Removing manual processes
    6. Deployment ecosystem
      1. Infrastructure tooling
      2. Cloud providers and tooling
    7. Summary
    8. Questions
    9. Further reading
  9. The Business Value of CI/CD
    1. Technical requirements
    2. Problems with communication
      1. Miscommunication of requirements
      2. Lack of proper documentation
      3. Timezone differences
      4. Lack of trust and mutual respect
      5. Cultural differences and language barriers
      6. Long feedback loop cycles
    3. Communicating pain points to team members
      1. Waiting for requirements information
      2. Undocumented steps in a deployment pipeline
      3. Keys to the kingdom to select few
      4. Too many communication channels
      5. Pain-Driven Development (PDD)
    4. Sharing responsibilities among different teams
      1. Rotating team members
      2. Asking for feedback on development practices
      3. Creating cross-functional teams
    5. Knowing your stakeholders
      1. Project managers
      2. Executive leadership team
      3. End users
    6. Demonstrating why CI/CD is important
      1. Metrics and reporting
      2. Educating leadership on automation
    7. Getting approval for CI/CD from your stakeholders
      1. Starting a skunkworks project
      2. Starting CI/CD on your local machine
      3. Company presentation
      4. Lunch and learn
    8. Summary
    9. Questions
    10. Further reading
  10. Installation and Basics of Jenkins
    1. Technical requirements
    2. Windows installation
      1. Prerequisites to installing Jenkins
        1. Finding your version of Windows
        2. Installing Java
      2. Windows installer
      3. Installing Jenkins in Windows
      4. Running the Jenkins Installer in Windows
        1. Installing Jenkins with the Chocolatey Package Manager
      5. Starting and stopping Jenkins in Windows in Command Prompt
    3. Linux installation
      1. Installing Jenkins on Ubuntu
      2. Starting the Jenkins service in Ubuntu
      3. Opening network traffic firewalls
      4. Unlocking Jenkins for the first login
    4. macOS installation
      1. Jenkins downloads package
      2. Unlocking Jenkins for the first login
      3. Installing Jenkins via Homebrew
    5. Running Jenkins locally
      1. Creating a new item
      2. Console Output
    6. Managing Jenkins
      1. Configuring environment variables and tools
      2. Configuring a job to poll the GitHub version control repository
    7. Summary
    8. Questions
    9. Further reading
  11. Writing Freestyle Scripts
    1. Technical requirements
    2. Creating a simple freestyle script
      1. Jenkins dashboard navigation
        1. Jenkins login screen
        2. Jenkins dashboard
      2. Adding a new build job item
      3. Build configuration options
    3. Configuring a freestyle job
      1. General
      2. Source Code Management
      3. Build Triggers
      4. Build environment
      5. Build
      6. Post-build actions
    4. Adding environment variables
      1. Global environment variable configuration
      2. EnvInject Plugin
    5. Debugging issues with a freestyle job
      1. Build Project View
      2. Debugging issues with freestyle script
    6. Summary
    7. Questions
    8. Further reading
  12. Developing Plugins
    1. Technical requirements
    2. Jenkins plugins explained
      1. Why are plugins useful?
      2. Jenkins plugin documentation
      3. Installing plugins in Jenkins
    3. Building a simple Jenkins plugin
      1. Java installation
      2. Maven installation instructions
        1. Windows installation
          1. Maven installation through the Chocolatey package manager
          2. Maven installation through the Maven source code
          3. Environment variables setup for the Windows operating system
        2. macOS installation
          1. Maven installation through the Homebrew package manager
        3. Unix installation
          1. Maven installation through the apt-get package manager
    4. Jenkins plugin development
      1. Maven settings file
      2. HelloWorld Jenkins plugin
      3. Folder layout explanation
      4. Jenkins plugin source code explanation
      5. Building a Jenkins plugin
      6. Installing a Jenkins plugin
    5. Jenkins plugin ecosystem
      1. List of available plugins
    6. Summary
    7. Questions
    8. Further reading
  13. Building Pipelines with Jenkins
    1. Technical requirements
    2. Jenkins 2.0
      1. Why move to Jenkins 2.0?
      2. Installing the Blue Ocean plugin on an existing instance
      3. Installing the Blue Ocean plugin through a Jenkins Docker Image 
        1. Docker prerequisites
        2. Installing Docker images
      4. Accessing Blue Ocean Jenkins view
    3. The Jenkins pipeline
      1. Creating a Jenkins pipeline
        1. Providing a personal access token for GitHub
        2. Picking your GitHub organization
        3. Picking the GitHub repository
      2. Creating a pipeline with the pipeline editor
    4. Navigating in Jenkins Blue Ocean
      1. Pipelines view
      2. Pipeline detail view
      3. Pipeline build view
      4. Pipeline stage view
      5. Other views in Jenkins pipelines
    5. Pipeline Syntax
      1. Pipeline Editor
        1. Jenkinsfile
          1. Pipeline keyword
          2. Agent keyword
          3. Stages keyword
      2. Pipeline Syntax documentation
    6. Summary
    7. Questions
    8. Further reading
  14. Installation and Basics of Travis CI
    1. Technical requirements
    2. Travis CI introduction
      1. Comparing Travis CI and Jenkins
    3. Travis CI prerequisites
      1. Creating a GitHub account
      2. Creating a Travis CI account
      3. Adding an SSH Key to your new GitHub Account
    4. Adding a simple Travis CI YAML configuration script
      1. Travis CI YML script contents
      2. Adding a Github repository into your Travis CI account
        1. Travis CI job log
    5. Travis CI script breakdown
      1. Select a programming language
      2. Selecting infrastructure
        1. Virtual image with Ubuntu Precise (12.04) infrastructure
        2. Default infrastructure
        3. Virtual Image with Ubuntu Trusty (14.04) infrastructure
        4. Container based infrastructure
        5. macOS infrastructure
      3. Build customization
        1. The build life cycle
          1. before_install step
          2. install step
          3. before_script step
          4. script step
          5. after_script step
          6. List of build life cycle
        2. Build failure behavior
        3. Installing a secondary programming language for a CI build
          1. Example Travis CI YML script with multiple languages
        4. Docker in Travis CI
          1. Example Travis CI YML script with a Dockerfile
        5. GUI and headless browsers in Travis CI
          1. Example Travis yml script with headless Chrome, Puppeteer, and the Jest Testing Library
    6. Summary
    7. Questions
    8. Further reading
  15. Travis CI CLI Commands and Automation
    1. Technical requirements
    2. Travis CLI installation
      1. Windows installation
      2. Linux installation
      3. macOS installation
    3. Travis CLI commands
      1. Non-API commands
        1. Printing help information
        2. Printing version information
      2. API commands
        1. Logging in to Travis CI
        2. Displaying the current access token
        3. Logging out of Travis CI
        4. Displaying accounts information
          1. Displaying help information for Travis commands
        5. Interactive console session
        6. Printing API endpoint Information
        7. Conducting live monitoring with all the CI builds that are currently running
        8. Initiating a Travis CI API call 
          1. API V3 REST call with curl
        9. Printing out important system configuration information
        10. Listing all the repositories that the currently logged in user has access to
        11. Initiating synchronization with Travis CI for any new or outdated repositories in GitHub
        12. lint - a Travis YML script
        13. Obtaining current build information for the organization or user
        14. Finding the currently logged-in user information
      3. Repository commands
        1. Displaying the most recent build information for each branch in Git version control
        2. Listing cache information for all the repositories
        3. Deleting cache information for a given repository
        4. Enabling a repository in Travis CI
        5. Disabling a repository in Travis CI
        6. Cancelling the latest build in Travis CI
        7. Encrypting environment variables or deployment keys
          1. Adding environment variables in Travis CI
        8. Encrypting a file
        9. Listing environment information
        10. Setting an environment variable
        11. Deleting an environment variable
        12. Clearing out all environment variables
        13. List history information for recent builds
        14. Initializing Travis CLI on a project
        15. Printing out CI build log information
        16. Opening the Travis web interface for the project
        17. Printing out public key information for the repository
        18. Restarting the latest CI build in Travis CI
        19. Printing out current build requests in Travis CI
        20. Printing out particular repository settings
        21. Configuring Travis CI add-ons
        22. Displaying general information for the current CI build
        23. Listing SSH key in Travis CI
        24. Displaying status information for the current build
      4. Travis CI options for Pro version and Enterprise version
        1. Option to display the information for Pro version
        2. Option to display information for Enterprise version
    4. Summary
    5. Questions
    6. Further reading
  16. Travis CI UI Logging and Debugging
    1. Technical requirements
    2. Travis web client overview
      1. Main dashboard overview
      2. Job log overview
    3. Debugging build locally with Docker
    4. Running build in debug mode
      1. Getting an API token from the profile page
        1. Getting  access to a token using Travis CLI
      2. Obtaining a job ID from the build log
      3. Obtaining the job ID from the URL in the View config button 
      4. Obtaining the job ID via an API call to the /builds endpoint
      5. API call to start a build job in debug mode
      6. Fetching an SSH session for the debug mode
      7. Travis debug mode convenience Bash functions
      8. tmate shell session actions
    5. Travis Web UI logging
      1. Steps that Travis CI takes to protect your environment-specific variables
    6. Travis CI deployment overview and debugging
      1. Supported Providers in Travis CI
      2. Heroku setup in Travis CI
      3. Debug failure in Travis YML script
    7. Summary
    8. Questions
    9. Further reading
  17. Installation and Basics of CircleCI
    1. Technical requirements
    2. CircleCI
    3. Comparing CircleCI and Jenkins
    4. CircleCI prerequisites
      1. Creating a GitHub account
      2. Creating a Bitbucket account
      3. Creating a CircleCI account
    5. Setting up CircleCI in GitHub
    6. Setting up CircleCI in Bitbucket
      1. Setting up a new Java project in Bitbucket with CircleCI build
    7. CircleCI configuration overview
      1. Concept overview of CircleCI configuration
      2. Adding source files to the new repository
      3. CircleCI build job for the new repository
    8. Summary
    9. Questions
    10. Further reading
  18. CircleCI CLI Commands and Automation
    1. Technical requirements
    2. CircleCI CLI installation
      1. Installing CircleCI CLI on macOS/Linux
      2. Installing nightly build versions of CircleCI through GitHub releases
    3. CircleCI CLI commands
      1. Version command
      2. Help command
      3. Config command
      4. Build command
      5. step command
      6. configure command
        1. Setting an API token with CircleCI
        2. Setting the API token and API endpoint in interactive mode
      7. Tests command
    4. Using Workflows in CircleCI
      1. Workflows in action in CircleCI Web UI
      2. Sequential workflows example
        1. Adding environment variables to a project
        2. Updated workflows section and config YML script
    5. Working with the CircleCI API
      1. Test CircleCI API connection
      2. Using the CircleCI API to get build summary for a single Git repo
      3. Using the jq utility to compute some metrics of our CircleCI builds
    6. Summary
    7. Questions
    8. Further reading
  19. CircleCI UI Logging and Debugging
    1. Technical requirements
    2. Job log overview
      1. Run steps in job with a default build job
        1. Adding a project to CircleCI via the API
          1. Using curl as a REST client
        2. Parse build_url attribute from the JSON response
        3. CircleCI Web UI job log analysis
        4. Best practices for environment variable safe usage
      2. Run steps in job with workflows
        1. Adding a workflows section to the config YML script
      3. Using the CircleCI API to find the most recent build URL
    3. Debugging slow builds in CircleCI
    4. Logging and troubleshooting techniques
      1. Running a build using a local config YML script to troubleshoot
    5. Summary
    6. Questions
    7. Further reading
  20. Best Practices
    1. Technical requirements
    2. Best practices for different types of testing in CI/CD
      1. Smoke testing
        1. Smoke test example
      2. Unit testing
        1. Unit test example
      3. Integration testing
        1. Integration test example
          1. API testing example using the supertest Node.js library
          2. API testing example with the baloo Golang library
          3. API testing example with curl, bash, and jq
      4. System testing
        1. System test example
      5. Acceptance testing
        1. Acceptance test example
      6. Best practices for running different tests in a CI/CD pipeline
    3. Best practices in password and secrets storage
      1. Vault installation
        1. Starting the dev server for Vault
        2. Checking the status of the Vault server
        3. Setting  an API secret in Vault
        4. Using the Vault RESTful API
          1. Vault RESTful API endpoint GET /v1/sys/raw/logical
      2. Overall best practices for secrets management
    4. Best practices in deployment
      1. Creating a deployment checklist
        1. Collaboration among developers and operations
      2. Releasing automation
      3. Deployment script example
        1. Automated GitHub release example
          1. Golang script example
      4. Best practices for a deployment script
    5. Summary
    6. Questions
    7. Further reading
  21. Assessments
    1. Chapter 1: CI/CD with Automated Testing
    2. Chapter 2: Basics of Continuous Integration
    3. Chapter 3: Basics of Continuous Delivery
    4. Chapter 4: The Business Value of CI/CD
    5. Chapter 5: Installation and Basics of Jenkins
    6. Chapter 6: Writing Freestyle Scripts
    7. Chapter 7: Developing Plugins
    8. Chapter 8: Building Pipelines with Jenkins
    9. Chapter 9: Installation and Basics of Travis CI
    10. Chapter 10: Travis CI CLI Commands and Automation
    11. Chapter 11: Travis CI UI Logging and Debugging
    12. Chapter 12: Installation and Basics of CircleCI
    13. Chapter 13: Circle CI CLI Commands and Automation
    14. Chapter 14: Circle CI UI Logging and Debugging
    15. Chapter 15: Best Practices
  22. Other Books You May Enjoy
    1. Leave a review - let other readers know what you think

Product information

  • Title: Hands-On Continuous Integration and Delivery
  • Author(s): Jean-Marcel Belmont
  • Release date: August 2018
  • Publisher(s): Packt Publishing
  • ISBN: 9781789130485