Beautiful Testing

Leading Professionals Reveal How They Improve Software

Larger Cover

Beautiful Testing

Leading Professionals Reveal How They Improve Software

By Tim Riley, Adam Goucher

Publisher: O'Reilly Media

Released: October 2009

Pages: 352

Successful software depends as much on scrupulous testing as it does on solid architecture or elegant code. But testing is not a routine process, it's a constant exploration of methods and an evolution of good ideas.

Beautiful Testing offers 23 essays from 27 leading testers and developers that illustrate the qualities and techniques that make testing an art. Through personal anecdotes, you'll learn how each of these professionals developed beautiful ways of testing a wide range of products -- valuable knowledge that you can apply to your own projects.

Here's a sample of what you'll find inside:

Microsoft's Alan Page knows a lot about large-scale test automation, and shares some of his secrets on how to make it beautiful
Scott Barber explains why performance testing needs to be a collaborative process, rather than simply an exercise in measuring speed
Karen Johnson describes how her professional experience intersected her personal life while testing medical software
Rex Black reveals how satisfying stakeholders for 25 years is a beautiful thing
Mathematician John D. Cook applies a classic definition of beauty, based on complexity and unity, to testing random number generators

All author royalties will be donated to the Nothing But Nets campaign to save lives by preventing malaria, a disease that kills millions of children in Africa each year.

This book includes contributions from:

Adam Goucher
Linda Wilkinson
Rex Black
Martin Schröder
Clint Talbert
Scott Barber
Kamran Khan
Emily Chen
Brian Nitz
Remko Tronçon
Alan Page
Neal Norwitz
Michelle Levesque
Jeffrey Yasskin
John D. Cook
Murali Nandigama
Karen N. Johnson
Chris McMahon
Jennitta Andrea
Lisa Crispin
Matt Heusser
Andreas Zeller
David Schuler
Tomasz Kojm
Adam Christian
Tim Riley
Isaac Clerencia

Beautiful Testers
1. Chapter 1 Was It Good for You?
2. Chapter 2 Beautiful Testing Satisfies Stakeholders
  1. For Whom Do We Test?
  2. What Satisfies?
  3. What Beauty Is External?
  4. What Beauty Is Internal?
  5. Conclusions
3. Chapter 3 Building Open Source QA Communities
  1. Communication
  2. Volunteers
  3. Coordination
  4. Events
  5. Conclusions
4. Chapter 4 Collaboration Is the Cornerstone of Beautiful Performance Testing
  1. Setting the Stage
  2. 100%?!? Fail
  3. The Memory Leak That Wasn’t
  4. Can’t Handle the Load? Change the UI
  5. It Can’t Be the Network
  6. Wrap-Up
Beautiful Process
1. Chapter 5 Just Peachy: Making Office Software More Reliable with Fuzz Testing
  1. User Expectations
  2. What Is Fuzzing?
  3. Why Fuzz Test?
  4. Fuzz Testing
  5. Future Considerations
2. Chapter 6 Bug Management and Test Case Effectiveness
  1. Bug Management
  2. The First Step in Managing a Defect Is Defining It
  3. Test Case Effectiveness
  4. Case Study of the OpenSolaris Desktop Team
  5. Conclusions
  6. Acknowledgments
  7. References
3. Chapter 7 Beautiful XMPP Testing
  1. Introduction
  2. XMPP 101
  3. Testing XMPP Protocols
  4. Unit Testing Simple Request-Response Protocols
  5. Unit Testing Multistage Protocols
  6. Testing Session Initialization
  7. Automated Interoperability Testing
  8. Diamond in the Rough: Testing XML Validity
  9. Conclusions
  10. References
4. Chapter 8 Beautiful Large-Scale Test Automation
  1. Before We Start
  2. What Is Large-Scale Test Automation?
  3. The First Steps
  4. Automated Tests and Test Case Management
  5. The Automated Test Lab
  6. Test Distribution
  7. Failure Analysis
  8. Reporting
  9. Putting It All Together
5. Chapter 9 Beautiful Is Better Than Ugly
  1. The Value of Stability
  2. Ensuring Correctness
  3. Conclusions
6. Chapter 10 Testing a Random Number Generator
  1. What Makes Random Number Generators Subtle to Test?
  2. Uniform Random Number Generators
  3. Nonuniform Random Number Generators
  4. A Progression of Tests
  5. Conclusions
7. Chapter 11 Change-Centric Testing
  1. How to Set Up the Document-Driven, Change-Centric Testing Framework?
  2. Change-Centric Testing for Complex Code Development Models
  3. What Have We Learned So Far?
  4. Conclusions
8. Chapter 12 Software in Use
  1. A Connection to My Work
  2. From the Inside
  3. Adding Different Perspectives
  4. Exploratory, Ad-Hoc, and Scripted Testing
  5. Multiuser Testing
  6. The Science Lab
  7. Simulating Real Use
  8. Testing in the Regulated World
  9. At the End
9. Chapter 13 Software Development Is a Creative Process
  1. Agile Development As Performance
  2. Practice, Rehearse, Perform
  3. Evaluating the Ineffable
  4. Two Critical Tools
  5. Software Testing Movements
  6. The Beauty of Agile Testing
  7. QA Is Not Evil
  8. Beauty Is the Nature of This Work
  9. References
10. Chapter 14 Test-Driven Development: Driving New Standards of Beauty
  1. Beauty As Proportion and Balance
  2. Agile: A New Proportion and Balance
  3. Test-Driven Development
  4. Examples Versus Tests
  5. Readable Examples
  6. Permanent Requirement Artifacts
  7. Testable Designs
  8. Tool Support
  9. Team Collaboration
  10. Experience the Beauty of TDD
  11. References
11. Chapter 15 Beautiful Testing As the Cornerstone of Business Success
  1. The Whole-Team Approach
  2. Automating Tests
  3. Driving Development with Tests
  4. Delivering Value
  5. A Success Story
  6. Post Script
12. Chapter 16 Peeling the Glass Onion at Socialtext
  1. It’s Not Business…It’s Personal
  2. Tester Remains On-Stage; Enter Beauty, Stage Right
  3. Come Walk with Me, The Best Is Yet to Be
  4. Automated Testing Isn’t
  5. Into Socialtext
  6. A Balanced Breakfast Approach
  7. Regression and Process Improvement
  8. The Last Pieces of the Puzzle
  9. Acknowledgments
13. Chapter 17 Beautiful Testing Is Efficient Testing
  1. SLIME
  2. Scripting
  3. Discovering Developer Notes
  4. Oracles and Test Data Generation
  5. Mindmaps
  6. Efficiency Achieved
Beautiful Tools
1. Chapter 18 Seeding Bugs to Find Bugs: Beautiful Mutation Testing
  1. Assessing Test Suite Quality
  2. Watching the Watchmen
  3. An AspectJ Example
  4. Equivalent Mutants
  5. Focusing on Impact
  6. The Javalanche Framework
  7. Odds and Ends
  8. Acknowledgments
  9. References
2. Chapter 19 Reference Testing As Beautiful Testing
  1. Reference Test Structure
  2. Reference Test Extensibility
  3. Building Community
3. Chapter 20 Clam Anti-Virus: Testing Open Source with Open Tools
  1. The Clam Anti-Virus Project
  2. Testing Methods
  3. Summary
  4. Credits
4. Chapter 21 Web Application Testing with Windmill
  1. Introduction
  2. Overview
  3. Writing Tests
  4. The Project
  5. Comparison
  6. Conclusions
  7. References
5. Chapter 22 Testing One Million Web Pages
  1. In the Beginning…
  2. The Tools Merge and Evolve
  3. The Nitty-Gritty
  4. Summary
  5. Acknowledgments
6. Chapter 23 Testing Network Services in Multimachine Scenarios
  1. The Need for an Advanced Testing Tool in eBox
  2. Development of ANSTE to Improve the eBox QA Process
  3. How eBox Uses ANSTE
  4. How Other Projects Can Benefit from ANSTE

Appendix Contributors
Colophon

Title:

Beautiful Testing

Edited By:

Tim Riley, Adam Goucher

Publisher:

O'Reilly Media

Formats:

Print
Ebook
Safari Books Online

Print:

October 2009

Ebook:

October 2009

Pages:

352

Print ISBN:

978-0-596-15981-8

| ISBN 10:

0-596-15981-1

Ebook ISBN:

978-0-596-80810-5

| ISBN 10:

0-596-80810-0

Description

Table of Contents

Product Details

Colophon

Recommended for You

Recently Viewed

Hello, Android, 2nd Edition

Introducing Google's Mobile Development Platform

By Ed Burnette

November 2009

Managing Projects with make, 2nd Edition

The Power of GNU make for Building Anything

By Andy Oram, Steve Talbott

October 1991

.NET Compact Framework Pocket Guide

By Wei-Meng Lee

May 2004

Ebook: $7.99

Print & Ebook: $10.95

Print: $9.95

Customer Reviews

REVIEW SNAPSHOT®

by PowerReviews

O'Reilly Media Beautiful Testing

4.3

(based on 4 reviews)

100%

of respondents would recommend this to a friend.

Pros

Well-written (3)

Cons

Best Uses

Intermediate (4)
Novice (3)

- Reviewer Profile:
- Developer (4)

REVIEWS

Reviewed by 4 customers

Sort by

Displaying reviews 1-4

12/27/2011

4.0

My First 'Beautiful' Compilation Book

By Jon

from Las Vegas, NV

About Me Developer

Pros

Well-written

Cons

Best Uses

Intermediate
Novice

Comments about O'Reilly Media Beautiful Testing:

I finally sat down to read my first essay compilation book from O'Reilly that deals with Software Development, "Beautiful Testing." The essays are grouped into one of three categories, "Testers", "Process", or "Tools. I found the group of essays around Testers to be the most interesting as the essays focused more on the testing and Software Quality Assurance community as a whole rather than actually the implementation of testing processes and tools.

The first essay, "Was it good for you?" by Linda Wilkinson focuses on the need for a quality assurance program that balances the needs of software testing but not at the expense of the overall project. The author warns against making the quality assurance engineer as the gatekeeper of the project as in many cases it is not cost effective to fix every bug identified within an application. I have personally seen the use of testers as gatekeepers in many organizations. Often the development teams find routes that go around a difficult process to avoid testing when the process itself is too structured and rigorous.

Overall, I would recommend reading "Beautiful Testing" to any Project Manager, Software Development Manager, or QA Engineer. Also anyone that is looking to get their feet wet with XMPP, there is a nice short essay introducing the protocols. I found this to be helpful since I had not really read anything about the protocol until picking up this book.

I also plan to read more of the essay compilation series books like Beautiful Code and Data in the near future.

7/27/2010

(1 of 1 customers found this review helpful)

5.0

Beautiful case studies and essays

By N. Martin

from Columbus, OH

About Me Developer

Pros

Easy to understand
Something for everyone
Well-written

Cons

Sometimes too focused

Best Uses

Any software professional
Expert
Intermediate
Novice
Student

Comments about O'Reilly Media Beautiful Testing:

As a software developer, I see the importance of good testing on a day to day basis. It is the essential piece of the puzzle that ensures that your efforts bare fruit in terms of business value. However, I also see it as the most neglected and poorly implemented aspect of producing software - at least with some of the projects and environments I worked on (and, sadly, I certainly haven't always been an exception).

Having read this book my drive to improve the testing practices, procedures, and tools has been dramatically invigorated. Of particular interest to me, because they tie more directly with the work I do on a day to day basis, were the chapters on agile development, continuous integration, automated testing, and the associated tools. Chapter 14, "Test-Driven Development: Driving New Standards of Beauty" and "Beautiful Testing as the Cornerstone of Business Success" were truly awe-inspiring. They drew a picture in my mind of the perfect development environment, and even give somewhat of a road-map to get there.

Some of the chapters were excellent for their high level breadth. Chapter 2, "Beautiful Testing Satisfies Stakeholders" and chapter 6, "Bug Management and Test Case Effectiveness" gives you a good sense of the value proposition of a good testing foundation.

Others were a real joy to read (well...I guess if you're a geek). Chapter 1 was an entertaining, yet insightful, view into the mind of a tester. Chapter 12 was dramatic and really made you feel the weight of the pressure to make sure the testing was impeccable. One of my favorite chapters was "Software Development is a Creative Process", chapter 13. This chapter distilled how I feel about development.

There were a few chapter that didn't give me much value. Many of the chapters were very narrowly focused on a specific technology or project, and didn't do enough, in my opinion, to relate the chapter to the larger scope. Chapter 21, "Web Application Testing With Windmill" read almost like marketing material for the product. I also didn't get much out of "Beautiful XMPP Testing" (chapter 6) and "Testing a Random Number Generator" (chapter 10). These chapter, however, really didn't detract from the overall superb quality of the writing or high value of the information contained in this book - not to mention that it is probably a matter of perspective.

If you are involved in any aspect of software development, you should consider this required reading. The knowledge and insight taken out of this book will certainly make me more effective professionally.

2/16/2010

(2 of 2 customers found this review helpful)

3.0

An extensive overview

By Java Artisan

from Hasselt, Belgium

About Me Designer, Developer, Educator

Pros

Concise
Helpful examples

Cons

Sometimes too heavy
Too basic

Best Uses

Intermediate

Comments about O'Reilly Media Beautiful Testing:

I would recommend this book to experienced people involved in software development; willing to take the leap towards developing quality software. Whether they're managers, developers, analysts, etc... It's the right book to read when you're looking for a one-volume overview of the various aspects and techniques available today. I wouldn't use this book as a reference however. It's too high-level for that.

Each of the 23 sections is a story about the implementation of software testing in a particular project; in a way that is "beautiful" to the author(s) of the section. The level of the sections varies from academic (ie. testing pseudo-random number generation) to specific (ie. QA of the Python language) and from abstract (ie quality statistical metrics) to practical (ie implementing TDD). Hence, depending on the reader's level, some sections are easy- and others will be difficult to read.

Testing is not only about tools and methods. It's also about people and organisations implementing it. Also about the fact the will to invest in quality and perseverance is required to achieve quality.

The book does a really good job in showing the reader the many aspects of software testing. Personally I didn't suspect testing is that extensive. However, practical sections of the book should have been longer to allow the author(s) to elaborate. Ten pages on average may be enough for abstract subjects but they're not for technical matters.

What I learned from this book:
* Testing is an extensive domain.
* Implementing quality will get you nowhere unless it is backed by the management and part of your state of mind.
* People have to be trained to do it.
* There are references I'll be investigating.

11/9/2009

(2 of 2 customers found this review helpful)

5.0

Great book on testing

By Frank Stepanski

from Philadelphia, PA

About Me Designer, Developer, Educator

Pros

Accurate
Concise
Easy to understand
Well-written

Cons

Best Uses

Intermediate
Novice

Comments about O'Reilly Media Beautiful Testing:

Beautiful TEsting is written by a number of talented testers, developers and engineers. It is unique book since it does not focus one area of testing but gives the readers lot of different points of view with case studies, experience reports, and war stories that give valuable insight a normal book could not have given.

Personally, I have always thought as testng as something you have to do when you finish a software build but its just a necessary evil and does not really contribute to the real work which is building software for our client(s). But the theme of many of these chapters in this book is how you can make testing a part of your development life cycle and really incorporate the data that has been gathered in testing to you next release.

There is so much information in this book, it is hard to grasp it all so I recommend focusing on a few sections where it relates to your job position at work. Some essays are general that be applied to almost any developer or tester, then other chpaters are more specific such as the chapter on large scale automation, or XMPP, etc. But even so, the book reads so well rom chapter to chapter you will find yourself continuing to read as it leads itself to learning more and more about the different aspects of testing like an unfolding story.

I found that after reading a good part of the book, I really started to value testing a lot more and understood how valuable it really can be. The book is not written like a pure technical book, but the authors write in clear-easy to understand way that non-techinical people like testers (and not-software developers) can understand easily. I found myself stopping and starting as I was reading writing down lots of links to software and tools that I want to check out for my own testing needs. So I never thought this book was a hard-read, more like an interesting story.

This type of writing style by multiple authors I wish was done more with other books as most read like a boring college textbookor reference book that puts you to sleep after 30 minutes. I never thought I'd enjoy reading a book on testing, but I did, and I highly recommend it for anybody who does testing now or as part of their job (software or web developer, engineer, etc.)

Another extra bonus of buying this book is that all author royalties will be donated to the Nothing But Nets campaign to save lives by preventing malaria, a disease that kills millions of children in Africa each year. So its a win-win for everybody...

Get it!

Chapter break down:

Chapter 1 : Was It Good for You?
Chapter 2 : Beautiful Testing Satisfies Stakeholders
Chapter 3 : Building Open Source QA Communities
Chapter 4 : Collaboration Is the Cornerstone of Beautiful Performance Testing
Chapter 5 : Just Peachy: Making Office Software More Reliable with Fuzz Testing
Chapter 6 : Bug Management and Test Case Effectiveness
Chapter 7 : Beautiful XMPP Testing
Chapter 8 : Beautiful Large-Scale Test Automation
Chapter 9 : Beautiful Is Better Than Ugly
Chapter 10 : Testing a Random Number Generator
Chapter 11 : Change-Centric Testing
Chapter 12 : Software in Use
Chapter 13 : Software Development Is a Creative Process
Chapter 14 : Test-Driven Development: Driving New Standards of Beauty
Chapter 15 : Beautiful Testing As the Cornerstone of Business Success
Chapter 16 : Peeling the Glass Onion at Socialtext
Chapter 17 : Beautiful Testing Is Efficient Testing
Chapter 18 : Seeding Bugs to Find Bugs: Beautiful Mutation Testing
Chapter 19 : Reference Testing As Beautiful Testing
Chapter 20 : Clam Anti-Virus: Testing Open Source with Open Tools
Chapter 21 : Web Application Testing with Windmill
Chapter 22 : Testing One Million Web Pages
Chapter 23 : Testing Network Services in Multimachine Scenarios

Displaying reviews 1-4

Save a Tree - Go Digital what is this?

Ebook: $39.99

Formats: DAISY, ePub, Mobi, PDF

Print & Ebook: $54.99

Print: $49.99

Safari Books Online - Read now >

Beautiful Testers

Chapter 1 Was It Good for You?

Chapter 2 Beautiful Testing Satisfies Stakeholders

Chapter 3 Building Open Source QA Communities

Chapter 4 Collaboration Is the Cornerstone of Beautiful Performance Testing

Beautiful Process

Chapter 5 Just Peachy: Making Office Software More Reliable with Fuzz Testing

Chapter 6 Bug Management and Test Case Effectiveness

Chapter 7 Beautiful XMPP Testing

Chapter 8 Beautiful Large-Scale Test Automation

Chapter 9 Beautiful Is Better Than Ugly

Chapter 10 Testing a Random Number Generator

Chapter 11 Change-Centric Testing

Chapter 12 Software in Use

Chapter 13 Software Development Is a Creative Process

Chapter 14 Test-Driven Development: Driving New Standards of Beauty

Chapter 15 Beautiful Testing As the Cornerstone of Business Success

Chapter 16 Peeling the Glass Onion at Socialtext

Chapter 17 Beautiful Testing Is Efficient Testing

Beautiful Tools

Chapter 18 Seeding Bugs to Find Bugs: Beautiful Mutation Testing

Chapter 19 Reference Testing As Beautiful Testing

Chapter 20 Clam Anti-Virus: Testing Open Source with Open Tools

Chapter 21 Web Application Testing with Windmill

Chapter 22 Testing One Million Web Pages

Chapter 23 Testing Network Services in Multimachine Scenarios

Appendix Contributors

Colophon

About O'Reilly

Community

More O'Reilly Sites

Partner Sites

Shop O'Reilly