Many big data-driven companies today are moving to protect certain types of data against intrusion, leaks, or unauthorized eyes. But how do you lock down data while granting access to people who need to see it? In this practical book, authors Ted Dunning and Ellen Friedman offer two novel and practical solutions that you can implement right away.
Ideal for both technical and non-technical decision makers, group leaders, developers, and data scientists, this book shows you how to:
Share original data in a controlled way so that different groups within your organization only see part of the whole. You’ll learn how to do this with the new open source SQL query engine Apache Drill.
Provide synthetic data that emulates the behavior of sensitive data. This approach enables external advisors to work with you on projects involving data that you can't show them.
If you’re intrigued by the synthetic data solution, explore the log-synth program that Ted Dunning developed as open source code (available on GitHub), along with how-to instructions and tips for best practice. You’ll also get a collection of use cases.
Providing lock-down security while safely sharing data is a significant challenge for a growing number of organizations. With this book, you’ll discover new options to share data safely without sacrificing security.
Chapter 1So Secure It’s Lost
Safe Access in Secure Big Data Systems
Chapter 2The Challenge: Sharing Data Safely
Surprising Outcomes with Anonymity
The Netflix Prize
Unexpected Results from the Netflix Contest
Implications of Breaking Anonymity
Be Alert to the Possibility of Cross-Reference Datasets
New York Taxicabs: Threats to Privacy
Sharing Data Safely
Chapter 3Data on a Need-to-Know Basis
Views: A Secure Way to Limit What Is Seen
Why Limit Access?
Apache Drill Views for Granular Security
How Views Work
Summary of Need-to-Know Methods
Chapter 4Fake Data Gives Real Answers
The Surprising Thing About Fake Data
Keep It Simple: log-synth
Log-synth Use Case 1: Broken Large-Scale Hive Query
Log-synth Use Case 2: Fraud Detection Model for Common Point of Compromise
Summary: Fake Data and log-synth to Safely Work with Secure Data
Chapter 5Fixing a Broken Large-Scale Query
A Description of the Problem
Determining What the Synthetic Data Needed to Be
Schema for the Synthetic Data
Generating the Synthetic Data
Tips and Caveats
What to Do from Here?
Chapter 6Fraud Detection
What Is Really Important?
The User Model
Sampler for the Common Point of Compromise
How the Breach Model Works
Results of the Entire System Together
Chapter 7A Detailed Look at log-synth
Maintaining Simplicity: The Role of JSON in log-synth
Ted Dunning is Chief Applications Architect at MapR Technologies and active in the open source community.
He currently serves as VP for Incubator at the Apache Foundation, as a champion and mentor for a large number of projects, and as committer and PMC member of the Apache ZooKeeper and Drill projects. He developed the t-digest algorithm used to estimate extreme quantiles. T-digest has been adopted by several open source projects. He also developed the open source log-synth project described in this book.
Ted was the chief architect behind the MusicMatch (now Yahoo Music)and Veoh recommendation systems, built fraud-detection systems forID Analytics (LifeLock), and has issued 24 patents to date. Ted has aPhD in computing science from University of Sheffield. When he’s notdoing data science, he plays guitar and mandolin. Ted is on Twitter as@ted_dunning.
Ellen Friedman is a solutions consultant and well-known speaker and author, currently writing mainly about big data topics. She is a committer for the Apache Drill and Apache Mahout projects. With a PhD in Biochemistry, she has years of experience as a research scientist and has written about a variety of technical topics including molecular biology, nontraditional inheritance, and oceanography. Ellen is also co-author of a book of magic-themed cartoons, A Rabbit Under the Hat. Ellen is on Twitter as @Ellen_Friedman.