Principles of Data Wrangling

Table of contents

1. Foreword
2. 1. Introduction
   1. Magic Thresholds, PYMK, and User Growth at Facebook
3. 2. A Data Workflow Framework
   1. How Data Flows During and Across Projects
   2. Connecting Analytic Actions to Data Movement: A Holistic Workflow Framework for Data Projects
   3. Raw Data Stage Actions: Ingest Data and Create Metadata
      1. Ingesting Known and Unknown Data
      2. Creating Metadata
   4. Refined Data Stage Actions: Create Canonical Data and Conduct Ad Hoc Analyses
      1. Designing Refined Data
      2. Refined Stage Analytical Actions
   5. Production Data Stage Actions: Create Production Data and Build Automated Systems
      1. Creating Optimized Data
      2. Designing Regular Reports and Automated Products/Services
   6. Data Wrangling within the Workflow Framework
4. 3. The Dynamics of Data Wrangling
   1. Data Wrangling Dynamics
      1. Additional Aspects: Subsetting and Sampling
   2. Core Transformation and Profiling Actions
   3. Data Wrangling in the Workflow Framework
      1. Ingesting Data
      2. Describing Data
      3. Assessing Data Utility
      4. Designing and Building Refined Data
      5. Ad Hoc Reporting
      6. Exploratory Modeling and Forecasting
      7. Building an Optimized Dataset
      8. Regular Reporting and Building Data-Driven Products and Services
5. 4. Profiling
   1. Overview of Profiling
   2. Individual Value Profiling: Syntactic Profiling
   3. Individual Value Profiling: Semantic Profiling
   4. Set-Based Profiling
   5. Profiling Individual Values in the Candidate Master File
      1. Syntactic Profiling in the Candidate Master File
      2. Set-Based Profiling in the Candidate Master File
6. 5. Transformation: Structuring
   1. Overview of Structuring
   2. Intrarecord Structuring: Extracting Values
      1. Positional Extraction
      2. Pattern Extraction
      3. Complex Structure Extraction
   3. Intrarecord Structuring: Combining Multiple Record Fields
   4. Interrecord Structuring: Filtering Records and Fields
   5. Interrecord Structuring: Aggregations and Pivots
      1. Simple Aggregations
      2. Column-to-Row Pivots
      3. Row-to-Column Pivots
7. 6. Transformation: Enriching
   1. Unions
   2. Joins
   3. Inserting Metadata
   4. Derivation of Values
      1. Generic
      2. Proprietary
8. 7. Using Transformation to Clean Data
   1. Addressing Missing/NULL Values
   2. Addressing Invalid Values
9. 8. Roles and Responsibilities
   1. Skills and Responsibilities
      1. Data Engineer
      2. Data Architect
      3. Data Scientist
      4. Analyst
   2. Roles Across the Data Workflow Framework
   3. Organizational Best Practices
10. 9. Data Wrangling Tools
    1. Data Size and Infrastructure
    2. Data Structures
       1. Excel
       2. SQL
       3. Trifacta Wrangler
    3. Transformation Paradigms
       1. Excel
       2. SQL
       3. Trifacta Wrangler
    4. Choosing a Data Wrangling Tool