Books & Videos

Table of Contents

  1. Fundamental Concepts and Techniques

    1. Chapter 1 What’s in It for You?

      1. Uncovering the Internet’s True Potential
      2. What’s in It for Developers?
      3. What’s in It for Business Leaders?
      4. Final Thoughts
    2. Chapter 2 Ideas for Webbot Projects

      1. Inspiration from Browser Limitations
      2. A Few Crazy Ideas to Get You Started
      3. Final Thoughts
    3. Chapter 3 Downloading Web Pages

      1. Think About Files, Not Web Pages
      2. Downloading Files with PHP’s Built-in Functions
      3. Introducing PHP/CURL
      4. Installing PHP/CURL
      5. LIB_http
      6. Final Thoughts
    4. Chapter 4 Basic Parsing Techniques

      1. Content Is Mixed with Markup
      2. Parsing Poorly Written HTML
      3. Standard Parse Routines
      4. Using LIB_parse
      5. Useful PHP Functions
      6. Final Thoughts
    5. Chapter 5 Advanced Parsing with Regular Expressions

      1. Pattern Matching, the Key to Regular Expressions
      2. PHP Regular Expression Types
      3. Learning Patterns Through Examples
      4. Regular Expressions of Particular Interest to Webbot Developers
      5. When Regular Expressions Are (or Aren’t) the Right Parsing Tool
      6. Final Thoughts
    6. Chapter 6 Automating Form Submission

      1. Reverse Engineering Form Interfaces
      2. Form Handlers, Data Fields, Methods, and Event Triggers
      3. Unpredictable Forms
      4. Analyzing a Form
      5. Final Thoughts
    7. Chapter 7 Managing Large Amounts of Data

      1. Organizing Data
      2. Making Data Smaller
      3. Thumbnailing Images
      4. Final Thoughts
  2. Projects

    1. Chapter 8 Price-Monitoring Webbots

      1. The Target
      2. Designing the Parsing Script
      3. Initialization and Downloading the Target
      4. Further Exploration
    2. Chapter 9 Image-Capturing Webbots

      1. Example Image-Capturing Webbot
      2. Creating the Image-Capturing Webbot
      3. Further Exploration
      4. Final Thoughts
    3. Chapter 10 Link-Verification Webbots

      1. Creating the Link-Verification Webbot
      2. Running the Webbot
      3. Further Exploration
    4. Chapter 11 Search-Ranking Webbots

      1. Description of a Search Result Page
      2. What the Search-Ranking Webbot Does
      3. Running the Search-Ranking Webbot
      4. How the Search-Ranking Webbot Works
      5. The Search-Ranking Webbot Script
      6. Final Thoughts
      7. Further Exploration
    5. Chapter 12 Aggregation Webbots

      1. Choosing Data Sources for Webbots
      2. Example Aggregation Webbot
      3. Adding Filtering to Your Aggregation Webbot
      4. Further Exploration
    6. Chapter 13 FTP Webbots

      1. Example FTP Webbot
      2. PHP and FTP
      3. Further Exploration
    7. Chapter 14 Webbots That Read Email

      1. The POP3 Protocol
      2. Executing POP3 Commands with a Webbot
      3. Further Exploration
    8. Chapter 15 Webbots That Send Email

      1. Email, Webbots, and Spam
      2. Sending Mail with SMTP and PHP
      3. Writing a Webbot That Sends Email Notifications
      4. Further Exploration
    9. Chapter 16 Converting a Website into a Function

      1. Writing a Function Interface
      2. Final Thoughts
  3. Advanced Technical Considerations

    1. Chapter 17 Spiders

      1. How Spiders Work
      2. Example Spider
      3. LIB_simple_spider
      4. Experimenting with the Spider
      5. Adding the Payload
      6. Further Exploration
    2. Chapter 18 Procurement Webbots and Snipers

      1. Procurement Webbot Theory
      2. Sniper Theory
      3. Testing Your Own Webbots and Snipers
      4. Further Exploration
      5. Final Thoughts
    3. Chapter 19 Webbots and Cryptography

      1. Designing Webbots That Use Encryption
      2. A Quick Overview of Web Encryption
      3. Final Thoughts
    4. Chapter 20 Authentication

      1. What Is Authentication?
      2. Example Scripts and Practice Pages
      3. Basic Authentication
      4. Session Authentication
      5. Final Thoughts
    5. Chapter 21 Advanced Cookie Management

      1. How Cookies Work
      2. PHP/CURL and Cookies
      3. How Cookies Challenge Webbot Design
      4. Further Exploration
    6. Chapter 22 Scheduling Webbots and Spiders

      1. Preparing Your Webbots to Run as Scheduled Tasks
      2. The Windows XP Task Scheduler
      3. The Windows 7 Task Scheduler
      4. Non-calendar-based Triggers
      5. Final Thoughts
    7. Chapter 23 Scraping Difficult Websites with Browser Macros

      1. Barriers to Effective Web Scraping
      2. Overcoming Webscraping Barriers with Browser Macros
      3. Final Thoughts
    8. Chapter 24 Hacking iMacros

      1. Hacking iMacros for Added Functionality
      2. Further Exploration
    9. Chapter 25 Deployment and Scaling

      1. One-to-Many Environment
      2. One-to-One Environment
      3. Many-to-Many Environment
      4. Many-to-One Environment
      5. Scaling and Denial-of-Service Attacks
      6. Creating Multiple Instances of a Webbot
      7. Managing a Botnet
      8. Further Exploration
  4. Larger Considerations

    1. Chapter 26 Designing Stealthy Webbots and Spiders

      1. Why Design a Stealthy Webbot?
      2. Stealth Means Simulating Human Patterns
      3. Final Thoughts
    2. Chapter 27 Proxies

      1. What Is a Proxy?
      2. Proxies in the Virtual World
      3. Why Webbot Developers Use Proxies
      4. Using a Proxy Server
      5. Types of Proxy Servers
      6. Final Thoughts
    3. Chapter 28 Writing Fault-Tolerant Webbots

      1. Types of Webbot Fault Tolerance
      2. Error Handlers
      3. Further Exploration
    4. Chapter 29 Designing Webbot-Friendly Websites

      1. Optimizing Web Pages for Search Engine Spiders
      2. Web Design Techniques That Hinder Search Engine Spiders
      3. Designing Data-Only Interfaces
      4. Final Thoughts
    5. Chapter 30 Killing Spiders

      1. Asking Nicely
      2. Building Speed Bumps
      3. Setting Traps
      4. Final Thoughts
    6. Chapter 31 Keeping Webbots out of Trouble

      1. It’s All About Respect
      2. Copyright
      3. Trespass to Chattels
      4. Internet Law
      5. Final Thoughts
  1. Appendix PHP/CURL Reference

    1. Creating a Minimal PHP/CURL Session

    2. Initiating PHP/CURL Sessions

    3. Setting PHP/CURL Options

    4. Executing the PHP/CURL Command

    5. Closing PHP/CURL Sessions

  2. Appendix Status Codes

    1. HTTP Codes

    2. NNTP Codes

  3. Appendix SMS Gateways

    1. Sending Text Messages

    2. Reading Text Messages

    3. A Sampling of Text Message Email Addresses