Python Cookbook

Larger Cover

Python Cookbook

By Alex Martelli, David Ascher

Publisher: O'Reilly Media

Released: July 2002

Pages: 608

The Python Cookbook is a collection of problems, solutions, and practical examples for Python programmers, written by Python programmers. Over the past year, members of the Python community have contributed material to an online repository of Python recipes hosted by ActiveState. This book contains the best of those recipes, accompanied by overviews and background material by key Python figures.

The recipes in the Python Cookbook range from simple tasks, such as working with dictionaries and list comprehensions, to entire modules that demonstrate templating systems and network monitoring. This book contains over 200 recipes on the following topics:

Searching and sorting
Manipulating text
Working with files and the filesystem
Object-oriented programming
Dealing with threads and processes
System administration
Interacting with databases
Creating user interfaces
Network and web programming
Processing XML
Distributed programming
Debugging and testing
Extending Python

This book is a treasure trove of useful code for all Python programmers, from novices to advanced practitioners, with contributions from such Python luminaries as Guido Van Rossum, David Ascher, Tim Peters, Paul Prescod, Mark Hammond, and Alex Martelli, as well as over 100 other Python programmers. The recipes highlight Python best practices and can be used directly in day-to-day programming tasks, as a source of ideas, or as a way to learn more about Python.

The recipes in the Python Cookbook were edited by David Ascher, who is on the board of the Python Software Foundation and is the co-author of Learning Python, and Alex Martelli, who is known for his numerous and exhaustive postings on the Python mailing list. The book contains a foreword by Guido van Rossum, the creator of Python.

Chapter 1 Python Shortcuts
1. Introduction
2. Swapping Values WithoutUsing a Temporary Variable
3. Constructing a Dictionary Without Excessive Quoting
4. Getting a Value from a Dictionary
5. Adding an Entry to a Dictionary
6. Associating Multiple Values with Each Key in a Dictionary
7. Dispatching Using a Dictionary
8. Collecting a Bunch of Named Items
9. Finding the Intersection of Two Dictionaries
10. Assigning and Testing with One Statement
11. Using List Comprehensions Instead of map and filter
12. Unzipping Simple List-Like Objects
13. Flattening a Nested Sequence
14. Looping in Parallel over Index and Sequence Items
15. Looping Through Multiple Lists
16. Spanning a Range Defined by Floats
17. Transposing Two-Dimensional Arrays
18. Creating Lists of Lists Without Sharing References
Chapter 2 Searching and Sorting
1. Introduction
2. Sorting a Dictionary
3. Processing Selected Pairs of Structured Data Efficiently
4. Sorting While Guaranteeing Sort Stability
5. Sorting by One Field, Then by Another
6. Looking for Items in a Sorted Sequence Using Binary Search
7. Sorting a List of Objects by an Attribute of the Objects
8. Sorting by Item or by Attribute
9. Selecting Random Elements from a List Without Repetition
10. Performing Frequent Membership Tests on a Sequence
11. Finding the Deep Index of an Item in an Embedded Sequence
12. Showing Off Quicksort in Three Lines
13. Sorting Objects Using SQL’s ORDER BY Syntax
Chapter 3 Text
1. Introduction
2. Processing a String One Character at a Time
3. Testing if an Object Is String-Like
4. Aligning Strings
5. Trimming Space from the Ends of a String
6. Combining Strings
7. Checking Whether a String Contains a Set of Characters
8. Filtering a String for a Set of Characters
9. Controlling Case
10. Reversing a String by Words or Characters
11. Accessing Substrings
12. Changing the Indentation of a Multiline String
13. Testing Whether a String Represents an Integer
14. Expanding and Compressing Tabs
15. Replacing Multiple Patterns in a Single Pass
16. Converting Between Different Naming Conventions
17. Converting Between Characters and Values
18. Converting Between Unicode and Plain Strings
19. Printing Unicode Characters to Standard Output
20. Dispatching Based on Pattern Matches
21. Evaluating Code Inside Strings
22. Replacing Python Code with the Results of Executing That Code
23. Module: Yet Another Python Templating Utility (YAPTU)
24. Module: Roman Numerals
Chapter 4 Files
1. Introduction
2. Reading from a File
3. Writing to a File
4. Searching and Replacing Text in a File
5. Reading a Particular Line from a File
6. Retrieving a Line at Random from a File of Unknown Size
7. Counting Lines in a File
8. Processing Every Word in a File
9. Reading a Text File by Paragraphs
10. Reading Lines with Continuation Characters
11. Reading Data from ZIP Files
12. Reading INI Configuration Files
13. Sending Binary Data to Standard Output Under Windows
14. Using Random-Access Input/Output
15. Updating a Random-Access File
16. Splitting a Path into All of Its Parts
17. Treating Pathnames as Objects
18. Creating Directories Including Necessary Parent Directories
19. Walking Directory Trees
20. Swapping One File Extension for Another Throughout a Directory Tree
21. Finding a File Given an Arbitrary Search Path
22. Finding a File on the Python Search Path
23. Dynamically Changing the Python Search Path
24. Computing Directory Sizes in a Cross-Platform Way
25. File Locking Using a Cross-Platform API
26. Versioning Filenames
27. Module: Versioned Backups
Chapter 5 Object-Oriented Programming
1. Introduction
2. Overriding a Built-In Method
3. Getting All Members of a Class Hierarchy
4. Calling a Superclass _ _init_ _ Method if It Exists
5. Calling a Superclass Implementation of a Method
6. Implementing Properties
7. Implementing Static Methods
8. Implementing Class Methods
9. Delegating Automatically as an Alternative to Inheritance
10. Decorating an Object with Print-Like Methods
11. Checking if an Object Has Necessary Attributes
12. Making a Fast Copy of an Object
13. Adding Methods to a Class at Runtime
14. Modifying the Class Hierarchy of an Instance
15. Keeping References to Bound Methods Without Inhibiting Garbage Collection
16. Defining Constants
17. Managing Options
18. Implementing a Set Class
19. Implementing a Ring Buffer
20. Implementing a Collection
21. Delegating Messages to Multiple Objects
22. Implementing the Singleton Design Pattern
23. Avoiding the Singleton Design Pattern with the Borg Idiom
24. Implementing the Null Object Design Pattern
Chapter 6 Threads, Processes, and Synchronization
1. Introduction
2. Storing Per-Thread Information
3. Terminating a Thread
4. Allowing Multithreaded Read Access While Maintaining a Write Lock
5. Running Functions in the Future
6. Synchronizing All Methods in an Object
7. Capturing the Output and Error Streams from a Unix Shell Command
8. Forking a Daemon Process on Unix
9. Determining if Another Instance of a Script Is Already Running in Windows
10. Processing Windows Messages Using MsgWaitForMultipleObjects
Chapter 7 System Administration
1. Introduction
2. Running a Command Repeatedly
3. Generating Random Passwords
4. Generating Non-Totally Random Passwords
5. Checking the Status of a Unix Network Interface
6. Calculating Apache Hits per IP Address
7. Calculating the Rate of Client Cache Hits on Apache
8. Manipulating the Environment on Windows NT/2000/XP
9. Checking and Modifying the Set of Tasks Windows Automatically Runs at Logon
10. Examining the Microsoft Windows Registry for a List of Name Server Addresses
11. Getting Information About the Current User on Windows NT/2000
12. Getting the Windows Service Name from Its Long Name
13. Manipulating Windows Services
14. Impersonating Principals on Windows
15. Changing a Windows NT Password Using ADSI
16. Working with Windows Scripting Host (WSH) from Python
17. Displaying Decoded Hotkeys for Shortcuts in Windows
Chapter 8 Databases and Persistence
1. Introduction
2. Serializing Data Using the marshal Module
3. Serializing Data Using the pickle and cPickle Modules
4. Using the cPickle Module on Classes and Instances
5. Mutating Objects with shelve
6. Accessing a MySQL Database
7. Storing a BLOB in a MySQL Database
8. Storing a BLOB in a PostgreSQL Database
9. Generating a Dictionary Mapping from Field Names to Column Numbers
10. Using dtuple for Flexible Access to Query Results
11. Pretty-Printing the Contents of Database Cursors
12. Establishing Database Connections Lazily
13. Accessing a JDBC Database from a Jython Servlet
14. Module: jet2sql—Creating a SQL DDL from an Access Database
Chapter 9 User Interfaces
1. Introduction
2. Avoiding lambda in Writing Callback Functions
3. Creating Menus with Tkinter
4. Creating Dialog Boxes with Tkinter
5. Supporting Multiple Values per Row in a Tkinter Listbox
6. Embedding Inline GIFs Using Tkinter
7. Combining Tkinter and Asynchronous I/O with Threads
8. Using a wxPython Notebook with Panels
9. Giving the User Unobtrusive Feedback During Data Entry with Qt
10. Building GUI Solutions Independent of the Specific GUI Toolkit
11. Creating Color Scales
12. Using Publish/Subscribe Broadcasting to Loosen the Coupling Between GUI and Business Logic Systems
13. Module: Building GTK GUIs Interactively
Chapter 10 Network Programming
1. Introduction
2. Writing a TCP Client
3. Writing a TCP Server
4. Passing Messages with Socket Datagrams
5. Finding Your Own Name and Address
6. Converting IP Addresses
7. Grabbing a Document from the Web
8. Being an FTP Client
9. Sending HTML Mail
10. Sending Multipart MIME Email
11. Bundling Files in a MIME Message
12. Unpacking a Multipart MIME Message
13. Module: PyHeartBeat—Detecting Inactive Computers
14. Module: Interactive POP3 Mailbox Inspector
15. Module: Watching for New IMAP Mail Using a GUI
Chapter 11 Web Programming
1. Introduction
2. Testing Whether CGI Is Working
3. Writing a CGI Script
4. Using a Simple Dictionary for CGI Parameters
5. Handling URLs Within a CGI Script
6. Resuming the HTTP Download of a File
7. Stripping Dangerous Tags and Javascript from HTML
8. Running a Servlet with Jython
9. Accessing Netscape Cookie Information
10. Finding an Internet Explorer Cookie
11. Module: Fetching Latitude/Longitude Data from the Web
Chapter 12 Processing XML
1. Introduction
2. Checking XML Well-Formedness
3. Counting Tags in a Document
4. Extracting Text from an XML Document
5. Transforming an XML Document Using XSLT
6. Transforming an XML Document Using Python
7. Parsing an XML File with xml.parsers.expat
8. Converting Ad-Hoc Text into XML Markup
9. Normalizing an XML Document
10. Controlling XSLT Stylesheet Loading
11. Autodetecting XML Encoding
12. Module: XML Lexing (Shallow Parsing)
13. Module: Converting a List of Equal-Length Lists into XML
Chapter 13 Distributed Programming
1. Introduction
2. Making an XML-RPC Method Call
3. Serving XML-RPC Requests
4. Using XML-RPC with Medusa
5. Writing a Web Service That Supports Both XML-RPC and SOAP
6. Implementing a CORBA Client and Server
7. Performing Remote Logins Using telnetlib
8. Using Publish/Subscribe in a Distributed Middleware Architecture
9. Using Request/Reply in a Distributed Middleware Architecture
Chapter 14 Debugging and Testing
1. Introduction
2. Reloading All Loaded Modules
3. Tracing Expressions and Comments in Debug Mode
4. Wrapping Tracebacks in HTML
5. Getting More Information from Tracebacks
6. Starting the Debugger Automatically After an Uncaught Exception
7. Logging and Tracing Across Platforms
8. Determining the Name of the Current Function
9. Introspecting the Call Stack with Older Versions of Python
10. Debugging the Garbage-Collection Process
11. Tracking Instances of Particular Classes
Chapter 15 Programs About Programs
1. Introduction
2. Colorizing Python Source Using the Built-in Tokenizer
3. Importing a Dynamically Generated Module
4. Importing from a Module Whose Name Is Determined at Runtime
5. Importing Modules with Automatic End-of-Line Conversions
6. Simulating Enumerations in Python
7. Modifying Methods in Place
8. Associating Parameters with a Function (Currying)
9. Composing Functions
10. Adding Functionality to a Class
11. Adding a Method to a Class Instance at Runtime
12. Defining a Custom Metaclass to Control Class Behavior
13. Module: Allowing the Python Profiler to Profile C Modules
Chapter 16 Extending and Embedding
1. Introduction
2. Implementing a Simple Extension Type
3. Translating a Python Sequence into a C Array with the PySequence_Fast Protocol
4. Accessing a Python Sequence Item-by-Item with the Iterator Protocol
5. Returning None from a Python-Callable C Function
6. Coding the Methods of a Python Class in C
7. Implementing C Function Callbacks to a Python Function
8. Debugging Dynamically Loaded C Extensions with gdb
9. Debugging Memory Problems
10. Using SWIG-Generated Modules in a Multithreaded Environment
Chapter 17 Algorithms
1. Introduction
2. Testing if a Variable Is Defined
3. Evaluating Predicate Tests Across Sequences
4. Removing Duplicates from a Sequence
5. Removing Duplicates from a Sequence While Maintaining Sequence Order
6. Simulating the Ternary Operator in Python
7. Counting Items and Sorting by Incidence (Histograms)
8. Memoizing (Caching) the Return Values of Functions
9. Looking Up Words by Sound Similarity
10. Computing Factorials with lambda
11. Generating the Fibonacci Sequence
12. Wrapping an Unbounded Iterator to Restrict Its Output
13. Operating on Iterators
14. Rolling Dice
15. Implementing a First-In First-Out Container
16. Modeling a Priority Queue
17. Converting Numbers to Rationals via Farey Fractions
18. Evaluating a Polynomial
19. Module: Finding the Convex Hull of a Set of 2D Points
20. Module: Parsing a String into a Date/Time Object Portably
Chapter 18 List of Contributors
1. A
2. B
3. C
4. D
5. F
6. G
7. H
8. J
9. K
10. L
11. M
12. N
13. P
14. Q
15. R
16. S
17. T
18. U
19. V
20. W
21. Y
22. Z

Colophon

Title:

Python Cookbook

By:

Alex Martelli, David Ascher

Publisher:

O'Reilly Media

Formats:

Print
Safari Books Online

Print:

July 2002

Pages:

608

Print ISBN:

978-0-596-00167-4

| ISBN 10:

0-596-00167-3

Alex Martelli

Alex Martelli spent 8 years with IBM Research, winning three Outstanding Technical Achievement Awards. He then spent 13 as a Senior Software Consultant at think3 inc, developing libraries, network protocols, GUI engines, event frameworks, and web access frontends. He has also taught programming languages, development methods, and numerical computing at Ferrara University and other venues. He's a C++ MVP for Brainbench, and a member of the Python Software Foundation. He currently works for AB Strakt, a Python-centered software house in Gteborg, Sweden, mostly by telecommuting from his home in Bologna, Italy. Alex's proudest achievement is the articles that appeared in Bridge World (January/February 2000), which were hailed as giant steps towards solving issues that had haunted contract bridge theoreticians for decades.

View Alex Martelli's full profile page.
David Ascher

David Ascher is the lead for Python projects at ActiveState, including Komodo, ActiveState's integrated development environment written mostly in Python. David has taught courses about Python to corporations, in universities, and at conferences. He also organized the Python track at the 1999 and 2000 O'Reilly Open Source Conventions, and was the program chair for the 10th International Python Conference. In addition, he co-wrote Learning Python (both editions) and serves as a director of the Python Software Foundation. David holds a B.S. in physics and a Ph.D. in cognitive science, both from Brown University.

View David Ascher's full profile page.

Colophon

Our look is the result of reader comments, our own experimentation, and feedback from distribution channels. Distinctive covers complement our distinctive approach to technical topics, breathing personality and life into potentially dry subjects. The animal on the cover of Python Cookbook is a springhaas (Pedetes capensis), also known as a spring hare. Springhaas are not hares at all, but rather the only member of the family Pedetidae in the order Rodentia. They are not marsupials, but they are vaguely kangaroo-like, with small front legs, powerful hind legs designed for hopping, jumping, and leaping, and long, strong, bushy (but not prehensile) tails they use for balance and as a brace when sitting. They grow to be about 14-18 inches long, with tails as long as their bodies, and can weigh approximately 8 pounds. Springhaas have rich, glossy, tawny or golden-reddish coats with long, soft fur and white underbellies. Their heads are disproportionately large, and they have long ears (with a flap of skin at the base they can close to prevent sand from getting inside while they are digging) and large, dark brown eyes.

Springhaas mate throughout the year and have a gestation period of about 78-82 days. Females generally give birth to only one young (which stays with its mother until it is approximately seven weeks old) per litter but have three or four litters each year. Babies are born with teeth and fully furred, with their eyes closed and ears open.

Springhaas are terrestrial and well-adapted for digging, and they tend to spend their days in the small networks of their burrows and tunnels. They are nocturnal and primarily herbivorous, feeding on bulbs, roots, grains, and occasionally insects. While they are foraging, they move about on all fours, but they are able to move 10-25 feet in a single horizontal leap and are capable of quick getaways when frightened. Although they are often seen foraging in groups in the wild, they do not form an organized social unit and usually nest alone or in breeding pairs. Springhaas can live up to 15 years in captivity. They are found in Zaire, Kenya, and South Africa, in dry, desert, or semiarid areas, and they are a favorite and important food source in South Africa. Matt Hutchinson was the production editor for Python Cookbook. Matt Hutchinson and Rachel Wheeler copyedited the book. Colleen Gorman and Mary Anne Weeks Mayo provided quality control. John Bickelhaupt wrote the index.

Pam Spremulli designed the cover of this book, based on a series design by Edie Freedman. The cover image is from Animal Creation: Mammalia. Emma Colby produced the cover layout with QuarkXPress 4.1 using Adobe's ITC Garamond font.

David Futato designed the interior layout. This book was converted to FrameMaker 5.5.6 with a format conversion tool created by Erik Ray, Jason McIntosh, Neil Walls, and Mike Sierra that uses Perl and XML technologies. The text font is Linotype Birka; the heading font is Adobe Myriad Condensed; and the code font is Lucas-Font's TheSans Mono Condensed. This colophon was written by Rachel Wheeler.

Description

Table of Contents

Product Details

About the Author

Colophon

Recommended for You

Chapter 1 Python Shortcuts

Introduction

Swapping Values WithoutUsing a Temporary Variable

Constructing a Dictionary Without Excessive Quoting

Getting a Value from a Dictionary

Adding an Entry to a Dictionary

Associating Multiple Values with Each Key in a Dictionary

Dispatching Using a Dictionary

Collecting a Bunch of Named Items

Finding the Intersection of Two Dictionaries

Assigning and Testing with One Statement

Using List Comprehensions Instead of map and filter

Unzipping Simple List-Like Objects

Flattening a Nested Sequence

Looping in Parallel over Index and Sequence Items

Looping Through Multiple Lists

Spanning a Range Defined by Floats

Transposing Two-Dimensional Arrays

Creating Lists of Lists Without Sharing References

Chapter 2 Searching and Sorting

Introduction

Sorting a Dictionary

Processing Selected Pairs of Structured Data Efficiently

Sorting While Guaranteeing Sort Stability

Sorting by One Field, Then by Another

Looking for Items in a Sorted Sequence Using Binary Search

Sorting a List of Objects by an Attribute of the Objects

Sorting by Item or by Attribute

Selecting Random Elements from a List Without Repetition

Performing Frequent Membership Tests on a Sequence

Finding the Deep Index of an Item in an Embedded Sequence

Showing Off Quicksort in Three Lines

Sorting Objects Using SQL’s ORDER BY Syntax

Chapter 3 Text

Introduction

Processing a String One Character at a Time

Testing if an Object Is String-Like

Aligning Strings

Trimming Space from the Ends of a String

Combining Strings

Checking Whether a String Contains a Set of Characters

Filtering a String for a Set of Characters

Controlling Case

Reversing a String by Words or Characters

Accessing Substrings

Changing the Indentation of a Multiline String

Testing Whether a String Represents an Integer

Expanding and Compressing Tabs

Replacing Multiple Patterns in a Single Pass

Converting Between Different Naming Conventions

Converting Between Characters and Values

Converting Between Unicode and Plain Strings

Printing Unicode Characters to Standard Output

Dispatching Based on Pattern Matches

Evaluating Code Inside Strings

Replacing Python Code with the Results of Executing That Code

Module: Yet Another Python Templating Utility (YAPTU)

Module: Roman Numerals

Chapter 4 Files

Introduction

Reading from a File

Writing to a File

Searching and Replacing Text in a File

Reading a Particular Line from a File

Retrieving a Line at Random from a File of Unknown Size

Counting Lines in a File

Processing Every Word in a File

Reading a Text File by Paragraphs

Reading Lines with Continuation Characters

Reading Data from ZIP Files

Reading INI Configuration Files

Sending Binary Data to Standard Output Under Windows

Using Random-Access Input/Output

Updating a Random-Access File

Splitting a Path into All of Its Parts

Treating Pathnames as Objects

Creating Directories Including Necessary Parent Directories

Walking Directory Trees

Swapping One File Extension for Another Throughout a Directory Tree

Finding a File Given an Arbitrary Search Path