book

Python Cookbook, 3rd Edition

by David Beazley, Brian K. Jones

May 2013

Intermediate to advanced

706 pages

14h 42m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Who This Book Is ForWho This Book Is Not ForConventions Used in This BookOnline Code ExamplesUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgments
1.1. Unpacking a Sequence into Separate Variables1.2. Unpacking Elements from Iterables of Arbitrary Length1.3. Keeping the Last N Items1.4. Finding the Largest or Smallest N Items1.5. Implementing a Priority Queue1.6. Mapping Keys to Multiple Values in a Dictionary1.7. Keeping Dictionaries in Order1.8. Calculating with Dictionaries1.9. Finding Commonalities in Two Dictionaries1.10. Removing Duplicates from a Sequence while Maintaining Order1.11. Naming a Slice1.12. Determining the Most Frequently Occurring Items in a Sequence1.13. Sorting a List of Dictionaries by a Common Key1.14. Sorting Objects Without Native Comparison Support1.15. Grouping Records Together Based on a Field1.16. Filtering Sequence Elements1.17. Extracting a Subset of a Dictionary1.18. Mapping Names to Sequence Elements1.19. Transforming and Reducing Data at the Same Time1.20. Combining Multiple Mappings into a Single Mapping
2.1. Splitting Strings on Any of Multiple Delimiters2.2. Matching Text at the Start or End of a String2.3. Matching Strings Using Shell Wildcard Patterns2.4. Matching and Searching for Text Patterns2.5. Searching and Replacing Text2.6. Searching and Replacing Case-Insensitive Text2.7. Specifying a Regular Expression for the Shortest Match2.8. Writing a Regular Expression for Multiline Patterns2.9. Normalizing Unicode Text to a Standard Representation2.10. Working with Unicode Characters in Regular Expressions2.11. Stripping Unwanted Characters from Strings2.12. Sanitizing and Cleaning Up Text2.13. Aligning Text Strings2.14. Combining and Concatenating Strings2.15. Interpolating Variables in Strings2.16. Reformatting Text to a Fixed Number of Columns2.17. Handling HTML and XML Entities in Text2.18. Tokenizing Text2.19. Writing a Simple Recursive Descent Parser2.20. Performing Text Operations on Byte Strings
3.1. Rounding Numerical Values3.2. Performing Accurate Decimal Calculations3.3. Formatting Numbers for Output3.4. Working with Binary, Octal, and Hexadecimal Integers3.5. Packing and Unpacking Large Integers from Bytes3.6. Performing Complex-Valued Math3.7. Working with Infinity and NaNs3.8. Calculating with Fractions3.9. Calculating with Large Numerical Arrays3.10. Performing Matrix and Linear Algebra Calculations3.11. Picking Things at Random3.12. Converting Days to Seconds, and Other Basic Time Conversions3.13. Determining Last Friday’s Date3.14. Finding the Date Range for the Current Month3.15. Converting Strings into Datetimes3.16. Manipulating Dates Involving Time Zones
4.1. Manually Consuming an Iterator4.2. Delegating Iteration4.3. Creating New Iteration Patterns with Generators4.4. Implementing the Iterator Protocol4.5. Iterating in Reverse4.6. Defining Generator Functions with Extra State4.7. Taking a Slice of an Iterator4.8. Skipping the First Part of an Iterable4.9. Iterating Over All Possible Combinations or Permutations4.10. Iterating Over the Index-Value Pairs of a Sequence4.11. Iterating Over Multiple Sequences Simultaneously4.12. Iterating on Items in Separate Containers4.13. Creating Data Processing Pipelines4.14. Flattening a Nested Sequence4.15. Iterating in Sorted Order Over Merged Sorted Iterables4.16. Replacing Infinite while Loops with an Iterator
5.1. Reading and Writing Text Data5.2. Printing to a File5.3. Printing with a Different Separator or Line Ending5.4. Reading and Writing Binary Data5.5. Writing to a File That Doesn’t Already Exist5.6. Performing I/O Operations on a String5.7. Reading and Writing Compressed Datafiles5.8. Iterating Over Fixed-Sized Records5.9. Reading Binary Data into a Mutable Buffer5.10. Memory Mapping Binary Files5.11. Manipulating Pathnames5.12. Testing for the Existence of a File5.13. Getting a Directory Listing5.14. Bypassing Filename Encoding5.15. Printing Bad Filenames5.16. Adding or Changing the Encoding of an Already Open File5.17. Writing Bytes to a Text File5.18. Wrapping an Existing File Descriptor As a File Object5.19. Making Temporary Files and Directories5.20. Communicating with Serial Ports5.21. Serializing Python Objects
6.1. Reading and Writing CSV Data6.2. Reading and Writing JSON Data6.3. Parsing Simple XML Data6.4. Parsing Huge XML Files Incrementally6.5. Turning a Dictionary into XML6.6. Parsing, Modifying, and Rewriting XML6.7. Parsing XML Documents with Namespaces6.8. Interacting with a Relational Database6.9. Decoding and Encoding Hexadecimal Digits6.10. Decoding and Encoding Base646.11. Reading and Writing Binary Arrays of Structures6.12. Reading Nested and Variable-Sized Binary Structures6.13. Summarizing Data and Performing Statistics
7.1. Writing Functions That Accept Any Number of Arguments7.2. Writing Functions That Only Accept Keyword Arguments7.3. Attaching Informational Metadata to Function Arguments7.4. Returning Multiple Values from a Function7.5. Defining Functions with Default Arguments7.6. Defining Anonymous or Inline Functions7.7. Capturing Variables in Anonymous Functions7.8. Making an N-Argument Callable Work As a Callable with Fewer Arguments7.9. Replacing Single Method Classes with Functions7.10. Carrying Extra State with Callback Functions7.11. Inlining Callback Functions7.12. Accessing Variables Defined Inside a Closure
8.1. Changing the String Representation of Instances8.2. Customizing String Formatting8.3. Making Objects Support the Context-Management Protocol8.4. Saving Memory When Creating a Large Number of Instances8.5. Encapsulating Names in a Class8.6. Creating Managed Attributes8.7. Calling a Method on a Parent Class8.8. Extending a Property in a Subclass8.9. Creating a New Kind of Class or Instance Attribute8.10. Using Lazily Computed Properties8.11. Simplifying the Initialization of Data Structures8.12. Defining an Interface or Abstract Base Class8.13. Implementing a Data Model or Type System8.14. Implementing Custom Containers8.15. Delegating Attribute Access8.16. Defining More Than One Constructor in a Class8.17. Creating an Instance Without Invoking init8.18. Extending Classes with Mixins8.19. Implementing Stateful Objects or State Machines8.20. Calling a Method on an Object Given the Name As a String8.21. Implementing the Visitor Pattern8.22. Implementing the Visitor Pattern Without Recursion8.23. Managing Memory in Cyclic Data Structures8.24. Making Classes Support Comparison Operations8.25. Creating Cached Instances
9.1. Putting a Wrapper Around a Function9.2. Preserving Function Metadata When Writing Decorators9.3. Unwrapping a Decorator9.4. Defining a Decorator That Takes Arguments9.5. Defining a Decorator with User Adjustable Attributes9.6. Defining a Decorator That Takes an Optional Argument9.7. Enforcing Type Checking on a Function Using a Decorator9.8. Defining Decorators As Part of a Class9.9. Defining Decorators As Classes9.10. Applying Decorators to Class and Static Methods9.11. Writing Decorators That Add Arguments to Wrapped Functions9.12. Using Decorators to Patch Class Definitions9.13. Using a Metaclass to Control Instance Creation9.14. Capturing Class Attribute Definition Order9.15. Defining a Metaclass That Takes Optional Arguments9.16. Enforcing an Argument Signature on *args and **kwargs9.17. Enforcing Coding Conventions in Classes9.18. Defining Classes Programmatically9.19. Initializing Class Members at Definition Time9.20. Implementing Multiple Dispatch with Function Annotations9.21. Avoiding Repetitive Property Methods9.22. Defining Context Managers the Easy Way9.23. Executing Code with Local Side Effects9.24. Parsing and Analyzing Python Source9.25. Disassembling Python Byte Code

10.1. Making a Hierarchical Package of Modules10.2. Controlling the Import of Everything10.3. Importing Package Submodules Using Relative Names10.4. Splitting a Module into Multiple Files10.5. Making Separate Directories of Code Import Under a Common Namespace10.6. Reloading Modules10.7. Making a Directory or Zip File Runnable As a Main Script10.8. Reading Datafiles Within a Package10.9. Adding Directories to sys.path10.10. Importing Modules Using a Name Given in a String10.11. Loading Modules from a Remote Machine Using Import Hooks10.12. Patching Modules on Import10.13. Installing Packages Just for Yourself10.14. Creating a New Python Environment10.15. Distributing Packages
11.1. Interacting with HTTP Services As a Client11.2. Creating a TCP Server11.3. Creating a UDP Server11.4. Generating a Range of IP Addresses from a CIDR Address11.5. Creating a Simple REST-Based Interface11.6. Implementing a Simple Remote Procedure Call with XML-RPC11.7. Communicating Simply Between Interpreters11.8. Implementing Remote Procedure Calls11.9. Authenticating Clients Simply11.10. Adding SSL to Network Services11.11. Passing a Socket File Descriptor Between Processes11.12. Understanding Event-Driven I/O11.13. Sending and Receiving Large Arrays
12.1. Starting and Stopping Threads12.2. Determining If a Thread Has Started12.3. Communicating Between Threads12.4. Locking Critical Sections12.5. Locking with Deadlock Avoidance12.6. Storing Thread-Specific State12.7. Creating a Thread Pool12.8. Performing Simple Parallel Programming12.9. Dealing with the GIL (and How to Stop Worrying About It)12.10. Defining an Actor Task12.11. Implementing Publish/Subscribe Messaging12.12. Using Generators As an Alternative to Threads12.13. Polling Multiple Thread Queues12.14. Launching a Daemon Process on Unix
13.1. Accepting Script Input via Redirection, Pipes, or Input Files13.2. Terminating a Program with an Error Message13.3. Parsing Command-Line Options13.4. Prompting for a Password at Runtime13.5. Getting the Terminal Size13.6. Executing an External Command and Getting Its Output13.7. Copying or Moving Files and Directories13.8. Creating and Unpacking Archives13.9. Finding Files by Name13.10. Reading Configuration Files13.11. Adding Logging to Simple Scripts13.12. Adding Logging to Libraries13.13. Making a Stopwatch Timer13.14. Putting Limits on Memory and CPU Usage13.15. Launching a Web Browser
14.1. Testing Output Sent to stdout14.2. Patching Objects in Unit Tests14.3. Testing for Exceptional Conditions in Unit Tests14.4. Logging Test Output to a File14.5. Skipping or Anticipating Test Failures14.6. Handling Multiple Exceptions14.7. Catching All Exceptions14.8. Creating Custom Exceptions14.9. Raising an Exception in Response to Another Exception14.10. Reraising the Last Exception14.11. Issuing Warning Messages14.12. Debugging Basic Program Crashes14.13. Profiling and Timing Your Program14.14. Making Your Programs Run Faster
15.1. Accessing C Code Using ctypes15.2. Writing a Simple C Extension Module15.3. Writing an Extension Function That Operates on Arrays15.4. Managing Opaque Pointers in C Extension Modules15.5. Defining and Exporting C APIs from Extension Modules15.6. Calling Python from C15.7. Releasing the GIL in C Extensions15.8. Mixing Threads from C and Python15.9. Wrapping C Code with Swig15.10. Wrapping Existing C Code with Cython15.11. Using Cython to Write High-Performance Array Operations15.12. Turning a Function Pointer into a Callable15.13. Passing NULL-Terminated Strings to C Libraries15.14. Passing Unicode Strings to C Libraries15.15. Converting C Strings to Python15.16. Working with C Strings of Dubious Encoding15.17. Passing Filenames to C Extensions15.18. Passing Open Files to C Extensions15.19. Reading File-Like Objects from C15.20. Consuming an Iterable from C15.21. Diagnosing Segmentation Faults
Online ResourcesBooks for Learning PythonAdvanced Books

Content preview from Python Cookbook, 3rd Edition

Chapter 4. Iterators and Generators

Iteration is one of Python’s strongest features. At a high level, you might simply view iteration as a way to process items in a sequence. However, there is so much more that is possible, such as creating your own iterator objects, applying useful iteration patterns in the itertools module, making generator functions, and so forth. This chapter aims to address common problems involving iteration.

4.1. Manually Consuming an Iterator

Problem

You need to process items in an iterable, but for whatever reason, you can’t or don’t want to use a for loop.

Solution

To manually consume an iterable, use the next() function and write your code to catch the StopIteration exception. For example, this example manually reads lines from a file:

with open('/etc/passwd') as f:
    try:
        while True:
            line = next(f)
            print(line, end='')
    except StopIteration:
        pass

Normally, StopIteration is used to signal the end of iteration. However, if you’re using next() manually (as shown), you can also instruct it to return a terminating value, such as None, instead. For example:

with open('/etc/passwd') as f:
     while True:
         line = next(f, None)
         if line is None:
             break
         print(line, end='')

Discussion

In most cases, the for statement is used to consume an iterable. However, every now and then, a problem calls for more precise control over the underlying iteration mechanism. Thus, it is useful to know what actually happens.

The following interactive example illustrates the basic mechanics of what happens ...