Mastering Python

Mastering Python

by Rick van Hattem
Released April 2016
Publisher(s): Packt Publishing
ISBN: 9781785289729

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Master the art of writing beautiful and powerful Python by using all of the features that Python 3.5 offers

About This Book

  • Become familiar with the most important and advanced parts of the Python code style
  • Learn the trickier aspects of Python and put it in a structured context for deeper understanding of the language
  • Offers an expert's-eye overview of how these advanced tasks fit together in Python as a whole along with practical examples

Who This Book Is For

Almost anyone can learn to write working script and create high quality code but they might lack a structured understanding of what it means to be 'Pythonic'. If you are a Python programmer who wants to code efficiently by getting the syntax and usage of a few intricate Python techniques exactly right, this book is for you.

What You Will Learn

  • Create a virtualenv and start a new project
  • Understand how and when to use the functional programming paradigm
  • Get familiar with the different ways the decorators can be written in
  • Understand the power of generators and coroutines without digressing into lambda calculus
  • Create metaclasses and how it makes working with Python far easier
  • Generate HTML documentation out of documents and code using Sphinx
  • Learn how to track and optimize application performance, both memory and cpu
  • Use the multiprocessing library, not just locally but also across multiple machines
  • Get a basic understanding of packaging and creating your own libraries/applications

In Detail

Python is a dynamic programming language. It is known for its high readability and hence it is often the first language learned by new programmers. Python being multi-paradigm, it can be used to achieve the same thing in different ways and it is compatible across different platforms. Even if you find writing Python code easy, writing code that is efficient, easy to maintain, and reuse is not so straightforward.

This book is an authoritative guide that will help you learn new advanced methods in a clear and contextualised way. It starts off by creating a project-specific environment using venv, introducing you to different Pythonic syntax and common pitfalls before moving on to cover the functional features in Python. It covers how to create different decorators, generators, and metaclasses. It also introduces you to functools.wraps and coroutines and how they work. Later on you will learn to use asyncio module for asynchronous clients and servers. You will also get familiar with different testing systems such as py.test, doctest, and unittest, and debugging tools such as Python debugger and faulthandler. You will learn to optimize application performance so that it works efficiently across multiple machines and Python versions. Finally, it will teach you how to access C functions with a simple Python call. By the end of the book, you will be able to write more advanced scripts and take on bigger challenges.

Style and Approach

This book is a comprehensive guide that covers advanced features of the Python language, and communicate them with an authoritative understanding of the underlying rationale for how, when, and why to use them.

Table of contents

  1. Mastering Python
    1. Table of Contents
    2. Mastering Python
    3. Credits
    4. About the Author
    5. About the Reviewers
    6. www.PacktPub.com
      1. eBooks, discount offers, and more
        1. Why subscribe?
    7. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
        1. Downloading the example code
        2. Errata
        3. Piracy
        4. Questions
    8. 1. Getting Started – One Environment per Project
      1. Creating a virtual Python environment using venv
        1. Creating your first venv
        2. venv arguments
        3. Differences between virtualenv and venv
      2. Bootstrapping pip using ensurepip
        1. ensurepip usage
        2. Manual pip install
      3. Installing C/C++ packages
        1. Debian and Ubuntu
        2. Red Hat, CentOS, and Fedora
        3. OS X
        4. Windows
      4. Summary
    9. 2. Pythonic Syntax, Common Pitfalls, and Style Guide
      1. Code style – or what is Pythonic code?
        1. Formatting strings – printf-style or str.format?
        2. PEP20, the Zen of Python
          1. Beautiful is better than ugly
          2. Explicit is better than implicit
          3. Simple is better than complex
          4. Flat is better than nested
          5. Sparse is better than dense
          6. Readability counts
          7. Practicality beats purity
          8. Errors should never pass silently
          9. In the face of ambiguity, refuse the temptation to guess
          10. One obvious way to do it
          11. Now is better than never
          12. Hard to explain, easy to explain
          13. Namespaces are one honking great idea
          14. Conclusion
        3. Explaining PEP8
          1. Duck typing
          2. Differences between value and identity comparisons
          3. Loops
          4. Maximum line length
        4. Verifying code quality, pep8, pyflakes, and more
          1. flake8
            1. Pep8
            2. pyflakes
            3. McCabe
            4. flake8
          2. Pylint
      2. Common pitfalls
        1. Scope matters!
          1. Function arguments
          2. Class properties
          3. Modifying variables in the global scope
        2. Overwriting and/or creating extra built-ins
        3. Modifying while iterating
        4. Catching exceptions – differences between Python 2 and 3
        5. Late binding – be careful with closures
        6. Circular imports
        7. Import collisions
      3. Summary
    10. 3. Containers and Collections – Storing Data the Right Way
      1. Time complexity – the big O notation
      2. Core collections
        1. list – a mutable list of items
        2. dict – unsorted but a fast map of items
        3. set – like a dict without values
        4. tuple – the immutable list
      3. Advanced collections
        1. ChainMap – the list of dictionaries
        2. counter – keeping track of the most occurring elements
        3. deque – the double ended queue
        4. defaultdict – dictionary with a default value
        5. namedtuple – tuples with field names
        6. enum – a group of constants
        7. OrderedDict – a dictionary where the insertion order matters
        8. heapq – the ordered list
        9. bisect – the sorted list
      4. Summary
    11. 4. Functional Programming – Readability Versus Brevity
      1. Functional programming
      2. list comprehensions
      3. dict comprehensions
      4. set comprehensions
      5. lambda functions
        1. The Y combinator
      6. functools
        1. partial – no need to repeat all arguments every time
        2. reduce – combining pairs into a single result
          1. Implementing a factorial function
          2. Processing trees
      7. itertools
        1. accumulate – reduce with intermediate results
        2. chain – combining multiple results
        3. combinations – combinatorics in Python
        4. permutations – combinations where the order matters
        5. compress – selecting items using a list of Booleans
        6. dropwhile/takewhile – selecting items using a function
        7. count – infinite range with decimal steps
        8. groupby – grouping your sorted iterable
        9. islice – slicing any iterable
      8. Summary
    12. 5. Decorators – Enabling Code Reuse by Decorating
      1. Decorating functions
        1. Why functools.wraps is important
        2. How are decorators useful?
        3. Memoization using decorators
        4. Decorators with (optional) arguments
        5. Creating decorators using classes
      2. Decorating class functions
        1. Skipping the instance – classmethod and staticmethod
        2. Properties – smart descriptor usage
      3. Decorating classes
        1. Singletons – classes with a single instance
        2. Total ordering – sortable classes the easy way
      4. Useful decorators
        1. Single dispatch – polymorphism in Python
        2. Contextmanager, with statements made easy
        3. Validation, type checks, and conversions
        4. Useless warnings – how to ignore them
      5. Summary
    13. 6. Generators and Coroutines – Infinity, One Step at a Time
      1. What are generators?
        1. Advantages and disadvantages of generators
        2. Pipelines – an effective use of generators
        3. tee – using an output multiple times
        4. Generating from generators
        5. Context managers
      2. Coroutines
        1. A basic example
        2. Priming
        3. Closing and throwing exceptions
        4. Bidirectional pipelines
        5. Using the state
      3. Summary
    14. 7. Async IO – Multithreading without Threads
      1. Introducing the asyncio library
        1. The async and await statements
          1. Python 3.4
          2. Python 3.5
          3. Choosing between the 3.4 and 3.5 syntax
        2. A simple example of single-threaded parallel processing
        3. Concepts of asyncio
          1. Futures and tasks
          2. Event loops
            1. Event loop implementations
            2. Event loop policies
            3. Event loop usage
          3. Processes
        4. Asynchronous servers and clients
          1. Basic echo server
      2. Summary
    15. 8. Metaclasses – Making Classes (Not Instances) Smarter
      1. Dynamically creating classes
        1. A basic metaclass
        2. Arguments to metaclasses
        3. Accessing metaclass attributes through classes
      2. Abstract classes using collections.abc
        1. Internal workings of the abstract classes
        2. Custom type checks
        3. Using abc.ABC before Python 3.4
      3. Automatically registering a plugin system
        1. Importing plugins on-demand
        2. Importing plugins through configuration
        3. Importing plugins through the file system
      4. Order of operations when instantiating classes
        1. Finding the metaclass
        2. Preparing the namespace
        3. Executing the class body
        4. Creating the class object (not instance)
        5. Executing the class decorators
        6. Creating the class instance
        7. Example
      5. Storing class attributes in definition order
        1. The classic solution without metaclasses
        2. Using metaclasses to get a sorted namespace
      6. Summary
    16. 9. Documentation – How to Use Sphinx and reStructuredText
      1. The reStructuredText syntax
        1. Getting started with reStructuredText
        2. Inline markup
        3. Headers
        4. Lists
          1. Enumerated list
          2. Bulleted list
          3. Option list
          4. Definition list
          5. Nested lists
        5. Links, references, and labels
        6. Images
        7. Substitutions
        8. Blocks, code, math, comments, and quotes
        9. Conclusion
      2. The Sphinx documentation generator
        1. Getting started with Sphinx
          1. Using sphinx-quickstart
          2. Using sphinx-apidoc
        2. Sphinx directives
          1. The table of contents tree directive (toctree)
          2. Autodoc, documenting Python modules, classes, and functions
        3. Sphinx roles
      3. Documenting code
        1. Documenting a class with the Sphinx style
        2. Documenting a class with the Google style
        3. Documenting a class with the NumPy style
        4. Which style to choose
      4. Summary
    17. 10. Testing and Logging – Preparing for Bugs
      1. Using examples as tests with doctest
        1. A simple doctest example
        2. Writing doctests
        3. Testing with pure documentation
        4. The doctest flags
          1. True and False versus 1 and 0
          2. Normalizing whitespace
          3. Ellipsis
        5. Doctest quirks
          1. Testing dictionaries
          2. Testing floating-point numbers
          3. Times and durations
      2. Testing with py.test
        1. The difference between the unittest and py.test output
        2. The difference between unittest and py.test tests
          1. Simplifying assertions
          2. Parameterizing tests
          3. Automatic arguments using fixtures
            1. Cache
            2. Custom fixtures
          4. Print statements and logging
          5. Plugins
            1. pytest-cov
            2. pytest-pep8 and pytest-flakes
            3. Configuring plugins
      3. Mock objects
        1. Using unittest.mock
        2. Using py.test monkeypatch
      4. Logging
        1. Configuration
          1. Basic logging configuration
          2. Dictionary configuration
          3. JSON configuration
          4. Ini file configuration
          5. The network configuration
        2. Logger
          1. Usage
      5. Summary
    18. 11. Debugging – Solving the Bugs
      1. Non-interactive debugging
        1. Inspecting your script using trace
        2. Debugging using logging
        3. Showing call stack without exceptions
        4. Debugging asyncio
        5. Handling crashes using faulthandler
      2. Interactive debugging
        1. Console on demand
        2. Debugging using pdb
          1. Breakpoints
          2. Catching exceptions
          3. Commands
        3. Debugging using ipdb
        4. Other debuggers
        5. Debugging services
      3. Summary
    19. 12. Performance – Tracking and Reducing Your Memory and CPU Usage
      1. What is performance?
      2. Timeit – comparing code snippet performance
      3. cProfile – finding the slowest components
        1. First profiling run
        2. Calibrating your profiler
        3. Selective profiling using decorators
        4. Using profile statistics
      4. Line profiler
      5. Improving performance
        1. Using the right algorithm
        2. Global interpreter lock
        3. Try versus if
        4. Lists versus generators
        5. String concatenation
        6. Addition versus generators
        7. Map versus generators and list comprehensions
        8. Caching
        9. Lazy imports
        10. Using optimized libraries
        11. Just-in-time compiling
        12. Converting parts of your code to C
      6. Memory usage
        1. Tracemalloc
        2. Memory profiler
        3. Memory leaks
        4. Reducing memory usage
          1. Generators versus lists
          2. Recreating collections versus removing items
          3. Using slots
      7. Performance monitoring
      8. Summary
    20. 13. Multiprocessing – When a Single CPU Core Is Not Enough
      1. Multithreading versus multiprocessing
      2. Hyper-threading versus physical CPU cores
      3. Creating a pool of workers
      4. Sharing data between processes
      5. Remote processes
        1. Distributed processing using multiprocessing
        2. Distributed processing using IPyparallel
          1. ipython_config.py
          2. ipython_kernel_config.py
          3. ipcontroller_config.py
          4. ipengine_config.py
          5. ipcluster_config.py
      6. Summary
    21. 14. Extensions in C/C++, System Calls, and C/C++ Libraries
      1. Introduction
        1. Do you need C/C++ modules?
        2. Windows
        3. OS X
        4. Linux/Unix
      2. Calling C/C++ with ctypes
        1. Platform-specific libraries
          1. Windows
          2. Linux/Unix
          3. OS X
          4. Making it easy
        2. Calling functions and native types
        3. Complex data structures
        4. Arrays
        5. Gotchas with memory management
      3. CFFI
        1. Complex data structures
        2. Arrays
        3. ABI or API?
        4. CFFI or ctypes?
      4. Native C/C++ extensions
        1. A basic example
        2. C is not Python – size matters
        3. The example explained
          1. static
          2. PyObject*
          3. Parsing arguments
        4. C is not Python – errors are silent or lethal
        5. Calling Python from C – handling complex types
      5. Summary
    22. 15. Packaging – Creating Your Own Libraries or Applications
      1. Installing packages
      2. Setup parameters
      3. Packages
      4. Entry points
        1. Creating global commands
        2. Custom setup.py commands
      5. Package data
      6. Testing packages
        1. Unittest
        2. py.test
        3. Nosetests
      7. C/C++ extensions
        1. Regular extensions
        2. Cython extensions
      8. Wheels – the new eggs
        1. Distributing to the Python Package Index
      9. Summary
    23. Index

Product information

  • Title: Mastering Python
  • Author(s): Rick van Hattem
  • Release date: April 2016
  • Publisher(s): Packt Publishing
  • ISBN: 9781785289729