Mastering IPython 4.0

Table of contents

1. Mastering IPython 4.0
   1. Table of Contents
   2. Mastering IPython 4.0
   3. Credits
   4. About the Author
   5. About the Reviewer
   6. www.PacktPub.com
      1. eBooks, discount offers, and more
         1. Why subscribe?
   7. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
         1. Downloading the example code
         2. Downloading the color images of this book
         3. Errata
         4. Piracy
         5. Questions
   8. 1. Using IPython for HPC
      1. The need for speed
      2. FORTRAN to the rescue – the problems FORTRAN addressed
         1. Readability
         2. Portability
         3. Efficiency
         4. The computing environment
      3. Choosing between IPython and Fortran
         1. Fortran
         2. IPython
            1. Object-orientation
            2. Ease of adoption
         3. Popularity – Fortran versus IPython
            1. Useful libraries
         4. The cost of building (and maintaining) software
            1. Requirements and specification gathering
            2. Development
            3. Execution
            4. Testing and maintenance
         5. Alternatives
            1. Cross-language development
            2. Prototyping and exploratory development
      4. An example case – Fast Fourier Transform
         1. Fast Fourier Transform
            1. Fortran
            2. Python
         2. Performance concerns
         3. Software engineering concerns
            1. Complexity-based metrics
            2. Size-based metrics
         4. Where we stand now
      5. High Performance Computing
         1. The HPC learning curve
         2. Cloudy with a chance of parallelism (or Amazon's computer is bigger than yours)
         3. HPC and parallelism
         4. Clouds and HPC
      6. Going parallel
         1. Terminology
         2. A parallel programming example
            1. A serial program
            2. A parallel equivalent
            3. Discussion
      7. Summary
   9. 2. Advanced Shell Topics
      1. What is IPython?
      2. Installing IPython
      3. All-in-one distributions
         1. Package management with conda
         2. Canopy Package Manager
      4. What happened to the Notebook?
      5. Starting out with the terminal
      6. IPython beyond Python
         1. Shell integration
         2. History
      7. Magic commands
         1. Creating custom magic commands
      8. Cython
      9. Configuring IPython
      10. Debugging
          1. Post-mortem debugging
          2. Debugging at startup
          3. Debugger commands
      11. Read-Eval-Print Loop (REPL) and IPython architecture
      12. Alternative development environments
          1. Spyder
          2. Canopy
          3. PyDev
          4. Others
      13. Summary
   10. 3. Stepping Up to IPython for Parallel Computing
       1. Serial processes
          1. Program counters and address spaces
          2. Batch systems
          3. Multitasking and preemption
          4. Time slicing
       2. Threading
          1. Threading in Python
          2. Example
          3. Limitations of threading
          4. Global Interpreter Lock
             1. What happens in an interpreter?
             2. CPython
             3. Multi-core machines
             4. Kill GIL
       3. Using multiple processors
       4. The IPython parallel architecture
          1. Overview
          2. Components
             1. The IPython Engine
             2. The IPython Controller
             3. The IPython Hub
             4. The IPython Scheduler
       5. Getting started with ipyparallel
          1. ipcluster
          2. Hello world
          3. Using map_sync
          4. Asynchronous calls
          5. Synchronizing imports
       6. Parallel magic commands
          1. %px
          2. %%px
          3. %pxresult
          4. %pxconfig
          5. %autopx
       7. Types of parallelism
          1. SIMD
          2. SPMD
             1. ipcluster and mpiexec/mpirun
             2. ipcluster and PBS
                1. Starting the engines
                2. Starting the controller
                3. Using the scripts
          3. MapReduce
             1. Scatter and gather
             2. A more sophisticated method
          4. MIMD
          5. MPMD
          6. Task farming and load balancing
             1. The @parallel function decorator
       8. Data parallelism
          1. No data dependence
          2. External data dependence
       9. Application steering
          1. Debugging
          2. First to the post
          3. Graceful shutdown
       10. Summary
   11. 4. Messaging with ZeroMQ and MPI
       1. The storage hierarchy
          1. Address spaces
          2. Data locality
       2. ZeroMQ
          1. A sample ZeroMQ program
             1. The server
             2. The client
          2. Messaging patterns in ZeroMQ
             1. Pairwise
                1. Server
                2. Client
                3. Discussion
             2. Client/server
                1. Server 1
                2. Server 2
                3. Client
                4. Discussion
             3. Publish/subscribe
                1. Publisher
                2. Subscriber
                3. Discussion
             4. Push/Pull
                1. Ventilator
                2. Worker
                3. Sink
                4. Discussion
          3. Important ZeroMQ features
          4. Issues using ZeroMQ
             1. Startup and shutdown
             2. Discovery
       3. MPI
          1. Hello World
          2. Rank and role
          3. Point-to-point communication
          4. Broadcasting
          5. Reduce
          6. Discussion
             1. Change the configuration
             2. Divide the work
             3. Parcel out the work
          7. Process control
             1. Master
             2. Worker
       4. ZeroMQ and IPython
          1. ZeroMQ socket types
          2. IPython components
             1. Client
             2. Engine(s)
             3. Controller
             4. Hub
             5. Scheduler
             6. Connection diagram
          3. Messaging use cases
             1. Registration
             2. Heartbeat
             3. IOPub
       5. Summary
   12. 5. Opening the Toolkit – The IPython API
       1. Performance profiling
          1. Using utils.timing
          2. Using %%timeit
          3. Using %%prun
       2. The AsyncResult class
          1. multiprocessing.pool.Pool
             1. Blocking methods
             2. Nonblocking methods
             3. Obtaining results
             4. An example program using various methods
          2. mp.pool.AsyncResult
             1. Getting results
             2. An example program using various methods
          3. AsyncResultSet metadata
             1. Metadata keys
             2. Other metadata
       3. The Client class
          1. Attributes
          2. Methods
       4. The View class
          1. View attributes
          2. Calling Python functions
             1. Synchronous calls
             2. Asynchronous calls
             3. Configurable calls
             4. Job control
          3. DirectView
             1. Data movement
                1. Dictionary-style data access
                2. Scatter and gather
                3. Push and pull
             2. Imports
                1. Discussion
          4. LoadBalancedView
             1. Data movement
             2. Imports
       5. Summary
   13. 6. Works Well with Others – IPython and Third-Party Tools
       1. R
          1. The rpy2 module/extension
             1. Installing rpy2
          2. Using Rmagic
             1. The %R magic
             2. The %%R magic
             3. Pulling and pushing
          3. Graphics
          4. Using rpy2.robjects
             1. The basics
             2. Interpreting a string as R
       2. Octave
          1. The oct2py module/extension
             1. Installing oct2py
          2. Using Octave magic
             1. The %octave magic
                1. Tricky issues
          3. The %%octave magic
          4. Pushing and pulling
          5. Graphics
          6. Using the Octave module
             1. Pushing and pulling
             2. Running Octave code
       3. Hy
          1. The hymagic module/extension
             1. Installing hymagic
          2. Using hymagic
             1. The %hylang magic
             2. The %%hylang magic
          3. A quick introduction to Hy
             1. Hello world!
             2. Get used to parentheses
             3. Arithmetic operations are in the wrong place
             4. Function composition is everywhere
             5. Control structures in Hy
                1. Setting variable values
                2. Defining functions
                3. if statements
                4. Conditionals
                5. Loops
             6. Calling Python
       4. Summary
   14. 7. Seeing Is Believing– Visualization
       1. Matplotlib
          1. Starting matplotlib
          2. An initial graph
          3. Modifying the graph
          4. Controlling interactivity
       2. Bokeh
          1. Starting Bokeh
          2. An initial graph
          3. Modifying the graph
             1. Customizing graphs
          4. Interactive plots
             1. An example interactive plot
       3. R
          1. Installing ggplot2 and pandas
             1. Using DataFrames
          2. An initial graph
             1. Modifying the graph
             2. A different view
       4. Python-nvd3
          1. Starting Python-nvd3
          2. An initial graph
          3. Putting some tools together
          4. A different type of plot
       5. Summary
   15. 8. But It Worked in the Demo! – Testing
       1. Unit testing
          1. A quick introduction
             1. Assertions
             2. Environmental issues
                1. Before it starts – setup
                2. While it is running – mocks
                3. After it finishes – teardown
             3. Writing to be tested
       2. unittest
          1. Important concepts
          2. A test using setUp and tearDown
             1. One-time setUp and tearDown
          3. Decorators
       3. pytest
          1. Installation
          2. Back compatibility
          3. Test discovery
          4. Organizing test files
          5. Assertions
          6. A test using setUp and tearDown
             1. Classic xUnit-style
                1. Being verbose
          7. Using fixtures
             1. Skipping and failing
          8. Monkeypatching
       4. nose2
          1. Installation
          2. Back compatibility
          3. Test discovery
             1. Running individual tests
          4. Assertions, setup, and teardown
             1. Modified xUnit-style
          5. Using decorators
          6. Plugins
             1. Generating XML with the junitxml plugin
       5. Summary
   16. 9. Documentation
       1. Inline comments
          1. Using inline comments
          2. Function annotations
             1. Syntax
                1. Parameters
                2. Return values
             2. Semantics
          3. Type hints
             1. Syntax
             2. Semantics
       2. Docstrings
          1. Example
             1. Inheriting docstrings
          2. Recommended elements
             1. One-line docstrings
             2. Syntax
                1. Semantics
             3. Multiline docstrings
                1. Syntax
                2. Semantics
                   1. The API
                   2. Inputs
                   3. Functionality
                   4. Outputs
                   5. Error conditions
                   6. Relationship with other parts of the system
                   7. Example uses
          3. Example
       3. reStructuredText
          1. History and goals
          2. Customers
          3. The solution
          4. Overview
             1. Paragraphs
             2. Text styles
                1. Literal blocks
             3. Lists
                1. Enumerated lists
                2. Bulleted lists
                3. Definition lists
             4. Hyperlinks
             5. Sections
       4. Docutils
          1. Installation
          2. Usage
          3. Documenting source files
       5. Sphinx
          1. Installation and startup
          2. Specifying the source files
       6. Summary
   17. 10. Visiting Jupyter
       1. Installation and startup
       2. The Dashboard
       3. Creating a notebook
       4. Interacting with Python scripts
       5. Working with cells
          1. Cell tricks
             1. Cell scoping
             2. Cell execution
             3. Restart & Run All
             4. Magics
          2. Cell structure
             1. Code cells
             2. Markdown cells
             3. Raw cells
             4. Heading cells
       6. Being graphic
          1. Using matplotlib
          2. Using Bokeh
          3. Using R
          4. Using Python-nvd3
       7. Format conversion
          1. Other formats
          2. nbviewer
       8. Summary
   18. 11. Into the Future
       1. Some history
       2. The Jupyter project
          1. The Notebook
          2. The console
          3. Jupyter Client
          4. The future of Jupyter
             1. Official roadmap
             2. Official subprojects
                1. Direct creation
                2. Incorporation
             3. Incubation
                1. External incubation
       3. IPython
          1. Current activity
       4. The rise of parallelism
          1. The megahertz wars end
          2. The problem
          3. A parallel with the past
          4. The present
             1. Problems are getting bigger and harder
             2. Computers are becoming more parallel
             3. Clouds are rolling in
             4. There is no Big Idea
                1. Pragmatic evolution of techniques
                2. Better tools
                3. The Next Big Idea
       5. Growing professionalism
          1. The NSF
             1. Software Infrastructure for Sustained Innovation
       6. Summary
   19. Index