IPython + [Jupyter] NotebookEueung Mulyana

http://eueung.github.io/python/ipython-introHint: Navigate with Arrow Keys | Attribution-ShareAlike CC BY-SA

1 / 34

Agenda1. Introduction to IPython2. IPython QtConsole3. Jupyter Notebook4. Notebook: Getting Started

2 / 34

Introduction to IPython

3 / 34

One of Python’s most useful features is its interactiveinterpreter. It allows for very fast testing of ideas without theoverhead of creating test files as is typical in most programminglanguages. However, the interpreter supplied with the standardPython distribution is somewhat limited for extended interactiveuse.

IPythonA comprehensive environment for interactive and exploratorycomputing

Three Main Components:An enhanced interactive Python shell.A decoupled two-process communication model, whichallows for multiple clients to connect to a computationkernel, most notably the web-based notebook.An architecture for interactive parallel computing.

Some of the many useful features of IPython includes:

Command history, which can be browsed with the up anddown arrows on the keyboard.Tab auto-completion.In-line editing of code.Object introspection, and automatic extract ofdocumentation strings from python objects like classesand functions.Good interaction with operating system shell.Support for multiple parallel back-end processes, that canrun on computing clusters or cloud services like AmazonEC2.

4 / 34

IPython provides a rich architecture for interactive computingwith:

A powerful interactive shell.A kernel for Jupyter.Easy to use, high performance tools for parallelcomputing.Support for interactive data visualization and use of GUItoolkits.Flexible, embeddable interpreters to load into your ownprojects.

IPythonIPython is an interactive shell that addresses the limitation of thestandard python interpreter, and it is a work-horse for scientificuse of python. It provides an interactive prompt to the pythoninterpreter with a greatly improved user-friendliness.

It comes with a browser-based notebook with support for code,text, mathematical expressions, inline plots and other richmedia.

You don’t need to know anything beyond Python to start usingIPython – just type commands as you would at the standardPython prompt. But IPython can do much more than thestandard prompt...

5 / 34

IPythonBeyond the Terminal ...

The REPL as a Network ProtocolKernels

Execute CodeClients

Read inputPresent Output

Simple abstractions enable rich, sophisticated clients

6 / 34

The Four Most Helpful CommandsThe four most helpful commands is shown to you in a banner,every time you start IPython:

Command Description

?Introduction and overview of

IPython’s features.

%quickref Quick reference.

help Python’s own help system.

object?Details about object, useobject?? for extra details.

Tab CompletionTab completion, especially for attributes, is a convenient way toexplore the structure of any object you’re dealing with. Simplytype object_name.<TAB> to view the object’s attributes.Besides Python objects and keywords, tab completion alsoworks on file and directory names.

Running and EditingThe %run magic command allows you to run any python scriptand load all of its data directly into the interactive namespace.Since the file is re-read from disk each time, changes you maketo it are reflected immediately (unlike imported modules, whichhave to be specifically reloaded). IPython also includesdreload, a recursive reload function.

%run has special flags for timing the execution of your scripts(-t), or for running them under the control of either Python’spdb debugger (-d) or profiler (-p).

The %edit command gives a reasonable approximation ofmultiline editing, by invoking your favorite editor on the spot.IPython will execute the code you type in there as if it weretyped interactively.

7 / 34

Magic Functions ...The following examples show how to call the builtin %timeitmagic, both in line and cell mode:

In [1]: %timeit range(1000)100000 loops, best of 3: 7.76 us per loop

In [2]: %%timeit x = range(10000) ...: max(x) ...:1000 loops, best of 3: 223 us per loop

The builtin magics include:

Functions that work with code: %run, %edit, %save,%macro, %recall, etc.Functions which affect the shell: %colors, %xmode,%autoindent, %automagic, etc.Other functions such as %reset, %timeit,%%writefile, %load, or %paste.

Exploring your ObjectsTyping object_name? will print all sorts of details about anyobject, including docstrings, function definition lines (for callarguments) and constructor details for classes. To get specificinformation on an object, you can use the magic commands%pdoc, %pdef, %psource and %pfile.

Magic FunctionsIPython has a set of predefined magic functions that you can callwith a command line style syntax. There are two kinds ofmagics, line-oriented and cell-oriented.

Line magics are prefixed with the % character and workmuch like OS command-line calls: they get as anargument the rest of the line, where arguments arepassed without parentheses or quotes.Cell magics are prefixed with a double %%, and they arefunctions that get as an argument not only the rest of theline, but also the lines below it in a separate argument.

8 / 34

System Shell CommandsTo run any command at the system shell, simply prefix it with !.You can capture the output into a Python list. To pass the valuesof Python variables or expressions to system commands, prefixthem with $.

System Aliases

It’s convenient to have aliases to the system commands you usemost often. This allows you to work seamlessly from insideIPython with the same commands you are used to in yoursystem shell. IPython comes with some pre-defined aliases anda complete system for changing directories, both via a stack(%pushd, %popd and %dhist) and via direct %cd. The latterkeeps a history of visited directories and allows you to go to anypreviously visited one.

Magic Functions ...You can always call them using the % prefix, and if you’re callinga line magic on a line by itself, you can omit even that: runthescript.py. You can toggle this behavior by running the%automagic magic.

Cell magics must always have the %% prefix.

A more detailed explanation of the magic system can beobtained by calling %magic, and for more details on any magicfunction, call %somemagic? to read its docstring. To see allthe available magic functions, call %lsmagic.

System Shell Commands ...

!ping www.bbc.co.uk

files = !ls # capture

!grep -rF $pattern ipython/* # passing vars

9 / 34

History ...Input and output history are kept in variables called In and Out,keyed by the prompt numbers. The last three objects in outputhistory are also kept in variables named _, __ and ___.

You can use the %history magic function to examine pastinput and output. Input history from previous sessions is savedin a database, and IPython can be configured to save outputhistory.

Several other magic functions can use your input history,including %edit, %rerun, %recall, %macro, %save and%pastebin. You can use a standard format to refer to lines:

%pastebin 3 18-20 ~1/1-5

This will take line 3 and lines 18 to 20 from the current session,and lines 1-5 from the previous session.

HistoryIPython stores both the commands you enter, and the results itproduces. You can easily go through previous commands withthe up- and down-arrow keys, or access your history in moresophisticated ways.

DebuggingAfter an exception occurs, you can call %debug to jump into thePython debugger (pdb) and examine the problem. Alternatively,if you call %pdb, IPython will automatically start the debuggeron any uncaught exception. You can print variables, see code,execute statements and even walk up and down the call stack totrack down the true source of the problem. This can be anefficient way to develop and debug code, in many caseseliminating the need for print statements or external debuggingtools.

You can also step through a program from the beginning bycalling %run -d theprogram.py.

10 / 34

IPython QtConsole

11 / 34

To force multiline input, hit Ctrl-Enter at the end ofthe first line instead of EnterShift-Enter to execute!

IPython QtConsolea version of IPython, using the new two-process ZeroMQKernel, running in a PyQt GUIa very lightweight widget that largely feels like a terminal,but provides a number of enhancements only possible ina GUI, such as inline figures, proper multiline editing withsyntax highlighting, graphical calltips, and much more.

# To start the Qt Consoleipython qtconsole

# To see a quick introduction of the main features %guiref

See: Qt Console @ RTD

12 / 34

IPython QtConsoleMF %loadThe new %load magic takes any script, and pastes its contentsas your next input, so you can edit it before executing. The scriptmay be on your machine, but you can also specify an historyrange, or a url, and it will download the script from the web.This is particularly useful for playing with examples fromdocumentation, such as matplotlib.

In [6]: %load http://matplotlib.org/plot_directive/mpl_examples/mplot3d/contour3d_demo.py

In [7]: from mpl_toolkits.mplot3d import axes3d ...: import matplotlib.pyplot as plt ...: ...: fig = plt.figure() ...: ax = fig.add_subplot(111, projection='3d') ...: X, Y, Z = axes3d.get_test_data(0.05) ...: cset = ax.contour(X, Y, Z) ...: ax.clabel(cset, fontsize=9, inline=1) ...: ...: plt.show()

13 / 34

MF %load ...The %load magic can also load source code from objects in theuser or global namespace by invoking the -n option.

In [1]: import hello_world ...: %load -n hello_world.say_hello

In [3]: def say_hello() : ...: print("Hello World!")

Inline MatplotlibOne of the most exciting features of the QtConsole is embeddedmatplotlib figures. You can use any standard matplotlib GUIbackend to draw the figures, and since there is now a two-process model, there is no longer a conflict between user inputand the drawing eventloop.

14 / 34

If you have a reference to a matplotlib figure object, you canalways display that specific figure:

In [1]: f = plt.figure()

In [2]: plt.plot(np.rand(100))Out[2]: [<matplotlib.lines.Line2D at 0x7fc6ac03dd90>]

In [3]: display(f)

# Plot is shown here

In [4]: plt.title('A title')Out[4]: <matplotlib.text.Text at 0x7fc6ac023450>

In [5]: display(f)

# Updated plot with title is shown here.

Matplotlib: display()IPython provides a function display()for displaying richrepresentations of objects if they are available. The IPythondisplay system provides a mechanism for specifying PNG or SVG(and more) representations of objects for GUI frontends. Whenyou enable matplotlib integration via the %matplotlib magic,IPython registers convenient PNG and SVG renderers formatplotlib figures, so you can embed them in your document bycalling display() on one or more of them. This is especiallyuseful for saving your work.

In [4]: from IPython.display import display

In [5]: plt.plot(range(5)) # plots in the matplotlib window

In [6]: display(plt.gcf()) # embeds the current figure in the qtconsole

In [7]: display(*getfigs()) # embeds all active figures in the qtconsole

15 / 34

--matplotlib inline

If you want to have all of your figures embedded in yoursession, instead of calling display(), you can specify --matplotlib inline when you start the console, and eachtime you make a plot, it will show up in your document, as if youhad called display(fig)().

The inline backend can use either SVG or PNG figures (PNGbeing the default). To switch between them, set theInlineBackend.figure_format configurable in a config file, or viathe %config magic:

In [10]: %config InlineBackend.figure_format = 'svg'

Changing the inline figure format also affects calls to display()above, even if you are not using the inline backend for allfigures.

In [13]: [ fig.close() for fig in getfigs() ]

By default, IPython closes all figures at the completion of eachexecution. It also means that the first matplotlib call in each cellwill always create a new figure. However, it does prevent the listof active figures surviving from one input cell to the next, so ifyou want to continue working with a figure, you must hold on toa reference to it.

In [11]: plt.plot(range(100))<single-line plot>

In [12]: plt.plot([1,3,2])<another single-line plot># -----In [11]: fig = gcf() ....: fig.plot(rand(100))<plot>In [12]: fig.title('Random Title')<redraw plot with title>

This behavior is controlled by the InlineBackend.close_figuresconfigurable, and if you set it to False, via %config or config file,then IPython will not close figures, and tools like gcf(), gca(),getfigs() will behave the same as they do with other backends.You will, however, have to manually close figures:

16 / 34

Jupyter Notebook

17 / 34

Jupyter NotebookThe Jupyter Notebook is a web application for interactive datascience and scientific computing.

Using the Jupyter Notebook, you can author engagingdocuments that combine live-code with narrative text,equations, images, video, and visualizations. By encoding acomplete and reproducible record of a computation, thedocuments can be shared with others on GitHub, Dropbox, andthe Jupyter Notebook Viewer.

# Installsudo apt-get install build-essential python-devpip install jupyter

# Startjupyter notebook

# Previouslypip install "ipython[notebook]"ipython notebook

See: Jupyter@RTD

18 / 34

Open source, interactive data science and scientificcomputing across over 40 programming languages.The Jupyter Notebook is a web application that allows youto create and share documents that contain live code,equations, visualizations and explanatory text.Uses include: data cleaning and transformation,numerical simulation, statistical modeling, machinelearning and much more.

Language of ChoiceThe Notebook has support for over 40 programming languages,including those popular in Data Science such as Python, R, Juliaand Scala.

Share NotebooksNotebooks can be shared with others using email, Dropbox,GitHub and the Jupyter Notebook Viewer.

Interactive WidgetsCode can produce rich output such as images, videos, LaTeX,and JavaScript. Interactive widgets can be used to manipulateand visualize data in realtime.

Big-Data IntegrationLeverage big data tools, such as Apache Spark, from Python, Rand Scala. Explore that same data with pandas, scikit-learn,ggplot2, dplyr, etc.

See: Jupyter.ORG Website

19 / 34

Project JupyterThe IPython Notebook (2011)

Rich Web ClientText & MathCodeResultsShare, Reproduce.

See: Fernando Perez, IPython & Project Jupyter

20 / 34

Project JupyterIPython

Interactive Python shell at the terminalKernel for this protocol in PythonTools for Interactive Parallel computing

JupyterNetwork protocol for interactive computingClients for protocol

ConsoleQt ConsoleNotebook

Notebook file format & tools (nbconvert...)Nbviewer

What’s in a name?Inspired by the open languages of science:

Julia, Python & RNot an acronym: all languages equal class citizens.

Astronomy and Scientific Python: A long and fruitfulcollaborationGalileo's notebooks:

The original, open science, data-and-narrativepapersAuthorea: “Science was Always meant to be Open”

See: Fernando Perez, IPython & Project Jupyter

21 / 34

Project JupyterFrom IPython to Project Jupyter

Not just about Python: Kernels in any languageIPython : "Official"IJulia, IRKernel, IHaskell, IFSharp, Ruby, IScala, IErlang, ..Lots more! ~37 and countingWhy is it called IPython, if it can do Julia, R, Haskell, Ruby,… ?”

TL;DRSeparation of the language-agnostic componentsJupyter: protocol, format, multi-user serverIPython: interactive Python console, Jupyter kernelJupyter kernels = Languages which can be used from thenotebook (37 and counting)

A Simple and Generic Architecture 

See: Fernando Perez, IPython & Project Jupyter

22 / 34

ConventionIn this document, we use the terms Jupyter and IPythonNotebooks interchangeably. It might refer to the previousversion of the Notebook (IPython).

23 / 34

The NotebookNotebook mode supports literate computing and reproduciblesessions

Allows to store chunks of python along side the resultsand additional comments (HTML, Latex, MarkDown)Can be exported in various file formats

Notebook are the de-facto standard for sharing python sessions.

The Notebook: “Literate Computing”Computational Narratives

Computers deal with code and data.Humans deal with narratives that communicate.

Literate Computing (not Literate Programming)

Narratives anchored in a live computation, thatcommunicate a story based on data and results.

24 / 34

Seamless Notebook Sharing

nbviewerZero-install reading of notebooksJust share a URLnbviewer.ipython.org

25 / 34

Jupyter EcosystemReproducible ResearchPaper, Notebooks and Virtual MachineScientific BloggingExecutable booksMOOCs and University CoursesExecutable Papers...

Jose Unpingco

Python for Signal ProcessingSpringer hardcover bookChapters: IPython NotebooksPosted as a blog entryAll available as a Github repo

26 / 34

Notebook Workflows 

Credit: Joshua Barrat (via F. Perez)

27 / 34

Notebook: Getting Started

28 / 34

ReviewIPython NotebookIPython notebook is an HTML-based notebook environment forPython. It is based on the IPython shell, but provides a cell-based environment with great interactivity, where calculationscan be organized and documented in a structured way.

Although using a web browser as graphical interface, IPythonnotebooks are usually run locally, from the same computer thatrun the browser.

IPython NotebookWeb-based user interface to IPython, Interactive Pythoninterpreter in the browserLiterate computing, Open format combining executablecode, text and multimediaPretty graphsVersion controlled science!

To start a new IPython notebook session, run the followingcommand:

ipython notebook# or jupyter notebook

from a directory where you want the notebooks to be stored.

This will open a new browser window (or a new tab in anexisting window) with an index page where existing notebooksare shown and from which new notebooks can be created.

29 / 34

Up and RunningAn IPython notebook lets you write and execute Python code inyour web browser. IPython notebooks make it very easy to tinkerwith code and execute it in bits and pieces; for this reasonIPython notebooks are widely used in scientific computing.

Once IPython is running, point your web browser athttp://localhost:8888 to start using IPython notebooks. Ifeverything worked correctly, you should see a screen showingall available IPython notebooks in the current directory.

If you click through to a notebook file, it will be executed anddisplayed on a new page.

See: CS231n: IPython Tutorial

30 / 34

Up and RunningAn IPython notebook is made up of a number of cells. Each cellcan contain Python code. You can execute a cell by clicking on itand pressing Shift-Enter. When you do so, the code in thecell will run, and the output of the cell will be displayed beneaththe cell. See example.

Global variables are shared between cells. See the notebookafter executing the second cell.

By convention, IPython notebooks are expected to be run fromtop to bottom. Failing to execute some cells or executing cellsout of order can result in errors.

See: CS231n: IPython Tutorial

31 / 34

Collections and LinksA gallery of interesting IPython Notebooks

Notebooks for LearningPython Fundamentals | DLAB @ BerkeleyPython Lectures | Rajath Kumar MPIntro Programming | Eric MatthesPython Crash Course | Eric MatthesIPython Minibook | Cyrille RossantIPython & Project Jupyter | Fernando Perez

32 / 34

References1. IPython Documentation @ readthedocs.org2. Jupyter Documentation @ readthedocs.org3. Fernando Perez, IPython & Project Jupyter | A language-independent architecture for open computing and

data science4. Juan Luis Cano Rodriguez, IPython: How a notebook is changing science | Python as a real alternative to

MATLAB, Mathematica and other commercial software5. Olivier Hervieu: Introduction to scientific programming in python6. CS231n: IPython Tutorial, http://cs231n.github.io/ipython-tutorial/7. J.R. Johansson: Introduction to scientific computing with Python

33 / 34

ENDEueung Mulyana

http://eueung.github.io/python/ipython-introHint: Navigate with Arrow Keys | Attribution-ShareAlike CC BY-SA

34 / 34