def incx(self):
    self.x += 1

class Point(object):
    def __init__(self, x): self.x = x
    incx = incx

We can then create a point at the origin and increment x:

In [2]: p = Point(0)

In [3]: p.incx()

In [4]: print p.x
1

The same code which defines Point continues to work if we move incx to another file, say demo.py, and import it using from demo import incx.

But if we were to put incx in demo.pyx and compile it (using python setup.py build_ext --inplace), we get a strange error when running our simple test:

$ python bad.py
Traceback (most recent call last):
  File "bad.py", line 10, in <module>
    p.incx()
TypeError: incx() takes exactly one argument (0 given)

Read on to learn about instance methods and see how I fixed this.

My first thought was to write a simple wrapper function, using functools.wraps to preserve the docstring and name. This proved to be a clumsy solution.

A better approach turned out to be to use types.MethodType to convert our built-in function into an (unbound) instance method. The docstring for MethodType is pretty short:

instancemethod(function, instance, class)

The first and third arguments are obvious. For the second argument we’ll use None since we have no instance of the class yet, making this an unbound instance method.

So to fix our simple example, we need only replace the incx = incx line in the class definition with a short snippet after the class is defined:

from types import MethodType
Point.incx = MethodType(incx, None, Point)

We can give this new version a try and see that it works:
$ python good.py
1


As always, the entire example is available as a Gist.

$ python2.7
Python 2.7.2+ (...)
...
>>> 1/2
0

$ python3.2
Python 3.2.2 (...)
...
>>> 1/2
0.5

In this post, we’ll discuss how to configure the behavior of the division operator in Python 2.7 and IPython to behave like the Python 3.x counterpart.

We’ll call the division in Python 2.x “classic” or “old.” It does division in the C sense: an integer divided by an integer yields an integer, rounding down if needed.

The “new” division in Python 3.x instead computes the true division result. In this case, an integer divided by an integer yields a float. The floor division can be accessed in a new operator “//” as in:

>>> 1 // 2
0

To allow for a transition period, two compatibility features are provided. First, module developers can add

from __future__ import division

to the top of any .py file to indicate that “/” should mean true division in that file.

In addition, the Python interpreter can be passed a flag to indicate the desired division behavior:

-Q arg : division options: -Qold (default), -Qwarn, -Qwarnall, -Qnew

So for example, we can run:

$ python2.7 -Qnew
...
>>> 1/2
0.5

But what about IPython? There isn’t a corresponding flag on the ipython script, yet. In discussing adding this feature to IPython, Fernando Perez provided a useful tip for accomplishing this using the IPython configuration files.

To begin, you’ll need to create an IPython profile by running

$ ipython profile create

Then edit the configuration file, which for me was located at ~/.config/ipython/profile_default/ipython_config.py, and uncomment and add the following:

# lines of code to run at IPython startup.
c.InteractiveShellApp.exec_lines = [
    'from __future__ import division'
]

We can test it out and see that it does in fact work as desired:

$ ipython
Python 2.7.2+ (default, Oct  4 2011, 20:06:09)
Type "copyright", "credits" or "license" for more information.

IPython 0.13.dev -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: 1/2
Out[1]: 0.5

But when I tried to run an existing script in the new IPython Qt Console, the usual overly verbose set of diagnostic messages did not appear! The symtoms were:

• Regular print statements in Python worked fine.
• printf statements in C-code wrapped with Boost.Python did not appear.

Read on to see a solution.

This seemed quite odd until I read more about the architecture of IPython Qt Console. In particular, IPython Qt Console is a two-process architecture whereby a background IPython process executes code and frontend client handles all user input and output. So our printf statements were being sent to stdout in the background process and therefore never appear in the Qt Console window.

Somebody must have experienced this before, but a few quick Google searched turned up nothing. How would Cython handle this? In particular, we could use Cython to compile a simple Python print statement. So consider the simple printf.pyx file:

def printf():
    print "Hello"

which we compile into C using cython printf.pyx. The resulting C file is quite lengthy, but the relevant bits are in the __Pyx_PrintOne() function:

static int __Pyx_PrintOne(PyObject* f, PyObject *o) {
    if (!f) {
        if (!(f = __Pyx_GetStdout()))
            return -1;
    }
    if (PyFile_SoftSpace(f, 0)) {
        if (PyFile_WriteString(" ", f) < 0)
            return -1;
    }
    if (PyFile_WriteObject(o, f, Py_PRINT_RAW) < 0)
        return -1;
    if (PyFile_WriteString("\n", f) < 0)
        return -1;
    return 0;
    /* the line below is just to avoid compiler
     * compiler warnings about unused functions */
    return __Pyx_Print(f, NULL, 0);
}

The __Pyx_GetStdout() function is nothing more than a thin wrapper to PySys_GetObject((char *)"stdout") with a bit of error checking.

Putting this all together, a simple reporting function (without any error checking) becomes:

void report(std::string msg) {
  PyObject *f = PySys_GetObject((char *)"stdout");
  PyFile_WriteString(msg.c_str(), f);
  PyFile_WriteString("\n", f);
}

For an example module using this code, see gist 1284927.

Update: After writing this, I learned of PySys_WriteStdout() which is a nearly drop-in replacement of printf. However the above method is still be useful if you need to print more than 1000 bytes at a time.

However if you need to wrap a function which takes a FILE* argument the previous approach will not prove fruitful. But Boost.Python is clearly capable of handling a similar situation, namely the implicit conversion from a Python string type to a char* type. Since the conversion from the internal PyObject* to char* is likely to be done with the PyString_AsString function, I went off in search of that code.

There were no references in the installed headers, but it did appear in the source code in converter/builtin_converters.cpp in the convert_to_cstring() function:

// An lvalue conversion function which extracts a char const* from a
// Python String.
void* convert_to_cstring(PyObject* obj)
{
  return PyString_Check(obj) ? PyString_AsString(obj) : 0;
}

… which was later called as …

// Add an lvalue converter for char which gets us char const*
registry::insert
  (convert_to_cstring, type_id(),
   &converter::wrap_pytype<&PyString_Type>::get_pytype);

Jackpot! We should be able to mimic these lines to create a Python file to FILE* converter using PyFile_AsFile():

#include <boost/python.hpp>

namespace {
  void *convert_to_FILEptr(PyObject* obj) {
    return PyFile_Check(obj) ? PyFile_AsFile(obj) : 0;
  }
}

BOOST_PYTHON_MODULE(file_wrapper) {
  boost::python::converter::registry::insert
    (convert_to_FILEptr,
     boost::python::type_id<FILE>(),
     &boost::python::converter::wrap_pytype<&PyFile_Type>::get_pytype);
}

For a complete example including test code, see gist: 1265889.

Edit: Of course this also works with boost::python::extract<FILE*>. I’ve updated the gist to demonstrate the relevant usage.

However debugging MPI scripts can be challenging. Here are some useful ways to run your programs for debugging.

• Open an xterm window for each MPI process, with the script running in iPython.

  $ mpirun -np 4 xterm -e "ipython script.py"

• Open an xterm window for each MPI process, with gdb attached to each python process. The -x flag tells gdb to run the commands given in the specified file. This is often a good place to add additional breakpoints.

  $ echo "run script.py" > gdb.in
  $ mpirun -np 4 xterm -e "gdb -x gdb.in python"

• Open each MPI process within screen, then open a gnome-terminal with one tab for each screen.

  $ mpirun -np 4 screen -L -m -D -S mpi \
      ipython script.py &
  $ gnome-terminal --tab -e "screen -RR -p mpi" \
                   --tab -e "screen -RR -p mpi" \
                   --tab -e "screen -RR -p mpi" \
                   --tab -e "screen -RR -p mpi"

  If your program dies unexpectedly, it is probably because LD_LIBRARY_PATH is stripped by glibc since screen is a setgid/setuid program. You can work around this my modifying the mpirun call as

  $ mpirun -np 4 screen -L -m -D -S mpi \
      env LD_LIBRARY_PATH=$LD_LIBRARY_PATH ipython script.py

For this post we’ll use the CppDuck.cpp and PyDuck.py from previous posts, changing only the main Bird module.

Most of the code can remain the same, however we’ll need to now change setQuack() to identify whether the passed object is a callable Python method (using PyCallable_Check) or a capsule.

void setQuack(object obj) {
  if (PyCallable_Check(obj.ptr())) {
    quacker = obj;
  }
  else {
    quacker = reinterpret_cast<quacker_t*>(PyCapsule_GetPointer(obj.ptr(), "quacker"));
  }
  if (PyErr_Occurred())
    throw_error_already_set();
}

The remainder of Bird.cpp can be as in Calling Python from C++. Usage is easy too:

>>> import Bird
>>> import CppDuck
>>> import PyDuck
>>>
>>> Bird.callQuack()
No noise came out.
>>>
>>> Bird.setQuack(CppDuck.getQuack())
>>> Bird.callQuack()
C++ Quack
>>>
>>> Bird.setQuack(PyDuck.quack)
>>> Bird.callQuack()
Python Quack

On some versions of Python this may cause the program to crash at exit since Python interpreter may have already shut down before the quacker Boost.Function object is destroyed.  We can fix this by defining a simple finalize function:

void finalize() {
  quacker.clear();
}

And adding to the module:

BOOST_PYTHON_MODULE(Bird) {
  // ...

  def("_finalize", &finalize);
  // The following code executes:
  // import atexit
  // atexit.register(_finalize)
  object atexit = object(handle<>(PyImport_ImportModule("atexit")));
  object finalize = scope().attr("_finalize");
  atexit.attr("register")(finalize);
}

This code ensures that the quacker object is cleared before Python is unloaded, preventing any potential crashes.

The full Bird.cpp program then looks like:

#include <boost/python.hpp>
#include <boost/function.hpp>
#include <iostream>

using namespace boost::python;

typedef void (quacker_t)(void);
boost::function<quacker_t> quacker;

void default_quacker() {
  std::cout << "No noise came out." << std::endl;
}

void setQuack(object obj) {
  if (PyCallable_Check(obj.ptr())) {
    quacker = obj;
  }
  else {
    quacker = reinterpret_cast<quacker_t*>(PyCapsule_GetPointer(obj.ptr(), "quacker"));
  }
  if (PyErr_Occurred())
    throw_error_already_set();
}

void callQuack() {
  quacker();
}

void finalize() {
  quacker.clear();
}

BOOST_PYTHON_MODULE(Bird) {
  quacker = &default_quacker;

  def("setQuack", &setQuack);
  def("callQuack", &callQuack);

  // Instead we register the function with the atexit module which
  // will be called _before_ the Python C-API is unloaded.
  def("_finalize", &finalize);
  object atexit = object(handle<>(PyImport_ImportModule("atexit")));
  object finalize = scope().attr("_finalize");
  atexit.attr("register")(finalize);
}

Here we describe how to use the with statement to create a simple timer with the syntax:

>>> with TicToc():
...     some_slow_operations
...
Elapsed time is 2.000073 seconds.

Readers familiar with MATLAB will recognize the similarity to the familiar timing mechanism:

> tic ; some_slow_operations; toc
Elapsed time is 10.020349 seconds.

Before we describe the TicToc class, let’s review the with statement. The documentation for reading files suggests that it is a good way to ensure files are properly closed, even if the code reading them raises an exception.

>>> with open('/tmp/workfile', 'r') as f:
...     read_data = f.read()
>>> f.closed
True

The with statement uses two methods, __enter__, which is run before the block executes, and __exit__, which is run after the block executes even if an exception is raised.  With this in mind, the previous code is mostly equivalent to:

>>> f = open('/tmp/workfile', 'r')
>>> f.__enter__()
<open file '/tmp/workfile', mode 'r' at 0x10041d810>
>>> read_data = f.read()
>>> f.__exit__(None, None, None)
>>> f.closed
True

Here the method file.__exit__ has automatically closed the file.

TicToc

The timer will be implemented as a class defining two methods:

1. __enter__ which records the start time.
2. __exit__ which prints the difference between the current time and the start time.

import time
class TicToc(object):
    """
    A simple code timer.

    Example
    -------
    >>> with TicToc():
    ...     time.sleep(2)
    Elapsed time is 2.000073 seconds.
    """
    def __init__(self, do_print = True):
        self.do_print = do_print
    def __enter__(self):
        self.start_time = time.time()
        return self
    def __exit__(self, type, value, traceback):
        self.elapsed = time.time() - self.start_time
        if self.do_print:
            print "Elapsed time is %f seconds." % self.elapsed

>>> from simpletimer import TicToc
>>> import time
>>> with TicToc():
...     time.sleep(2)
...
Elapsed time is 2.000115 seconds.
>>> with TicToc(False) as t:
...     time.sleep(1)
...
>>> t.elapsed
1.0001189708709717

What if instead we want to call a Python function directly from C? We’ll stick with the same Bird / Quacker metaphor as in the previous post, but we’ll now implement our duck class in PyDuck.py:

"""
PyDuck.py
"""

def quack():
    print "Python Quack"

Now how can we call Python from C++? It turns out that Boost.Python defines operator() for the object class, so calling Python is relatively easy.  But things become more difficult if we want to maintain a default implementation in C.

For example, one could imagine testing whether the Python object is set each time…

object quacker;

void callQuack() {
  if (quacker) {
    quacker();
  } else {
    default_quacker();
  }
}

But a cleaner approach is to instead use Boost.Function. This will allow all calls to use the same syntax as before, both when the quacker is set to a pure C function (i.e., default_quacker) or a callable Python object.

The final Bird.cpp file follows:

#include <boost/python.hpp>
#include <boost/function.hpp>
#include <iostream>

using namespace boost::python;

typedef void (quacker_t)(void);
boost::function<quacker_t> quacker;

void default_quacker() {
  std::cout << "No noise came out." << std::endl;
}

void setQuack(object obj) {
  quacker = obj;
}

void callQuack() {
  quacker();
}

BOOST_PYTHON_MODULE(Bird) {
  quacker = &default_quacker;

  def("setQuack", &setQuack);
  def("callQuack", &callQuack);
}

Testing it out:

>>> import Bird
>>> import PyDuck
>>>
>>> Bird.callQuack()
No noise came out.
>>>
>>> Bird.setQuack(PyDuck.quack)
>>> Bird.callQuack()
Python Quack

To begin, suppose we have two Boost.Python modules:

1. Bird, which has methods setQuack() and callQuack().
2. CppDuck, which has a method getQuack().

Our goal will be to get a function pointer with CppDuck.getQuack(), set it in another module in Bird.setQuack(), and finally call it with Bird.callQuack().

First let’s look at the code for CppDuck.cpp:

#include <boost/python.hpp>
#include <iostream>

using namespace boost::python;

void quack() {
  std::cout << "C++ Quack" << std::endl;
}

boost::python::object getQuack() {
  PyObject *result = PyCapsule_New((void*)&quack, "quacker", NULL);
  return object(boost::python::detail::new_reference(result));
}

BOOST_PYTHON_MODULE(CppDuck) {
  def("getQuack", &getQuack);
}

Note that we define a void (void) function “quack” which we will eventually pass to the Bird module. The method “getQuack” encapsulates it, and returns it as a Boost.Python object. Since PyCapsule_New() returns a new reference, we use boost::python::detail::new_reference to get the reference counting correct.

After compiling the module, we can try it out:

>>> import CppDuck
>>> CppDuck.getQuack()
<capsule object "quacker" at 0x100464930>

Let’s move on and see now how we can unpack the function pointer and call it in Bird.cpp:

#include <boost/python.hpp>
#include <iostream>

using namespace boost::python;

typedef void (quacker_t)(void);
quacker_t *quacker;

void default_quacker() {
  std::cout << "No noise came out." << std::endl;
}

void setQuack(object obj) {
  quacker = reinterpret_cast<quacker_t*>(PyCapsule_GetPointer(obj.ptr(), "quacker"));
  if (PyErr_Occurred())
    throw_error_already_set();
}

BOOST_PYTHON_MODULE(Bird) {
  quacker = &default_quacker;

  def("setQuack", &setQuack);
  def("callQuack", quacker);
}

Note the following conveniences in the code:

1. We create a typedef the function type — in this case void (void) — since we use it in multiple places.
2. We set a default quack function when the Python module is initialized. Without this, the Python interpreter will crash if we call callQuack before setting a valid quack.
3. PyCapsule_GetPointer gets passed the same string (i.e., “quacker”) as we passed to PyCapsule_New in Bird.cpp.
4. We use boost::python::throw_error_already_set() to display error messages. For example

   >>> Bird.setQuack(map)
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
   ValueError: PyCapsule_GetPointer called with invalid PyCapsule object

With both modules compiled, we can try it:

>>> import Bird
>>> import CppDuck
>>>
>>> Bird.callQuack()
No noise came out.
>>>
>>> quack = CppDuck.getQuack()
>>> Bird.setQuack(quack)
>>>
>>> Bird.callQuack()
No noise came out.

/application
    __init__.py
    submodule.py

I found a helpful post which uses PyImport_AddModule(“application.submodule”) to initialize a new Python module and boost::python::scope to add classes and method definitions to that module.

Unfortunately in my use “application” is not known at compile time — instead the C++ code is compiled into a static library which is reused for many different applications.  However, the same method will work if we first use the __name__ attribute to get the application name.

#include <boost/python.hpp>
using namespace boost::python;

/**
 * Methods and classes exposed by application.submodule.
 */
void import_application_submodule() {
  def("method1", &method1);
  // ...
}
/**
 * Methods and classes exposed by application.
 */
void import_application() {
  scope current;
  std::string submoduleName(extract<const char*>(current.attr("__name__")));
  submoduleName.append(".submodule");

  // Create the submodule, and attach it to the current scope.
  object submodule(borrowed(PyImport_AddModule(submoduleName.c_str())));
  current.attr("submodule") = submodule;

  {
    // Switch the scope to the submodule, add methods and classes.
    scope submoduleScope = submodule;
    import_application_submodule();
  }
}