Thanks for the writeup, Emanuele Rocca. I think this shows the power of open source software. An obscure bug affecting only a handful of people can still get fixed, even without much help from the original developers.

pandoc --from=markdown --to=rst --output=README.rst README.md

The key to getting around this problem is to provide RealVNC with a complete list of screen resolutions you would like to have available when starting RealVNC. For example, my .vnc/config file contains

# Additional Resolutions
-randr 800x600,1024x768,1280x800,1280x960,1280x1024,1344x756,1680x1050,1920x1080,1920x1200,3360x1050,1024x700,1200x740,1600x1000,3200x1000,1680x1020,768x1024

After restarting RealVNC, you can easily change the screen resolution using the xrandr command. You can list screen resolutions by running xrandr:

$ xrandr
 SZ:    Pixels          Physical       Refresh
 0    800 x 600    ( 203mm x 152mm )   0
 1   1024 x 768    ( 260mm x 195mm )   0
 2   1280 x 800    ( 325mm x 203mm )   0
 3   1280 x 960    ( 325mm x 244mm )   0
 4   1280 x 1024   ( 325mm x 260mm )   0
 5   1344 x 756    ( 341mm x 192mm )   0
 6   1680 x 1050   ( 427mm x 267mm )   0
*7   1920 x 1080   ( 488mm x 274mm )  *0
 8   1920 x 1200   ( 488mm x 305mm )   0
 9   3360 x 1050   ( 853mm x 267mm )   0
 10  1024 x 700    ( 260mm x 178mm )   0
 11  1200 x 740    ( 305mm x 188mm )   0
 12  1600 x 1000   ( 406mm x 254mm )   0
 13  3200 x 1000   ( 813mm x 254mm )   0
 14  1680 x 1020   ( 427mm x 259mm )   0
 15   768 x 1024   ( 195mm x 260mm )   0
Current rotation - normal
Current reflection - none
Rotations possible - normal
Reflections possible - none

To switch to a different resolution just run xrandr -s <resolution>, where the resolution is either the item number, like 12, or resolution, like 1600x1000.

Afterwards, especially in recent versions of Ubuntu (12.04 and later), you may find that the background Nautilus desktop didn’t get the memo that the screen resolution was changed. To work around this, it’s often easiest to quit and restart Nautilus:

$ nautilus -q; sleep 1; nautilus -n > /dev/null 2>&1 & disown %

If you find yourself doing this a lot, as I did, you may want to consider writing a little script to automate the task. I named my script xres and have posted the source as a gist.

1. Disable the X Server Render extension. Copy /etc/vnc/config to /etc/vnc/config.custom and add

   -extension RENDER

2. Create a custom xstartup file. Copy /etc/vnc/xstartup to /etc/vnc/xstartup.custom and make the following changes:
   • Remove the line

     [ -x /etc/vnc/xstartup.custom ] && exec /etc/vnc/xstartup.custom

   • Add "gnome-fallback" to the list of valid sessions by replacing

     for SESSION in "ubuntu-2d" "2d-gnome"; do

     with

     for SESSION in "ubuntu-2d" "2d-gnome" "gnome-fallback"; do

   • Add to the top of the file:

     unset XDG_RUNTIME_DIR

I found it necessary to unset XDG_RUNTIME_DIR because I could not figure out a way to guarantee the directory it points to actually exists. When the directory did not exist I experienced 100% CPU utilization bugs in indicator-datetime-service and gnome-settings-daemon.

Briefly, the XDG_RUNTIME_DIR variable and directory that it points to are managed by a PAM module in libpam-xdg-support. When jsmith logs in over SSH, the PAM module creates a directory /run/users/jsmith and increments a session count stored in /run/users/.jsmith.lock. When jsmith later logs out, the session count is decremented. When the session count reaches zero the directory is deleted. Since RealVNC does not appear to use the PAM session machinery, there is no way to guarantee that the session count remains positive if only a VNC session is active.

class A(object):
    """Docstring for class."""
    def __init__(self, x):
        self.x = x
    @property
    def __doc__(self):
        return "My value of x is %s." % self.x

This gives the desired result:

>>> a = A(10)
>>> print(a.__doc__)
My value of x is 10.

But hides the docstring for the class A:

>>> A.__doc__
<property at 0x2083db8>

Read on to see how we can fix this.

To fix this, we can implement a custom class implementing the descriptor protocol. The default property descriptor behaves as:

class Property(object):
    ...
    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        if self.fget is None:
            raise AttributeError, "unreadable attribute"
        return self.fget(obj)

Here we see that if `obj is None`, that is, if we attempt to access `__doc__` on the class itself the property descriptor returns itself. To solve our problem we can instead return a string, perhaps something like:

class DocstringProperty(object):
    def __init__(self, fget, class_doc):
        self.fget = fget
        self.class_doc = class_doc
    def __get__(self, obj, objtype=None):
        if obj is None:
            return self.class_doc
        else:
            return self.fget(obj)

I’ve provided a fuller implementation and a function decorator in gist #4041015.

For example:

>>> class A(object):
...     """Main docstring"""
...     def __init__(self, x):
...         self.x = x
...     @docstring_property(__doc__)
...     def __doc__(self):
...         return "My value of x is %s." % self.x
>>> A.__doc__
'Main docstring'
>>> a = A(10)
>>> print(a.__doc__)
My value of x is 10.

def incx(self):
    self.x += 1

class Point(object):
    def __init__(self, x): self.x = x
    incx = incx

We can then create a point at the origin and increment x:

In [2]: p = Point(0)

In [3]: p.incx()

In [4]: print p.x
1

The same code which defines Point continues to work if we move incx to another file, say demo.py, and import it using from demo import incx.

But if we were to put incx in demo.pyx and compile it (using python setup.py build_ext --inplace), we get a strange error when running our simple test:

$ python bad.py
Traceback (most recent call last):
  File "bad.py", line 10, in <module>
    p.incx()
TypeError: incx() takes exactly one argument (0 given)

Read on to learn about instance methods and see how I fixed this.

My first thought was to write a simple wrapper function, using functools.wraps to preserve the docstring and name. This proved to be a clumsy solution.

A better approach turned out to be to use types.MethodType to convert our built-in function into an (unbound) instance method. The docstring for MethodType is pretty short:

instancemethod(function, instance, class)

The first and third arguments are obvious. For the second argument we’ll use None since we have no instance of the class yet, making this an unbound instance method.

So to fix our simple example, we need only replace the incx = incx line in the class definition with a short snippet after the class is defined:

from types import MethodType
Point.incx = MethodType(incx, None, Point)

We can give this new version a try and see that it works:
$ python good.py
1


As always, the entire example is available as a Gist.

$ python2.7
Python 2.7.2+ (...)
...
>>> 1/2
0

$ python3.2
Python 3.2.2 (...)
...
>>> 1/2
0.5

In this post, we’ll discuss how to configure the behavior of the division operator in Python 2.7 and IPython to behave like the Python 3.x counterpart.

We’ll call the division in Python 2.x “classic” or “old.” It does division in the C sense: an integer divided by an integer yields an integer, rounding down if needed.

The “new” division in Python 3.x instead computes the true division result. In this case, an integer divided by an integer yields a float. The floor division can be accessed in a new operator “//” as in:

>>> 1 // 2
0

To allow for a transition period, two compatibility features are provided. First, module developers can add

from __future__ import division

to the top of any .py file to indicate that “/” should mean true division in that file.

In addition, the Python interpreter can be passed a flag to indicate the desired division behavior:

-Q arg : division options: -Qold (default), -Qwarn, -Qwarnall, -Qnew

So for example, we can run:

$ python2.7 -Qnew
...
>>> 1/2
0.5

But what about IPython? There isn’t a corresponding flag on the ipython script, yet. In discussing adding this feature to IPython, Fernando Perez provided a useful tip for accomplishing this using the IPython configuration files.

To begin, you’ll need to create an IPython profile by running

$ ipython profile create

Then edit the configuration file, which for me was located at ~/.config/ipython/profile_default/ipython_config.py, and uncomment and add the following:

# lines of code to run at IPython startup.                                                                                                                 
c.InteractiveShellApp.exec_lines = [
    'from __future__ import division'
]

We can test it out and see that it does in fact work as desired:

$ ipython
Python 2.7.2+ (default, Oct  4 2011, 20:06:09) 
Type "copyright", "credits" or "license" for more information.

IPython 0.13.dev -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: 1/2
Out[1]: 0.5

But when I tried to run an existing script in the new IPython Qt Console, the usual overly verbose set of diagnostic messages did not appear! The symtoms were:

• Regular print statements in Python worked fine.
• printf statements in C-code wrapped with Boost.Python did not appear.

Read on to see a solution.

This seemed quite odd until I read more about the architecture of IPython Qt Console. In particular, IPython Qt Console is a two-process architecture whereby a background IPython process executes code and frontend client handles all user input and output. So our printf statements were being sent to stdout in the background process and therefore never appear in the Qt Console window.

Somebody must have experienced this before, but a few quick Google searched turned up nothing. How would Cython handle this? In particular, we could use Cython to compile a simple Python print statement. So consider the simple printf.pyx file:

def printf():
    print "Hello"

which we compile into C using cython printf.pyx. The resulting C file is quite lengthy, but the relevant bits are in the __Pyx_PrintOne() function:

static int __Pyx_PrintOne(PyObject* f, PyObject *o) {
    if (!f) {
        if (!(f = __Pyx_GetStdout()))
            return -1;
    }
    if (PyFile_SoftSpace(f, 0)) {
        if (PyFile_WriteString(" ", f) < 0)
            return -1;
    }
    if (PyFile_WriteObject(o, f, Py_PRINT_RAW) < 0)
        return -1;
    if (PyFile_WriteString("\n", f) < 0)
        return -1;
    return 0;
    /* the line below is just to avoid compiler
     * compiler warnings about unused functions */
    return __Pyx_Print(f, NULL, 0);
}

The __Pyx_GetStdout() function is nothing more than a thin wrapper to PySys_GetObject((char *)"stdout") with a bit of error checking.

Putting this all together, a simple reporting function (without any error checking) becomes:

void report(std::string msg) {
  PyObject *f = PySys_GetObject((char *)"stdout");
  PyFile_WriteString(msg.c_str(), f);
  PyFile_WriteString("\n", f);
}

For an example module using this code, see gist 1284927.

Update: After writing this, I learned of PySys_WriteStdout() which is a nearly drop-in replacement of printf. However the above method is still be useful if you need to print more than 1000 bytes at a time.

However if you need to wrap a function which takes a FILE* argument the previous approach will not prove fruitful. But Boost.Python is clearly capable of handling a similar situation, namely the implicit conversion from a Python string type to a char* type. Since the conversion from the internal PyObject* to char* is likely to be done with the PyString_AsString function, I went off in search of that code.

There were no references in the installed headers, but it did appear in the source code in converter/builtin_converters.cpp in the convert_to_cstring() function:

// An lvalue conversion function which extracts a char const* from a
// Python String.
void* convert_to_cstring(PyObject* obj)
{
  return PyString_Check(obj) ? PyString_AsString(obj) : 0;
}

… which was later called as …

// Add an lvalue converter for char which gets us char const*
registry::insert
  (convert_to_cstring, type_id(),
   &converter::wrap_pytype<&PyString_Type>::get_pytype);

Jackpot! We should be able to mimic these lines to create a Python file to FILE* converter using PyFile_AsFile():

#include <boost/python.hpp>

namespace {
  void *convert_to_FILEptr(PyObject* obj) {
    return PyFile_Check(obj) ? PyFile_AsFile(obj) : 0;
  }
}

BOOST_PYTHON_MODULE(file_wrapper) {
  boost::python::converter::registry::insert
    (convert_to_FILEptr,
     boost::python::type_id<FILE>(),
     &boost::python::converter::wrap_pytype<&PyFile_Type>::get_pytype);
}

For a complete example including test code, see gist: 1265889.

Edit: Of course this also works with boost::python::extract<FILE*>. I’ve updated the gist to demonstrate the relevant usage.

However debugging MPI scripts can be challenging. Here are some useful ways to run your programs for debugging.

• Open an xterm window for each MPI process, with the script running in iPython.

  $ mpirun -np 4 xterm -e "ipython script.py"

• Open an xterm window for each MPI process, with gdb attached to each python process. The -x flag tells gdb to run the commands given in the specified file. This is often a good place to add additional breakpoints.

  $ echo "run script.py" > gdb.in
  $ mpirun -np 4 xterm -e "gdb -x gdb.in python"

• Open each MPI process within screen, then open a gnome-terminal with one tab for each screen.

  $ mpirun -np 4 screen -L -m -D -S mpi \
      ipython script.py &
  $ gnome-terminal --tab -e "screen -RR -p mpi" \
                   --tab -e "screen -RR -p mpi" \
                   --tab -e "screen -RR -p mpi" \
                   --tab -e "screen -RR -p mpi"

  If your program dies unexpectedly, it is probably because LD_LIBRARY_PATH is stripped by glibc since screen is a setgid/setuid program. You can work around this my modifying the mpirun call as

  $ mpirun -np 4 screen -L -m -D -S mpi \
      env LD_LIBRARY_PATH=$LD_LIBRARY_PATH ipython script.py