Monthly Archives: September 2011

Debugging MPI + Python

Python is increasingly becoming a popular language for controlling large numerical simulations due to its scripting abilities and easy bindings with C, C++, and Fortran as provided by ctypes, Boost.Python, SWIG, etc. In addition, there are some nice convenience wrappers for MPI, including mpi4py.

However debugging MPI scripts can be challenging. Here are some useful ways to run your programs for debugging.

  • Open an xterm window for each MPI process, with the script running in iPython.

    $ mpirun -np 4 xterm -e "ipython"
  • Open an xterm window for each MPI process, with gdb attached to each python process. The -x flag tells gdb to run the commands given in the specified file. This is often a good place to add additional breakpoints.

    $ echo "run" >
    $ mpirun -np 4 xterm -e "gdb -x python"
  • Open each MPI process within screen, then open a gnome-terminal with one tab for each screen.

    $ mpirun -np 4 screen -L -m -D -S mpi 
        ipython &
    $ gnome-terminal --tab -e "screen -RR -p mpi" 
                     --tab -e "screen -RR -p mpi" 
                     --tab -e "screen -RR -p mpi" 
                     --tab -e "screen -RR -p mpi"

    If your program dies unexpectedly, it is probably because LD_LIBRARY_PATH is stripped by glibc since screen is a setgid/setuid program. You can work around this my modifying the mpirun call as

    $ mpirun -np 4 screen -L -m -D -S mpi 
        env LD_LIBRARY_PATH=$LD_LIBRARY_PATH ipython