Python is increasingly becoming a popular language for controlling large numerical simulations due to its scripting abilities and easy bindings with C, C++, and Fortran as provided by ctypes, Boost.Python, SWIG, etc. In addition, there are some nice convenience wrappers for MPI, including mpi4py.

However debugging MPI scripts can be challenging. Here are some useful ways to run your programs for debugging.

• Open an xterm window for each MPI process, with the script running in iPython.

  $ mpirun -np 4 xterm -e "ipython script.py"

• Open an xterm window for each MPI process, with gdb attached to each python process. The -x flag tells gdb to run the commands given in the specified file. This is often a good place to add additional breakpoints.

  $ echo "run script.py" > gdb.in
  $ mpirun -np 4 xterm -e "gdb -x gdb.in python"

• Open each MPI process within screen, then open a gnome-terminal with one tab for each screen.

  $ mpirun -np 4 screen -L -m -D -S mpi \
      ipython script.py &
  $ gnome-terminal --tab -e "screen -RR -p mpi" \
                   --tab -e "screen -RR -p mpi" \
                   --tab -e "screen -RR -p mpi" \
                   --tab -e "screen -RR -p mpi"

  If your program dies unexpectedly, it is probably because LD_LIBRARY_PATH is stripped by glibc since screen is a setgid/setuid program. You can work around this my modifying the mpirun call as

  $ mpirun -np 4 screen -L -m -D -S mpi \
      env LD_LIBRARY_PATH=$LD_LIBRARY_PATH ipython script.py