Debugging MPI + Python

Python is increasingly becoming a popular language for controlling large numerical simulations due to its scripting abilities and easy bindings with C, C++, and Fortran as provided by ctypes, Boost.Python, SWIG, etc. In addition, there are some nice convenience wrappers for MPI, including mpi4py.

However debugging MPI scripts can be challenging. Here are some useful ways to run your programs for debugging.

  • Open an xterm window for each MPI process, with the script running in iPython.

    $ mpirun -np 4 xterm -e "ipython script.py"
  • Open an xterm window for each MPI process, with gdb attached to each python process. The -x flag tells gdb to run the commands given in the specified file. This is often a good place to add additional breakpoints.

    $ echo "run script.py" > gdb.in
    $ mpirun -np 4 xterm -e "gdb -x gdb.in python"
  • Open each MPI process within screen, then open a gnome-terminal with one tab for each screen.

    $ mpirun -np 4 screen -L -m -D -S mpi 
        ipython script.py &
    $ gnome-terminal --tab -e "screen -RR -p mpi" 
                     --tab -e "screen -RR -p mpi" 
                     --tab -e "screen -RR -p mpi" 
                     --tab -e "screen -RR -p mpi"

    If your program dies unexpectedly, it is probably because LD_LIBRARY_PATH is stripped by glibc since screen is a setgid/setuid program. You can work around this my modifying the mpirun call as

    $ mpirun -np 4 screen -L -m -D -S mpi 
        env LD_LIBRARY_PATH=$LD_LIBRARY_PATH ipython script.py
About these ads

One thought on “Debugging MPI + Python

  1. I just found out that newer versions of gdb have an -ex flag which can be used to pass commands for gdb to run without creating a file.

    So these days I’m often using:
    mpirun -np 8 xterm -e gdb -ex run --args python script.py

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s