Skip over navigation

keyword search: ITC

For:

• Students
• Instructors
• Researchers
• Staff
• IT Professionals

TotalView Support

TotalView is a full-featured, source-level, graphical debugger for applications written in C, C++, Fortran (77 and 90), assembler, and mixed source/assembler codes. It is a multiprocess, multithread debugger that supports multiple parallel programming paradigms including MPI, PVM and OpenMP.

The University has 2 floating network licenses for TotalView on the Sun Solaris, SGI IRIX, and IBM AIX platforms and 2 floating network licenses for the Linux platform.

Upgrade Announcement
Announcement concerning the most recent upgrade of TotalView.

Your code must be compiled appropriately to use Totalview. For most Unix compilers, the debug flag -g must be added to the compilation options, just as it would be for less sophisticated debuggers such as gdb. Optimization should also be suppressed, since optimization can change the code in ways that make it difficult for the debugger to interpret. Once the code has been recompiled and an executable generated, you are ready to invoke Totalview.

Access From Unix/Linux Platforms

1. The filesystem /common should be mounted from the Unix server jeeves.itc.virginia.edu to the Unix platforms (Sun Solaris, SGI IRIX, IBM AIX, Linux) that you are logged into. In addition, the directory for the appropriate platform (e.g. /solaris or /linux) should also be mounted from jeeves.itc and a symbolic link created from the uva subdirectory of the platform directory (e.g. /linux/i386/glibc2.2/uva) to /uva on the host machine. This is done by default on all ITC maintained Unix machines.

   The most recent version of TotalView is installed in the /common/toolworks/totalview directory.

2. To start TotalView, use the command totalview (/uva/bin should be part of your PATH variable by default).
   

3. If you are running TotalView remotely, make sure you use ssh to log into the remote machine so that the TotalView graphic dispay is properly directed to your machine, as well as for security reasons.

Using Totalview to Debug MPI Codes on the Linux Clusters

One of the most powerful features of Totalview is its ability to debug parallel codes. ITC's license permits debugging of MPI jobs on up to 2 nodes on any Linux cluster, Aspen, Birch, or Cedar.

Totalview's advanced features are available only under its X interface; hence you must be running an X server on your local workstation. Unix systems, such as Solaris or Linux, automatically include an X server. Under Windows you must run an X server such as eXceed.

If you can debug your code with short runs, you can use the Aspen or Birch frontends. In this case, only Ethernet can be used as the communications layer, so it may be necessary to recompile the code. To run an MPI job on the frontend interactively, mpirun must be used since mpiexec works only with PBS. To launch such a debugging run on the frontend (lc0, lc1, or lc2), use

mpirun -dbg=totalview -np 4 -machinefile myhosts mybinary

The file myhosts can be called anything desired, but it must consist of a single line of the form

localhost:4 

In this example, two processes will be placed on each cpu of the frontend, which may slow down the executable, although in most cases the penalty is not severe. Please use the frontends only for short debugging runs. If the job crashes, please run the commands

killall yourbinary 
cleanipcs 

afterward, where "yourbinary" is the name of the binary you ran.

In order to perform longer or more complex debugging of MPI applications on more than two processors, or on any number if using Myrinet on birch.itc, you must run Totalview from the nodes. The cluster nodes are behind a VPN (virtual private network) and therefore the usual method for setting the X display, X forwarding over ssh, does not work. The display must be enabled and set explicitly.

1. Under Unix: find out the fully-qualified name of your host by running hostname. As an example, suppose that the name returned is jellybean.itc.virginia.edu. From a terminal on jellybean, type

   xauth extract - jellybean.itc.virginia.edu:0 | ssh blue.unix xauth merge -
   
   

   Note: The TCP port used by X, usually 6000, must be open for the machines in use. For Linux users, if you are using the iptables recommended by ITC Linux support, add a rule such as

   -A UVAfw -s 128.143.0.0/16 -p tcp -m tcp --dport 6000 -m state --state NEW -j ACCEPT
   

   to /etc/sysconfig/iptables and restart iptables. (Have your system administrator do this for you if you do not administer your machine).

2. With eXceed under Windows: open a console window and at the prompt, type

   ipconfig
   

   This will return your IP address. For the purposes of this example, suppose it is 128.143.2.7

   If you are using Windows XP with the firewall enabled, which in general should be the case, you must also open the X11 TCP port. To do this, right-click your network connection and select Properties. Select the Advanced tab. This will bring up a window. Find the Settings button in the lower right and click on it. You must add a new connection so select Add. Name it X11. Always enter 127.0.0.1 as the Name or IP address of the computer hosting this service. Enter an number of 6000 for both internal and external ports and make sure that TCP is selected. Click OK and then OK again. Reboot your system to be sure the change takes effect. For more detailed instructions, see here.

Now log on to the frontend, e.g. birch.itc. The compute nodes can be accessed by requesting an interactive session from the queue manager, PBS. To do this, you can construct a trivial PBS script such as the following, which for purposes of this example will be called debug.sh:

#!/bin/sh
#PBS -l nodes=2:ppn=2

Now submit debug.sh with

qsub -I debug.sh

Another option is to provide resource options explicitly on the command line:

qsub -I -l nodes=2:ppn=2

If all resource requests are specified as options to qsub, no debug script is required; however, it is frequently more convenient to have a debug job script to reduce typing, and to make it easier to change resource requests. By whatever method you choose to submit your interactive job, you will be placed on a node in interactive mode, in your top-level home directory. Change to the directory where your executable and any input files reside. Now you must set your environment display. If you use ksh or bash, type

export DISPLAY=jellybean.itc.virginia.edu:0
or
export DISPLAY=128.143.2.7:0

depending on whether you know the name or only the IP address of your local machine. If you use tcsh, type

setenv DISPLAY jellybean.itc.virginia.edu:0

and similarly for an IP address. For an MPI job, Totalview must be invoked through the mpiexec (or mpirun) command. Please use mpiexec if possible; instructions for mpirun are included for completeness. If using mpirun, be sure your path is set correctly. If using Ethernet on either Aspen or Birch, type

mpiexec -totalview -comm mpich-p4 mybinary
or
mpirun -dbg=totalview -np 4 -machinefile $PBS_NODEFILE mybinary

If using Myrinet on Birch, type

mpiexec -totalview -comm mpich-gm mybinary

After you have completed your debugging session and have exited Totalview, be sure to exit from the shell on the compute node so that the node can be returned to PBS for other jobs. Exiting the shell terminates your interactive PBS job.

All TotalView documentation can be accessed online in HTML. The User's Guide and Reference Guide are also available in PDF format at the Etnus site.

Tutorial Documents

Getting Started with TotalView"
TotalView online tutorial accessed from the Etnus website.

LLNL TotalView Tutorial"
TotalView online tutorial accessed from Lawrence Livermore National Lab.

NERSC TotalView Tutorial"
TotalView online tutorial accessed from National Energy Research Supercomputing Center.

If users cannot find the answers to their questions in the online documentation, they can email their questions to ITC Research Computing Support.

TotalView FAQ
TotalView's FAQ at the Etnus website