Parallel Programming with Matrix Distributed Processing
Massimo Di Pierro
%EndAName {{\footnotesize School of Computer Science
T% elecommunications and Information Systems}}
Matrix Distributed Processing (MDP)\ is a C++ library for fast development of efficient parallel algorithms. MDP enables programmers to focus on algorithms, while parallelization is dealt with automatically and transparently. Here we present a brief overview of MDP and examples of applications in Computer Science (Cellular Automata), Engineering (PDE Solver) and Physics (Ising Model).
hipc05.tex

\documentclass[11pt]{article}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{times}

%TCIDATA{Created=Thu Mar 24 17:57:52 2005}
%TCIDATA{LastRevised=Sun May 08 23:58:29 2005}
%TCIDATA{Language=American English}

\setlength{\textwidth}{6.5in}
\setlength{\textheight}{9.0in}
\setlength{\topmargin}{-.5in}
\setlength{\oddsidemargin}{-.0600in}
\setlength{\evensidemargin}{.0625in}
\input{tcilatex}
\begin{document}

\title{Parallel Programming with Matrix Distributed Processing}
\author{Massimo Di Pierro \\
%EndAName
{{\footnotesize School of \textbf{C}omputer Science, \textbf{T}%
elecommunications and \textbf{I}nformation Systems}}\\
{\footnotesize DePaul University, 243 S. Wabash Av., Chicago, IL 60604, USA}}
\maketitle

\begin{abstract}
Matrix Distributed Processing (MDP)\ is a C++ library for fast development
of efficient parallel algorithms. \texttt{MDP} enables programmers to focus
on algorithms, while parallelization is dealt with automatically and
transparently. Here we present a brief overview of \texttt{MDP} and examples
of applications in Computer Science (Cellular Automata), Engineering (PDE
Solver) and Physics (Ising Model).
\end{abstract}

\section{Introduction}

Matrix Distributed Processing (\texttt{MDP}) \cite{mdp1}\cite{mdp2} is a
collection of classes and functions written in C++ for fast development of
parallel algorithms such as solvers for partial differential equations,
mesh-like algorithms, and various types of graph-based problems. These
algorithms find frequent application in many sectors of physics,
engineering, electronics and computational finance.

\texttt{MDP} includes:

\begin{itemize}
\item  a natural syntax for the algorithms that is transparent to the
underlying parallelization;

\item  parallelization based optimization algorithms implemented in MPI;

\item  functions for linear algebra computations with support of a
Maple-like syntax;

\item  statistical functions;

\item  fitting functions;

\item  a Parallel SIMulator (PSIM) which allows \texttt{MDP} algorithms to
run on single processor machines without MPI (uses fork for creating the
processes and socket pairs for communication).
\end{itemize}

\texttt{MDP} was originally developed for and still constitutes the core of
FermiQCD\cite{fermiqcd2002}\cite{fermiqcd2003}, a library for Lattice QCD
computations. Lattice QCD is a Monte Carlo based numerical approach to the
physics of composite particles made of quarks (e.g., protons and neutrons)
and it is considered one of the most computationally intensive projects of
modern physics.

FermiQCD, was developed by the author at the University of Southampton (UK)
and Fermilab (Department of Energy) and it is currently used by other
physics departments around the world\cite{app1}\cite{app2}\cite{app3}\cite
{app4}\cite{app5}.

While Lattice QCD is currently one of the main applications of \texttt{MDP},
its range of applicability is not limited to it as we show in the following
examples.

\section{Example: Parallel Game of Life}

As a first example we show here how to write, in a few lines, a parallel
program to find stable configurations for the Game Of Life. The game works
as follows:

\begin{itemize}
\item  It is defined on a board ($N\times N$ cells) with periodic boundary
conditions.

\item  Each cell can be alive (1) or dead (0).

\item  The game consists of a series of iterations and, at each iteration,
if a cell is dead and the number of beighbor cells who are alive is greater
or equal 3, the cell dies; if a cell is alive and the number of beighbor
cells who are alive is 2 or 3, the cell remains alive; otherwise the cell
will be dead.

\item  Given a random starting configuration for the board, we want to
determine its evolution and whether it reaches a stable configuration.
\end{itemize}

Here is the entire parallel code:
\begin{verbatim}
00  #include "mdp.h"         
01
02  const int alive=1, dead=0;         // some constants
03
04  int rulesofgame(int a00, int a01, int a02,
05                  int a10, int a11, int a12,
06                  int a20, int a21, int a22) {
07    int sum=a00+a01+a02+a10+a12+a20+a21+a22;
08    if(a11==dead && sum==3) return alive;
09    else if(a11==alive && (sum==2 || sum==3)) return alive;
10    return read;
11  }                                  // rules of the game         
12
13  int main(int argc, char **argv) {
14    mdp.open_wormholes(argc,argv);   // open communication
15    int L[]={10,10};                 // declare board size
16    mdp_lattice board(2,L);         // declare board
17    mdp_field<int> C(board);         // declare cells
18    mdp_field<int> newC(board);      // declare new set of cells
19    mdp_site x(board);             
20
21    forallsites(x) C(x)=(board.random(x).plain()>0.5)?alive:dead;
22
23    while(1) {                       // game iteration
24      C.update();                    // communicate!
25      forallsites(x)                 // parallel loop
26        newC(x)=rulseofgame(C((x-1)-0),C(x-1),C((x-1)+0),
27                            C(x-0),    C(x),  C(x+0),
28                            C((x+1)-0),C(x+1),C((x+1)+0));  
29      int diff=0;
30      forallsites(x) {
31        diff+=abs(C(x)-newC(x));     // count changed cells
32        C(x)=newC(x);                // store new cells
33      }
34      if(diff==0) break;             // exit if cells didn't change
35    }
36    C.save("cells.dat")              // save cells
37    mdp.close_wormholes();
38    return 0;
39  }
\end{verbatim}

\begin{itemize}
\item  Line 00 includes the basic library.

\item  Lines 04-11 define the rule of the game. The values of a cell (%
\texttt{a11}) and its neighbors are passed as a $3\times 3$ table (\texttt{%
a00-a22}). The function returns the new value for \texttt{a11}.

\item  Line 14 opens the parallel communication channels, line 37 closes
them.

\item  Line 15 declares a 2D array containing the board size, \texttt{L}.

\item  Line 16 declares a lattice object (board) that represents the set of
board sites and its topology. By default a mesh topology with toroid
boundary conditions. Two arguments are passed to the constructor: the
dimension, and size of the board. For each board site, it is possible to
specify on which parallel process it is going to be allocated as well as
alternative topologies.

\item  Line 17 declares a field of cells (\texttt{C}) on the board (the
cells that live on it).

\item  Line 18 declares an auxiliary field that will be necessary for the
computation.

\item  Line 19 declares a \texttt{x} a variable that represents a generic
site of the board and will be used for looping over the sites and therefore
the cells that live on them.

\item  Line 21 sets the initial configuration of the board by looping over
all sites \texttt{x} and setting the corresponding cell \texttt{C(x)} to a
random \texttt{alive} or \texttt{dead}. Each process loops only over the
sites that are allocated locally by the process. Notice how MDP provides a
local random number generator for each site of the board, \texttt{%
board.random(x)}, which is vital in order to be able to reproduce results of
stochastic algorithms.

\item  Lines 23-35 loop over the iterations of the game.

\item  Line 24 is performs a critical operation; it informs MDP that the
value of the field \texttt{C} has changed and parallel communication is
needed to synchronize those buffers that contain copies of non-local sites (%
\emph{synchronization}).

\item  Lines 25-28 apply the rules of the game at each site. The new values
of the states for the cells are stored in the auxiliary field \texttt{newC}.
Each process loops over the local sites only. For each local site (\texttt{x}%
), \texttt{x+0} represents the neighbor site when coordinate 0 is
incremented by one, \texttt{x-0} represents the neighbor site when
coordinate 0 is decremented by one, \texttt{x+1} represents the neighbor
site when coordinate 1 is incremented by one, etc.\footnote{%
This notation may appear bizzarre but it is nothing more than a sum ($%
\overrightarrow{x}+\overrightarrow{i}$) or difference ($\overrightarrow{x}-%
\overrightarrow{i}$) of vectors where $\overrightarrow{x}$ is a vector that
represents a site on the board and the integer $\overrightarrow{i}$
represents a unit verson in direction $i.$}.

\item  Lines 29-33 perform two operations in a single parallel loop: the new
states for the cells (\texttt{newC}) are copied back into \texttt{C}; the
number of cells that have changed state are counted and the number is stored
in \texttt{diff}.

\item  Line 34 terminates the iteration if no cell has changed its state.

\item  Line 36 saves the board configuration in a file \texttt{cells.dat}.
\end{itemize}

Most of the parallel work is done by the constructor (\texttt{mdp\_lattice})
of the \texttt{board} which, from the board topology, determines how to
partition it and determines the optimal communication patterns; by the
constructor (\texttt{mdp\_field}) of the field of cells (\texttt{C}) which
allocates the local cells and the buffers to store copies of non-local
cells; and by the method \texttt{update} which performs communication to
copy remote cell values into the local buffers.

We'll show in the next few examples that is equally easy to implement fields
of any class of objects. In the next example we'll consider a field of
matrices.

\section{Example: Parallel PDE Solver}

Consider here, as a different example that presents many similarities with
the one above, the following Laplace equation: 
\begin{equation}
\nabla ^2\varphi (x)=f(x)  \label{eq1}
\end{equation}
where $\varphi (x)$ is a field of $2\times 2$ complex matrices defined on a
3D space (\texttt{space}), $x=(x_0,x_1,x_2)$ limited by $0\leq x_i<L_i$, and 
\begin{eqnarray}
L &=&\{10,10,10\} \\
f(x) &=&A\sin (2\pi x_1/L_1)  \nonumber \\
A &=&\left( 
\begin{array}{ll}
1 & i \\ 
3 & 1
\end{array}
\right)  \nonumber
\end{eqnarray}
The initial conditions are $\varphi _{initial}(x)=0.$ We will also assume
that $x_i+L_i=x_i$ (torus topology).

\textbf{Solution:} In order to solve eq.~(\ref{eq1}) we first discretize the
Laplacian ($\nabla ^2=\partial _0^2+\partial _1^2+\partial _2^2$) and
rewrite it as 
\begin{equation}
\sum_{\mu =0,1,2}\left[ \varphi (x+\widehat{\mu })-2\varphi (x)+\varphi (x-%
\widehat{\mu })\right] =f(x)
\end{equation}

where $\widehat{\mu }$ is a unit vector in the discretized space in
direction $\mu $. Hence we solve for $\varphi (x)$ and obtain the following
recurrence relation 
\begin{equation}
\varphi (x)=\frac{\sum_{\mu =0,1,2}\left[ \varphi (x+\widehat{\mu })+\varphi
(x-\widehat{\mu })\right] -f(x)}6  \label{eq3}
\end{equation}

The following is a typical \texttt{MDP} program that solves eq.~(\ref{eq1})
by recursively iterating eq.~(\ref{eq3}). Notice how the program is parallel
but there are no explicit calls to communication functions:
\begin{verbatim}
00  #include "mdp.h"
01
02  void main(int argc, char** argv) {
03     mdp.open_wormholes(argc,argv);    // open communications
04     int L[]={10,10,10};               // declare volume
05     mdp_lattice      space(3,L);      // declare lattice
06     mdp_site         x(space);        // declare site variable
07     mdp_matrix_field phi(space,2,2);  // declare field of 2x2
08     mdp_matrix       A(2,2);          // declare matrix A
09     A(0,0)=1;  A(0,1)=I;
10     A(1,0)=3;  A(1,1)=1;
11     forallsites(x)                    // loop (in parallel)
12        phi(x)=0;                      // initialize the field
13     phi.update();                     // communicate!
14
15     for(int i=0; i<1000; i++) {       // iterate 1000 times
16        forallsites(x)                 // loop (in parallel)
17           phi(x)=(phi(x+0)+phi(x-0)+
18                   phi(x+1)+phi(x-1)+
19                   phi(x+2)+phi(x-2)-
20                   A*sin(2.0*Pi*x(1)/L[1]))/6;  // the equation
21        phi.update();                  // communicate!
22     }
23     phi.save("field_phi.mdp");        // save field
24     mdp.close_wormholes();            // close communications
25  }
\end{verbatim}

\textbf{Notes:}

\begin{itemize}
\item[$\bullet$]  Line 04 declares the size of the box used to approximate
the space $L=\{L_0,L_1,L_2\}$.

\item[$\bullet$]  Line 05 declares a 3-dimensional lattice, called \texttt{%
space}, on the box $L$. \texttt{MDP} supports up to 10-dimensional lattices.
By default, a lattice object is a mesh with torus topology.

\item[$\bullet$]  Line 06 declares a site variable site \texttt{x} that will
be used to loop over the lattice.

\item[$\bullet$]  Line 07 declares a field of $2\times 2$ matrices, called 
\texttt{phi}, over the lattice \texttt{space}.

\item[$\bullet$]  Lines 08-10 define the matrix A.

\item[$\bullet$]  Lines 11-12 initialize the field \texttt{phi}. Notice that 
\texttt{phi} is distributed over the parallel processes and \texttt{%
forallsites} is a parallel loop.

\item[$\bullet$]  Line 13 performs \emph{synchronization}.

\item[$\bullet$]  Lines 15 through 23 perform 1000 iterations to guarantee
convergence.

\item[$\bullet$]  Line 16 loops over all sites in parallel.

\item[$\bullet$]  Lines 17 through 20 implement eq.~(\ref{eq3}). Notice the
similarity in notation. Here \texttt{phi(x)} is a $2\times 2$ complex matrix.

\item[$\bullet$]  Line 21 performs \emph{synchronization}.

\item[$\bullet$]  Line 23 saves the field. Notice that all fields, including
user-defined ones, inherit \texttt{save} and \texttt{load} methods from the
basic \texttt{mdp\_field} class.
\end{itemize}

\section{Example: Parallel Ising Model}

As one more example of usage of \texttt{MDP} we report here a simple program
for the Ising model.
\begin{verbatim}
00  #include "mdp.h"
01  void main(int argc, char** argv) {
02    mdp.open_wormholes(argc,argv);
03    int L[]={100};
04    mdp_lattice line(1,L);          // declare the lattice
05    mdp_field<int> spin(line);      // declare the spin variables
06    mdp_site x(line);
07    int dE=0, M=L[0], dM=0;         // E for Energy, M for Magnetization
08    float kappa=2.0;                // inverse temperature
09    forallsites(x) spin(x)=+1;      // set initial conditions
10    while(1) {
11      dM=0;
12      for(int parity=EVEN; parity<=ODD; parity++) {
13        forallsitesofparity(x,parity) {
14          dE=2*spin(x)*(spin(x-0)+spin(x+0));   // compute energy variation
15          if(exp(-kappa*dE)>mdp_random.plain()) // Monte Carlo accept-reject
16            { spin(x)*=-1; dM=dM+2*spin(x); }
17        }
18        spin.update(parity);       // communicate
19      }
20      mdp.add(dM);
21      M=M+dM;                      // compute new value for the energy
22      mdp << "magnetization=" << M << endl;
23    }
24    mdp.close_wormholes();
25  }
\end{verbatim}

In this example:

\begin{itemize}
\item  Lines 3-4 declare a 1D lattice of 100 points (\texttt{line}).

\item  Line 5 declares a field of integers (\texttt{spin}) on the lattice.

\item  Line 7 sets the total magnetization M for this initial spin
configuration.

\item  Line 9 sets the initial configuration: all field variables equal to 1.

\item  Line 14 computes the energy variation (\texttt{dE}) of each Markov
Chain Monte Carlo (MCMC)\ step.

\item  Lines 15-16 perform the Monte Carlo accept-reject. If a change is
accepted the spin at site \texttt{x} is flipped and the total magnetization 
\texttt{M} changes (line 16).
\end{itemize}

Note how at each MCMC step, first the code tries to flip the spins at even
locations then, after it updates the lattice sites, it tries to flip the
spins at odd locations (line 13). This guarantees computation results are
independent on parallelization of the lattice line.

Since this even-odd distinction is common in many lattice algorithms, 
\texttt{MDP} stores all even lattice sites and all odd lattice sites close
together. This speeds up loops over one of the two subsets and also speeds
up communication. In fact, in this example, we are able to limit the
synchronization (update) to the site of a given parity (line 18).

\section{Linear Algebra and Other Tools}

\texttt{MDP} includes a Linear Algebra package and other tools. Some of the
most important classes are:

\begin{itemize}
\item  class \texttt{mdp\_real}, that should be used in place of float or
double;

\item  class \texttt{mdp\_complex}, for complex numbers;

\item  class \texttt{mdp\_array}, for vectors and/or multidimensional
tensors;

\item  class \texttt{mdp\_matrix}, for any kind of complex rectangular
matrix;

\item  class \texttt{mdp\_measure}, for error propagation;

\item  class \texttt{mdp\_jackboot}, a container for jackknife and bootstrap
algorithms.
\end{itemize}

The most notable difference between our linear algebra package and other
existing packages is its natural syntax.

For example:
\begin{verbatim}
mdp_matrix A,B;
A=Random.SU(7);
B=exp(A)+inv(A)*hermitian(A)+5;
\end{verbatim}

reads like

$
\begin{tabular}{l}
$A$ and $B$ are matrices \\ 
$A$ is a random $SU(7)$ matrix \\ 
$B=e^A+A^{-1}A^H+5\cdot \mathbf{1}$%
\end{tabular}
$

Note that each matrix can be resized at will and is resized automatically
when a value is assigned.

\texttt{MDP} includes functions for fitting such as the
Levenberger-Marquardt algorithm.

\section{Lattice, Site, and Field}

An \texttt{mdp\_lattice} is the class that describes the space on which
fields are defined; it stores the \emph{topology} of the space (by default
that of a torus in $d$ dimensions) and information about \emph{partioning}
of the space over the parallel processes. In \texttt{MDP}, a lattice is a
graph, defined as a collection of points (lattice \emph{sites}) connected by
links (they specify the topology). Each site is uniquely mapped to one of
the parallel processes.

The only restriction is that the graph must have a degree less than 20. From
now on we will assume the default topology of a torus; therefore the lattice
should be thought of as a mesh in $d\leq 10$ dimensions.

The constructor class \texttt{mdp\_lattice} determines on which process to
store each site, determines the neighbors of each site, and determines the
sizes of the buffers where each process keeps copies of those sites that are
non-local but are neighbors of the local sites.

The constructor also allocates a parallel random number generator so that
each site of the lattice has its own independent random number generator.
This is important for parallel Monte Carlo applications of \texttt{MDP} and
ensures reproducibility of computations on different architectures.

On each lattice it is possible to allocate fields. Some fields are built-in,
for example
\begin{verbatim}
mdp_complex_field
\end{verbatim}

i.e. the field of complex numbers.\ The user can declare any type of field.
For example a field of 5 float per lattice site:
\begin{verbatim}
class S {
 public: float S[5];
};
int L[]={10,10,10};
mdp_lattice cube(3,L);
mdp_field<S> psi(cube);
\end{verbatim}

This code declares a $10\times 10\times 10$ lattice (\texttt{cube}) and a
field (\texttt{psi}), that lives on the \texttt{cube}. The site variables of 
\texttt{psi}, \texttt{psi(x)} belong to class \texttt{S} (assuming \texttt{x}
is an \texttt{mdp\_site} on the \texttt{cube}).

User-defined fields can be saved, loaded, and synchronized
\begin{verbatim}
psi.save("filename");
psi.load("filename");
psi.update();
\end{verbatim}

Synchronization means that all processes will perform MPI communications to
make sure all buffers that contain copies of non-local site variables are
updated with the proper values. The method update should be called
immediately after the local site variables of a field have been changed.

Once a field object is declared in the field constructor, each process
dynamically allocates memory for the buffers that store the copies of those
sites that are non-local, but are neighbors of the local sites. These
buffers are created in such as way to ensure optimal communication patterns.

Every time a field changes, for example in a parallel loop such as
\begin{verbatim}
forallsites(x) phi(x)=0;
\end{verbatim}

the program notifies the field that its values have been changed by calling
\begin{verbatim}
phi.update();
\end{verbatim}

The method \texttt{update} performs all required communication to copy site
variables that need to be synchronized between each couple of overlapping
processes.

Lattice sites are represented by objects of class \texttt{mdp\_site}. Site
objects can be looped over in parallel loops (such as \texttt{forallsites})
but it also possible to explicitly address a specific site by specifying the
site coordinates. Obviously, only the process that stores a site locally
should address a specific site. Class \texttt{mdp\_site} has methods to
check if a site is local, if it is non-local but a local copy is present,
and which process stores the site locally.

\section{Optimal Communication Patterns}

In \texttt{MDP}, the lattice objects, according to the lattice topology and
the parallel partitioning, determine the optimal way to store site variables
in memory and performing parallel communication. This information is then
used by the field method \texttt{update} that performs the synchronization
of the field variables.

Note that \texttt{MDP} does not attempt to overlap computation and
communication. By ``optimal communication'' pattern we mean that, under the
assumptions below, the method update minimizes network traffic and data
copies.

Current optimizations are based on the assumption that each processing node
has one and only one network card and that the network is isotropic (latency
and bandwidth for each couple of nodes is the same). This assumption is
generally true for Ethernet and Myrinet clusters.

Communications are optimal in the sense that:

\begin{itemize}
\item  Each process retrieves all non-local site variables in a single
send/recv for each process that contains sites which are neighbors of local
sites.

\item  Two processes that do not store neighbor sites do not communicate
with each other.

\item  No process is involved in more than a single send and a single
receive at one time.

\item  Each process stores close in memory those copies of non-local site
variables which are local to the same process. In this way synchronization
does not require the use of additional buffers receiving buffers.
\end{itemize}

This technque is particularly efficient for algorithms that only require
next-neighbor communication and run on a all-to-all network topology such as
Ethernet or Myrinet.

It is possible, in principle, to change the above communication patterns to
optimize communication for other network topologies.

Although communications are currently based on MPI, they do not make use of
communication tags. It is therefore possible, in principle, to speed-up
communication by using a faster tagless and bufferless protocol such as
Myricom GM.

Our communication patterns have the effect of making communication almost
insensitive to network latency, and communication speed is dominated by
network bandwidth. Benchmarks are very much application dependent since
parallel efficiency is greatly affected by the lattice size, by the amount
of computation performed per site, processor speed and type of
interconnection. In many typical applications, like the one described in the
preceding example, the drop in efficiency is less than 10\% up to 8 nodes
(processes) and less than 20\% up to 32 (our tests are usually performed on
a cluster of Pentium 4 PCs (2.2GHz) running Linux and connected by Myrinet).

\section{MDP and PSIM}

For portability reasons, \texttt{MDP} is based on MPI. Nevertheless it is
desirable to be able to run, test and debug \texttt{MDP}\ programs on a
single node (with single or multi processor architecture) without having to
install MPI. The latest version of \texttt{MDP} includes a Parallel
SIMulator (PSIM). Despite the name, this is not quite a simulator but an
emulator, i.e., a message passing library that uses local (unix/posix)
socket pairs. PSIM is Objected Oriented and is not based on MPI.

When compiling with PSIM, the parallel processes are created at start-up by
forking. The number of parallel processes is specified at runtime by passing
the following command line argument to any \texttt{MDP} executable program,
\begin{verbatim}
-PSIM_NPROCS=4
\end{verbatim}

(this makes 4 parallel processes).

PSIM also creates a communication log that can be used for debugging. 
\texttt{MDP} with PSIM has been tested on Linux, Mac and Windows (with
cygwin).

For single processor node, using PSIM does not introduce any speed-up, but,
for a small number of processes (2-16) it does not slow down the code
either. For multi-process shared memory architecture, parallelization of
PSIM should produce a speed-up comparable with MPI. We have not been yet
performed such tests.

Moreover, PSIM should perform well on openMosix clusters as soon as
openMosix starts supporting migratable sockets since all communication
between the parallel processes will be done by the operating system.
Unfortunately openMosix does not support migratable sockets yet.

\section{Conclusions}

\texttt{MDP}\ is an easy, powerful, and reliable tool for developing
efficient parallel numerical applications. We have shown here examples in
Computer Science (Cellular Automata), Mathematics (PDE solver) and Physics\
(Ising model).

\texttt{MDP} enables the programmer to focus on algorithm design while
parallelization issues are dealt with automatically in a way transparent to
the programmer.

The underlying communication functions are written in MPI but it is possible
also to compile it without MPI using the built-in PSIM emulator which
enables one to run parallel processes on a single node and/or single
processor architectures, such as a PC. This is useful for testing and
debugging purposes.

All of the features here described are fully functional and have been tested
in real-life applications such as FermiQCD, developed by the University of
Southampton (UK) and Fermilab (Department of Energy).

\smallskip \textbf{\smallskip Acknowledgements}

The author wishes to thank DePaul University for supporting his research and
Fermilab for the fruitful collaboration based on it. He also thanks John
Novak for insightful comments about this manuscript.

\smallskip \textbf{Code download}

\smallskip MDP, PSIM and FermiQCD are currently distributed together and can
be downloaded from:

\texttt{http://www.fermiqcd.net}

\begin{thebibliography}{9}
\bibitem{mdp1}  M. Di Pierro, Computer Physics Communcations 141:98-148, 2001

\bibitem{mdp2}  M. Di Pierro. Proceedings ICCSA 2003

\bibitem{fermiqcd2002}  M. Di Pierro, Nucl. Phys. Proc. Suppl. 106:1034-1036%
\textbf{,} 2002

\bibitem{fermiqcd2003}  M. Di Pierro et al., www.fermiqcd.net,
Nucl.Phys.Proc.Suppl.129:832-834,2004 [hep-lat/0311027]

\bibitem{app1}  J. Simone at al., Nucl.Phys.Proc.Suppl.140:443-445,2005

\bibitem{app2}  M. Okamoto et al., Nucl.Phys.Proc.Suppl.140:461-463,2005

\bibitem{app3}  C. Aubin et al. Phys.Rev.Lett.94:011601,2005

\bibitem{app4}  L. Li and Y. Meurice, Phys.Rev.D71:054509,2005

\bibitem{app5}  J. Flynn et al. JHEP 0411:049,2004
\end{thebibliography}

\end{document}

tcilatex.tex

%%%%%%%%%%%%%%%%%%%% Start \macros\tci\tcilatex.tex %%%%%%%%%%%%%%%%%%%

% Macros for Scientific Word 2.5 documents saved with the LaTeX filter.
%Copyright (C) 1994-95 TCI Software Research, Inc.
\typeout{TCILATEX Macros for Scientific Word 2.5 <22 Dec 95>.}
\typeout{NOTICE:  This macro file is NOT proprietary and may be 
freely copied and distributed.}
%
\makeatletter
%
%%%%%%%%%%%%%%%%%%%%%%
% macros for time
\newcount\@hour\newcount\@minute\chardef\@x10\chardef\@xv60
\def\tcitime{
\def\@time{%
  \@minute\time\@hour\@minute\divide\@hour\@xv
  \ifnum\@hour<\@x 0\fi\the\@hour:%
  \multiply\@hour\@xv\advance\@minute-\@hour
  \ifnum\@minute<\@x 0\fi\the\@minute
  }}%

%%%%%%%%%%%%%%%%%%%%%%
% macro for hyperref
\@ifundefined{hyperref}{\def\hyperref#1#2#3#4{#2\ref{#4}#3}}{}

% macro for external program call
\@ifundefined{qExtProgCall}{\def\qExtProgCall#1#2#3#4#5#6{\relax}}{}
%%%%%%%%%%%%%%%%%%%%%%
%
% macros for graphics
%
\def\FILENAME#1{#1}%
%
\def\QCTOpt[#1]#2{%
  \def\QCTOptB{#1}
  \def\QCTOptA{#2}
}
\def\QCTNOpt#1{%
  \def\QCTOptA{#1}
  \let\QCTOptB\empty
}
\def\Qct{%
  \@ifnextchar[{%
    \QCTOpt}{\QCTNOpt}
}
\def\QCBOpt[#1]#2{%
  \def\QCBOptB{#1}
  \def\QCBOptA{#2}
}
\def\QCBNOpt#1{%
  \def\QCBOptA{#1}
  \let\QCBOptB\empty
}
\def\Qcb{%
  \@ifnextchar[{%
    \QCBOpt}{\QCBNOpt}
}
\def\PrepCapArgs{%
  \ifx\QCBOptA\empty
    \ifx\QCTOptA\empty
      {}%
    \else
      \ifx\QCTOptB\empty
        {\QCTOptA}%
      \else
        [\QCTOptB]{\QCTOptA}%
      \fi
    \fi
  \else
    \ifx\QCBOptA\empty
      {}%
    \else
      \ifx\QCBOptB\empty
        {\QCBOptA}%
      \else
        [\QCBOptB]{\QCBOptA}%
      \fi
    \fi
  \fi
}
\newcount\GRAPHICSTYPE
%\GRAPHICSTYPE 0 is for TurboTeX
%\GRAPHICSTYPE 1 is for DVIWindo (PostScript)
%%%(removed)%\GRAPHICSTYPE 2 is for psfig (PostScript)
\GRAPHICSTYPE=\z@
\def\GRAPHICSPS#1{%
 \ifcase\GRAPHICSTYPE%\GRAPHICSTYPE=0
   \special{ps: #1}%
 \or%\GRAPHICSTYPE=1
   \special{language "PS", include "#1"}%
%%%\or%\GRAPHICSTYPE=2
%%%  #1%
 \fi
}%
%
\def\GRAPHICSHP#1{\special{include #1}}%
%
% \graffile{ body }                                  %#1
%          { contentswidth (scalar)  }               %#2
%          { contentsheight (scalar) }               %#3
%          { vertical shift when in-line (scalar) }  %#4
\def\graffile#1#2#3#4{%
%%% \ifnum\GRAPHICSTYPE=\tw@
%%%  %Following if using psfig
%%%  \@ifundefined{psfig}{\input psfig.tex}{}%
%%%  \psfig{file=#1, height=#3, width=#2}%
%%% \else
  %Following for all others
  % JCS - added BOXTHEFRAME, see below
    \leavevmode
    \raise -#4 \BOXTHEFRAME{%
        \hbox to #2{\raise #3\hbox to #2{\null #1\hfil}}}%
}%
%
% A box for drafts
\def\draftbox#1#2#3#4{%
 \leavevmode\raise -#4 \hbox{%
  \frame{\rlap{\protect\tiny #1}\hbox to #2%
   {\vrule height#3 width\z@ depth\z@\hfil}%
  }%
 }%
}%
%
\newcount\draft
\draft=\z@
\let\nographics=\draft
\newif\ifwasdraft
\wasdraftfalse

%  \GRAPHIC{ body }                                  %#1
%          { draft name }                            %#2
%          { contentswidth (scalar)  }               %#3
%          { contentsheight (scalar) }               %#4
%          { vertical shift when in-line (scalar) }  %#5
\def\GRAPHIC#1#2#3#4#5{%
 \ifnum\draft=\@ne\draftbox{#2}{#3}{#4}{#5}%
  \else\graffile{#1}{#3}{#4}{#5}%
  \fi
 }%
%
\def\addtoLaTeXparams#1{%
    \edef\LaTeXparams{\LaTeXparams #1}}%
%
% JCS -  added a switch BoxFrame that can 
% be set by including X in the frame params.
% If set a box is drawn around the frame.

\newif\ifBoxFrame \BoxFramefalse
\newif\ifOverFrame \OverFramefalse
\newif\ifUnderFrame \UnderFramefalse

\def\BOXTHEFRAME#1{%
   \hbox{%
      \ifBoxFrame
         \frame{#1}%
      \else
         {#1}%
      \fi
   }%
}


\def\doFRAMEparams#1{\BoxFramefalse\OverFramefalse\UnderFramefalse\readFRAMEparams#1\end}%
\def\readFRAMEparams#1{%
 \ifx#1\end%
  \let\next=\relax
  \else
  \ifx#1i\dispkind=\z@\fi
  \ifx#1d\dispkind=\@ne\fi
  \ifx#1f\dispkind=\tw@\fi
  \ifx#1t\addtoLaTeXparams{t}\fi
  \ifx#1b\addtoLaTeXparams{b}\fi
  \ifx#1p\addtoLaTeXparams{p}\fi
  \ifx#1h\addtoLaTeXparams{h}\fi
  \ifx#1X\BoxFrametrue\fi
  \ifx#1O\OverFrametrue\fi
  \ifx#1U\UnderFrametrue\fi
  \ifx#1w
    \ifnum\draft=1\wasdrafttrue\else\wasdraftfalse\fi
    \draft=\@ne
  \fi
  \let\next=\readFRAMEparams
  \fi
 \next
 }%
%
%Macro for In-line graphics object
%   \IFRAME{ contentswidth (scalar)  }               %#1
%          { contentsheight (scalar) }               %#2
%          { vertical shift when in-line (scalar) }  %#3
%          { draft name }                            %#4
%          { body }                                  %#5
%          { caption}                                %#6


\def\IFRAME#1#2#3#4#5#6{%
      \bgroup
      \let\QCTOptA\empty
      \let\QCTOptB\empty
      \let\QCBOptA\empty
      \let\QCBOptB\empty
      #6%
      \parindent=0pt%
      \leftskip=0pt
      \rightskip=0pt
      \setbox0 = \hbox{\QCBOptA}%
      \@tempdima = #1\relax
      \ifOverFrame
          % Do this later
          \typeout{This is not implemented yet}%
          \show\HELP
      \else
         \ifdim\wd0>\@tempdima
            \advance\@tempdima by \@tempdima
            \ifdim\wd0 >\@tempdima
               \textwidth=\@tempdima
               \setbox1 =\vbox{%
                  \noindent\hbox to \@tempdima{\hfill\GRAPHIC{#5}{#4}{#1}{#2}{#3}\hfill}\\%
                  \noindent\hbox to \@tempdima{\parbox[b]{\@tempdima}{\QCBOptA}}%
               }%
               \wd1=\@tempdima
            \else
               \textwidth=\wd0
               \setbox1 =\vbox{%
                 \noindent\hbox to \wd0{\hfill\GRAPHIC{#5}{#4}{#1}{#2}{#3}\hfill}\\%
                 \noindent\hbox{\QCBOptA}%
               }%
               \wd1=\wd0
            \fi
         \else
            %\show\BBB
            \ifdim\wd0>0pt
              \hsize=\@tempdima
              \setbox1 =\vbox{%
                \unskip\GRAPHIC{#5}{#4}{#1}{#2}{0pt}%
                \break
                \unskip\hbox to \@tempdima{\hfill \QCBOptA\hfill}%
              }%
              \wd1=\@tempdima
           \else
              \hsize=\@tempdima
              \setbox1 =\vbox{%
                \unskip\GRAPHIC{#5}{#4}{#1}{#2}{0pt}%
              }%
              \wd1=\@tempdima
           \fi
         \fi
         \@tempdimb=\ht1
         \advance\@tempdimb by \dp1
         \advance\@tempdimb by -#2%
         \advance\@tempdimb by #3%
         \leavevmode
         \raise -\@tempdimb \hbox{\box1}%
      \fi
      \egroup%
}%
%
%Macro for Display graphics object
%   \DFRAME{ contentswidth (scalar)  }               %#1
%          { contentsheight (scalar) }               %#2
%          { draft label }                           %#3
%          { name }                                  %#4
%          { caption}                                %#5
\def\DFRAME#1#2#3#4#5{%
 \begin{center}
     \let\QCTOptA\empty
     \let\QCTOptB\empty
     \let\QCBOptA\empty
     \let\QCBOptB\empty
     \ifOverFrame 
        #5\QCTOptA\par
     \fi
     \GRAPHIC{#4}{#3}{#1}{#2}{\z@}
     \ifUnderFrame 
        \nobreak\par #5\QCBOptA
     \fi
 \end{center}%
 }%
%
%Macro for Floating graphic object
%   \FFRAME{ framedata f|i tbph x F|T }              %#1
%          { contentswidth (scalar)  }               %#2
%          { contentsheight (scalar) }               %#3
%          { caption }                               %#4
%          { label }                                 %#5
%          { draft name }                            %#6
%          { body }                                  %#7
\def\FFRAME#1#2#3#4#5#6#7{%
 \begin{figure}[#1]%
  \let\QCTOptA\empty
  \let\QCTOptB\empty
  \let\QCBOptA\empty
  \let\QCBOptB\empty
  \ifOverFrame
    #4
    \ifx\QCTOptA\empty
    \else
      \ifx\QCTOptB\empty
        \caption{\QCTOptA}%
      \else
        \caption[\QCTOptB]{\QCTOptA}%
      \fi
    \fi
    \ifUnderFrame\else
      \label{#5}%
    \fi
  \else
    \UnderFrametrue%
  \fi
  \begin{center}\GRAPHIC{#7}{#6}{#2}{#3}{\z@}\end{center}%
  \ifUnderFrame
    #4
    \ifx\QCBOptA\empty
      \caption{}%
    \else
      \ifx\QCBOptB\empty
        \caption{\QCBOptA}%
      \else
        \caption[\QCBOptB]{\QCBOptA}%
      \fi
    \fi
    \label{#5}%
  \fi
  \end{figure}%
 }%
%
%
%    \FRAME{ framedata f|i tbph x F|T }              %#1
%          { contentswidth (scalar)  }               %#2
%          { contentsheight (scalar) }               %#3
%          { vertical shift when in-line (scalar) }  %#4
%          { caption }                               %#5
%          { label }                                 %#6
%          { name }                                  %#7
%          { body }                                  %#8
%
%    framedata is a string which can contain the following
%    characters: idftbphxFT
%    Their meaning is as follows:
%             i, d or f : in-line, display, or floating
%             t,b,p,h   : LaTeX floating placement options
%             x         : fit contents box to contents
%             F or T    : Figure or Table. 
%                         Later this can expand
%                         to a more general float class.
%
%
\newcount\dispkind%

\def\makeactives{
  \catcode`\"=\active
  \catcode`\;=\active
  \catcode`\:=\active
  \catcode`\'=\active
  \catcode`\~=\active
}
\bgroup
   \makeactives
   \gdef\activesoff{%
      \def"{\string"}
      \def;{\string;}
      \def:{\string:}
      \def'{\string'}
      \def~{\string~}
      %\bbl@deactivate{"}%
      %\bbl@deactivate{;}%
      %\bbl@deactivate{:}%
      %\bbl@deactivate{'}%
    }
\egroup

\def\FRAME#1#2#3#4#5#6#7#8{%
 \bgroup
 \@ifundefined{bbl@deactivate}{}{\activesoff}
 \ifnum\draft=\@ne
   \wasdrafttrue
 \else
   \wasdraftfalse%
 \fi
 \def\LaTeXparams{}%
 \dispkind=\z@
 \def\LaTeXparams{}%
 \doFRAMEparams{#1}%
 \ifnum\dispkind=\z@\IFRAME{#2}{#3}{#4}{#7}{#8}{#5}\else
  \ifnum\dispkind=\@ne\DFRAME{#2}{#3}{#7}{#8}{#5}\else
   \ifnum\dispkind=\tw@
    \edef\@tempa{\noexpand\FFRAME{\LaTeXparams}}%
    \@tempa{#2}{#3}{#5}{#6}{#7}{#8}%
    \fi
   \fi
  \fi
  \ifwasdraft\draft=1\else\draft=0\fi{}%
  \egroup
 }%
%
% This macro added to let SW gobble a parameter that
% should not be passed on and expanded. 

\def\TEXUX#1{"texux"}

%
% Macros for text attributes:
%
\def\BF#1{{\bf {#1}}}%
\def\NEG#1{\leavevmode\hbox{\rlap{\thinspace/}{$#1$}}}%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%
% macros for user - defined functions
\def\func#1{\mathop{\rm #1}}%
\def\limfunc#1{\mathop{\rm #1}}%

%
% miscellaneous 
%\long\def\QQQ#1#2{}%
\long\def\QQQ#1#2{%
     \long\expandafter\def\csname#1\endcsname{#2}}%
%\def\QTP#1{}% JCS - this was changed becuase style editor will define QTP
\@ifundefined{QTP}{\def\QTP#1{}}{}
\@ifundefined{QEXCLUDE}{\def\QEXCLUDE#1{}}{}
%\@ifundefined{Qcb}{\def\Qcb#1{#1}}{}
%\@ifundefined{Qct}{\def\Qct#1{#1}}{}
\@ifundefined{Qlb}{\def\Qlb#1{#1}}{}
\@ifundefined{Qlt}{\def\Qlt#1{#1}}{}
\def\QWE{}%
\long\def\QQA#1#2{}%
%\def\QTR#1#2{{\em #2}}% Always \em!!!
%\def\QTR#1#2{\mbox{\begin{#1}#2\end{#1}}}%cb%%%
\def\QTR#1#2{{\csname#1\endcsname #2}}%(gp) Is this the best?
\long\def\TeXButton#1#2{#2}%
\long\def\QSubDoc#1#2{#2}%
\def\EXPAND#1[#2]#3{}%
\def\NOEXPAND#1[#2]#3{}%
\def\PROTECTED{}%
\def\LaTeXparent#1{}%
\def\ChildStyles#1{}%
\def\ChildDefaults#1{}%
\def\QTagDef#1#2#3{}%
%
% Macros for style editor docs
\@ifundefined{StyleEditBeginDoc}{\def\StyleEditBeginDoc{\relax}}{}
%
% Macros for footnotes
\def\QQfnmark#1{\footnotemark}
\def\QQfntext#1#2{\addtocounter{footnote}{#1}\footnotetext{#2}}
%
% Macros for indexing.
\def\MAKEINDEX{\makeatletter\input gnuindex.sty\makeatother\makeindex}%	
\@ifundefined{INDEX}{\def\INDEX#1#2{}{}}{}%
\@ifundefined{SUBINDEX}{\def\SUBINDEX#1#2#3{}{}{}}{}%
\@ifundefined{initial}%  
   {\def\initial#1{\bigbreak{\raggedright\large\bf #1}\kern 2\p@\penalty3000}}%
   {}%
\@ifundefined{entry}{\def\entry#1#2{\item {#1}, #2}}{}%
\@ifundefined{primary}{\def\primary#1{\item {#1}}}{}%
\@ifundefined{secondary}{\def\secondary#1#2{\subitem {#1}, #2}}{}%
%
%
\@ifundefined{ZZZ}{}{\MAKEINDEX\makeatletter}%
%
% Attempts to avoid problems with other styles
\@ifundefined{abstract}{%
 \def\abstract{%
  \if@twocolumn
   \section*{Abstract (Not appropriate in this style!)}%
   \else \small 
   \begin{center}{\bf Abstract\vspace{-.5em}\vspace{\z@}}\end{center}%
   \quotation 
   \fi
  }%
 }{%
 }%
\@ifundefined{endabstract}{\def\endabstract
  {\if@twocolumn\else\endquotation\fi}}{}%
\@ifundefined{maketitle}{\def\maketitle#1{}}{}%
\@ifundefined{affiliation}{\def\affiliation#1{}}{}%
\@ifundefined{proof}{\def\proof{\noindent{\bfseries Proof. }}}{}%
\@ifundefined{endproof}{\def\endproof{\mbox{\ \rule{.1in}{.1in}}}}{}%
\@ifundefined{newfield}{\def\newfield#1#2{}}{}%
\@ifundefined{chapter}{\def\chapter#1{\par(Chapter head:)#1\par }%
 \newcount\c@chapter}{}%
\@ifundefined{part}{\def\part#1{\par(Part head:)#1\par }}{}%
\@ifundefined{section}{\def\section#1{\par(Section head:)#1\par }}{}%
\@ifundefined{subsection}{\def\subsection#1%
 {\par(Subsection head:)#1\par }}{}%
\@ifundefined{subsubsection}{\def\subsubsection#1%
 {\par(Subsubsection head:)#1\par }}{}%
\@ifundefined{paragraph}{\def\paragraph#1%
 {\par(Subsubsubsection head:)#1\par }}{}%
\@ifundefined{subparagraph}{\def\subparagraph#1%
 {\par(Subsubsubsubsection head:)#1\par }}{}%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% These symbols are not recognized by LaTeX
\@ifundefined{therefore}{\def\therefore{}}{}%
\@ifundefined{backepsilon}{\def\backepsilon{}}{}%
\@ifundefined{yen}{\def\yen{\hbox{\rm\rlap=Y}}}{}%
\@ifundefined{registered}{%
   \def\registered{\relax\ifmmode{}\r@gistered
                    \else$\m@th\r@gistered$\fi}%
 \def\r@gistered{^{\ooalign
  {\hfil\raise.07ex\hbox{$\scriptstyle\rm\text{R}$}\hfil\crcr
  \mathhexbox20D}}}}{}%
\@ifundefined{Eth}{\def\Eth{}}{}%
\@ifundefined{eth}{\def\eth{}}{}%
\@ifundefined{Thorn}{\def\Thorn{}}{}%
\@ifundefined{thorn}{\def\thorn{}}{}%
% A macro to allow any symbol that requires math to appear in text
\def\TEXTsymbol#1{\mbox{$#1$}}%
\@ifundefined{degree}{\def\degree{{}^{\circ}}}{}%
%
% macros for T3TeX files
\newdimen\theight
\def\Column{%
 \vadjust{\setbox\z@=\hbox{\scriptsize\quad\quad tcol}%
  \theight=\ht\z@\advance\theight by \dp\z@\advance\theight by \lineskip
  \kern -\theight \vbox to \theight{%
   \rightline{\rlap{\box\z@}}%
   \vss
   }%
  }%
 }%
%
\def\qed{%
 \ifhmode\unskip\nobreak\fi\ifmmode\ifinner\else\hskip5\p@\fi\fi
 \hbox{\hskip5\p@\vrule width4\p@ height6\p@ depth1.5\p@\hskip\p@}%
 }%
%
\def\cents{\hbox{\rm\rlap/c}}%
\def\miss{\hbox{\vrule height2\p@ width 2\p@ depth\z@}}%
%\def\miss{\hbox{.}}%        %another possibility 
%
\def\vvert{\Vert}%           %always translated to \left| or \right|
%
\def\tcol#1{{\baselineskip=6\p@ \vcenter{#1}} \Column}  %
%
\def\dB{\hbox{{}}}%                 %dummy entry in column 
\def\mB#1{\hbox{$#1$}}%             %column entry
\def\nB#1{\hbox{#1}}%               %column entry (not math)
%
%\newcount\notenumber
%\def\clearnotenumber{\notenumber=0}
%\def\note{\global\advance\notenumber by 1
% \footnote{$^{\the\notenumber}$}}
%\def\note{\global\advance\notenumber by 1
\def\note{$^{\dag}}%
%
%

\def\newfmtname{LaTeX2e}
\def\chkcompat{%
   \if@compatibility
   \else
     \usepackage{latexsym}
   \fi
}

\ifx\fmtname\newfmtname
  \DeclareOldFontCommand{\rm}{\normalfont\rmfamily}{\mathrm}
  \DeclareOldFontCommand{\sf}{\normalfont\sffamily}{\mathsf}
  \DeclareOldFontCommand{\tt}{\normalfont\ttfamily}{\mathtt}
  \DeclareOldFontCommand{\bf}{\normalfont\bfseries}{\mathbf}
  \DeclareOldFontCommand{\it}{\normalfont\itshape}{\mathit}
  \DeclareOldFontCommand{\sl}{\normalfont\slshape}{\@nomath\sl}
  \DeclareOldFontCommand{\sc}{\normalfont\scshape}{\@nomath\sc}
  \chkcompat
\fi

%
% Greek bold macros
% Redefine all of the math symbols 
% which might be bolded	 - there are 
% probably others to add to this list

\def\alpha{\Greekmath 010B }%
\def\beta{\Greekmath 010C }%
\def\gamma{\Greekmath 010D }%
\def\delta{\Greekmath 010E }%
\def\epsilon{\Greekmath 010F }%
\def\zeta{\Greekmath 0110 }%
\def\eta{\Greekmath 0111 }%
\def\theta{\Greekmath 0112 }%
\def\iota{\Greekmath 0113 }%
\def\kappa{\Greekmath 0114 }%
\def\lambda{\Greekmath 0115 }%
\def\mu{\Greekmath 0116 }%
\def\nu{\Greekmath 0117 }%
\def\xi{\Greekmath 0118 }%
\def\pi{\Greekmath 0119 }%
\def\rho{\Greekmath 011A }%
\def\sigma{\Greekmath 011B }%
\def\tau{\Greekmath 011C }%
\def\upsilon{\Greekmath 011D }%
\def\phi{\Greekmath 011E }%
\def\chi{\Greekmath 011F }%
\def\psi{\Greekmath 0120 }%
\def\omega{\Greekmath 0121 }%
\def\varepsilon{\Greekmath 0122 }%
\def\vartheta{\Greekmath 0123 }%
\def\varpi{\Greekmath 0124 }%
\def\varrho{\Greekmath 0125 }%
\def\varsigma{\Greekmath 0126 }%
\def\varphi{\Greekmath 0127 }%

\def\nabla{\Greekmath 0272 }
\def\FindBoldGroup{%
   {\setbox0=\hbox{$\mathbf{x\global\edef\theboldgroup{\the\mathgroup}}$}}%
}

\def\Greekmath#1#2#3#4{%
    \if@compatibility
        \ifnum\mathgroup=\symbold
           \mathchoice{\mbox{\boldmath$\displaystyle\mathchar"#1#2#3#4$}}%
                      {\mbox{\boldmath$\textstyle\mathchar"#1#2#3#4$}}%
                      {\mbox{\boldmath$\scriptstyle\mathchar"#1#2#3#4$}}%
                      {\mbox{\boldmath$\scriptscriptstyle\mathchar"#1#2#3#4$}}%
        \else
           \mathchar"#1#2#3#4% 
        \fi 
    \else 
        \FindBoldGroup
        \ifnum\mathgroup=\theboldgroup % For 2e
           \mathchoice{\mbox{\boldmath$\displaystyle\mathchar"#1#2#3#4$}}%
                      {\mbox{\boldmath$\textstyle\mathchar"#1#2#3#4$}}%
                      {\mbox{\boldmath$\scriptstyle\mathchar"#1#2#3#4$}}%
                      {\mbox{\boldmath$\scriptscriptstyle\mathchar"#1#2#3#4$}}%
        \else
           \mathchar"#1#2#3#4% 
        \fi     	    
	  \fi}

\newif\ifGreekBold  \GreekBoldfalse
\let\SAVEPBF=\pbf
\def\pbf{\GreekBoldtrue\SAVEPBF}%
%

\@ifundefined{theorem}{\newtheorem{theorem}{Theorem}}{}
\@ifundefined{lemma}{\newtheorem{lemma}[theorem]{Lemma}}{}
\@ifundefined{corollary}{\newtheorem{corollary}[theorem]{Corollary}}{}
\@ifundefined{conjecture}{\newtheorem{conjecture}[theorem]{Conjecture}}{}
\@ifundefined{proposition}{\newtheorem{proposition}[theorem]{Proposition}}{}
\@ifundefined{axiom}{\newtheorem{axiom}{Axiom}}{}
\@ifundefined{remark}{\newtheorem{remark}{Remark}}{}
\@ifundefined{example}{\newtheorem{example}{Example}}{}
\@ifundefined{exercise}{\newtheorem{exercise}{Exercise}}{}
\@ifundefined{definition}{\newtheorem{definition}{Definition}}{}


\@ifundefined{mathletters}{%
  %\def\theequation{\arabic{equation}}
  \newcounter{equationnumber}  
  \def\mathletters{%
     \addtocounter{equation}{1}
     \edef\@currentlabel{\theequation}%
     \setcounter{equationnumber}{\c@equation}
     \setcounter{equation}{0}%
     \edef\theequation{\@currentlabel\noexpand\alph{equation}}%
  }
  \def\endmathletters{%
     \setcounter{equation}{\value{equationnumber}}%
  }
}{}

%Logos
\@ifundefined{BibTeX}{%
    \def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
                 T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}}{}%
\@ifundefined{AmS}%
    {\def\AmS{{\protect\usefont{OMS}{cmsy}{m}{n}%
                A\kern-.1667em\lower.5ex\hbox{M}\kern-.125emS}}}{}%
\@ifundefined{AmSTeX}{\def\AmSTeX{\protect\AmS-\protect\TeX\@}}{}%
%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% NOTE: The rest of this file is read only if amstex has not been
% loaded.  This section is used to define amstex constructs in the
% event they have not been defined.
%
%
\ifx\ds@amstex\relax
   \message{amstex already loaded}\makeatother\endinput% 2.09 compatability
\else
   \@ifpackageloaded{amstex}%
      {\message{amstex already loaded}\makeatother\endinput}
      {}
   \@ifpackageloaded{amsgen}%
      {\message{amsgen already loaded}\makeatother\endinput}
      {}
\fi
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%
%
%
%  Macros to define some AMS LaTeX constructs when 
%  AMS LaTeX has not been loaded
% 
% These macros are copied from the AMS-TeX package for doing
% multiple integrals.
%
\let\DOTSI\relax
\def\RIfM@{\relax\ifmmode}%
\def\FN@{\futurelet\next}%
\newcount\intno@
\def\iint{\DOTSI\intno@\tw@\FN@\ints@}%
\def\iiint{\DOTSI\intno@\thr@@\FN@\ints@}%
\def\iiiint{\DOTSI\intno@4 \FN@\ints@}%
\def\idotsint{\DOTSI\intno@\z@\FN@\ints@}%
\def\ints@{\findlimits@\ints@@}%
\newif\iflimtoken@
\newif\iflimits@
\def\findlimits@{\limtoken@true\ifx\next\limits\limits@true
 \else\ifx\next\nolimits\limits@false\else
 \limtoken@false\ifx\ilimits@\nolimits\limits@false\else
 \ifinner\limits@false\else\limits@true\fi\fi\fi\fi}%
\def\multint@{\int\ifnum\intno@=\z@\intdots@                          %1
 \else\intkern@\fi                                                    %2
 \ifnum\intno@>\tw@\int\intkern@\fi                                   %3
 \ifnum\intno@>\thr@@\int\intkern@\fi                                 %4
 \int}%                                                               %5
\def\multintlimits@{\intop\ifnum\intno@=\z@\intdots@\else\intkern@\fi
 \ifnum\intno@>\tw@\intop\intkern@\fi
 \ifnum\intno@>\thr@@\intop\intkern@\fi\intop}%
\def\intic@{%
    \mathchoice{\hskip.5em}{\hskip.4em}{\hskip.4em}{\hskip.4em}}%
\def\negintic@{\mathchoice
 {\hskip-.5em}{\hskip-.4em}{\hskip-.4em}{\hskip-.4em}}%
\def\ints@@{\iflimtoken@                                              %1
 \def\ints@@@{\iflimits@\negintic@
   \mathop{\intic@\multintlimits@}\limits                             %2
  \else\multint@\nolimits\fi                                          %3
  \eat@}%                                                             %4
 \else                                                                %5
 \def\ints@@@{\iflimits@\negintic@
  \mathop{\intic@\multintlimits@}\limits\else
  \multint@\nolimits\fi}\fi\ints@@@}%
\def\intkern@{\mathchoice{\!\!\!}{\!\!}{\!\!}{\!\!}}%
\def\plaincdots@{\mathinner{\cdotp\cdotp\cdotp}}%
\def\intdots@{\mathchoice{\plaincdots@}%
 {{\cdotp}\mkern1.5mu{\cdotp}\mkern1.5mu{\cdotp}}%
 {{\cdotp}\mkern1mu{\cdotp}\mkern1mu{\cdotp}}%
 {{\cdotp}\mkern1mu{\cdotp}\mkern1mu{\cdotp}}}%
%
%
%  These macros are for doing the AMS \text{} construct
%
\def\RIfM@{\relax\protect\ifmmode}
\def\text{\RIfM@\expandafter\text@\else\expandafter\mbox\fi}
\let\nfss@text\text
\def\text@#1{\mathchoice
   {\textdef@\displaystyle\f@size{#1}}%
   {\textdef@\textstyle\tf@size{\firstchoice@false #1}}%
   {\textdef@\textstyle\sf@size{\firstchoice@false #1}}%
   {\textdef@\textstyle \ssf@size{\firstchoice@false #1}}%
   \glb@settings}

\def\textdef@#1#2#3{\hbox{{%
                    \everymath{#1}%
                    \let\f@size#2\selectfont
                    #3}}}
\newif\iffirstchoice@
\firstchoice@true
%
%    Old Scheme for \text
%
%\def\rmfam{\z@}%
%\newif\iffirstchoice@
%\firstchoice@true
%\def\textfonti{\the\textfont\@ne}%
%\def\textfontii{\the\textfont\tw@}%
%\def\text{\RIfM@\expandafter\text@\else\expandafter\text@@\fi}%
%\def\text@@#1{\leavevmode\hbox{#1}}%
%\def\text@#1{\mathchoice
% {\hbox{\everymath{\displaystyle}\def\textfonti{\the\textfont\@ne}%
%  \def\textfontii{\the\textfont\tw@}\textdef@@ T#1}}%
% {\hbox{\firstchoice@false
%  \everymath{\textstyle}\def\textfonti{\the\textfont\@ne}%
%  \def\textfontii{\the\textfont\tw@}\textdef@@ T#1}}%
% {\hbox{\firstchoice@false
%  \everymath{\scriptstyle}\def\textfonti{\the\scriptfont\@ne}%
%  \def\textfontii{\the\scriptfont\tw@}\textdef@@ S\rm#1}}%
% {\hbox{\firstchoice@false
%  \everymath{\scriptscriptstyle}\def\textfonti
%  {\the\scriptscriptfont\@ne}%
%  \def\textfontii{\the\scriptscriptfont\tw@}\textdef@@ s\rm#1}}}%
%\def\textdef@@#1{\textdef@#1\rm\textdef@#1\bf\textdef@#1\sl
%    \textdef@#1\it}%
%\def\DN@{\def\next@}%
%\def\eat@#1{}%
%\def\textdef@#1#2{%
% \DN@{\csname\expandafter\eat@\string#2fam\endcsname}%
% \if S#1\edef#2{\the\scriptfont\next@\relax}%
% \else\if s#1\edef#2{\the\scriptscriptfont\next@\relax}%
% \else\edef#2{\the\textfont\next@\relax}\fi\fi}%
%
%
%These are the AMS constructs for multiline limits.
%
\def\Let@{\relax\iffalse{\fi\let\\=\cr\iffalse}\fi}%
\def\vspace@{\def\vspace##1{\crcr\noalign{\vskip##1\relax}}}%
\def\multilimits@{\bgroup\vspace@\Let@
 \baselineskip\fontdimen10 \scriptfont\tw@
 \advance\baselineskip\fontdimen12 \scriptfont\tw@
 \lineskip\thr@@\fontdimen8 \scriptfont\thr@@
 \lineskiplimit\lineskip
 \vbox\bgroup\ialign\bgroup\hfil$\m@th\scriptstyle{##}$\hfil\crcr}%
\def\Sb{_\multilimits@}%
\def\endSb{\crcr\egroup\egroup\egroup}%
\def\Sp{^\multilimits@}%
\let\endSp\endSb
%
%
%These are AMS constructs for horizontal arrows
%
\newdimen\ex@
\ex@.2326ex
\def\rightarrowfill@#1{$#1\m@th\mathord-\mkern-6mu\cleaders
 \hbox{$#1\mkern-2mu\mathord-\mkern-2mu$}\hfill
 \mkern-6mu\mathord\rightarrow$}%
\def\leftarrowfill@#1{$#1\m@th\mathord\leftarrow\mkern-6mu\cleaders
 \hbox{$#1\mkern-2mu\mathord-\mkern-2mu$}\hfill\mkern-6mu\mathord-$}%
\def\leftrightarrowfill@#1{$#1\m@th\mathord\leftarrow
\mkern-6mu\cleaders
 \hbox{$#1\mkern-2mu\mathord-\mkern-2mu$}\hfill
 \mkern-6mu\mathord\rightarrow$}%
\def\overrightarrow{\mathpalette\overrightarrow@}%
\def\overrightarrow@#1#2{\vbox{\ialign{##\crcr\rightarrowfill@#1\crcr
 \noalign{\kern-\ex@\nointerlineskip}$\m@th\hfil#1#2\hfil$\crcr}}}%
\let\overarrow\overrightarrow
\def\overleftarrow{\mathpalette\overleftarrow@}%
\def\overleftarrow@#1#2{\vbox{\ialign{##\crcr\leftarrowfill@#1\crcr
 \noalign{\kern-\ex@\nointerlineskip}$\m@th\hfil#1#2\hfil$\crcr}}}%
\def\overleftrightarrow{\mathpalette\overleftrightarrow@}%
\def\overleftrightarrow@#1#2{\vbox{\ialign{##\crcr
   \leftrightarrowfill@#1\crcr
 \noalign{\kern-\ex@\nointerlineskip}$\m@th\hfil#1#2\hfil$\crcr}}}%
\def\underrightarrow{\mathpalette\underrightarrow@}%
\def\underrightarrow@#1#2{\vtop{\ialign{##\crcr$\m@th\hfil#1#2\hfil
  $\crcr\noalign{\nointerlineskip}\rightarrowfill@#1\crcr}}}%
\let\underarrow\underrightarrow
\def\underleftarrow{\mathpalette\underleftarrow@}%
\def\underleftarrow@#1#2{\vtop{\ialign{##\crcr$\m@th\hfil#1#2\hfil
  $\crcr\noalign{\nointerlineskip}\leftarrowfill@#1\crcr}}}%
\def\underleftrightarrow{\mathpalette\underleftrightarrow@}%
\def\underleftrightarrow@#1#2{\vtop{\ialign{##\crcr$\m@th
  \hfil#1#2\hfil$\crcr
 \noalign{\nointerlineskip}\leftrightarrowfill@#1\crcr}}}%
%%%%%%%%%%%%%%%%%%%%%

% 94.0815 by Jon:

\def\qopnamewl@#1{\mathop{\operator@font#1}\nlimits@}
\let\nlimits@\displaylimits
\def\setboxz@h{\setbox\z@\hbox}


\def\varlim@#1#2{\mathop{\vtop{\ialign{##\crcr
 \hfil$#1\m@th\operator@font lim$\hfil\crcr
 \noalign{\nointerlineskip}#2#1\crcr
 \noalign{\nointerlineskip\kern-\ex@}\crcr}}}}

 \def\rightarrowfill@#1{\m@th\setboxz@h{$#1-$}\ht\z@\z@
  $#1\copy\z@\mkern-6mu\cleaders
  \hbox{$#1\mkern-2mu\box\z@\mkern-2mu$}\hfill
  \mkern-6mu\mathord\rightarrow$}
\def\leftarrowfill@#1{\m@th\setboxz@h{$#1-$}\ht\z@\z@
  $#1\mathord\leftarrow\mkern-6mu\cleaders
  \hbox{$#1\mkern-2mu\copy\z@\mkern-2mu$}\hfill
  \mkern-6mu\box\z@$}


\def\projlim{\qopnamewl@{proj\,lim}}
\def\injlim{\qopnamewl@{inj\,lim}}
\def\varinjlim{\mathpalette\varlim@\rightarrowfill@}
\def\varprojlim{\mathpalette\varlim@\leftarrowfill@}
\def\varliminf{\mathpalette\varliminf@{}}
\def\varliminf@#1{\mathop{\underline{\vrule\@depth.2\ex@\@width\z@
   \hbox{$#1\m@th\operator@font lim$}}}}
\def\varlimsup{\mathpalette\varlimsup@{}}
\def\varlimsup@#1{\mathop{\overline
  {\hbox{$#1\m@th\operator@font lim$}}}}

%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
\def\tfrac#1#2{{\textstyle {#1 \over #2}}}%
\def\dfrac#1#2{{\displaystyle {#1 \over #2}}}%
\def\binom#1#2{{#1 \choose #2}}%
\def\tbinom#1#2{{\textstyle {#1 \choose #2}}}%
\def\dbinom#1#2{{\displaystyle {#1 \choose #2}}}%
\def\QATOP#1#2{{#1 \atop #2}}%
\def\QTATOP#1#2{{\textstyle {#1 \atop #2}}}%
\def\QDATOP#1#2{{\displaystyle {#1 \atop #2}}}%
\def\QABOVE#1#2#3{{#2 \above#1 #3}}%
\def\QTABOVE#1#2#3{{\textstyle {#2 \above#1 #3}}}%
\def\QDABOVE#1#2#3{{\displaystyle {#2 \above#1 #3}}}%
\def\QOVERD#1#2#3#4{{#3 \overwithdelims#1#2 #4}}%
\def\QTOVERD#1#2#3#4{{\textstyle {#3 \overwithdelims#1#2 #4}}}%
\def\QDOVERD#1#2#3#4{{\displaystyle {#3 \overwithdelims#1#2 #4}}}%
\def\QATOPD#1#2#3#4{{#3 \atopwithdelims#1#2 #4}}%
\def\QTATOPD#1#2#3#4{{\textstyle {#3 \atopwithdelims#1#2 #4}}}%
\def\QDATOPD#1#2#3#4{{\displaystyle {#3 \atopwithdelims#1#2 #4}}}%
\def\QABOVED#1#2#3#4#5{{#4 \abovewithdelims#1#2#3 #5}}%
\def\QTABOVED#1#2#3#4#5{{\textstyle 
   {#4 \abovewithdelims#1#2#3 #5}}}%
\def\QDABOVED#1#2#3#4#5{{\displaystyle 
   {#4 \abovewithdelims#1#2#3 #5}}}%
%
% Macros for text size operators:

%JCS - added braces and \mathop around \displaystyle\int, etc.
%
\def\tint{\mathop{\textstyle \int}}%
\def\tiint{\mathop{\textstyle \iint }}%
\def\tiiint{\mathop{\textstyle \iiint }}%
\def\tiiiint{\mathop{\textstyle \iiiint }}%
\def\tidotsint{\mathop{\textstyle \idotsint }}%
\def\toint{\mathop{\textstyle \oint}}%
\def\tsum{\mathop{\textstyle \sum }}%
\def\tprod{\mathop{\textstyle \prod }}%
\def\tbigcap{\mathop{\textstyle \bigcap }}%
\def\tbigwedge{\mathop{\textstyle \bigwedge }}%
\def\tbigoplus{\mathop{\textstyle \bigoplus }}%
\def\tbigodot{\mathop{\textstyle \bigodot }}%
\def\tbigsqcup{\mathop{\textstyle \bigsqcup }}%
\def\tcoprod{\mathop{\textstyle \coprod }}%
\def\tbigcup{\mathop{\textstyle \bigcup }}%
\def\tbigvee{\mathop{\textstyle \bigvee }}%
\def\tbigotimes{\mathop{\textstyle \bigotimes }}%
\def\tbiguplus{\mathop{\textstyle \biguplus }}%
%
%
%Macros for display size operators:
%

\def\dint{\mathop{\displaystyle \int}}%
\def\diint{\mathop{\displaystyle \iint }}%
\def\diiint{\mathop{\displaystyle \iiint }}%
\def\diiiint{\mathop{\displaystyle \iiiint }}%
\def\didotsint{\mathop{\displaystyle \idotsint }}%
\def\doint{\mathop{\displaystyle \oint}}%
\def\dsum{\mathop{\displaystyle \sum }}%
\def\dprod{\mathop{\displaystyle \prod }}%
\def\dbigcap{\mathop{\displaystyle \bigcap }}%
\def\dbigwedge{\mathop{\displaystyle \bigwedge }}%
\def\dbigoplus{\mathop{\displaystyle \bigoplus }}%
\def\dbigodot{\mathop{\displaystyle \bigodot }}%
\def\dbigsqcup{\mathop{\displaystyle \bigsqcup }}%
\def\dcoprod{\mathop{\displaystyle \coprod }}%
\def\dbigcup{\mathop{\displaystyle \bigcup }}%
\def\dbigvee{\mathop{\displaystyle \bigvee }}%
\def\dbigotimes{\mathop{\displaystyle \bigotimes }}%
\def\dbiguplus{\mathop{\displaystyle \biguplus }}%
%
%Companion to stackrel
\def\stackunder#1#2{\mathrel{\mathop{#2}\limits_{#1}}}%
%
%
% These are AMS environments that will be defined to
% be verbatims if amstex has not actually been 
% loaded
%
%
\begingroup \catcode `|=0 \catcode `[= 1
\catcode`]=2 \catcode `\{=12 \catcode `\}=12
\catcode`\\=12 
|gdef|@alignverbatim#1\end{align}[#1|end[align]]
|gdef|@salignverbatim#1\end{align*}[#1|end[align*]]

|gdef|@alignatverbatim#1\end{alignat}[#1|end[alignat]]
|gdef|@salignatverbatim#1\end{alignat*}[#1|end[alignat*]]

|gdef|@xalignatverbatim#1\end{xalignat}[#1|end[xalignat]]
|gdef|@sxalignatverbatim#1\end{xalignat*}[#1|end[xalignat*]]

|gdef|@gatherverbatim#1\end{gather}[#1|end[gather]]
|gdef|@sgatherverbatim#1\end{gather*}[#1|end[gather*]]

|gdef|@gatherverbatim#1\end{gather}[#1|end[gather]]
|gdef|@sgatherverbatim#1\end{gather*}[#1|end[gather*]]


|gdef|@multilineverbatim#1\end{multiline}[#1|end[multiline]]
|gdef|@smultilineverbatim#1\end{multiline*}[#1|end[multiline*]]

|gdef|@arraxverbatim#1\end{arrax}[#1|end[arrax]]
|gdef|@sarraxverbatim#1\end{arrax*}[#1|end[arrax*]]

|gdef|@tabulaxverbatim#1\end{tabulax}[#1|end[tabulax]]
|gdef|@stabulaxverbatim#1\end{tabulax*}[#1|end[tabulax*]]


|endgroup
  

  
\def\align{\@verbatim \frenchspacing\@vobeyspaces \@alignverbatim
You are using the "align" environment in a style in which it is not defined.}
\let\endalign=\endtrivlist
 
\@namedef{align*}{\@verbatim\@salignverbatim
You are using the "align*" environment in a style in which it is not defined.}
\expandafter\let\csname endalign*\endcsname =\endtrivlist




\def\alignat{\@verbatim \frenchspacing\@vobeyspaces \@alignatverbatim
You are using the "alignat" environment in a style in which it is not defined.}
\let\endalignat=\endtrivlist
 
\@namedef{alignat*}{\@verbatim\@salignatverbatim
You are using the "alignat*" environment in a style in which it is not defined.}
\expandafter\let\csname endalignat*\endcsname =\endtrivlist




\def\xalignat{\@verbatim \frenchspacing\@vobeyspaces \@xalignatverbatim
You are using the "xalignat" environment in a style in which it is not defined.}
\let\endxalignat=\endtrivlist
 
\@namedef{xalignat*}{\@verbatim\@sxalignatverbatim
You are using the "xalignat*" environment in a style in which it is not defined.}
\expandafter\let\csname endxalignat*\endcsname =\endtrivlist




\def\gather{\@verbatim \frenchspacing\@vobeyspaces \@gatherverbatim
You are using the "gather" environment in a style in which it is not defined.}
\let\endgather=\endtrivlist
 
\@namedef{gather*}{\@verbatim\@sgatherverbatim
You are using the "gather*" environment in a style in which it is not defined.}
\expandafter\let\csname endgather*\endcsname =\endtrivlist


\def\multiline{\@verbatim \frenchspacing\@vobeyspaces \@multilineverbatim
You are using the "multiline" environment in a style in which it is not defined.}
\let\endmultiline=\endtrivlist
 
\@namedef{multiline*}{\@verbatim\@smultilineverbatim
You are using the "multiline*" environment in a style in which it is not defined.}
\expandafter\let\csname endmultiline*\endcsname =\endtrivlist


\def\arrax{\@verbatim \frenchspacing\@vobeyspaces \@arraxverbatim
You are using a type of "array" construct that is only allowed in AmS-LaTeX.}
\let\endarrax=\endtrivlist

\def\tabulax{\@verbatim \frenchspacing\@vobeyspaces \@tabulaxverbatim
You are using a type of "tabular" construct that is only allowed in AmS-LaTeX.}
\let\endtabulax=\endtrivlist

 
\@namedef{arrax*}{\@verbatim\@sarraxverbatim
You are using a type of "array*" construct that is only allowed in AmS-LaTeX.}
\expandafter\let\csname endarrax*\endcsname =\endtrivlist

\@namedef{tabulax*}{\@verbatim\@stabulaxverbatim
You are using a type of "tabular*" construct that is only allowed in AmS-LaTeX.}
\expandafter\let\csname endtabulax*\endcsname =\endtrivlist

% macro to simulate ams tag construct


% This macro is a fix to eqnarray
\def\@@eqncr{\let\@tempa\relax
    \ifcase\@eqcnt \def\@tempa{& & &}\or \def\@tempa{& &}%
      \else \def\@tempa{&}\fi
     \@tempa
     \if@eqnsw
        \iftag@
           \@taggnum
        \else
           \@eqnnum\stepcounter{equation}%
        \fi
     \fi
     \global\tag@false
     \global\@eqnswtrue
     \global\@eqcnt\z@\cr}


% This macro is a fix to the equation environment
 \def\endequation{%
     \ifmmode\ifinner % FLEQN hack
      \iftag@
        \addtocounter{equation}{-1} % undo the increment made in the begin part
        $\hfil
           \displaywidth\linewidth\@taggnum\egroup \endtrivlist
        \global\tag@false
        \global\@ignoretrue   
      \else
        $\hfil
           \displaywidth\linewidth\@eqnnum\egroup \endtrivlist
        \global\tag@false
        \global\@ignoretrue 
      \fi
     \else   
      \iftag@
        \addtocounter{equation}{-1} % undo the increment made in the begin part
        \eqno \hbox{\@taggnum}
        \global\tag@false%
        $$\global\@ignoretrue
      \else
        \eqno \hbox{\@eqnnum}% $$ BRACE MATCHING HACK
        $$\global\@ignoretrue
      \fi
     \fi\fi
 } 

 \newif\iftag@ \tag@false
 
 \def\tag{\@ifnextchar*{\@tagstar}{\@tag}}
 \def\@tag#1{%
     \global\tag@true
     \global\def\@taggnum{(#1)}}
 \def\@tagstar*#1{%
     \global\tag@true
     \global\def\@taggnum{#1}%  
}

% Do not add anything to the end of this file.  
% The last section of the file is loaded only if 
% amstex has not been.



\makeatother
\endinput

%%%%%%%%%%%%%%%%%%%%% End \macros\tci\tcilatex.tex %%%%%%%%%%%%%%%%%%%%

hep-lat0505005.extracted_bib
\begin{thebibliography}{9}
\bibitem{mdp1}  M. Di Pierro, Computer Physics Communcations 141:98-148, 2001

\bibitem{mdp2}  M. Di Pierro. Proceedings ICCSA 2003

\bibitem{fermiqcd2002}  M. Di Pierro, Nucl. Phys. Proc. Suppl. 106:1034-1036%
\textbf{,} 2002

\bibitem{fermiqcd2003}  M. Di Pierro et al., www.fermiqcd.net,
Nucl.Phys.Proc.Suppl.129:832-834,2004 [hep-lat/0311027]

\bibitem{app1}  J. Simone at al., Nucl.Phys.Proc.Suppl.140:443-445,2005

\bibitem{app2}  M. Okamoto et al., Nucl.Phys.Proc.Suppl.140:461-463,2005

\bibitem{app3}  C. Aubin et al. Phys.Rev.Lett.94:011601,2005

\bibitem{app4}  L. Li and Y. Meurice, Phys.Rev.D71:054509,2005

\bibitem{app5}  J. Flynn et al. JHEP 0411:049,2004
\end{thebibliography}