Q:

When the left-right parenthesis is too big, how to make it smaller?

When the left-right parenthesis is too big as in the example below, how do you make the parenthesis smaller? 
         \documentclass[reqno]{amsart} 
         \usepackage{amssymb}
         \usepackage{mathrsfs}
         \usepackage{amsmath}
         \usepackage{amsthm}
         \usepackage{stackrel} 
        \usepackage{courier}
        \usepackage[all,cmtip]{xy} 
        \usepackage{mathtools} 
        \usepackage{hyperref}
        \usepackage{multicol}
        \usepackage{enumerate}
        \usepackage{bookmark}

        \begin{document}

            $ \left(
      \bordermatrix{
        &   &        \cr 
        &a_{11} & a_{12} \cr 
        &a_{21} & a_{22} \cr 
        },
      \bordermatrix{
        &   &        \cr 
        &b_{11} & b_{12} \cr 
        &b_{21} & b_{22} \cr 
        } 
             \right) $

              \end{document}

The command \small doesn't seem to work; I am getting errors. I would like the outer parenthesis to be the same size as the parenthesis around the matrices.

A:

\bordermatrix is a plain tex structure, and the fact that it is intended to
incorporate an extra row at the top is what is forcing the outer parentheses to be
much too large.
instead, you can use one of the amsmath "fenced" matrices.  \pmatrix is
what is recommended here:
\documentclass[reqno]{amsart} 

\begin{document}
\[
 \Biggl(
  \begin{pmatrix}
        a_{11} & a_{12} \\
        a_{21} & a_{22}
  \end{pmatrix} .
  \begin{pmatrix}
        b_{11} & b_{12} \\
        b_{21} & b_{22}
  \end{pmatrix}
 \Biggr)
\]

\end{document}

instead of \left and \right -- which in this case give outer parentheses the
same size as the inner ones, i've used \Bigg ones, which are just slightly
larger.
you're already using amsart, so you don't need to load amsmath.  (amsart also incorporates amsthm, so you don't need that either, although it isn't relevant
for this examples.)

A:

If you use the tabular or the matrix environment for your matrices, the \left and \right should always have the correct size.
\documentclass[reqno]{amsart}

\begin{document}

$\left(
  \left(
    \begin{tabular}{ll}
      $a_{11}$ & $a_{12}$ \\
      $a_{21}$ & $a_{22}$ \\
    \end{tabular}
  \right)
   ,
  \left(
    \begin{tabular}{ll}
      $b_{11}$ & $b_{12}$ \\
      $b_{21}$ & $b_{22}$ \\
    \end{tabular}
  \right)
\right)$

$\left(
  \left(
    \begin{matrix}
      a_{11} & a_{12} \\
      a_{21} & a_{22} \\
    \end{matrix}
  \right)
   ,
  \left(
    \begin{matrix}
      b_{11} & b_{12} \\
      b_{21} & b_{22} \\
    \end{matrix}
  \right)
\right)$

\end{document}

A:

\bordermatrix is intended to place additional annotations in the "border" of a matrix. The empty line is seen by \left and \right and the size of the delimiters is increases. Package amsmath (loaded in your preamble) provides several matrix environments, pmatrix which can be used here to set a simple matrix with parentheses:
\documentclass[reqno]{amsart}
\usepackage{amsmath}

\begin{document}
\[
  \left(
    \begin{pmatrix}
      a_{11} & a_{12} \\
      a_{21} & a_{22}
    \end{pmatrix},
    \begin{pmatrix}
      b_{11} & b_{12} \\
      b_{21} & b_{22}
    \end{pmatrix}
  \right)
\]
\end{document}

