Let $X$ be an $m \times n$ matrix. Define the orthogonal projection onto the column space of $X$ as


Also, define the linear span of a set of $m\times 1$ column vectors of $X$ as $\mbox{span}(X)$.

I'm wondering what the relation between $P(X)$ and vector space $\mbox{span}(X)$ is. Can you help me with an intuitive explanation? Intuitive and easy interpretation would be appreciated.

  • $\begingroup$ I am not following some things. $X$ gives us $n$ vectors of $\mathbb{R}^m$. We define the projection on $\mathbb{R}^m$ ? and we define the projection over the span of the columns of $X$ ? So I suppose $n\leq m$ ? Another thing, I don't catch what means $X'$. What do you refer as a set of $mx1$ vectors, which vectors? $\endgroup$ – HFKy Jul 20 '18 at 2:07
  • $\begingroup$ First, $X'$ is the transpose of $X$. It's assumed that $n \leq m$ holds. $m\times 1$ vectors refer to each $m\times 1$ columns of $X$. $\endgroup$ – user568810 Jul 20 '18 at 2:37
  • $\begingroup$ $m$x$1$ columns ? $X$ doesn't have $n$ column vectors, and $n \leq m ?$ $\endgroup$ – HFKy Jul 20 '18 at 2:39
  • $\begingroup$ Consider EACH column of $X$. $X$ has $n$ number of $m\times 1$ vectors. And yes, $n\leq m$. $\endgroup$ – user568810 Jul 20 '18 at 2:41
  • $\begingroup$ "mx1 vectors" ???. $X$ is mxn is impossible to get "mx1 vectors" whatever it means if $n \leq m$. Being $X$ mxn you have $n$ column vectors of $\mathbb{K}^m$ meaning that each of these vectors have $m$ components. $\endgroup$ – HFKy Jul 20 '18 at 2:45

You have used the "regression" tag, so I assume that the context in linear regression.

The columns of design matrix $X$ form a vector space if there is no intercept in the model $Y= X\beta$, in a case of an intercept this is an affine space. The intuitive relation is that the hat matrix $H = X(X'X)^{-1}X'$ projects the $n$ dimensional response vectors $y$ into the space that is spanned by your explanatory variables. Namely, $Hy=\hat{y}$ gives you the "closest" vector that can be uniquely represented by a linear combination of the columns of $X$ (explanatory variables).


Your Answer

