My question pertains to section 2, called "Multi-class Logistic regression", of this pdf, especially the update rules. (The entire section is only a couple of paragraphs.)

Everything seems to make sense, but I don't see how one is supposed to implement the gradient ascent algorithm.

  • What is $y_i^k$? What are its dimensions?
  • What is the dimension of the weights?
  • What is $x_i$? Say you have a matrix where each row is an observation and each column is a feature. What would $x_i$ be? Is it a row?
  • Is $p(k|x_i)$ the same thing as $p(y = k|x)$ for a particular observation?

Side note: I'm trying to implement a simple multi-class logistic regression in MATLAB.

  • 1
    $\begingroup$ Be wary of any field that invents new names for old ideas or that claims that logistic regression is a classifer. The proper terms for what you are seeking are polytomous logistic regression or multinomial logistic regression. $\endgroup$ – Frank Harrell Jun 6 '14 at 14:43

Suppose $Y$ is a multiclass output of dim $N \times 1$, i.e., the $i^{\rm th}$ row of $Y$, denoted $y_i$ can take values in $\{1,2,\ldots,K\}$. Since the author is using 1-of-K coding, $y_i^k = 1$ if $y_i = k$ and $0$ otherwise.

You also have feature matrix $X$ dim $N \times P$. The weight matrix ${\bf w_k}$ will be of dimension $(P + 1) \times 1$ where the $+ 1$ is for an intercept and will correspond to the $k^{\rm th}$ regression $\rightarrow p(y = k | x)$. Based on the notes, $x_i$ is the $i^{\rm th}$ row of $X$.


Your Answer

