I list 5 concerns here, with detailed discussion and questions for the authors below
W1: While theorems suggest "existence" of a linear transformation that will approximate the posterior, the actual construction procedure for the "recovered topic posterior" is unclear
W2: Many steps are difficult to understand / replicate from main paper
W3: Unclear what theorems can say about finite training sets
W4: Justification / intuition for Theorems is limited in the main paper
Responses to W1-W3 are most important for the rebuttal.
W1: Actual procedure for constructing the "recovered topic posterior" is unclear
In both synthetic and real experiments, the proposed self-supervised learning (SSL) method is used to produce a "recovered topic posterior" p( w x). However, the procedure used here is unclear... how do we estimate p( w x) using the learning function f(x)?
The theorems imply that a linear function exists with limited (or zero) approximation error for any chosen scalar summary of the doc-topic weights w. However, how such a linear function is constructed is unclear. The bottom of page four suggests that when t=1 and A is full rank, that "one can use the pseudoinverse of A to recover the posterior", however it seems (1) unclear what the procedure is in general and what its assumptions are, and (2) odd that the prior may not needed at all.
Can the authors clarify how to estimate the recovered topic posterior using the proposed SSL method?
W2: Many other steps are difficult to understand / replicate from main paper
Here's a quick list of questions on experimental steps I am confused about / would have trouble reproducing
For the toy experiments in Sec. 5:
Do you estimate the topic-word parameter A? Or assume the true value is given?
What is the format for document x provided as input to the neural networks that define f(x)? The top paragraph of page 7 makes it seem like you provide an ordered list of words. Wouldn't a bag-of-words count vector be a more robust choice?
How do you set t=1 (predict one word given others) but somehow also use "the last 6 words are chosen as the prediction target"?
How do you estimate the "recovered topic posterior" for each individual model (LDA, CTM, etc)? Is this also using HMC (which is used to infer the ground-truth posterior)?
Why use 2000 documents for the "pure" topic model but 500 in test set for other models? Wouldn't more complex models benefit from a larger test set?
For the real experiments in Sec. 6:
How many topics were used?
How did you get topic-word parameters for this "real" dataset?
How big is the AG news dataset? Main paper should at least describe how many documents in train/test, and how many vocabulary words.
W3: Unclear what theorems / methods can say about finite training sets
All the theorems seem to hold when considering terms that are expectations over a known distribution over observed-data x and missing-data y. However, in practical data analysis we do not know the true data generating distribution, we only have a finite training set.
I am wondering about this method's potential in practice for modest-size datasets. For the synthetic dataset with V=5000 (a modest vocabulary size), the experiments considered 0.72 million to 6 million documents, which seems quite large.
What practically must be true of the observed dataset for the presented methods to work well?
W4: Justification / intuition for Theorems is limited in the main paper
All 3 theorems in the main paper are presented without much intuition or justification about why they should be true, which I think limits their impact on the reader. (I'll try to wade thru the supplement, but did not have time before the review deadline).
Theorem 3 tries to give intuition for the t=1 case, but I think could be stronger: why should f(x) have an optimal form p ( y = v 1 x )
? Why should "these probabilities" have the form A E [ w x ]
? I know space is limited, but helping your reader figure things out a bit more explicitly will increase the impact.
Furthermore, the reader would benefit from understanding how tight the bounds in Theorem 4 are. Can we compute the bound quality for toy data and understand it more practically?
Detailed Feedback on Presentation
No need to reply to these in rebuttal but please do address as you see fit in any revision
Page 3:
"many topic models can be viewed"... should probably say "the generative process of many topic models can be viewed..."
the definition of A_ij is not quite right. I would not say "word i \in topic j", I would say "word i topic j". A word is not contained in a topic, Each word has a chance of being generated.
I'd really avoid writing Δ ( K )
and would just use Δ
throughout .... unclear why this needs to be a function of K
but the topic-word parameters (whose size also depends on K
) does not
Should we call the reconstruction objective a "partial reconstruction" or "masked reconstruction"? I'm used to reconstruction in an auto-encoder context, where the usual "reconstruction" objective is literally to recover all observed data, not a piece of observed data that we are pretending not to see
In Eq. 1, are you assuming an ordered or unordered representation of the words in x and y?
Page 4:
I would not reuse the variable y in both reconstruction and contrastive contexts. Find another variable. Same with theta.
Page 5:
I would use f ∗
to denote the exact minimizer, not just f
Figure 2 caption should clarify:
what is the takeaway for this figure? Does reader want to see low values? Does this figure suggest the approach is working as expected?
what procedure is used for the "recovered" posterior? Your proposed SSL method?
why does Pure have a non-monotonic trend as alpha gets larger?
6: How many topics were used? How did you get topic-word parameters for this "real" dataset? How big is the AG news dataset? Main paper should at least describe how many documents in train/test, and how many vocabulary words.