One thing I'm debating with myself is the novelty of this paper. If we view Eq. (2) more generally, it has the form P o u t = P i n ω where ω
is the weighting function. It seems to me that this is a general formulation of many recent step-wise controllable generation methods, including Dexperts and GeDi. Both Dexperts and GeDi rely on discriminators to guide generation. To improve computation efficiency, it is natural to discard discriminators entirely and also avoid fine-tuning. One of the only ways to achieve controllable generation without discriminators and fine-tuning is 1) define attributes with keywords, and 2) impose the control at every generation step. Both these two ideas are not new; 1) appears at least in PPLM (in topic control where each topic is defined with a list of keywords) and 2) is the approach taken by at least Dexperts and GeDi. Both 1) and 2) are used in earlier methods such as [1] (see for example Section 2.2 therein). Therefore, on one end, I think the proposed method does not add much new ideas to the existing line of work in (step-wise) controllable generation. On the other hand, even with the general form above, the design of the weighting function ω
still requires careful design, which is the focus of the present paper. The proposed form, e.g., applying a tangent function on the exp-sum of attribute-related words (tokens), is an interesting way I haven't seen before.
The other concern I have is on the experiments. The authors demonstrate improved performance over PPLM and CTRL, in particular on efficiency, which is kind of expected. However, I am also curious about the comparison of the proposed approach with other inference-time controllable generation methods, such as Dexperts and GeDi. The proposed method would outperform these two methods on efficiency for sure (albeit less significantly so as compared to PPLM and CTRL), but I'm interested in the trade-off between efficiency and performance - it would make a very interesting case if the proposed approach achieves similar performance in terms of the evaluation metrics that the authors have already considered with improved efficiency compared to other inference-time controllable generation methods. Such comparison is currently missing from the experimental results.
[1] https://aclanthology.org/P17-4008.pdf
1) appears at least in PPLM (in topic control where each topic is defined with a list of keywords) and