Weakness:
The writing should be improved for proofreading. 1.1. Some figures are far away from the paragraphs describing them. For example, the double selective activation is defined on page 5, while the illustration of this activation appears on page 3 (Fig 1(A)). 1.2. Mathematic formula should be more professional. In the 7th line of section 2, x^(k) are n-dimensional vectors and the relation "<" is not defined for them. Texts in math formula should not be oblique, such as "otherwise", "argmax", and "if".
The claimed contributions are minor or not supported. 2.1. Interpretable constructions for the universal approximation is not novel. Chapter 4 of [1] provides a visual (and can be proven rigorously) proof of the universal approximation, which indicates that neural networks can approximate functions using "almost" piecewise constant functions. 2.2. The proof of the universal approximation using TNN only holds for 1-dimensional inputs, as admitted by authors in the paragraph begins with "generalization to n-dimensions" at the end of page 4. 2.3. The claimed “universal approximation” is not exactly the one machine learning cares. This paper only considers fitting training samples, while the universal approximation in machine learning considers approximating a target function on a compact set. 2.4. The resistance to catastrophic forgetting is not supported by theorems. Authors claim that this contribution is supported by Proposition 1 on page 4, but Proposition 1 has nothing to do with catastrophic forgetting. Catastrophic forgetting is " the tendency for knowledge of previously learned task(s) (e.g. task A) to be abruptly lost as information relevant to the current task (e.g. task B) is incorporated"[2], which needs two sequential tasks. Although D and DUA are two training sets, Proposition 1 has nothing to do with catastrophic forgetting in the following sense: i) DUA contains D as a subset, thus the knowledge in the previous task (D) also exists in the current task and forgetting does not exist obviously; ii) the statement of Proposition 1 is trivial since we can always choose A=D’ and the conclusion in Proposition 1 is the same as Theorem 1. 2.5. The generalization is not the one machine learning cares about. The paragraph begins with “generalizability to test dataset” on page 4 claims that “they (test data) can be incorporated into the training dataset to create a better model with the above-said error upper-bound”. In machine learning, the label of test data should not be used for training.
[1] Nielsen, M. A. (2015). Neural networks and deep learning (Vol. 25). San Francisco, CA: Determination press. [2] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., ... & Hadsell, R. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13), 3521-3526.
25). San Francisco, CA: Determination press. [2] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., ... & Hadsell, R. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13), 3521-3526.