Weakness
1. The proposed model's parameterization depends on the number of events and predicates making it difficult to generalize to unseen events or required retraining.
2. The writing needs to be improved to clearly discuss the proposed approach.
3. The experiments baselines are of the authors' own design; it lacks a comparison to the literature baselines using the same dataset. If there is no such baseline, please discuss the criteria in choosing such baselines. Details:
1. Page 1, "causal mechanisms", causality is different from temporal relationship. Please use the terms carefully.
2. Page 3, it seems to me that M_T is defined over the probabilities of atomic events. The notation as it is used not making it difficult to make sense of this concept. Please consider providing examples to explain M_T.
3. Page 4, equation (2), it is not usual to feed probabilities to convolution.
a. Please discuss in section 3 how your framework can handle raw inputs, such as video or audio? Do you need an atomic event predictor or human label to use your proposed system? If so, is it possible to extend your framework to directly have video as input instead of event probability distributions? Can you do end2end training from raw inputs, such as video or audio? (although you mentioned Faster R-CNN in the experiment section, it is better to discuss the whole pipeline in the methodology).
b. Have you tried discrete event embeddings to represent the atomic and composite events so as the framework can learn distributional embedding representation of events so as to learn the temporal rules?
4. Page 4, please explain what you want to achieve with M_A = M_C \otimes M_D. It is unusual to multiple length by conv1D output. Also please define \otimes here. I am guessing it is elementwise multiplication from the context.
5. Page 4, "M_{D:,:,l}=l. This can be thought as a positional encoding. It is not clear to me why this can be taken as positional encoding?
6. Page 6, please detail how do you sample top c predicates. Please define what is s in a = softmax(s). It seems to me the dimension of s with \sum_i (c i) can be quite large making it softmax(s) very costly.
1. Page 1, "causal mechanisms", causality is different from temporal relationship. Please use the terms carefully.