Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

# Noether Embedding:

Efficient Learning of Temporal Regularities

 Chi Gao\({}^{\dagger}\) Zidong Zhou\({}^{\dagger}\) Luping Shi\({}^{*}\)

Center for Brain-Inspired Computing Research,

Optical Memory National Engineering Research Center,

Tsinghua University - China Electronics Technology HIK Group

Co. Joint Research Center for Brain-Inspired Computing,

IDG / McGovern Institute for Brain Research at Tsinghua University,

Department of Precision Instrument,

Tsinghua University, Beijing 100084, China.

{gaoc20, zzd21}@mails.tsinghua.edu.cn

lpshi@mail.tsinghua.edu.cn

Equal contribution.Corresponding author.The code is publicly available at: [https://github.com/KevinGao7/Noether-Embedding](https://github.com/KevinGao7/Noether-Embedding).

###### Abstract

Learning to detect and encode temporal regularities (TRs) in events is a prerequisite for human-like intelligence. These regularities should be formed from limited event samples and stored as easily retrievable representations. Existing event embeddings, however, cannot effectively decode TR validity with well-trained vectors, let alone satisfy the efficiency requirements. We develop Noether Embedding (NE) as the first efficient TR learner with event embeddings. Specifically, NE possesses the intrinsic time-translation symmetries of TRs indicated as conserved local energies in the embedding space. This structural bias reduces the calculation of each TR validity to embedding each event sample, enabling NE to achieve data-efficient TR formation insensitive to sample size and time-efficient TR retrieval in constant time complexity. To comprehensively evaluate the TR learning capability of embedding models, we define complementary tasks of TR detection and TR query, formulate their evaluation metrics, and assess embeddings on classic ICEWS14, ICEWS18, and GDELT datasets. Our experiments demonstrate that NE consistently achieves about double the F1 scores for detecting valid TRs compared to classic embeddings, and it provides over ten times higher confidence scores for querying TR intervals. Additionally, we showcase NE's potential applications in social event prediction, personal decision-making, and memory-constrained scenarios.

## 1 Introduction

Recall the last time you went to a restaurant but waited for half an hour after ordering dishes. You probably knew something was wrong and may have called the waitperson for help. This behavior is guided by the temporal regularity (TR) of 'order dishes -(about 10 minutes)-> have meals' stored in your brain as schemas (Ghosh & Gilboa, 2014). Such TRs play a significant role in enabling humans to exhibit flexible out-of-distribution and systematic generalization abilities (Goyal & Bengio, 2022), and are directly learned from experience through a statistical accumulation of common event structures (Pudhiyidath et al., 2020), as shown in Figure 1. Since there exist enormous potential TRs due to a large number of event types and time intervals, detecting valid TRs from allpotential ones is therefore necessary, serving as a prerequisite capability for humans to form more complex event schemas in the brain to support downstream cognitive functions (Schapiro et al., 2017; McClelland et al., 1995). To attain human-level abilities in event scenarios, it is crucial to possess two fundamental properties when learning TRs. Firstly, TRs should be formed from even a limited number of experiences, which humans achieve since childhood (Pudhiyjdath et al., 2020). Secondly, TRs should be stored as representations that can be instantly retrieved given appropriate cues, which is a central feature of human memory (Chaudhuri & Fiete, 2016).

It remains an open problem to jointly achieve the data-efficient formation and time-efficient retrieval of 1-1 TRs with event embeddings, although embedding models have achieved outstanding performance in various tasks such as event completion and event prediction (Cai et al., 2022; Zhao, 2021). Classic event embeddings can only encode patterns such as inversion and composition and decode the fitted event occurrences for better performance in completion tasks (Xu et al., 2020; Wang et al., 2020; Messner et al., 2022). However, we aim to encode TRs by directly training the embeddings of each event sample and decode TRs by calculating well-trained embeddings without first decoding the fitted event occurrences. Our primary challenge in achieving such a counterintuitive function is to design the inductive bias that automatically integrates the event statistics of each potential TR over time.

Symmetry governs regularities in nature (Tanaka & Kunin, 2021), long before Noether proved the equivalence between symmetries and conservation laws in a physical system (Noether, 1918). Inspired by Noether's theorem, we develop Noether Embedding (NE) with the intrinsic time-translation symmetries of TRs indicated as conserved local energies in the embedding space. Calculating the event statistics of each potential TR is therefore converted to reducing the training loss of all event embeddings. This allows the direct revelation of TR validity by decoding the corresponding local energies through calculating well-trained embeddings after training convergence.

Contributions are twofold. Firstly, we develop NE which for the first time jointly achieves the data-efficient formation and time-efficient retrieval of TRs solely by embedding event samples. Secondly, we define complementary tasks of TR detection and TR query, formulate their evaluation metrics, and adopt classic datasets for evaluations, aiming at complete evaluations of embeddings' TR learning capabilities. Both our tasks and method generalize to arbitrary forms of structured events.

## 2 Problem Formalization

### Definitions

Temporal Regularity (TR)An event item \(q\) can generally be represented by the basic symbolic form of \((ev,t)\), where \(ev\) is the event type, and \(t\) is the discrete occurrence time. Building on the interpretations from cognitive science literature (Ghosh & Gilboa, 2014; Pudhiyjdath et al., 2020), we formally define TRs as temporal associations that remain invariant to time shifts:

\[(ev_{b},t)\rightarrow(ev_{h},t+\tau)\quad\forall t\in\mathbb{T}_{a} \tag{1}\]

\(ev_{b}\neq ev_{h},t,t+\tau\) respectively refer to the body and head event type and their occurrence time. \(\mathbb{T}_{a}\) refers to the complete collection of the absolute time points in the whole event set, and \(\tau\) denotes the relative time. Note that \(\tau=0\) indicates the synchrony of body and head event occurrences.

Figure 1: Illustration of TR learning. TRs indicate temporal associations invariant to time shifts, and are learned from event items through statistical accumulation.

For example, a TR could be: Whenever someone orders dishes in a restaurant, he or she will have meals in around ten minutes, where \(\tau=10\).

Metrics for TR ValiditySince real-world data often contain noise, we introduce an adaptive \(\triangle=[\tau(1-\eta),\tau(1+\eta)]\) to replace \(\tau\) when evaluating statistically learned temporal regularities from events. For an evaluated TR abbreviated as \(tr:(ev_{b},ev_{h},\tau,\eta)\), if an event \(q:(ev,t)\) satisfies that \(ev=ev_{b}\), we denote it as \(b(q;tr)\); if \(ev=ev_{h}\), we denote it as \(h(q;tr)\). We define the support of a TR as the number of event pairs respectively satisfying the body and head:

\[sp(tr)=n(b(q;tr)\wedge h(q^{\prime};tr)\wedge(t^{\prime}-t)\in\triangle(\tau, \eta)) \tag{2}\]

\(\wedge\) denotes 'and', \(\in\) denotes 'in', and \(q:(ev,t),q^{\prime}:(ev^{\prime},t^{\prime})\) refer to arbitrary two different events in the event set. Note that when calculating \(sp(tr)\), we can only count one event once and in one pair to avoid overcounting when events occur in consecutive periods.

We respectively define the standard confidence, head coverage, and general confidence of a TR as:

\[sc(tr)=\frac{sp(tr)}{n(b(tr))},\quad hc(tr)=\frac{sp(tr)}{n(h(tr))},\quad gc( tr)=\frac{2}{sc(tr)}+\frac{1}{hc(tr)} \tag{3}\]

\(n(b(tr)),n(h(tr))\) respectively represent the number of events \(q:(ev,t)\) satisfying \(ev=ev_{b},ev=ev_{h}\) in the event set. Here we borrow the metrics \(sc,hc,gc\) generally used in the rule mining field (Galarraga et al., 2015) to ensure fair and reasonable evaluations. We modify them by introducing an adaptive \(\tau\) with \(\eta\) to evaluate TR validity. Intuitively, standard confidence \(sc\) can be viewed as the probability that the head event will occur within time \(t+\triangle\) once a body event occurs at time \(t\), whose statistical sufficiency is supported by \(sp\). \(hc\) and \(gc\) can be interpreted similarly.

Above some \(sp\), the higher the \(gc\), the more valid a TR is. For a potential TR \(tr:(ev_{b},ev_{h},\tau,\eta)\) with fixed event types \(ev_{b},ev_{h}\) and ratio \(\eta\), its general confidence can be written as a function of \(\tau\): \(gc(\tau)\).

### Tasks

For a fixed \(\eta\) in \(\triangle\), a potential TR can be an arbitrary \((ev_{i},ev_{j},\tau)\), where \(i,j\in\mathbb{P},\tau\in\mathbb{T}_{r}\) (\(\mathbb{P}\) is the set of event types, \(\mathbb{T}_{r}\) is the set of relative time points). Therefore, we define the two complementary tasks below to comprehensively evaluate the TR learning capabilities of event embeddings.

TR DetectionFor a query \((ev_{b},ev_{h})\), its ground truth confidence \(gc_{g}=\max\limits_{\tau\in\mathbb{T}_{r}}gc(\tau)\). Queries whose \(gc_{g}\geq\theta\) are considered to imply valid TRs that reveal good regularities, while queries whose \(gc_{g}<\theta\) are considered to imply invalid TRs, where \(\theta\) is a fixed threshold.

The task of TR detection is to identify valid TRs from all tested ones. The model is expected to determine whether a query \((ev_{b},ev_{h})\) implies a valid TR. The F1 score of all judgments is reported.

TR QueryOnly queries that imply valid TRs are considered for testing. The ground truth relative time \(\tau_{g}\) is set as what maximizes \(gc(\tau)\) in computing \(gc_{g}\). The model outputs \(\tau^{\prime}\).

The task of TR query is to output the correct \(\tau^{\prime}=\tau_{g}\) for valid TRs. For each tested query \((ev_{b},ev_{h})\), a ratio \(r^{\prime}=\frac{gc(\tau^{\prime})}{gc_{g}}\) is computed. The averaged ratio \(r\) of all queries is reported.

## 3 Noether Embedding

Noether's theoremIn 1915, mathematician Emmy Noether proved one of the most fundamental theorems in theoretical physics: every differentiable symmetry of the action of a physical system with conservative forces has a corresponding conservation law. Specifically, time-translation symmetry corresponds to energy conservation.

TRs indicate time-translation symmetriesAn event pair \((ev_{b},t)\rightarrow(ev_{h},t+\tau)\) can be regarded as a mapping of the body and the head event type over \(t\) with a parameter \(\tau\). Therefore, ideal TRs indicate the invariance of such mappings under the transformation of time translation since the mapping holds \(\forall t\in\mathbb{T}_{a}\) for TRs.

Construct embeddings with conserved local energiesDenote \(\mathbf{q}(t;ev)\) as the embedding of each event sample, and \(g(\mathbf{q}(t;ev_{b}),\mathbf{q}(t+\tau;ev_{h}))\) as the local energy of a corresponding body-and-head event pair of a potential TR. If \(g\) is innately conserved, meaning that \(g=g(\tau;ev_{b},ev_{h})\) is invariant to \(t\), it indicates time-translation symmetry. We can then use the value of \(g\) to approximate TR validity after training each event embedding \(\mathbf{q}(t;ev)\). A more strict correspondence between NE variables and those in a physical system is shown in Appendix A.1.1.

Noether-Inspired structural biasesAccordingly, the enabling factors of NE can be summarized as follows: (i) the event embedding \(\mathbf{q}\) should be constructed to make each local energy \(g\) remain invariant to \(t\); (ii) the training loss should be designed to make the value of \(g\) approximate TR validity; (iii) the local energy \(g\) should be used as the decoding function. We thus construct NE as below.

### NE's Framework and Formulas

FrameworkAs illustrated in Figure 2, NE uses a distributed storage of complex vectors to learn TRs. At the encoding stage, the event items are converted to NE representations through embedding each event sample, where TR validity is automatically calculated and stored in the embedding space. At the decoding stage, given each query \((ev_{b},ev_{h})\), potential TRs are detected and queried by directly calculating their relevant embeddings to derive the corresponding decoding results \(g(\tau)\).

#### The Encoding Stage

\[\mathbf{q}(t;ev)=\mathbf{u}(ev)\circ\mathbf{r}(t) \tag{4}\]

In the event embedding above, \(\circ\) denotes the Hadmard (or element-wise) product. \(\mathbf{u}(ev),\mathbf{r}(t)\in\mathbb{C}^{d}\) are complex vectors where \(\mathbf{u}(ev)=\frac{\mathbf{v}(ev)}{||\mathbf{v}(ev)||}\), \(\mathbf{v}(ev)\in\mathbb{C}^{d}\), \(\mathbf{r}(t)=e^{i\mathbf{\omega}t}\), \(\mathbf{\omega}\in\mathbb{R}^{d}\), and \(d\) is the dimension of vectors. Each event type \(ev\) corresponds to an independently trainable vector of \(\mathbf{u}(ev)\), while \(\mathbf{\omega}\) is a global time vector for \(\mathbf{r}(t)\) of all event embeddings. Note that the \(d\)\(\omega\)s in \(\mathbf{\omega}\) are fixed to a certain distribution. The event embedding \(\mathbf{q}(t;ev)\) can thus be depicted as a rotation of event-type vectors \(\mathbf{u}(ev)\) by time \(\mathbf{r}(t)\) in the d-dimensional complex space.

The score function and loss function of each event sample are defined as follows:

\[f(t;ev)=\sum_{i=1}^{d}Real(\mathbf{q}(t;ev))_{i} \tag{5}\]

\[L(\xi;C_{p},C_{n})=(\frac{1}{\sqrt{d}}f(\xi)-C_{p})^{2}+\frac{1}{N}\sum(\frac{ 1}{\sqrt{d}}f(\xi^{\prime})-C_{n})^{2} \tag{6}\]

We denote \(\xi\) as a positive event sample and \(\xi^{\prime}\)s as its generated negative samples whose number is \(N\). For a positive sample \(\xi:(ev,t)\), its negative samples \(\xi^{\prime}\)s are the whole set of \(\{(ev,t^{\prime}\neq t)\}\)

Figure 2: Illustration of NE. Solid lines and purple graphs in the middle jointly represent the data flow of NE. The red graphs below and the blue ones above demonstrate cases of a TR with significant temporal regularities and a TR with no. It is shown that each decoding result reveals an integrated temporal association of its relevant event pairs separated in time.

where \(t^{\prime}\in\mathbb{T}_{a}\). \(C_{p},C_{n}\) are two different constants for positive and negative samples, respectively. \(C_{p},C_{n}\in[-1,1]\) because \(||\mathbf{u}(ev)||=1\), and we generally set \(C_{p}=1,C_{n}=0\).

Until the training converges, events and TRs form a distributed storage, which includes the global time vector of \(\mathbf{\omega}\) and trainable event type vectors \(\mathbf{u}(ev)\). At the decoding stage, no training but only the inference of vectors is conducted, as described below.

The Decoding StageThe decoding function for a query \((ev_{b},ev_{h})\) is:

\[g(\tau)=||\mathbf{u}_{b}-\mathbf{u}_{h}\circ\mathbf{r}(\tau)||^{2} \tag{7}\]

\(\mathbf{u}_{b},\mathbf{u}_{h}\) are the event type vectors respectively of \(ev_{b},ev_{h}\). \(\tau\) is traversed through set \(\mathbb{T}_{r}\) of the relative time points such as \(\mathbb{T}_{r}:\{-\tau_{max},...,0,...,\tau_{max}\}\) to plot the decoding results. \(\min\limits_{\tau\in\mathbb{T}_{r}}g(\tau)\) is computed, which is compared with a global threshold \(g_{th}\) to decide whether a potential TR is valid or not (for TR detection). For a valid TR, the \(\tau^{\prime}\) which minimizes \(g(\tau),\tau\in\mathbb{T}_{r}\) is selected as the model output of the relative time (for TR query).

Since \(\mathbf{r}(t+\tau)=\mathbf{r}(t)\circ\mathbf{r}(\tau)\), \(\mathbf{r}(t)\circ\overline{\mathbf{r}(t)}=\mathbf{1},\forall t\in\mathbb{T}_{a}\), the decoding function exactly indicates the conserved local energy: \(g(\tau)=||\mathbf{u}_{b}-\mathbf{u}_{h}\circ\mathbf{r}(\tau)||^{2}=||\mathbf{u}_{b}\circ\mathbf{r} (t)-\mathbf{u}_{h}\circ\mathbf{r}(t+\tau)||^{2}=||\mathbf{q}_{b}(t)-\mathbf{q}_{h}(t+\tau)||^{ 2},\forall t\in\mathbb{T}_{a}\). This indicates that \(g=g(\tau;ev_{b},ev_{h})\) is invariant to \(t\), and the conserved energy \(g\) of arbitrary two event samples is of a quadratic form in the embedding space.

### Why NE is Efficient

Briefly speaking, the Fourier-like representations enable NE's large-capacity storage for TR validity and event occurrences, serving as a prerequisite for NE to learn TR effectively. The Noether-inspired structural biases further leads to NE's efficient TR learning capabilities. Here we only illustrate the effect of the structural biases. The reasons why NE learns TR effectively is explained in Appendix A.2.

Data-efficiency by Encoding Translation SymmetriesThe invariance of \(g(\tau)\) to \(t\) means that the value of \(g(\tau)\) is determined by a competitive effect between sample pairs across time. Considering event sample pairs \((ev_{b},t),(ev_{h},t+\tau)\) with varying \(t\) in the embedding space, if a sample pair are both positive or both negative samples, they will decrease \(g(\tau)\) since their score functions are mapped to the same constant. Otherwise, they will increase \(g(\tau)\). Since \(g(\tau)\) is invariant to \(t\), \(g(\tau)\) is trained to balance these two forces. Therefore, the value of \(g(\tau)\) after training convergence is generally determined by the ratio of sample pairs with increasing or decreasing forces. \(g(\tau)\) is thus insensitive to the number of sample pairs that are both positive. This results in a data-efficient TR formation in NE, even with limited event occurrences from which to learn a TR.

Time-efficiency by Decoding Conserved EnergiesBy calculating \(g(\tau)=||\mathbf{u}_{b}-\mathbf{u}_{h}\circ\mathbf{r}(\tau)||^{2},\tau\in\mathbb{T}_{r}\), we enable efficient TR querying for each query \((ev_{b},ev_{h})\). This process has a constant time complexity since \(\mathbb{T}_{r}\) is an arbitrary user-selected set of relative time points, and the vector dimension \(d\) can be effectively handled using GPUs. Importantly, the querying time is independent of the number of events in the entire event set and the relevant event occurrences supporting the queried TR.

## 4 Experiment

It is important to highlight that in our evaluation, we initially compare NE and classic embeddings in terms of learning effectiveness (Section 4.2), without considering the efficiency requirements. As classic embeddings are shown to be ineffective in learning TRs, we then focus on demonstrating the learning efficiency of NE in Section 4.3.

### Experimental Setting

DatasetA temporal knowledge graph (TKG) comprises \((s,p,o,t)\) quadruples (Leblay and Chekol, 2018), where \(s,p,o,t\) represent the subject, predicate, object, and time. TKG is widely used in a variety of fields to represent global political events (Trivedi et al., 2017), financial incidents (Yang et al., 2019), user-item behaviors (Xiao et al., 2020), etc. Notably, ICEWS (Boschee et al., 2015) and GDELT (Leetaru and Schrodt, 2013) are two popular data sources for TKG research (Cai et al., 2022).

In our experiments, we use ICEWS14 and ICEWS18, the same as in (Han et al., 2020). They contain global political events in 2014 and 2018, and we denote them as D14 and D18, respectively. We also use the GDELT released by (Jin et al., 2019), which contains global social events from 2018/1/1 to 2018/1/31. In the experiments, we denote each \((s,p,o)\) triple as a specific event type \(ev\). It is worth mentioning that alternative settings, such as representing each predicate \(p\) as an event type \(ev\), are also applicable to our model.

Model ImplementationFor NE, \(d=400,C_{p}=1,C_{n}=0\) and the global time vector \(\mathbf{\omega}\) is set as \(\omega_{k}=\frac{(2\pi\times\omega_{max})^{\frac{k}{2}}-1}{T_{n}},k=0,1,...,d-1\), where \(T_{a}\) is the number of absolute time points, and \(\omega_{max}\) is a tunable hyperparameter set as 600. The training details are in Appendix B.1. We compare NE with six classic and vastly different TKG embeddings, DE-SimplE (Goel et al., 2020), TeRo (Xu et al., 2020b), ATiSE (Xu et al., 2020a), TNTComplex (Lacroix et al., 2020), BoxTE (Messner et al., 2022), and TASTER (Wang et al., 2023) with their original parameter settings and \(d=400\) the same as NE. We set the queried time set \(\mathbb{T}_{r}=\{1-T_{a},...,0,...,T_{a}-1\}\) at the decoding stage. In this way, only one query needs to be decoded between each \((ev_{i},ev_{j})\) and \((ev_{j},ev_{i})\), \(i\neq j\).

Adaptations. All baselines can only output their score function \(f^{\prime}(t)\) to decode event occurrences but cannot directly decode TR validity by \(g(\tau)\) as NE does. For comparison, we add an interface \(g^{\prime}(\tau)=\frac{\sum_{t,t+v\in\mathbb{T}_{a}^{\prime}}f^{\prime}_{b}(t)f ^{\prime}_{b}(t+\tau)}{\sum_{t\in\mathbb{T}_{a}}f^{\prime}_{b}(t)\sum_{t\in \mathbb{T}_{a}}f^{\prime}_{b}(t)}\) to these models that indirectly compute TR validity from the decoded event occurrences. We also evaluate NE with \(g^{\prime}(\tau)\) to show the validity of \(g^{\prime}(\tau)\) itself.

Evaluation DetailsWe select \((ev_{b},ev_{h})\)s for tests whose event occurrences of \(ev_{b}\) and \(ev_{h}\) are both \(\geq 2\). Otherwise, their supports (\(sp\) in Definition 2) will be too small for evaluating TR validity. We set \(\eta=0.1\) in \(\triangle\)s for strict evaluations and take the upper integer \(\triangle=[\tau-\lceil\tau\eta\rceil,\tau+\lceil\tau\eta\rceil]\). Note that in the extreme situation where body and head event occurrences both \(=2\), stochastic noises are still quite unlikely to interfere with the evaluation of TR validity since \(\eta=0.1\) is strict. Only forward or reverse queries (\((ev_{b},ev_{h})\)s whose \(s_{b}=s_{h},o_{b}=o_{h}\) or \(s_{b}=o_{h},o_{b}=s_{h}\) for \((s,p,o,t)\) quadruples) are tested for better interpretability without sacrificing generality. We set \(\theta=0.8\) to distinguish between valid and invalid TRs. The fact that TRs whose \(gc_{g}\sim 0.8\) are of a tiny percentage of all tested TRs adds to the rationality of such metrics. Ablations where \(\theta=0.7,0.9\) are in Appendix B.3.3.

In comparative studies with baselines 4.2, we report the highest F1 in TR detection by tuning the global threshold \(g_{th}\) (defined in Section 3.2) after embedding the whole event set to achieve full evaluations. We also remove TRs whose \(\tau_{g}=0\) for TR query because they account for most valid TRs but can hardly reflect the query difficulty. In NE's demonstration studies 4.3 4.4, we first use D14 to derive the global threshold \(g_{th}\) with the highest F1 in TR detection and then apply the same \(g_{th}\) for evaluating NE's performance in D18. This setting better demonstrates NE's practicality.
excellent performance of NE with \(g^{\prime}(\tau)\) indicates that \(g^{\prime}(\tau)\) itself is valid and thus guarantees fair comparisons. It is worth noting that NE is intrinsically different from all existing baselines because only NE can directly decode TR validity by \(g(\tau)\). In contrast, baselines can only decode event occurrences \(f^{\prime}(t)\) from which to indirectly calculate TR validity (such as by \(g^{\prime}(\tau)\)). Detailed results of precision and recall rates with error bars are reported in Appendix B.2.

DiscussionFigure 3(a) shows NE's decoding distribution \(g_{m}=\min\limits_{\tau\in\mathbb{T}_{r}}g(\tau)\) by each query's ground truth \(gc_{g}=\max\limits_{\tau\in\mathbb{T}_{r}}gc(\tau)\). It can be observed that the decoded conserved local energy accurately reveals the TR validity, which enables NE to successfully distinguish between valid and invalid TRs, as demonstrated in the case shown in Figure 3(b). Table 10 shows that baselines with \(g^{\prime}(\tau)\) still perform poorly. This is mainly because their \(f(t)\)s do not fill well. Specifically, Figure 3(c) illustrates that TNTComplEx has much noise in its \(f^{\prime}(t)\) compared to NE. The reason is that baseline models are generally designed to achieve good performance in the completion task and, therefore, over-apply the generalization capabilities of distributed representations, which hinders the fit of event occurrences.

### NE's Superior Learning Capabilities

Data EfficiencyIn Figure 4(a) and 4(b), we group TRs by their number \(n\) of relevant events. It is shown that NE accurately detects valid TRs and reports correct \(\tau\)s with only two event pairs as positive samples. This performance is comparable to humans, able to form temporally associative memories with minimal experience (Hudson et al., 1992; Bauer and Mandler, 1989; Schapiro et al., 2017). Note that the maximum group in the test has \(n>400\), while we only show groups with \(n\leq 40\) for display considerations.

Time EfficiencyAs explained in 3.3, NE's specific decoding function \(g(\tau)\) enables NE to retrieve TRs in a constant time complexity by vector computations. Calculating \(g^{\prime}(\tau)\) of classic embeddings, however, requires an additional time complexity relevant to \(T_{a}\) (the number of absolute time points).

Storage EfficiencyIn addition to the data-efficient and time-efficient properties, NE is, in fact, also a storage-efficient memory for TRs and event occurrences. Here is a detailed analysis:

Figure 4: Grouped performances of NE

Figure 3: Illustrations of NE or baselines

The storage of NE vectors, denoted as \(S(NE)\), can be calculated as follows: \(S(NE)=S(ev-vector)+S(time-vector)=2*N*d*64bit+2*d*64bit\). In our experiments, we used torch.LongTensor and N represents the number of event-type vectors. On the other hand, the storage of exact counting, denoted as \(S(CT)\), can be calculated as follows: \(S(CT)=S(TR)+S(event)=N^{2}*T_{a}*log_{2}(n/N)bit+N*(n/N)*log_{2}(T_{a})bit\). Here, \(n\) represents the number of all event occurrences. We reserved the storage accuracy of TR validity to effectively distinguish different values, resulting in approximately \(log_{2}(n/N)bit\) for each TR validity \((ev_{b},ev_{h},\tau)\).

For the ICEWS14 and ICEWS18 datasets, where \(d=400,T_{a}=365\), and \(n=90730,468558,N=50295,266631\), we calculated the compression ratio \(\frac{S(CT)}{S(NE)}\) of NE as 421 and 2336, respectively. This remarkable capability of NE can be attributed to the fact that it separately stores the information of TR validities \((ev_{b},ev_{h},\tau)\) using event-type vectors and a global time vector. By representing the common information of related TRs efficiently in memory, NE achieves a compression ratio that is approximately linear to the number of event types.

FlexibilityIn Figure 4(c), we group valid TRs by their golden \(\tau_{g}\)s. NE is shown to be flexible for learning TRs with \(\tau\)s varying broadly, comparable to humans with stable memory codes for various time intervals in the hippocampus (Mankin et al., 2012).

### NE's Wide Potential Use

Potential Use in Social Event PredictionIn D18, NE successfully reports 21010 valid TRs with an F1 score of 0.83. The encoding stage takes around 1 hour, while decoding only takes less than 1 minute. Cases are presented below and in Figure 6, with additional cases available in Appendix B.4.

(1) Case 1. Citizen (India) will _Reject_ to Narendra Modi (events in day 23, 122, and 168) in around 87 days whenever Narendra Modi _Appeal for diplomatic cooperation (such as policy support)_ to Citizen (India) (events in day 102, 200, and 264).

(2) Case 2. Russia will _Meet at a 'third' location_ with Ukraine (events in day 31 and 138) in around 136 days whenever Ukraine _Use conventional military force_ to Russia (events in day 161 and 288).

Since the TRs mined can generalize across time, the results above imply NE's potential use in both reliable and interpretable event predictions urgently needed in the big data era (Zhao, 2021).

Figure 5: Social event prediction Figure 6: Personal decision making

Potential Use in Personal Decision MakingConsider an intelligent machine that has visited a restaurant four times, with the occurrence time of each event episode used as input for NE, as shown in Figure 6(a). After training all events, the decoded TR validity \(\min\limits_{\tau\in\mathbb{T}_{r}}g(\tau)\) is transformed linearly and demonstrated in Figure 6(b). Despite the recurrent TRs on the slash that can be set aside, valid TRs such as 'order dishes -(about 10 minutes)-> have meals' are well distinguished from invalid ones such as 'order dishes -(about 5 minutes) -> look at the floor'.

Combining NE with front-end methods that take unstructured videos as input and output event items in the form of \((ev,t)\), and with back-end methods that use the decoded valid TRs to guide decision-making, NE has the potential to aid intelligent machines in surviving in changing environments by generalizing from little experience, just as human beings (Goyal & Bengio, 2022; Xue et al., 2018).

Potential Use in Memory-constrained ScenariosAs discussed in 4.3, NE approximately reduces the required storage space from \(M^{2}\) to \(M\) in our experimental settings, where \(M\) is the number of event types. Therefore, NE holds significant potential for applications in memory-constrained scenarios like the edge. This is important when \(M\) is large, which is usual in the big-data era.

### Ablation Studies

Here we demonstrate ablation studies of loss constants \(C_{p},C_{n}\) and time vector \(\mathbf{\omega}\), while those for dimension \(d\) and event type vector \(\mathbf{u}\) are shown in Appendix B.3.

Loss ConstantsTable 2 shows that NE performs optimally when \(C_{p}=1\) and \(C_{n}=0\). In fact, as \(C_{p}\) approaches 1, the \(g(\tau)\) of perfect TRs (\(gc(\tau)=1,\eta=1\)) is enforced to converge to its minimum \(0\). This global constant for all potential TRs in the embedding space allows \(g(\tau)\) to reveal TR validity better. In terms of \(C_{n}\), setting it to \(0\) results in negative samples occupying the largest embedding space. Since negative samples comprise most of all trained event samples, this setting improves the fit of negative samples and optimizes NE's performance.

Maximal Frequency CoefficientTable 3 shows that NE performs optimally with different values of \(\omega_{max}\), respectively, in the TR detection and query task.

\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{D14 (90730 events)} & \multicolumn{3}{c}{D18 (468558 events)} & \multicolumn{3}{c}{GDELT (2278405 events)} \\ \cline{2-11} \(\{\omega_{k}\}\) & linear & exponential & linear & exponential & linear & exponential \\ \hline TR Query (r) & 0.81 & 0.82 & 0.75 & 0.82 & 0.24 & 0.85 \\ \hline \hline \end{tabular}
\end{table}
Table 4: NE on the three datasets with increasing events, and with different distributions of \(\{\omega_{k}\}\)

\begin{table}
\begin{tabular}{c|c c c c c|c c c c} \hline \hline \multirow{2}{*}{\(C_{p}\)} & \multicolumn{3}{c|}{**TR Detection (F1)**} & \multicolumn{3}{c}{**TR Query (r)**} \\  & 0.4 & 0.2 & 0 & -0.2 & -0.4 & 0.4 & 0.2 & 0 & -0.2 & -0.4 \\ \hline
1 & 0.78 & 0.80 & 0.82 & 0.81 & 0.80 & 0.86 & 0.87 & 0.87 & 0.87 & 0.87 \\
0.8 & 0.64 & 0.67 & 0.79 & 0.80 & 0.80 & 0.85 & 0.86 & 0.86 & 0.87 & 0.86 \\
0.6 & 0.29 & 0.40 & 0.68 & 0.79 & 0.79 & 0.44 & 0.82 & 0.86 & 0.86 & 0.85 \\
0.4 & 0.18 & 0.31 & 0.49 & 0.76 & 0.79 & 0.12 & 0.35 & 0.84 & 0.84 & 0.81 \\
0.2 & 0.53 & 0.19 & 0.27 & 0.71 & 0.78 & 0.27 & 0.05 & 0.28 & 0.79 & 0.71 \\ \hline \hline \end{tabular}
\end{table}
Table 2: NE on ICEWS14 in different \(C_{p}\) and \(C_{n}\) settings

\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \(\omega_{max}\) & 1 & 5 & 10 & 50 & 100 & 200 & 400 & 600 & 800 \\ \hline TR Query (r) & 0.15 & 0.93 & 0.92 & 0.85 & 0.85 & 0.85 & 0.85 & 0.85 & 0.85 & 0.85 \\ TR Detection (F1) & 0.22 & 0.45 & 0.55 & 0.56 & 0.54 & 0.53 & 0.53 & 0.51 & 0.53 \\ \hline \hline \end{tabular}
\end{table}
Table 3: NE on GDELT with different \(\omega_{max}\)sFrequency DistributionTable 4 shows that the larger the dataset, the more exponential distribution (\(\omega_{k}=\frac{(2\pi\times\omega_{max})^{\frac{k}{2}}-1}{I_{a}},k=0,1,...,d-1\)) surpasses linear distribution (\(\omega_{k}=\frac{2\pi\times k\times\omega_{max}}{d\times I_{a}},k=0,1,...,d-1\)) with the same parameters of \(d=400,\omega_{max}=600\). This suggests that real-world event occurrences depend more on low-frequency terms than high-frequency ones.

## 5 Related Work

Event Schema InductionIn the natural language processing (NLP) field, a significant research focus is on inducing event schemas from text (Huang et al., 2016; Li et al., 2021), including from language models (Dror et al., 2022), to support downstream applications such as search, question-answering, and recommendation (Guan et al., 2022). These NLP methods aim to organize known event regularities already given as priors for the extracting algorithm (such as extracting 'earthquake -> tsunami' from the sentence 'An earthquake causes a tsunami.') and focus on the schemas for use. In contrast, our tasks are designed to learn event regularities directly from experience without supervision. Specifically, the only prior models know is whether an event occurs, and models are required to detect valid TRs from all potential ones and report the correct relative time of valid TRs.

Temporal Rule MiningVarious temporal rules are mined from event sets to reveal regularities in industry, security, healthcare, etc (Segura-Delgado et al., 2020; Chen et al., 2007; Yoo and Shekhar, 2008; Namaki et al., 2017). Although the search methods used discover event regularities directly from events without supervision, both the mined rules and source events are generally stored as symbolic representations in list form. In contrast, by applying event embeddings, NE is a distributed and approximate memory for both TRs and event items. NE strikes a balance between storage efficiency and storage accuracy compared to exact counting, as detailedly discussed in 4.3.

Embedding Models of Structured DataWithin all embedding models of static and temporal knowledge graphs (Chen et al., 2020; Cai et al., 2022; Wang et al., 2020; Messner et al., 2022), three are most related to NE. RotatE (Sun et al., 2019) represents each entity and relationship as a complex vector to model relation patterns on knowledge graphs, and TeRo (Xu et al., 2020) represents time as a rotation of entities to model time relation patterns on temporal knowledge graphs. While both RotatE and TeRo introduce complex vectors for better completion performance, NE first explores using complex vectors for TR detection in events. In particular, the specific use of complex vectors in RotatE encodes inverse relations and in TeRo encodes asymmetric and reflexive relations. NE, instead, apply rotating complex unit vectors to encode time-translation symmetries of all potential TRs. IterE (Zhang et al., 2019) construct a decodable embedding model to discover rules for better knowledge graph completion performance. While we take functional inspiration from IterE that embedding models can jointly encode data and decode regularities, we focus on event data and define the new problems of TR detection and TR query. Specifically, while IterE focuses on discrete variables, NE focuses on the continuous variable of time that involves Fourier-like transformations.

To summarize, TR detection and TR query focus on achieving human-like schema learning capabilities rather than pursuing better support for NLP applications like the event schema induction task. Meanwhile, NE leverages the advantages of distributed representations over symbolic ones of search methods in temporal rule mining and is distinct from existing embedding models of structured data.

## 6 Conclusion

We have developed NE which for the first time enables data-efficient TR formation and time-efficient TR retrieval simply through embedding event samples. We have formally defined the tasks of TR detection and TR query to comprehensively evaluate the TR learning capabilities of embedding models. We have demonstrated NE's potential use in social event prediction, personal decision-making, and memory-constrained scenarios. We hope that we have facilitated the development of human-like event intelligence.

One limitation of NE is that when the vector dimension \(d\) is set much lower than the number of absolute time points \(T_{a}\), significant performance degradation of NE will occur as observed in the GDELT experiment. Future research is needed to improve this weakness. The privacy issues potentially brought about by TR detection and the causality of TRs should also be handled properly.

## Acknowledgements

We sincerely thank Rong Zhao, Hao Zheng, Dahu Feng, Lukai Li, Hui Zeng, Wenhao Zhou, Yu Du, Songchen Ma, and Faqiang Liu for their valuable comments and encouragements. This work was supported by the National Nature Science Foundation of China (nos. 62088102, 61836004) and the National Key Research and Development Program of China (no. 2021ZD0200300).

## References

* Bauer and Mandler (1989) Bauer, P. J. and Mandler, J. M. One thing follows another: Effects of temporal structure on 1-to 2-year-olds' recall of events. _Developmental psychology_, 25(2):197, 1989.
* Boschee et al. (2015) Boschee, E., Lautenschlager, J., O'Brien, S., Shellman, S., Starz, J., and Ward, M. Icews coded event data. _Harvard Dataverse_, 12, 2015.
* Bright et al. (2020) Bright, I. M., Meister, M. L., Cruzado, N. A., Tiganj, Z., Buffalo, E. A., and Howard, M. W. A temporal record of the past with a spectrum of time constants in the monkey entorhinal cortex. _Proceedings of the National Academy of Sciences_, 117(33):20274-20283, 2020.
* Cai et al. (2022) Cai, B., Xiang, Y., Gao, L., Zhang, H., Li, Y., and Li, J. Temporal knowledge graph completion: A survey. _arXiv preprint arXiv:2201.08236_, 2022.
* Chaudhuri and Fiete (2016) Chaudhuri, R. and Fiete, I. Computational principles of memory. _Nature neuroscience_, 19(3):394-403, 2016.
* Chen et al. (2007) Chen, M., Chen, S.-C., and Shyu, M.-L. Hierarchical temporal association mining for video event detection in video databases. In _2007 IEEE 23rd International Conference on Data Engineering Workshop_, pp. 137-145. IEEE, 2007.
* Chen et al. (2020) Chen, X., Jia, S., and Xiang, Y. A review: Knowledge reasoning over knowledge graph. _Expert Systems with Applications_, 141:112948, 2020.
* Dror et al. (2022) Dror, R., Wang, H., and Roth, D. Zero-shot on-the-fly event schema induction. _arXiv preprint arXiv:2210.06254_, 2022.
* Galarraga et al. (2015) Galarraga, L., Teflioudi, C., Hose, K., and Suchanek, F. M. Fast rule mining in ontological knowledge bases with amie \(+\). _The VLDB Journal_, 24(6):707-730, 2015.
* Ghosh and Gilboa (2014) Ghosh, V. E. and Gilboa, A. What is a memory schema? a historical perspective on current neuroscience literature. _Neuropsychologia_, 53:104-114, 2014.
* Goel et al. (2020) Goel, R., Kazemi, S. M., Brubaker, M., and Poupart, P. Diachronic embedding for temporal knowledge graph completion. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 34, pp. 3988-3995, 2020.
* Goyal and Bengio (2022) Goyal, A. and Bengio, Y. Inductive biases for deep learning of higher-level cognition. _Proceedings of the Royal Society A_, 478(2266):20210068, 2022.
* Guan et al. (2022) Guan, S., Cheng, X., Bai, L., Zhang, F., Li, Z., Zeng, Y., Jin, X., and Guo, J. What is event knowledge graph: A survey. _IEEE Transactions on Knowledge and Data Engineering_, 2022.
* Han et al. (2020) Han, Z., Chen, P., Ma, Y., and Tresp, V. serte: Explainable reasoning on temporal knowledge graphs for forecasting future links. _arXiv preprint arXiv:2012.15537_, 2020.
* Howard et al. (2014) Howard, M. W., MacDonald, C. J., Tiganj, Z., Shankar, K. H., Du, Q., Hasselmo, M. E., and Eichenbaum, H. A unified mathematical framework for coding time, space, and sequences in the hippocampal region. _Journal of Neuroscience_, 34(13):4692-4707, 2014.
* Huang et al. (2016) Huang, L., Cassidy, T., Feng, X., Ji, H., Voss, C., Han, J., and Sil, A. Liberal event extraction and event schema induction. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pp. 258-268, 2016.
* Hudson et al. (1992) Hudson, J. A., Fivush, R., and Kuebli, J. Scripts and episodes: The development of event memory. _Applied Cognitive Psychology_, 6(6):483-505, 1992.
* Huang et al. (2016)Jin, W., Jiang, H., Qu, M., Chen, T., Zhang, C., Szekely, P., and Ren, X. Recurrent event network: Global structure inference over temporal knowledge graph. 2019.
* Lacroix et al. (2020) Lacroix, T., Obozinski, G., and Usunier, N. Tensor decompositions for temporal knowledge base completion. _arXiv preprint arXiv:2004.04926_, 2020.
* Leblay & Chekol (2018) Leblay, J. and Chekol, M. W. Deriving validity time in knowledge graph. In _Companion Proceedings of the The Web Conference 2018_, pp. 1771-1776, 2018.
* Leetaru & Schrodt (2013) Leetaru, K. and Schrodt, P. A. Gdelt: Global data on events, location, and tone, 1979-2012. In _ISA annual convention_, volume 2, pp. 1-49. Citeseer, 2013.
* Li et al. (2021) Li, M., Li, S., Wang, Z., Huang, L., Cho, K., Ji, H., Han, J., and Voss, C. The future is not one-dimensional: Complex event schema induction by graph modeling for event prediction. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pp. 5203-5215, 2021.
* Mankin et al. (2012) Mankin, E. A., Sparks, F. T., Slayyeh, B., Sutherland, R. J., Leutgeb, S., and Leutgeb, J. K. Neuronal code for extended time in the hippocampus. _Proceedings of the National Academy of Sciences_, 109(47):19462-19467, 2012.
* McClelland et al. (1995) McClelland, J. L., McNaughton, B. L., and O'Reilly, R. C. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. _Psychological review_, 102(3):419, 1995.
* Messner et al. (2022) Messner, J., Abboud, R., and Ceylan, I. I. Temporal knowledge graph completion using box embeddings. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 36, pp. 7779-7787, 2022.
* Namaki et al. (2017) Namaki, M. H., Wu, Y., Song, Q., Lin, P., and Ge, T. Discovering graph temporal association rules. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_, pp. 1697-1706, 2017.
* Noether (1918) Noether, E. Invariante variationsprobleme. _Nachrichten von der Gesellschaft der Wissenschaften zu Gottingen, Mathematisch-Physikalische Klasse_, 1918:235-257, 1918. URL [http://eudml.org/doc/59024](http://eudml.org/doc/59024).
* Pudhiyidath et al. (2020) Pudhiyidath, A., Roome, H. E., Coughlin, C., Nguyen, K. V., and Preston, A. R. Developmental differences in temporal schema acquisition impact reasoning decisions. _Cognitive neuropsychology_, 37(1-2):25-45, 2020.
* Schapiro et al. (2017) Schapiro, A. C., Turk-Browne, N. B., Botvinick, M. M., and Norman, K. A. Complementary learning systems within the hippocampus: a neural network modelling approach to reconciling episodic memory with statistical learning. _Philosophical Transactions of the Royal Society B: Biological Sciences_, 372(1711):20160049, 2017.
* Segura-Delgado et al. (2020) Segura-Delgado, A., Gacto, M. J., Alcala, R., and Alcala-Fdez, J. Temporal association rule mining: An overview considering the time variable as an integral or implied component. _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_, 10(4):e1367, 2020.
* Sun et al. (2019) Sun, Z., Deng, Z.-H., Nie, J.-Y., and Tang, J. Rotate: Knowledge graph embedding by relational rotation in complex space. _arXiv preprint arXiv:1902.10197_, 2019.
* Tanaka & Kunin (2021) Tanaka, H. and Kunin, D. Noether's learning dynamics: Role of symmetry breaking in neural networks. _Advances in Neural Information Processing Systems_, 34:25646-25660, 2021.
* Trivedi et al. (2017) Trivedi, R., Dai, H., Wang, Y., and Song, L. Know-evolve: Deep temporal reasoning for dynamic knowledge graphs. In _international conference on machine learning_, pp. 3462-3471. PMLR, 2017.
* Tsao et al. (2018) Tsao, A., Sugar, J., Lu, L., Wang, C., Knierim, J. J., Moser, M.-B., and Moser, E. I. Integrating time from experience in the lateral entorhinal cortex. _Nature_, 561(7721):57-62, 2018.
* Wang et al. (2020) Wang, J., Zhang, W., Chen, X., Lei, J., and Lai, X. 3drte: 3d rotation embedding in temporal knowledge graph. _IEEE Access_, 8:207515-207523, 2020.
* Wang et al. (2018)Wang, X., Lyu, S., Wang, X., Wu, X., and Chen, H. Temporal knowledge graph embedding via sparse transfer matrix. _Information Sciences_, 623:56-69, 2023.
* Xiao et al. (2020) Xiao, C., Sun, L., and Ji, W. Temporal knowledge graph incremental construction model for recommendation. In _Asia-pacific web (apweb) and web-age information management (waim) joint international conference on web and big data_, pp. 352-359. Springer, 2020.
* Xu et al. (2020a) Xu, C., Nayyeri, M., Alkhoury, F., Yazdi, H., and Lehmann, J. Temporal knowledge graph completion based on time series gaussian embedding. In _International Semantic Web Conference_, pp. 654-671. Springer, 2020a.
* Xu et al. (2020b) Xu, C., Nayyeri, M., Alkhoury, F., Yazdi, H. S., and Lehmann, J. Tero: A time-aware knowledge graph embedding via temporal rotation. _arXiv preprint arXiv:2010.01029_, 2020b.
* Xue et al. (2018) Xue, J.-R., Fang, J.-W., and Zhang, P. A survey of scene understanding by event reasoning in autonomous driving. _Machine Intelligence Research_, 15(3):249-266, 2018.
* Yang et al. (2019) Yang, Y., Wei, Z., Chen, Q., and Wu, L. Using external knowledge for financial event prediction based on graph neural networks. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_, pp. 2161-2164, 2019.
* Yoo & Shekhar (2008) Yoo, J. S. and Shekhar, S. Similarity-profiled temporal association mining. _IEEE Transactions on Knowledge and Data Engineering_, 21(8):1147-1161, 2008.
* Zhang et al. (2019) Zhang, W., Paudel, B., Wang, L., Chen, J., Zhu, H., Zhang, W., Bernstein, A., and Chen, H. Iteratively learning embeddings and rules for knowledge graph reasoning. In _The World Wide Web Conference_, pp. 2366-2377, 2019.
* Zhao (2021) Zhao, L. Event prediction in the big data era: A systematic survey. _ACM Computing Surveys (CSUR)_, 54(5):1-37, 2021.

## Appendix A Theoretical Illustrations

### Interdisciplinary Correspondences

#### a.1.1 Physical Correspondence

Considering each event type as a particle, the event embedding \(\mathbf{q}(t;ev)\) can be viewed as its generalized coordinate at time \(t\) in the d-dimensional complex embedding space. Suppose that each two particles \(ev_{b},ev_{h}\) are connected by a spring with the force linear to their distance, their potential energy can then be expressed as \(||\mathbf{q}_{b}(t)-\mathbf{q}_{h}(t+\tau)||^{2}\), where \(\tau\) is a parameter. As \(\mathbf{q}(t;ev)\) changes with time, its kinetic energy also changes and can be viewed as being driven by external nonconservative forces. Using the extended Noether's theorem, we can see that the work of external forces and the kinetic energy offset, leading to a conserved quantity of time-translation symmetry that only includes the potential energy instead of the total energy. The inductive bias of Noether Embedding exactly enforces that such local potential energies are innately conserved, which means that \(||\mathbf{q}_{m}(t_{k})-\mathbf{q}_{h}(t_{k}+\tau_{r})||^{2}=||\mathbf{q}_{m}(t_{s})-\mathbf{q} _{n}(t_{s}+\tau_{r})||^{2},\forall ev_{m},ev_{n}\in\mathbb{P},t_{k},t_{s}\in \mathbb{T}_{a},\tau_{r}\in\mathbb{T}_{r}\), where \(\mathbb{P},\mathbb{T}_{a},\mathbb{T}_{r}\) refer to the set of event types, absolute time points, and relative time points, respectively.

#### a.1.2 Biological Correspondence

The formation of TRs relies on two central functions, namely measurement and integration. The measurement function involves comparing the temporal distance between each two events. It is claimed that Laplace transformation exists in the hippocampus for representing time (Howard et al., 2014). Specifically, the population of temporal context cells (or referred to as ramping cells (Tsao et al., 2018)) in the lateral entorhinal cortex is discovered to code time with a variety of rate constants (Bright et al., 2020). \(\mathbf{r}(t)\) in NE functionally corresponds to such a cell population, which stores the rate distribution in vector \(\mathbf{\omega}\) of \(\mathbf{r}(t)\). The integration function aggregates event pairs separated in time to form TRs. Statistical learning for schema formation is reported to occur in the pathway connecting EC to CA1 in the hippocampus (Schapiro et al., 2017). In NE, we achieve the same function through the time-translation symmetry introduced by \(\mathbf{r}(t)\): \(\mathbf{r}(t+\tau)=\mathbf{r}(t)\circ\mathbf{r}(\tau)\), \(\mathbf{r}(t)\circ\overline{\mathbf{r}(t)}=\mathbf{1},\forall t\in\mathbb{T}_{a}\).

### Revelation of TR Validity

#### a.2.1 Effectiveness in TR Query

NE is an effective method for TR query due to its distributed storage of cross-correlations.

The score function \(f(t)\) describes the occurrences of each event type, where its value is mapped to \(C_{p}\) for time points implying event occurrences and \(C_{n}\) for those that do not. The support \(sp(\tau)\) of two event types \(ev_{b},ev_{h}\) can be then approximated by calculating the time-lagged cross-correlation \(R_{f_{b},f_{h}}(\tau)=\sum_{t,t+\tau\in\mathbb{T}_{a}}f_{b}(t)f_{h}(t+\tau)\) after training convergence.

Denote \(F(\omega)\) as the Fourier expansion of \(f(t)\). Awaraing that in NE, \(f(t)=\sum_{j=1}^{d}Real(\mathbf{u}\circ e^{i\omega t})_{j}\), we can see that the encoding stage enables a \(d\)-dimensional Fourier-like expansion for each \(f(t)\). The time vector \(\mathbf{\omega}\) provides the expansion basis and \(\mathbf{u}\) stores the coefficients as \(F(\omega)\)s.

By Fourier expansions, the corresponding correlation \(R^{\prime}_{f_{b},f_{h}}(\tau)=\int_{-\infty}^{+\infty}f_{b}(t)f_{h}(t+\tau) dt=\int_{-\infty}^{+\infty}\overline{F_{b}(\omega)}F_{h}(\omega)e^{i\omega \tau}d\omega\). In NE, \(g(\tau)=||\mathbf{u}_{b}-\mathbf{u}_{h}\circ\mathbf{r}(\tau)||^{2}=2-2\sum_{j=1}^{d}Real( \overline{\mathbf{u}_{b}}\circ\mathbf{u}_{h}\circ e^{i\omega\tau})_{j}\). Therefore, each \(g(\tau)\) reveals exactly the fitted time-lagged cross-correlation and is thus proportional to the support \(sp(\tau)\) of a TR. This guarantees NE's effectiveness in TR query.

#### a.2.2 Effectiveness in TR Detection

NotationsWithin all event samples trained for a potential TR, suppose that \(m\) event pairs share the same relative time of \(\tau\), where \(m_{1}\) pairs are both positive or both negative samples. In each remaining \(m_{2}=m-m_{1}\) pair, one is a positive sample while the other is a negative one. Denote that \(\mathbb{M}=\{1,2,...,m\},\mathbb{M}_{1}=\{i_{1},i_{2},...,i_{m_{1}}\},\mathbb{ M}_{2}=\{j_{1},j_{2},...,j_{m_{2}}\}\). There exists such least upper bound \(\eta\) that \((\frac{1}{\sqrt{d}}f(t;ev)-C)^{2}<\eta\) for the score functions of all \(2m\) samples, where \(C=C_{p},C_{n}\)Denote that \(\mathbf{a}_{i}=(cos(\mathbf{\omega}_{1}t_{i}),cos(\mathbf{\omega}_{2}t_{i}),...,cos(\mathbf{ \omega}_{d}t_{i}),sin(\mathbf{\omega}_{1}t_{i}),sin(\mathbf{\omega}_{2}t_{i}),...,sin( \mathbf{\omega}_{d}t_{i}))^{T}\), \(i\in\mathbb{M}\), where \(d\) is the dimension of NE vectors, \(t_{i}\) is time of the \(i\)th body event in \(m\) sample pairs. Denote that in the decoding function, \(\mathbf{u}_{b}-\mathbf{u}_{h}\circ\mathbf{r}(\tau)=\mathbf{\alpha}-i\mathbf{\beta}\) where \(\mathbf{\alpha},\mathbf{\beta}\) are both real vectors, and \(\mathbf{x}=(\mathbf{\alpha}_{1},\mathbf{\alpha}_{2},...,\mathbf{\alpha}_{d},\mathbf{\beta}_{1}, \mathbf{\beta}_{2},...,\mathbf{\beta}_{d})^{T}\). Denote that \(c_{1}=\underset{i\in\mathbb{M}_{1}}{max}(cos(\mathbf{a}_{i},\mathbf{x}))^{2},c_{2}= \underset{j\in\mathbb{M}_{2}}{min}(cos(\mathbf{a}_{j},\mathbf{x}))^{2}\).

Using the trigonometric inequality, we can derive the following conclusions:

Theorem 1Within the \(m_{1}\) sample pairs, if \(c_{1}>0\), then \(g(\tau)<\frac{4\eta}{c_{1}}\).

Theorem 2Within the \(m_{2}\) sample pairs, if \(c_{2}>0\) and \(|C_{p}-C_{n}|>2\sqrt{\eta}\), then \(g(\tau)>\frac{(|C_{p}-C_{n}|-2\sqrt{\eta})^{2}}{c_{2}}\).

Implications\(d,T_{a},\mathbf{\omega},C_{p},C_{n}\) jointly result in the probability distributions of \(c_{1},c_{2}\), where \(c_{1},c_{2}>0\) is generally guaranteed by experimental settings. Two conclusions can then be drawn from the two theorems. (1) Convergence. We can see from theorem 1 that \(g(\tau)\to 0\) as \(\eta\to 0\). This implies that the \(g(\tau)\) of perfect TRs (\(gc(\tau)=1,\eta=1\)) will converge to its minimum of 0. (2) Competition. Since \(g(\tau)\) is invariant to \(t\), comparing these two theorems also tells us about the competing effect of well-trained sample pairs for the value of \(g(\tau)\), generally affected by the ratio \(\frac{m_{1}}{m_{2}}\). These two conclusions, along with the fact that \(g(\tau)\) is proportional to the support \(sp(\tau)\) (as illustrated in A.2.1), jointly make \(g(\tau)\) reveal the TR validity \(gc(\tau)\) and thus guarantees NE's effectiveness in TR detection.

It is worth noting that \(d\) is generally set as \(d>T_{a}\) to control the values of \(c_{1},c_{2}\), where \(T_{a}\) is the number of absolute time points of the whole event set. Otherwise, NE's TR detection performance will be interfered. For example, if \(d\ll T_{a}\), then \(d\ll m\) in most cases. It will thus be very likely that \(c_{1}=0\) so that theorem 1 can not be applied in NE.

## Appendix B Experimental Supplements

### Training Details

All models are trained for 100 epochs on each dataset using the Adagrad optimizer (with a learning rate of 0.01) and the StepLR learning rate scheduler (with a step size of 10 and gamma of 0.9). The experiments are conducted on a single GPU (GeForce RTX 3090). The hyper-parameters of NE are fine-tuned using a grid search to achieve relatively optimal results.

### Main Results with Error Bars

To ensure the reliability of the results, the experiments are repeated three times, and the error bars are derived accordingly. Table 5 and 6 show that the main results are quite stable with small error bars. Note that the precision and recall rates in Table 6 correspond exactly to the F1 scores in Table 5. The reason why the recall rate of NE is lower than that of TASTER is that we report the highest F1 score of each model in comparative studies by tuning their respective global threshold, denoted as \(g_{th}\). As the F1 score is calculated as the harmonic mean of precision and recall rates, TASTER achieves its highest F1 score by reporting many false positives, resulting in a relatively high recall rate but an extremely low precision rate.

### Additional Ablations

Unless otherwise specified, the experiments below adopt the original parameter settings as described in the main text.

#### b.3.1 Normalization of Event Type Vectors

Table 7 demonstrates that normalized event type vectors slightly outperform unnormalized ones in the TR detection task.

#### b.3.2 Dimension of Embedding Vectors

Table 8 and 9 demonstrate that \(d\) scarcely affects the performance of NE as long as it is more than some certain value. It is worth noting that NE's detection performance in GDELT may be further improved with larger values of \(d\)s, as illustrated in A.2.2, because \(T_{a}=2976\) in GDELT, which is much larger than the tested dimensions.

#### b.3.3 Threshold for Valid TRs

In the main text, the threshold for distinguishing valid and invalid TRs is chosen as \(\theta=0.8\). Here we report NE's results on D14 with varying \(\theta\)s in Table 10. It is shown that NE still has an overwhelming advantage over all baselines.

### Mined TRs

Additional cases of mined TRs are shown in Table 11 as below.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{**Embedding**} & \multicolumn{3}{c}{**Precision**} & \multicolumn{3}{c}{**Recall**} \\  & D14 & D18 & GDELT & D14 & D18 & GDELT \\ \hline TNTComplEx & 0.22\(\pm\)0.01 & 0.11\(\pm\)0.00 & 0.04\(\pm\)0.00 & 0.33\(\pm\)0.02 & 0.50\(\pm\)0.01 & 1.00\(\pm\)0.01 \\ DE-SimplE & 0.16\(\pm\)0.00 & 0.14\(\pm\)0.00 & - & 0.35\(\pm\)0.03 & 0.33\(\pm\)0.00 & - \\ TASTER & 0.10\(\pm\)0.00 & 0.08\(\pm\)0.00 & 0.04\(\pm\)0.00 & 0.99\(\pm\)0.01 & 0.99\(\pm\)0.01 & 0.97\(\pm\) 0.03 \\ TeRo & 0.51\(\pm\)0.04 & 0.91\(\pm\)0.01 & 0.20\(\pm\)0.04 & 0.37\(\pm\)0.02 & 0.49\(\pm\)0.00 & 0.14\(\pm\)0.02 \\ BoxTE & 0.40\(\pm\)0.02 & 0.39\(\pm\)0.05 & 0.15\(\pm\)0.01 & 0.41\(\pm\)0.02 & 0.41\(\pm\)0.02 & 0.22\(\pm\)0.03 \\ ATISE & 0.35\(\pm\)0.01 & 0.49\(\pm\)0.03 & 0.15\(\pm\)0.01 & 0.47\(\pm\)0.01 & 0.40\(\pm\)0.00 & 0.21\(\pm\)0.01 \\ \hline NE with \(g^{\prime}(\tau)\) & 0.99\(\pm\)0.00 & 0.98\(\pm\)0.00 & 0.90\(\pm\)0.00 & 0.64\(\pm\)0.00 & 0.66\(\pm\)0.00 & 0.32\(\pm\)0.00 \\ NE with \(g(\tau)\) & 0.99\(\pm\)0.00 & 0.99\(\pm\)0.00 & 0.83\(\pm\)0.00 & 0.70\(\pm\)0.00 & 0.72\(\pm\)0.00 & 0.37\(\pm\)0.00 \\ \hline \hline \end{tabular}
\end{table}
Table 6: TR Detection by precision and recall rates

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{**Embedding**} & \multicolumn{3}{c}{**TR Detection (F1)**} & \multicolumn{3}{c}{**TR Query (r)**} \\  & D14 & D18 & GDELT & D14 & D18 & GDELT \\ \hline TNTComplEx & 0.26\(\pm\)0.01 & 0.18\(\pm\)0.00 & 0.08\(\pm\)0.01 & 0.08\(\pm\)0.00 & 0.08\(\pm\)0.00 & 0.01\(\pm\)0.00 \\ DE-SimplE & 0.22\(\pm\)0.00 & 0.20\(\pm\)0.00 & - & 0.09\(\pm\)0.00 & 0.09\(\pm\)0.00 & - \\ TASTER & 0.18\(\pm\)0.00 & 0.15\(\pm\)0.00 & 0.08\(\pm\)0.00 & 0.09\(\pm\)0.00 & 0.09\(\pm\)0.00 & 0.00\(\pm\)0.00 \\ TeRo & 0.43\(\pm\)0.02 & 0.64\(\pm\)0.01 & 0.16\(\pm\)0.00 & 0.08\(\pm\)0.00 & 0.08\(\pm\)0.00 & 0.01\(\pm\)0.00 \\ BoxTE & 0.40\(\pm\)0.01 & 0.40\(\pm\)0.02 & 0.18\(\pm\)0.01 & 0.08\(\pm\)0.00 & 0.08\(\pm\)0.00 & 0.01\(\pm\)0.00 \\ ATISE & 0.40\(\pm\)0.01 & 0.44\(\pm\)0.01 & 0.18\(\pm\)0.01 & 0.08\(\pm\)0.00 & 0.08\(\pm\)0.00 & 0.01\(\pm\)0.00 \\ \hline NE with \(g^{\prime}(\tau)\) & 0.78\(\pm\)0.00 & 0.79\(\pm\)0.00 & 0.48\(\pm\)0.00 & 0.85\(\pm\)0.00 & 0.83\(\pm\)0.00 & 0.83\(\pm\)0.00 \\ NE with \(g(\tau)\) & **0.82\(\pm\)**0.00 & **0.83\(\pm\)**0.00 & **0.51\(\pm\)**0.00 & **0.87\(\pm\)**0.00 & **0.86\(\pm\)**0.00 & **0.85\(\pm\)**0.00 \\ \hline \hline \end{tabular}
\end{table}
Table 5: TR detection by F1 scores and TR query by confidence ratios

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Embedding**} & \multicolumn{3}{c}{**Detection(F1)**} & \multicolumn{3}{c}{**Query(r)**} \\  & \(\theta\)=0.7 & \(\theta\)=0.8 & \(\theta\)=0.9 & \(\theta\)=0.7 & \(\theta\)=0.8 & \(\theta\)=0.9 \\ \hline TNTCompIEx & 0.29 & 0.26 & 0.25 & 0.07 & 0.08 & 0.06 \\ DE-SimplE & 0.28 & 0.22 & 0.21 & 0.09 & 0.09 & 0.21 \\ TASTER & 0.28 & 0.18 & 0.17 & 0.09 & 0.09 & 0.08 \\ TeRo & 0.35 & 0.43 & 0.40 & 0.08 & 0.08 & 0.06 \\ BoxTE & 0.38 & 0.40 & 0.41 & 0.08 & 0.18 & 0.06 \\ ATISE & 0.28 & 0.35 & 0.33 & 0.08 & 0.08 & 0.06 \\ \hline NE with \(g^{\prime}(\tau)\) & 0.63 & 0.78 & 0.80 & 0.81 & 0.85 & 0.86 \\ NE with \(g(\tau)\) & **0.71** & **0.82** & **0.84** & **0.87** & **0.87** & **0.87** \\ \hline \hline \end{tabular}
\end{table}
Table 10: Statistical results with different thresholds \(\theta\) on ICEWS14

\begin{table}
\begin{tabular}{c c c c c} \hline \hline \(d\) & 100 & 200 & 300 & 400 & 500 \\ \hline TR Query (r) & 0.83 & 0.84 & 0.85 & 0.85 & 0.84 \\ TR Detection (F1) & 0.22 & 0.45 & 0.50 & 0.51 & 0.51 \\ \hline \hline \end{tabular}
\end{table}
Table 8: NE on GDELT with different dimensions

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \(d\) & 50 & 100 & 200 & 400 & 600 & 800 \\ \hline TR Query (r) & 0.81 & 0.86 & 0.87 & 0.87 & 0.87 & 0.87 \\ TR Detection (F1) & 0.29 & 0.55 & 0.76 & 0.82 & 0.83 & 0.83 \\ \hline \hline \end{tabular}
\end{table}
Table 9: NE on ICEWS14 with different dimensions

Response:
## Review

### Summary

This paper introduces Noether Embedding (NE) for efficiently detecting and retrieving temporal regularities (TRs) from event data. It defines two tasks, TR detection and TR query, alongside their evaluation metrics. The authors argue that existing models lack the capability to learn temporal regularities. NE is evaluated on benchmark datasets and demonstrates improved performance compared to traditional methods. The paper also discusses the significance of time-translation symmetries in its approach. However, concerns are raised regarding the novelty of the tasks, clarity of implementation details, and the overall originality of the contribution when compared to existing event embedding methods.

### Strengths

- The proposed temporal regularity problem is important, and many existing models lack TR learning capabilities.
- The introduction of NE provides a novel approach for encoding temporal regularities and facilitates efficient retrieval.
- The tasks of TR detection and TR query are well-defined with corresponding evaluation metrics.
- Experiments on benchmark datasets demonstrate the effectiveness of the proposed framework and its superiority over classic embeddings.
- The paper is well-written and presents a clear overview of the problem and solution.

### Weaknesses

- The novelty of the TR mining task is questionable, as it appears to extend existing methodologies rather than introduce a fundamentally new paradigm.
- The implementation details, particularly regarding event embeddings, are insufficiently clear for reproducibility.
- Existing event embedding methods are not adequately discussed in terms of their applicability to the proposed tasks.
- The connection between Noether's theorem and the proposed method lacks clarity and is poorly presented.
- Certain evaluation metrics and approximations used in the experiments may not be justified as the best comparisons.

### Questions

- What is the rationale behind the choice of different embedding strategies for event representation?
- How does the framework handle relative time conditioned on multiple events?
- Can the authors clarify the reasons for the differences in recall and confidence scores observed in the experiments?
- What is the significance of the predefined thresholds and parameters used in the detection process?
- Could the authors elaborate on the societal impacts and limitations of the proposed method?

### Soundness

**Score:** 2

**Description:** 2 = fair; the methodology has merit but is not thoroughly validated or lacks clarity in some aspects.

### Presentation

**Score:** 2

**Description:** 2 = fair; while the writing is generally clear, several sections require better explanation and organization for improved understanding.

### Contribution

**Score:** 2

**Description:** 2 = fair; the contribution is valuable but presents concerns regarding originality and the significance of the proposed tasks.

### Rating

**Score:** 5

**Description:** 5 = Borderline accept; the paper has technical strengths but also substantial weaknesses that require addressing before final acceptance.

### Paper Decision

**Decision:** Accept (poster)

**Reasons:** The paper presents a significant contribution to the understanding and mining of temporal regularities, with a novel approach using Noether embeddings. Despite some weaknesses in originality and clarity, the detailed responses from the authors and improvements made during the review process have persuaded reviewers to recommend acceptance. The results demonstrate a solid performance on benchmark datasets and hold potential for practical applications, warranting further exploration and refinement in future work.